Deep learning questions that come up in ML engineer and data scientist interviews — short answers you can say out loud, with pointers to the topic that explains each one.
Fundamentals
What is the difference between machine learning and deep learning?
Why do neural networks need activation functions?
What does the universal approximation theorem say — and not say?
How many parameters does a Conv2d(64, 128, kernel_size=3) have?
Training
Explain backpropagation.
Batch vs mini-batch vs stochastic gradient descent?
How does Adam work? Why AdamW?
What happens if the learning rate is too high or too low?
Why use cross-entropy loss for classification, and what should the initial loss be?
Stability & generalisation
What causes vanishing/exploding gradients and how do you fix them?
How does batch normalization behave differently at train and test time?
How does dropout work?
Your training accuracy is 99% but validation is 75%. What do you do?
Your training loss is not decreasing. How do you debug?
Architectures
Why do CNNs work well for images?
What problem do residual connections solve?
RNN vs LSTM vs Transformer?
Explain scaled dot-product attention. Why the scaling?
Encoder-only vs decoder-only Transformers?
When would you use transfer learning?
Generative models
Compare VAEs, GANs and diffusion models.
What is the reparameterization trick in VAEs?
Practical
What's the difference between model.eval() and torch.no_grad()?
How do you train with a batch that doesn't fit in GPU memory?
What is mixed precision training?
Quick-fire checklist
Neuron & bias · XOR & hidden layers · ReLU vs sigmoid · Softmax & logits · Cross-entropy · Mini-batch SGD · Chain rule · Gradient check · Momentum · Adam vs AdamW · Warmup + cosine · He / Xavier init · Vanishing gradients · Gradient clipping · BatchNorm vs LayerNorm · Dropout · Weight decay · Early stopping · Data augmentation · zero_grad → backward → step · train() / eval() · state_dict · Overfit one batch · Mixed precision · Conv output size · Receptive field · ResNet skip · Transfer learning · LSTM gates · Embeddings · Self-attention · √d_k scaling · Multi-head · Positional encoding · VAE & KL · GAN minimax · Diffusion denoising