Training & Optimization — Interview Q&A

Answer: Repeatedly update parameters θ by stepping opposite the gradient of the loss L: smaller loss along steepest descent direction (locally). Requires differentiable objective (or subgradients).

Answer: Full-batch: gradient over all data—accurate but expensive. Mini-batch: noisy estimate, GPU-friendly. SGD often means mini-batch in practice; strict one-sample SGD is very noisy.

Answer: Too large: oscillation or divergence. Too small: painfully slow progress and risk of stalling in flat regions. Schedules and adaptive methods address this.

Answer: Non-convex losses have many local minima and saddle points. In high dimensions saddles are common; “bad” local minima are less universal than folklore suggests, but optimization is still hard.

Answer: A point where the gradient is zero but it is neither minimum nor maximum—curvature positive in some directions, negative in others. GD can slow down near saddles; noise (minibatches) or momentum helps escape.

Answer: Accumulate a velocity vector damped by friction so updates continue through noisy or ill-conditioned directions—like a ball rolling in a valley. Helps damp oscillations in narrow valleys.

Answer: For convex smooth functions with appropriate step sizes, GD converges to a global minimizer. Deep neural network losses are generally not convex—this theorem does not apply directly.

Answer: Larger batch → lower variance gradient, more stable steps, more memory. Smaller batch → noisier updates that can act like regularization and help generalization (with caveats).

Answer: One epoch is a full pass over the training set (possibly shuffled). One iteration is one parameter update (often one mini-batch). Multiple iterations per epoch.

Answer: Data arrives as a stream; each (or few) example(s) trigger an update—natural fit for stochastic style updates. Same GD template with a non-stationary distribution.

Answer: A nearly flat region where gradients are tiny—progress slows. Learning-rate warmup, restarts, or second-order hints (in advanced optimizers) can help.

Answer: Small-batch stochasticity can help escape sharp minima and explore the landscape—linked to flat minima hypotheses. Not a guarantee; interaction with batch norm and scale matters.

Answer: Line search picks step size along the descent direction by evaluating the objective—common in classical optimization, rare in large deep learning (too expensive); deep learning favors hand-tuned or scheduled η.

Answer: Rescaling coordinates so the landscape is more isotropic—Newton-like methods use inverse Hessian; Adam/RMSprop use diagonal adaptive scaling as a cheap approximation.

Answer: Backpropagation computes ∇L; gradient descent uses that vector to update weights. They answer different questions: “which direction?” vs “how to step?”

Answer: The standard method to compute ∂L/∂θ for all parameters by applying the chain rule backward from the loss through each layer. It is reverse-mode automatic differentiation on the network graph.

Answer: If y = f(g(x)), then dy/dx = (df/dg)(dg/dx). In many dimensions, derivatives become Jacobians and products become appropriate matrix-vector multiplies.

Answer: Loss is a scalar; we need one gradient vector w.r.t. millions of parameters. Reverse mode gives all partials in roughly one forward + one backward pass cost. Forward mode would repeat for each parameter.

Answer: Traverse layers from output toward input, propagating adjoint (gradient w.r.t. downstream activations). At each node, multiply local Jacobians into the incoming upstream gradient.

Answer: Given upstream g = ∂L/∂z: ∂L/∂b = sum over batch of g, ∂L/∂W = xᵀg (layout-dependent), ∂L/∂x = gWᵀ. Interviewers check you know dimensions line up.

Answer: Pass gradient through if pre-activation > 0, else zero. At exactly zero, use convention 0 or 1 (subgradient). Elementwise mask.

Answer: JVP pushes perturbations forward (forward mode). VJP pulls loss gradient backward—what each layer implements in backprop. Efficiency: we want VJPs for scalar loss.

Answer: To compute local derivatives at each layer you need forward activations (and sometimes intermediate tensors). Memory scales with network width, depth, and batch size.

Answer: Don’t store every activation; recompute some during backward. Less memory, more compute—used for large models (Transformers).

Answer: Gradients from all paths add (multivariate chain rule). Same weight used twice → two contribution terms to ∂L/∂w.

Answer: Backprop multiplies Jacobians layer by layer; if many factors are < 1 (saturating activations), the signal shrinks toward early layers—same math, architectural fix (ReLU, ResNet, gates).

Answer: The network is a DAG of ops; forward evaluates nodes, backward applies chain rule along edges. Frameworks build this graph dynamically (eager with autograd) or statically.

Answer: SGD + backprop uses first-order gradients. Second-order (Hessian) methods exist but are expensive; some approximations (K-FAC, etc.) are niche.

Answer: Traverses the autograd graph from loss, accumulating .grad on leaf tensors that require grad. Must call zero_grad between iterations unless gradients add intentionally.

Answer: For many networks, backward is roughly 2× the multiply-add cost of forward (same order—rule of thumb). Constant factors depend on fusion and framework.

Answer: Uses gradients (from backprop) to update parameters each step—choosing direction and step size (and possibly per-parameter scaling).

Answer: θ ← θ − η g where g is minibatch gradient and η is learning rate. Simple, well-understood baseline.

Answer: Maintain velocity v: v ← β v + g, then θ ← θ − η v. Accelerates along consistent gradient directions, dampens oscillations in ravines.

Answer: Looks ahead: gradient evaluated at θ − β v (approx.)—more responsive near curvature changes. Often “NAG” in frameworks.

Answer: Accumulate sum of squared gradients per parameter; divide update by growing denominator—adaptive smaller steps for frequent features. Learning rate can shrink to zero too aggressively.

Answer: Use exponential moving average of squared gradients (EMA) instead of full sum—prevents denominator from exploding, works better for non-convex deep nets.

Answer: First moment m: EMA of gradient (like momentum). Second moment v: EMA of squared gradient (like RMSprop). Bias-correct both early in training; divide step by √(v)+ε.

Answer: AdamW decouples weight decay: L2 penalty applied directly to weights, not mixed into the adaptive gradient—often better generalization and standard in Transformers.

Answer: Typical β₁=0.9 (gradient EMA decay), β₂=0.999 (squared-gradient EMA). Control memory horizon of first and second moment estimates.

Answer: Numerical stability—avoid division by zero when second moment is tiny. Usually 1e-8 scale.

Answer: Some vision tasks generalize slightly better with tuned SGD+momentum + schedule; Adam can find sharper minima in some studies—not universal, dataset and tuning matter.

Answer: No full Hessian—only diagonal curvature estimates from squared-gradient EMA. Much cheaper than Newton methods.

Answer: Lazy Adam / sparse Adam updates only touched parameters; useful in embedding-heavy models with huge vocabularies.

Answer: Adam stores two moment buffers per parameter (m and v)—roughly 2× optimizer state vs SGD with momentum (one velocity). Matters for large models.

Answer: AdamW with weight decay, warmup + decay schedule, β defaults above—standard in BERT/GPT-style training.

Answer: Scalar η scaling the gradient step: controls speed vs stability—too large diverges, too small is slow. Same symbol in Adam as base step size (per-parameter scaling aside).

Answer: A function of step or epoch that changes η during training—e.g. decay after plateaus, cosine to zero, warmup then constant.

Answer: Multiply η by constant γ every N epochs or when metric plateaus—classic CV recipe; simple and interpretable.

Answer: η_t = η_0 · γ^t or continuous η_0 e^(−kt)—smooth decrease; need tune decay rate.

Answer: η follows a cosine curve from max to min over T steps—smooth decay used in many modern trainers (SGDR extends with restarts).

Answer: Early steps have unreliable moment estimates; large η can destabilize. Linear warmup ramps η so early updates are smaller—standard in BERT/GPT recipes.

Answer: Increase η to a maximum then decrease in one cycle—idea from Smith: helps fast training and can aid generalization; used with cyclical LR ideas.

Answer: Increase η each batch, plot loss—find region where loss drops fastest before explosion; heuristic to pick max LR for one-cycle or training.

Answer: When batch size ×k, some recipes scale η ×k to keep gradient noise similar—works as heuristic for SGD in some regimes; not universal (warmup, BN, Adam complicate).

Answer: When validation metric stops improving, multiply η by factor < 1—adaptive to training dynamics, no fixed epoch list.

Answer: Schedulers often clamp η ≥ η_min so training doesn’t stall completely; cosine schedules use explicit minimum.

Answer: No—Adam adapts per-parameter scale but global η still matters; Transformers use warmup + decay with AdamW routinely.

Answer: Periodically reset η to high value (cosine cycles)—helps escape flat regions; related to ensemble of snapshots near restarts.

Answer: Random search on log-uniform η often more efficient than grid—covers orders of magnitude; use validation score.

Answer: Peak learning rate, warmup steps, total steps / decay shape (cosine or linear)—cite paper recipe then tune.

Answer: In backprop, gradients are products of terms through layers. If many factors are < 1 (e.g. saturated sigmoid derivatives), early-layer gradients → 0 and those weights barely update.

Answer: Same product picture: factors > 1 repeatedly → huge gradients, unstable updates, NaNs. Common in RNNs over long sequences if unrolled.

Answer: Derivatives are ≤ 0.25 (sigmoid) or small in saturation—each layer shrinks the backward signal when stacked deeply.

Answer: Derivative is 1 for active neurons—less shrinkage than sigmoid. Caveat: dead ReLUs still pass zero gradient.

Answer: y = F(x)+x adds an identity path; gradient can bypass stacked layers via +1 shortcut—eases training of very deep nets.

Answer: LSTM’s cell state and additive updates with gates allow better long-range gradient flow than simple tanh recurrence where each step multiplies Jacobians.

Answer: GRU has fewer gates but similar idea: gating and update blending to mitigate vanishing in sequences—often comparable performance with less compute.

Answer: Clip by norm: if ||g|| > threshold, scale g down—common in RNNs. Value clipping caps each element—less common. Stops one batch from destroying weights.

Answer: It stabilizes activations and can help optimization indirectly—not a guarantee; deep nets still benefit from good init, ReLU, residuals.

Answer: Highways use learned gates on skip vs transform; ResNet uses identity skip + simpler F(x). ResNet won for simplicity and performance in vision.

Answer: Depth is finite and paths include residual + LN; attention mixes tokens in O(1) depth per layer—not the same T-step product as unrolled RNN, but very deep stacks still need design care.

Answer: Good init keeps activations in a range where derivatives aren’t tiny everywhere—reduces extreme products at start of training.

Answer: Sudden NaN loss, gradient norm spikes, weights blow up—watch global norm per step.

Answer: Small gradients can underflow in fp16; loss scaling multiplies loss before backward so gradients stay in representable range, then unscales for the optimizer.

Answer: Use ReLU, good init, BN/LN, residuals, gated RNNs or attention, and gradient clipping when training recurrent or unstable nets.