Neural Network Fundamentals — Interview Q&A

Answer: An early binary linear classifier: it forms a weighted sum of inputs plus a bias, then applies a threshold (step) to decide between two classes. It is historically important as a simple trainable “neuron” and the starting point for multi-layer networks.

Answer: Compute the pre-activation (margin) s = w·x + b. Predict Å· = sign(s) (with a convention for s = 0, e.g. treat as +1 or define a tie rule). Training adjusts w, b only when ŷ ≠ y.

Answer: Two classes are linearly separable if there exists a hyperplane w·x + b = 0 that puts all examples of one class strictly on one side and the other class on the other. The perceptron can learn such a separator when data are separable.

Answer: XOR in 2D is not linearly separable: no single line separates (0,0)/(1,1) from (0,1)/(1,0). A perceptron is exactly one linear decision boundary, so it cannot fit XOR without adding features or hidden layers (e.g. MLP).

Answer: For labels y ∈ {−1, +1}, when (x, y) is misclassified, update w ← w + η y x and b ← b + η y (learning rate η > 0; often η = 1 in the classic algorithm). Correct points receive no update.

Answer: If the data are linearly separable, the perceptron rule converges in a finite number of mistakes (Novikoff-style bounds). If data are not separable, updates can cycle indefinitely—need pocket algorithm, averages, or a different model/loss.

Answer: Without b, every separating hyperplane must pass through the origin in feature space. The bias shifts the decision boundary so it can separate offset clouds of points. Often implemented as an extra input fixed at 1 with a weight wâ‚€.

Answer: The step gives a hard decision and zero gradient almost everywhere—classic perceptron uses discrete updates, not backprop through the step. Sigmoid is smooth, yields probabilities, and supports gradient-based training (logistic regression / neural nets with continuous loss).

Answer: Both use a linear score w·x + b. Logistic regression outputs sigmoid(score) as probability and minimizes log loss with gradients. The perceptron uses a hard threshold and mistake-driven updates; same geometry, different output and training objective.

Answer: With y ∈ {−1, +1}, the (signed) margin is often written y (w·x + b). It is positive when the prediction is correct; larger values mean the point is farther from the decision boundary. Convergence proofs bound the number of mistakes using margin and norm of a separating vector.

Answer: The decision boundary is still linear, but scale differences across features can slow convergence or make updates dominated by large-magnitude inputs. Standardizing or scaling features often helps iterative algorithms behave more evenly in practice.

Answer: Common reductions: one-vs-rest (one perceptron per class vs all others) or one-vs-one (pairwise classifiers). At prediction time, combine votes or scores. This is not softmax—mention softmax as the smooth multi-class alternative in neural nets.

Answer: On noisy or non-separable data, standard perceptron updates may not stabilize. The pocket variant keeps the weight vector that achieved the lowest training error so far (“in your pocket”) while continuing updates, returning the best snapshot instead of the last iterate.

Answer: Both learn linear separators. SVM maximizes margin (often with slack for soft-margin) and yields a unique solution under convex optimization. Perceptron finds any separating hyperplane if one exists; many solutions possible. SVM generalizes better with kernels; perceptron is simple and historically foundational.

Answer: One perceptron = one linear boundary. Hidden layers of non-linear units compose boundaries: early layers can fold or combine half-spaces so later layers separate XOR-like patterns. That is the core idea of an MLP: depth + non-linearity overcomes single-layer limits.

Answer: An MLP is a feedforward neural network: layers of neurons where each layer is typically fully connected to the next, with non-linear activations between affine transforms. Information flows input → hidden(s) → output without cycles.

Answer: Activations are computed in one direction only—from input to output. There are no recurrent edges that feed a layer’s output back into earlier layers in the same forward pass (that would be an RNN or similar).

Answer: Hidden layers let the network build intermediate representations. Stacking non-linear layers composes functions so the model can approximate non-linear boundaries (e.g. XOR) that a single linear layer cannot.

Answer: Depth (more layers) increases compositional power and hierarchical features; can help sample efficiency for some tasks but risks optimization issues. Width (more units per layer) increases capacity per layer; very wide shallow nets can also approximate functions. Trade-offs: data, compute, vanishing gradients, and inductive bias.

Answer: Weights: d_in × d_out. Bias: d_out. Total d_in × d_out + d_out. Mention this scales quickly for large fully connected layers.

Answer: A composition of linear maps is still linear. The entire deep stack collapses to a single affine transform—no extra expressive power vs one linear layer.

Answer: With a suitable non-linearity and enough hidden units, a single-hidden-layer MLP can approximate many continuous functions on compact domains (universal approximation theme). In practice depth, data, and optimization matter—not only width.

Answer: A hidden layer can form new features (e.g. AND-like combinations) so the output layer becomes linearly separable in that feature space. Classic example: 2 inputs → small hidden layer with non-linearity → output.

Answer: For batch size B, input is B × d_in. Linear layer: Y = XW + b (broadcast bias). Same weights for all batch rows—this is why matrix multiply is efficient on GPUs.

Answer: Images have local structure and translation patterns. CNNs use shared local filters—far fewer parameters and better inductive bias. A flat MLP on pixels ignores locality and scales poorly with resolution.

Answer: High parameter count vs data lets the network memorize noise. Mitigate with regularization (L2, dropout), more data, early stopping, or architecture better matched to the problem.

Answer: Poor scaling can make activations explode or vanish layer-to-layer, giving useless gradients. Schemes like Xavier/He set variance based on fan-in/fan-out to keep signal scale stable at initialization.

Answer: Linear logits followed by softmax (often applied inside loss for numerical stability). Training uses cross-entropy on probabilities or logits with log-softmax.

Answer: Often a linear output (no squashing) with MSE or Huber loss for real-valued targets. For bounded outputs you might use sigmoid scaling or tanh to a range.

Answer: Tabular or fixed-length feature vectors without strong spatial or sequential structure, baselines, or as a component inside larger models. For sequences use RNN/Transformer; for grids use CNN.

Answer: Computing the network’s output from input by applying each layer in order: affine transforms, biases, activations, pooling, etc.—no weight updates. It is prediction / loss input during training and pure inference at deploy time.

Answer: Forward: compute outputs and (usually) cache intermediates for loss. Backward: apply chain rule to get gradients for learning. Forward does not change weights; backward supplies the update signal.

Answer: z = Wx + b, then a = f(z) for activation f. For a batch, X is stacked rows and the same W applies to each.

Answer: X: B × d_in, W: d_in × d_out, bias b: d_out (broadcast). Output Y: B × d_out with Y = XW + b (row-wise).

Answer: Dominant term is O(m·k·n) multiply-adds (often quoted as ~2mkn FLOPs if counting mul+add separately). Used to reason about layer cost in forward pass.

Answer: Each layer’s input is the previous layer’s output. Reordering changes the composed function entirely unless the architecture is specially designed (e.g. parallel branches with merge).

Answer: Pre-activations z and post-activations a (and BN stats inputs) so backprop can compute local gradients without recomputing everything. Frameworks handle this in autograd.

Answer: Dropout disabled (or scaled). BatchNorm uses running mean/var not batch stats. No gradient tracking needed—saves memory and compute.

Answer: Logits can be large; e^z overflows. z' = z − max(z) shifts logits without changing softmax output (invariant) but keeps exponentials bounded—numerical stability.

Answer: Usually yes—B=1 is a degenerate batch; matrix ops still work. Some ops (e.g. BN) behave differently with tiny batch size; that’s a practical caveat.

Answer: Storing activations for backprop, plus optimizer state if updating. Wider/deeper nets and larger batch increase activation memory—often the bottleneck before weights.

Answer: Applying ops with explicit weight tensors passed in (e.g. F.linear(x, W, b)) instead of nn.Module parameters—same math, useful for meta-learning or custom graphs.

Answer: Many ops run in float16/bfloat16 for speed; sensitive reductions (loss, BN) may stay in float32. Loss scaling can help with small gradients in low precision.

Answer: It is a frozen forward computation graph (no backward), optimized for deployment—same layer order as training forward, possibly fused ops.

Answer: x → h1 = f(W1x+b1) → h2 = f(W2h1+b2) → logits = W3h2+b3 → probs = softmax(logits). Mention where nonlinearity stops (before softmax).

Answer: A directed acyclic graph (DAG) representing a function: nodes are variables or operations, edges show data flow. Used to evaluate the function and (with autodiff) derivatives systematically.

Answer: Nodes: tensors after an op, or the op itself (depends on framework representation). Edges: which outputs feed which inputs. The graph encodes dependencies for topological order.

Answer: Computes exact derivatives (up to floating point) by applying chain rule along the graph—not numerical finite differences, not full symbolic algebra on the whole expression tree by hand.

Answer: Pushes directional derivatives forward; costs scale with number of inputs. Useful when few inputs and many outputs (rare for standard NN training vs reverse mode).

Answer: One scalar loss, millions of parameters—reverse mode gets the full gradient in O(graph size) time, same order as one forward pass (roughly). This is backpropagation.

Answer: For standard autodiff you need a clear topological order. RNNs unroll in time creating a DAG over steps; true cycles need special handling (implicit differentiation / BPTT structure).

Answer: Eager: build graph as Python runs (PyTorch default). Static: trace or compile a full graph first (older TF graphs, torch.compile, XLA)—enables fusion and deployment optimizations.

Answer: Leaves are parameters or inputs you optimize; intermediates are non-leaf. .grad fills on leaves by default; retain_graph keeps graph for multiple backward calls.

Answer: Breaks the graph from that tensor onward—no gradient flows through. Used to freeze parts of the model or treat values as constants.

Answer: Yes—block gradients through that path; common in GANs, reinforcement learning tricks, or fixed targets.

Answer: Forward computes outputs; backward receives gradient w.r.t. outputs and returns gradients w.r.t. each differentiable input—must be mathematically consistent with the forward op.

Answer: Frameworks can build a graph over gradient computations (create_graph=True in PyTorch) for Hessian-vector products; memory and cost grow quickly.

Answer: They can overwrite values still needed for backward. Frameworks error or warn when versions mismatch—prefer out-of-place when tensors require grad.

Answer: Symbolic: algebra rules, expression swell. Finite diff: cheap to code, inaccurate and slow for high-D. Autodiff: exact, efficient for ML-scale graphs.

Answer: Inference drops backward nodes and anything only needed for gradients—smaller, faster. Export formats (ONNX, TorchScript) target forward-only execution.