Object Tracking — Interview Q&A

Answer: Estimating object state over time in video—position, size, sometimes 3D pose—while preserving identity across frames.

Answer: Single-object tracking: one target given init box. Multi-object tracking: many objects with IDs—requires association across detections.

Answer: Run detector each frame, link boxes into trajectories via association—modular and strong with good detectors.

Answer: Decide which detection belongs to which track—classic bipartite matching with cost matrix (IoU, appearance, motion).

Answer: Greedy or Hungarian match predicted boxes to tracks by highest IoU above threshold—simple baseline for MOT.

Answer: Solves assignment problem in O(n³)—global optimum for linear cost—used in SORT / DeepSORT association.

Answer: Tracker swaps identities between objects—common near crossings; penalized in MOTA metric.

Answer: Predict motion during missing detections (Kalman), re-identify with appearance when visible again, or joint optimization over windows.

Answer: Small errors accumulate updating from own predictions—mitigated by periodic re-detection or robust loss.

Answer: Predict box between frames with constant-velocity model; update when measurements arrive—cheap smooth motion prior.

Answer: Cosine distance on embedding reduces ID switches when IoU ambiguous (similar DeepSORT).

Answer: Multiple Object Tracking Accuracy—combines false positives, misses, and ID switches vs ground truth trajectories.

Answer: Uses only past and current frames—needed for robotics/live video; batch methods use future frames (smoother but not causal).

Answer: Track corner features with local flow—fast but fragile to appearance change; less common alone for generic objects now.

Answer: Template branch + search region CNN—fast SOT without online fine-tuning in early versions (SiamFC family).

Answer: Track queries attend across space-time (e.g. TrackFormer)—joint detection+association in one model trend.

Answer: Light detector + simple association (SORT) or specialized accelerators—appearance models add compute.

Answer: Track in bird’s-eye view from multi-camera or LiDAR—used in autonomous driving stacks.

Answer: Associate 3D boxes or point clusters—IoU in 3D or GIoU variants; Kalman in xyz + yaw.

Answer: MOTChallenge, KITTI tracking, nuScenes tracking—each defines detection input protocol and metrics.

Answer: Optimal recursive estimator for linear systems with Gaussian noise—alternates prediction from dynamics and correction from noisy observations.

Answer: x_{k+1} = F x_k + w_k (process noise), z_k = H x_k + v_k (measurement noise)—Kalman assumes linear F,H and Gaussian w,v.

Answer: Often [cx, cy, s, r, vx, vy, vs] (center, scale area-ish, aspect, velocities)—measurements update subset.

Answer: x̂− = F x̂, P− = F P Fᵀ + Q—propagate mean and covariance forward in time without new measurement.

Answer: Fuse measurement z using Kalman gain K: x̂ = x̂− + K(z − H x̂−), P = (I − K H) P−—reduce uncertainty along observed dimensions.

Answer: K balances trust in prediction vs measurement based on covariances—if R small (accurate sensor), K larger, trust measurement more.

Answer: Process noise covariance—higher Q = more model uncertainty, tracker follows measurements faster but noisier.

Answer: Measurement noise—higher R = smoother track, lag on maneuvers; lower R = jittery if detector noisy.

Answer: Assumes derivative of position constant between frames—simple, works for smooth motion; fails on sharp turns.

Answer: Adds acceleration state for more expressive motion—better for maneuvering targets, more parameters to tune.

Answer: Nonlinear dynamics or measurement—linearize with Jacobians around current estimate; no longer globally optimal but widely used.

Answer: Handle stronger nonlinearities—UKF uses sigma points; particle filters for non-Gaussian multimodal posteriors (rare in simple MOT).

Answer: Skip update; covariance grows with prediction-only steps until next match—standard in SORT when object temporarily not detected.

Answer: H maps state to observed variables (e.g. only position observed, not velocity directly inferred from motion over time).

Answer: State estimate covariance—uncertainty ellipsoid; should shrink after informative updates.

Answer: From finite differences of first two boxes or zero velocity with high initial P—tradeoff between fast lock vs overshoot.

Answer: Each track maintains Kalman state; Hungarian matches detections to predicted boxes—simple, fast MOT baseline.

Answer: cv2.KalmanFilter with transition/measurement matrices—set dt, Q, R for bbox tracking experiments.

import cv2
kf = cv2.KalmanFilter(4, 2)  # example dims

Answer: Use Joseph form for P update, symmetric enforcement, or square-root filtering if covariance becomes indefinite.

Answer: Highly nonlinear motion, multi-modal uncertainty (occlusions), or heavy-tailed detector noise—consider particle, IMM, or learning-based motion.

Answer: Simple online MOT: Kalman filter motion model + Hungarian assignment with IoU cost between predicted and detected boxes—very fast.

Answer: Predict all tracks → match detections to tracks by IoU → update matched with Kalman measurement → create new tracks for unmatched dets → delete stale tracks.

Answer: Optimal one-to-one assignment minimizing total cost—better than greedy max-IoU for competing hypotheses.

Answer: Often 1 − IoU or negative IoU with threshold—reject matches below IoU min (no assignment).

Answer: Delete track if unmatched for max_age frames; confirm birth only after min_hits to reduce spurious tracks from false positives.

Answer: CNN appearance embedding + cosine distance combined with motion Mahalanobis gate—reduces ID switches when IoU ambiguous.

Answer: Train embedding so same-ID images are closer than different-ID—used with triplet or classification losses on person crops.

Answer: First match high-confidence detections to tracks using appearance+motion; then lower-confidence in second stage—reduces clutter confusion.

Answer: Reject association if innovation (z − Hx) is unlikely under predicted covariance—filters physically impossible jumps.

Answer: Typical 128-D L2-normalized vector per detection crop—cosine distance = 1 − dot product.

Answer: Store recent embeddings per track for matching—manage length to balance memory and adaptability to appearance change.

Answer: IoU fails when overlapping—appearance helps reacquire correct ID after split; still hard in dense crowds.

Answer: Minimal overhead beyond detector—no heavy joint optimization per frame unlike some MHT approaches.

Answer: Also associates low-score detections in a second pass—recovers occluded objects SORT might drop.

Answer: Adds camera motion compensation + improved Re-ID—strong MOTChallenge scores.

Answer: IoU-only methods degrade—appearance, higher-order models, or transformer MOT help.

Answer: On person re-ID datasets (Market1501, etc.) separate from detector—domain gap to target scene matters.

Answer: Assumes good detector; IoU association weak under fast motion / low FPS; camera motion not modeled in vanilla SORT.

Answer: TrackFormer / MOTR predict tracks end-to-end—no hand-crafted association but need more data and compute.

Answer: Match detector FPS; batch Re-ID CNN; tune thresholds per scene; log ID switches for QA.