Linear Algebra for Data Science & Machine Learning

Vectors

A vector is an ordered list of numbers. Geometrically, a vector represents a point or direction in space. In Data Science, a feature vector contains all features for one observation (row).

• 1D vector: features of one sample, e.g. \([height, weight, age]\).
• n‑dimensional vectors: embeddings in NLP, image pixels, etc.
• Common operations: addition, scaling, dot product, norm (length).

import numpy as np

# Column and row vectors
v = np.array([1, 2, 3])          # row vector (shape: (3,))
v_col = v.reshape(-1, 1)         # column vector (shape: (3, 1))

# Vector addition and scaling
u = np.array([4, 5, 6])
sum_vec = v + u                  # [5, 7, 9]
scaled = 2 * v                   # [2, 4, 6]

# Dot product and norm
dot = np.dot(v, u)               # 1*4 + 2*5 + 3*6 = 32
norm = np.linalg.norm(v)         # length of vector

print("v:", v)
print("u:", u)
print("dot(v, u):", dot)
print("||v||:", norm)

Matrices & Matrix Multiplication

A matrix is a 2D array of numbers. In Data Science, a dataset is typically represented as a matrix \(X \in \mathbb{R}^{n \times d}\) where \(n\) is the number of rows (samples) and \(d\) is the number of features.

• Each row: one data point.
• Each column: one feature.
• Matrix multiplication is used heavily in neural networks and linear models.

# Design matrix X and parameter vector w
X = np.array([
    [1, 2],
    [3, 4],
    [5, 6]
])                              # shape: (3, 2)

w = np.array([0.1, 0.5])        # shape: (2,)
b = 0.3                         # bias term

# Linear model: y = Xw + b
y = X @ w + b                   # "@" is matrix multiplication in Python

print("X shape:", X.shape)
print("w shape:", w.shape)
print("Predictions y:", y)

Eigenvalues, Eigenvectors & SVD (Intuition)

Many dimensionality reduction techniques such as PCA rely on eigenvalues and singular values.

• Eigenvectors indicate important directions in the data.
• Eigenvalues tell you how much variance lies along each direction.
• SVD factorizes a matrix into \(U \Sigma V^T\) and is used under the hood in PCA.

# Covariance matrix example
X = np.array([[2.5, 2.4],
              [0.5, 0.7],
              [2.2, 2.9],
              [1.9, 2.2],
              [3.1, 3.0]])

X_centered = X - X.mean(axis=0)
cov = np.cov(X_centered, rowvar=False)

eig_vals, eig_vecs = np.linalg.eig(cov)

print("Covariance matrix:\n", cov)
print("Eigenvalues:", eig_vals)
print("Eigenvectors:\n", eig_vecs)

# SVD
U, S, Vt = np.linalg.svd(X_centered)
print("Singular values:", S)

In practice, libraries like scikit-learn hide these details, but understanding the Linear Algebra behind them helps you debug, tune and explain models better.