Neural Network Fundamentals

TensorFlow/Keras

import tensorflow as tf
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(4,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='mse')

PyTorch

import torch.nn as nn
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 8)
        self.fc2 = nn.Linear(8, 4)
        self.out = nn.Linear(4, 1)
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return torch.sigmoid(self.out(x))

Regression & Forecasting

Housing prices, stock trends, energy load.

Classification

Spam detection, credit risk, medical diagnosis.

Feature learning

Autoencoders, embeddings, representation learning.

ReLU (Rectified Linear Unit)

f(x) = max(0, x)

def relu(x):
    return np.maximum(0, x)
# derivative: 1 if x>0 else 0

Pros: Computationally cheap, sparse, no saturation for x>0. Cons: Dying ReLU (neurons stuck at 0).

Leaky ReLU

f(x) = x if x>0 else αx (α small, e.g., 0.01)

def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

Fixes dying ReLU; allows gradient flow for negative values.

ELU (Exponential Linear Unit)

f(x)= x if x>0 else α(e^x -1)

Smooth, negative values push mean closer to zero. Faster learning.

PReLU (Parametric ReLU)

α is learned during training.

# TensorFlow: tf.keras.layers.PReLU()

TensorFlow / Keras

import tensorflow as tf
# As activation string or layer
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(32, activation='leaky_relu'), 
    tf.keras.layers.Dense(10, activation='softmax')
])
# Advanced: tf.keras.activations.gelu, tf.nn.swish

PyTorch

import torch.nn as nn
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.act1 = nn.ReLU()
        self.fc2 = nn.Linear(128, 64)
        self.act2 = nn.LeakyReLU(0.02)
        self.out = nn.Linear(64, 10)
    def forward(self, x):
        x = self.act1(self.fc1(x))
        x = self.act2(self.fc2(x))
        return self.out(x)  # with CrossEntropyLoss, no softmax needed

Binary Cross-Entropy (BCE)

BCE = -[y log(Å·) + (1-y) log(1-Å·)]

def binary_crossentropy(y, y_pred):
    y_pred = np.clip(y_pred, 1e-7, 1-1e-7)  # stability
    return -np.mean(y * np.log(y_pred) + 
                    (1 - y) * np.log(1 - y_pred))

Use: Binary classification, sigmoid output.

Categorical Cross-Entropy (CCE)

CCE = -Σ y_i log(ŷ_i)

def categorical_crossentropy(y, y_pred):
    y_pred = np.clip(y_pred, 1e-7, 1.0)
    return -np.sum(y * np.log(y_pred)) / y.shape[0]

Use: Multi-class, softmax output.

CTC Loss

Connectionist Temporal Classification. Used in speech recognition, handwriting recognition. Aligns sequences without alignment labels.

tf.nn.ctc_loss

Contrastive Loss

L = y*d² + (1-y)*max(margin-d,0)²

Used in Siamese networks, similarity learning.

Triplet Loss

max(d(a,p)-d(a,n)+margin, 0)

Face recognition (FaceNet), embeddings.

TensorFlow / Keras

import tensorflow as tf

# Common losses
model.compile(loss='mse', optimizer='adam')  # regression
model.compile(loss='binary_crossentropy', ...)
model.compile(loss='categorical_crossentropy', ...)
model.compile(loss=tf.keras.losses.Huber(delta=1.5), ...)

# Custom loss function
def custom_mse(y_true, y_pred):
    return tf.reduce_mean(tf.square(y_true - y_pred))

PyTorch

import torch.nn as nn

criterion = nn.MSELoss()  # regression
criterion = nn.BCELoss()  # requires sigmoid
criterion = nn.BCEWithLogitsLoss()  # stable, from_logits
criterion = nn.CrossEntropyLoss()  # includes softmax
criterion = nn.KLDivLoss()  # KL divergence

# Custom
class CustomLoss(nn.Module):
    def forward(self, y_pred, y_true):
        return torch.mean((y_true - y_pred)**2)

TensorFlow

with tf.GradientTape() as tape:
    y_pred = model(X)
    loss = tf.keras.losses.MSE(y, y_pred)
grads = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))

PyTorch

y_pred = model(X)
loss = nn.MSELoss()(y_pred, y)
loss.backward()  # <-- one line backprop!
optimizer.step()

Custom Layers

Implement your own forward/backward in frameworks.

Debugging

Why are gradients NaN? Why isn't this layer learning?

Research

Modify gradient flow (e.g., reversible nets, synthetic gradients).

Batch GD

Uses entire dataset to compute gradient. θ = θ - lr · ∇L(θ; all data)

Slow Stable Not feasible for large datasets.

Stochastic GD (SGD)

θ = θ - lr · ∇L(θ; xᵢ, yᵢ)

Update per sample. High variance Online learning

Mini-batch GD

θ = θ - lr · ∇L(θ; batch)

Balanced Most common. Batch size 32-512.

AdaGrad

Gₜ = Gₜ₋₁ + (∇L(θₜ))²
θₜ₊₁ = θₜ - lr/√(Gₜ + ε) · ∇L(θₜ)

Adapts per-parameter learning rates. Good for sparse data. Learning rate decays monotonically.

Weakness: LR becomes infinitesimally small.

RMSprop

E[g²]ₜ = βE[g²]ₜ₋₁ + (1-β)(∇L)²
θₜ₊₁ = θₜ - lr/√(E[g²]ₜ + ε) · ∇L

Unpublished, but widely used. Fixes AdaGrad's decaying LR problem. β typically 0.9.

Lion (EvoLved Sign Momentum)

mₜ = β₁mₜ₋₁ + (1-β₁)∇L
θₜ₊₁ = θₜ - lr · sign(β₂mₜ + (1-β₂)∇L)

Discovered by symbolic search. More memory-efficient than Adam. Used in Google's latest models.

Adafactor

Memory-efficient Adam for large models. Factorizes second moment estimates. Used in T5.

LAMB & LARS

Layer-wise Adaptive Rate Scaling. For large-batch training (BERT, ResNet on TPUs).

TensorFlow / Keras

import tensorflow as tf

# Common optimizers
model.compile(optimizer='sgd', ...)
model.compile(optimizer='adam', ...)
model.compile(optimizer=tf.keras.optimizers.AdamW(learning_rate=1e-4, weight_decay=1e-4))

# Learning rate schedule
lr_schedule = tf.keras.optimizers.schedules.CosineDecay(1e-3, decay_steps=10000)
optimizer = tf.keras.optimizers.Adam(lr_schedule)

# Custom optimizer loop
@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        y_pred = model(x)
        loss = loss_fn(y, y_pred)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

PyTorch

import torch.optim as optim

# Optimizers
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
optimizer = optim.Adam(model.parameters(), lr=0.001)
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

# Learning rate schedulers
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=5)

# Training loop
for epoch in range(epochs):
    for x, y in dataloader:
        optimizer.zero_grad()
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        loss.backward()
        optimizer.step()
    scheduler.step()