Deep Learning

GANs & Autoencoders

Generative adversarial networks and autoencoder architectures for generation and compression.

Generative Adversarial Networks (GANs)

The Adversarial Game

GANs are a minimax game between two players: Generator (G) and Discriminator (D). G tries to fool D by generating realistic samples. D tries to distinguish real from fake.

z ~ N(0,1) → G(z) → D(x) → real/fake

D(x) = probability that x is real. G learns to maximize D(G(z)).

Minimax objective: min_G max_D V(D,G) = E_x[log D(x)] + E_z[log(1 - D(G(z)))]

Intuition: G is like a counterfeiter, D is the detective. Competition drives both to perfection.

Vanilla GAN – The Original

Generator

Maps latent vector z to data space. Typically MLP with ReLU + sigmoid/tanh for images.

def build_generator(latent_dim=100):
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(128, input_dim=latent_dim),
        tf.keras.layers.ReLU(),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Dense(784, activation='tanh')  # MNIST
    ])
    return model
Discriminator

Binary classifier. Outputs probability of real image.

def build_discriminator():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(128, input_shape=(784,)),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])
    return model
PyTorch Vanilla GAN Training Loop 
# Training loop (alternating updates)
for epoch in range(epochs):
    for real_imgs, _ in dataloader:
        batch_size = real_imgs.size(0)
        z = torch.randn(batch_size, latent_dim)
        
        # Train Discriminator
        fake_imgs = G(z)
        real_pred = D(real_imgs)
        fake_pred = D(fake_imgs.detach())
        d_loss = -torch.mean(torch.log(real_pred) + torch.log(1 - fake_pred))
        d_loss.backward()
        optimizer_D.step()
        
        # Train Generator
        z = torch.randn(batch_size, latent_dim)
        fake_imgs = G(z)
        fake_pred = D(fake_imgs)
        g_loss = -torch.mean(torch.log(fake_pred))
        g_loss.backward()
        optimizer_G.step()

DCGAN – Convolutional GAN

DCGAN brought CNNs to GANs with key architectural guidelines that stabilized training.

Guidelines
  • Replace pooling with strided conv (D) / fractional conv (G)
  • BatchNorm in both G and D
  • Remove fully connected layers
  • ReLU in G (except output tanh)
  • LeakyReLU in D
# Generator (DCGAN)
class DCGenerator(nn.Module):
    def __init__(self, latent_dim=100):
        super().__init__()
        self.deconv = nn.Sequential(
            nn.ConvTranspose2d(latent_dim, 512, 4, 1, 0, bias=False),
            nn.BatchNorm2d(512), nn.ReLU(True),
            nn.ConvTranspose2d(512, 256, 4, 2, 1, bias=False),
            nn.BatchNorm2d(256), nn.ReLU(True),
            nn.ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
            nn.BatchNorm2d(128), nn.ReLU(True),
            nn.ConvTranspose2d(128, 3, 4, 2, 1, bias=False),
            nn.Tanh()
        )
    
    def forward(self, z):
        return self.deconv(z.view(z.size(0), z.size(1), 1, 1))

Training Challenges & Stabilization

⚠️ Mode Collapse: Generator produces limited varieties. D gets stuck in a local optimum.

Solution: WGAN, minibatch discrimination, unrolled GANs.

⚠️ Vanishing Gradients: D becomes too good → G gradient ~0.

Solution: Use WGAN (Earth Mover distance), label smoothing, instance noise.

✅ Label Smoothing: Use 0.9/0.1 instead of 1/0. Prevents overconfidence.
✅ BatchNorm & Spectral Norm: Spectral normalization for D stabilizes training.
✅ Gradient Penalty: WGAN-GP enforces Lipschitz constraint via gradient norm.

Wasserstein GAN (WGAN)

WGAN replaces the binary discriminator with a critic that scores realness. Uses Earth Mover distance, more stable training.

WGAN Loss

D_loss = E[D(fake)] - E[D(real)]
G_loss = -E[D(fake)]

Critic weights clipped to [-c, c] (WGAN) or gradient penalty (WGAN-GP).

WGAN-GP (Gradient Penalty) 
def gradient_penalty(critic, real, fake, device):
    batch_size, c, h, w = real.shape
    epsilon = torch.rand(batch_size, 1, 1, 1).repeat(1, c, h, w).to(device)
    interpolated = epsilon * real + (1 - epsilon) * fake
    mixed_score = critic(interpolated)
    
    gradient = torch.autograd.grad(
        inputs=interpolated,
        outputs=mixed_score,
        grad_outputs=torch.ones_like(mixed_score),
        create_graph=True,
        retain_graph=True
    )[0]
    gradient = gradient.view(batch_size, -1)
    gradient_norm = gradient.norm(2, dim=1)
    gp = torch.mean((gradient_norm - 1) ** 2)
    return gp

Conditional GAN (cGAN)

Both generator and discriminator receive additional condition (class label, text, image). Enables controlled generation.

Architecture

Concatenate condition y to z (G) and to x (D).

# Generator
z = torch.randn(batch_size, latent_dim)
y = one_hot(labels)  # condition
gen_input = torch.cat([z, y], dim=1)
fake = G(gen_input)

# Discriminator
dis_input = torch.cat([image, y], dim=1)
score = D(dis_input)
Applications
  • Pix2Pix: Image-to-image translation (edges→photo)
  • CycleGAN: Unpaired translation (horse→zebra)
  • Text-to-Image: Generate images from descriptions
  • SRGAN: Super-resolution

StyleGAN & Progressive GANs

Progressive GAN

Start with low resolution (4x4), add layers as training progresses. Stabilizes high-res generation.

1024x1024 faces, cats, cars.

StyleGAN

Mapping network + AdaIN (adaptive instance normalization). Style mixing enables controllable synthesis (pose, identity, lighting).

Key idea: Noise injects stochastic variation (freckles, hair).

StyleGAN formula: w = MappingNetwork(z) → AdaIN(conv, w) → stochastic variation via noise. Separates high-level attributes from stochastic details.

Evaluating GANs – FID & Inception Score

Inception Score (IS)

Uses ImageNet-pretrained Inception. Measures:

  • High confidence predictions (realistic)
  • Diversity across samples

Criticism: Doesn't detect mode collapse if classes are diverse.

FID (Fréchet Inception Distance)

Compares statistics of real vs fake in Inception feature space.

FID = ||μ_r - μ_f||² + Tr(Σ_r + Σ_f - 2(Σ_rΣ_f)^½)

Lower is better Standard metric today.

# FID using torchmetrics
from torchmetrics.image.fid import FrechetInceptionDistance

fid = FrechetInceptionDistance(feature=2048)
fid.update(real_images, real=True)
fid.update(fake_images, real=False)
print(f"FID: {fid.compute():.2f}")

Production-Ready GAN Implementations

PyTorch Lightning GAN 
import pytorch_lightning as pl

class GAN(pl.LightningModule):
    def __init__(self, latent_dim=100):
        super().__init__()
        self.generator = Generator(latent_dim)
        self.discriminator = Discriminator()
        
    def training_step(self, batch, batch_idx, optimizer_idx):
        real_imgs, _ = batch
        z = torch.randn(real_imgs.size(0), self.latent_dim)
        
        if optimizer_idx == 0:  # train D
            fake_imgs = self.generator(z)
            d_loss = self.discriminator_loss(real_imgs, fake_imgs)
            return d_loss
        else:  # train G
            fake_imgs = self.generator(z)
            g_loss = self.generator_loss(fake_imgs)
            return g_loss
Keras TF-GAN 
# TensorFlow GAN with custom training
@tf.function
def train_step(real_images):
    z = tf.random.normal([BATCH_SIZE, latent_dim])
    
    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
        fake_images = generator(z, training=True)
        
        real_output = discriminator(real_images, training=True)
        fake_output = discriminator(fake_images, training=True)
        
        gen_loss = generator_loss(fake_output)
        disc_loss = discriminator_loss(real_output, fake_output)
    
    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)
    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)
    
    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))
    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))

GAN Family Comparison

Model Key Idea Stability Quality Use Case
Vanilla GANMinimax BCE❌ Low⭐Educational
DCGANConvolutional guidelines⭐⭐⭐⭐Small images
WGAN-GPWasserstein + gradient penalty⭐⭐⭐⭐⭐⭐⭐Default stable choice
cGANConditional generation⭐⭐⭐⭐⭐Labeled synthesis
StyleGANStyle modulation + noise⭐⭐⭐⭐⭐⭐⭐⭐High-res faces
CycleGANCycle consistency (unpaired)⭐⭐⭐⭐⭐⭐Unpaired translation

GANs & Responsible AI

⚠️ Deepfakes: GANs can generate realistic fake faces/videos. Critical to develop detection methods and watermarking.
⚠️ Bias amplification: GANs trained on biased datasets amplify stereotypes (e.g., gender, race). Use balanced, diverse data.
✅ Positive impact: Medical imaging (synthetic MRI), drug discovery, data augmentation, creative tools.
✅ Detection: GAN-generated image detection is active research. Use forensic tools.

GAN Cheatsheet

G generator
D discriminator
z latent vector
WGAN Wasserstein
GP gradient penalty
cGAN conditional
FID evaluation
Mode collapse failure

Autoencoders: Learning Efficient Representations

What is an Autoencoder?

An autoencoder is a neural network trained to copy its input to its output. Internally, it compresses the input into a latent-space representation (bottleneck), then reconstructs the output from this representation. This forces the model to learn the most salient features of the data distribution.

Input (x) → Encoder → z (bottleneck) → Decoder → Reconstruction (x̂)

Loss = ℒ(x, x̂) = MSE or Binary Cross-Entropy

Undercomplete Autoencoders

Bottleneck Dimension & Compression

dim(latent) < dim(input)

The network is forced to learn a compressed representation. Trained with reconstruction loss only (no regularization).

PCA analogy Non-linear manifold learning

Risk of Overcomplete

If dim(latent) ≥ dim(input), the network may learn identity function (copying). Regularization required.

Undercomplete → meaningful compression; Overcomplete → needs constraints

Undercomplete Autoencoder (PyTorch)
import torch.nn as nn

class UndercompleteAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, input_dim),
            nn.Sigmoid()  # for pixel values [0,1]
        )
    
    def forward(self, x):
        z = self.encoder(x)
        x_hat = self.decoder(z)
        return x_hat, z

Regularized Autoencoders

Sparse Autoencoder

ℒ = ||x - x̂||² + λ · Ω(z)

Penalize activations of hidden units (L1 regularization, KL divergence). Encourages specialized features.

Ω(z) = Σ|z| or KL(ρ||ρ̂)

Denoising Autoencoder (DAE)

Corrupt input with noise (e.g., Gaussian, dropout), reconstruct clean original.

Learns robust features, removes noise. x̃ = x + ε, minimize ||x - d(e(x̃))||²

Contractive Autoencoder (CAE)

ℒ = ||x - x̂||² + λ ||J_f(x)||²_F

Penalizes Frobenius norm of encoder's Jacobian. Encourages invariance to small input changes.

Practical Insight

Denoising + Sparse often combined. Contractive is computationally expensive; denoising is simpler and effective.

Denoising Autoencoder – adding noise 
def add_noise(x, noise_factor=0.3):
    noise = torch.randn_like(x) * noise_factor
    x_noisy = x + noise
    return torch.clamp(x_noisy, 0., 1.)

# Training loop
for x_batch in dataloader:
    x_noisy = add_noise(x_batch)
    x_hat, _ = model(x_noisy)
    loss = nn.functional.binary_cross_entropy(x_hat, x_batch)

Variational Autoencoders (VAE)

VAEs are generative models that learn a probabilistic latent space. Instead of encoding a point, encoder outputs parameters of a Gaussian distribution (μ, σ). Decoder samples from this distribution to generate data.

VAE Loss: Reconstruction + KL Divergence

ℒ = 𝔼[log p(x|z)] - β · D_KL(q(z|x) || p(z))

Reconstruction: make x̂ similar to x.
KL divergence: regularize latent distribution towards prior (usually N(0,1)).

Reparameterization Trick

z = μ + σ ⊙ ε, ε ~ N(0, I). Enables backpropagation through sampling.

μ (mean), log-var (σ²) from encoder; sample, then decode.

VAE Encoder with Reparameterization (PyTorch) 
class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=20):
        super().__init__()
        # Encoder -> μ and logvar
        self.fc1 = nn.Linear(input_dim, 400)
        self.fc21 = nn.Linear(400, latent_dim)  # μ
        self.fc22 = nn.Linear(400, latent_dim)  # logvar
        # Decoder
        self.fc3 = nn.Linear(latent_dim, 400)
        self.fc4 = nn.Linear(400, input_dim)
    
    def encode(self, x):
        h = torch.relu(self.fc1(x))
        return self.fc21(h), self.fc22(h)
    
    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std
    
    def decode(self, z):
        h = torch.relu(self.fc3(z))
        return torch.sigmoid(self.fc4(h))
    
    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 784))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

# VAE loss function
def vae_loss(recon_x, x, mu, logvar):
    BCE = nn.functional.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD
β-VAE: Increase β (>1) for stronger latent regularization, leading to more disentangled representations.

Modern & Advanced Autoencoders

Convolutional Autoencoders

Use Conv2D, TransposeConv2D for images. Essential for vision tasks.

VQ-VAE (Vector Quantized)

Discrete latent space. Used in high-fidelity generation (VQ-VAE-2, WaveNet).

Adversarial Autoencoders

GAN-based regularization to match prior distribution.

Variational Autoencoder (β-VAE)

Disentanglement: factorized latent representations.

Ladder VAE

Hierarchical latent variables.

Diffusion Autoencoders

Hybrid of diffusion and autoencoding.

Real-World Applications

Dimensionality Reduction

Non-linear PCA. Visualize high-dim data (t-SNE alternative). Pretraining for supervised tasks.

Anomaly Detection

Train on normal data; anomalies have high reconstruction error. Used in fraud, industrial inspection.

Image Denoising

DAEs remove noise from photographs, medical scans.

Inpainting

Fill missing regions in images.

Molecule Generation

VAEs generate novel molecular structures.

Feature Disentanglement

Separate content/style in images (β-VAE).

Anomaly detection pipeline: Train AE on normal samples → reconstruction error threshold → flag outliers.

Framework Implementations

TensorFlow / Keras
import tensorflow as tf

# Convolutional autoencoder
class ConvAE(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.encoder = tf.keras.Sequential([
            tf.keras.layers.Input(shape=(28,28,1)),
            tf.keras.layers.Conv2D(32, 3, strides=2, padding='same', activation='relu'),
            tf.keras.layers.Conv2D(64, 3, strides=2, padding='same', activation='relu'),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(16)  # latent
        ])
        self.decoder = tf.keras.Sequential([
            tf.keras.layers.Dense(7*7*64, activation='relu'),
            tf.keras.layers.Reshape((7,7,64)),
            tf.keras.layers.Conv2DTranspose(64, 3, strides=2, padding='same', activation='relu'),
            tf.keras.layers.Conv2DTranspose(32, 3, strides=2, padding='same', activation='relu'),
            tf.keras.layers.Conv2D(1, 3, padding='same', activation='sigmoid')
        ])
    
    def call(self, x):
        z = self.encoder(x)
        return self.decoder(z)

model = ConvAE()
model.compile(optimizer='adam', loss='mse')
PyTorch (Convolutional VAE)
class ConvVAE(nn.Module):
    def __init__(self, latent_dim=20):
        super().__init__()
        # Encoder: Conv2d -> μ, logvar
        self.enc_conv = nn.Sequential(
            nn.Conv2d(1, 32, 4, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(32, 64, 4, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(64, 128, 4, stride=2, padding=1), nn.ReLU(),
            nn.Flatten()
        )
        self.fc_mu = nn.Linear(128*4*4, latent_dim)
        self.fc_logvar = nn.Linear(128*4*4, latent_dim)
        
        # Decoder
        self.dec_fc = nn.Linear(latent_dim, 128*4*4)
        self.dec_conv = nn.Sequential(
            nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1), nn.ReLU(),
            nn.ConvTranspose2d(64, 32, 4, stride=2, padding=1), nn.ReLU(),
            nn.ConvTranspose2d(32, 1, 4, stride=2, padding=1), nn.Sigmoid()
        )
    # ... reparameterization, forward, loss

Latent Space Arithmetic & Interpolation

Autoencoders learn meaningful latent spaces. VAEs produce smooth, continuous manifolds.

🧮 Latent vector arithmetic: z = z_smiling - z_neutral + z_male → generates smiling male (word2vec style).
✨ Interpolation: z = (1-α)·z₁ + α·z₂. Decode to see smooth morphing between images.
Latent space interpolation (NumPy-style) 
def interpolate(model, x1, x2, steps=10):
    """x1, x2: input images, model: VAE"""
    model.eval()
    with torch.no_grad():
        z1 = model.encode(x1)[0]  # mu
        z2 = model.encode(x2)[0]
        alphas = torch.linspace(0, 1, steps)
        interpolates = []
        for alpha in alphas:
            z = (1 - alpha) * z1 + alpha * z2
            x_hat = model.decode(z)
            interpolates.append(x_hat)
    return interpolates

Autoencoder Variants – Cheatsheet

Undercomplete Compression
Sparse Feature selection
Denoising Robustness
Contractive Local invariance
VAE Generative
β-VAE Disentanglement
VQ-VAE Discrete latent
AAE Adversarial

Autoencoder Variants Comparison

Variant Latent Space Loss Terms Primary Use
UndercompleteContinuous, deterministicReconstructionDimensionality reduction
Sparse AEContinuous + sparsity penaltyReconstruction + L1/KLInterpretable features
Denoising AEContinuous, robustReconstruction (corrupted input)Noise removal, pretraining
Contractive AEContinuous + Jacobian normReconstruction + ||J_f||²Invariant representations
VAEProbabilistic (Gaussian)Reconstruction + KLGeneration, interpolation
VQ-VAEDiscrete codebookReconstruction + commitmentHigh-fidelity generation

Autoencoder Pitfalls & Debugging

⚠️ Overcomplete AE copies input: Use regularization (sparse, denoising, dropout) or smaller bottleneck.
⚠️ VAE posterior collapse: Decoder ignores latent code. Use KL annealing, free bits, or β < 1.
✅ Blurry VAE outputs: Use perceptual loss, adversarial training (VQ-VAE), or hierarchical VAEs.
✅ Monitor reconstruction error: Should decrease. Compare train/test error for anomaly detection threshold.
PreviousNext