Express.js Advanced Topics Guide

1. Pagination

What it is: A technique for splitting large datasets into smaller, manageable chunks (pages) to reduce response size, improve performance, and enhance user experience. Instead of returning 10,000 records at once, pagination returns 50 per page with navigation controls.

Why it matters

Performance: Reduces database load and network transfer
UX: Faster initial load, easier data consumption
Scalability: Prevents memory exhaustion on server/client
Cost: Less bandwidth usage, faster processing

Pagination strategies

Strategy	How it works	Best for	Pros	Cons
Offset/Limit	OFFSET 100 LIMIT 20	Small datasets, static lists	Simple, skip to any page	Inefficient for large offsets
Cursor-based	WHERE id > last_id LIMIT 20	Large, real-time data	Fast, consistent, no gaps	Can't skip pages
Keyset	Sort + filter on indexed column	Infinite scroll	Very fast, stable	Complex implementation
Page numbers	?page=3&size=50	UI tables	User-friendly	Offset performance issues

Offset/Limit pagination (basic but common):

JavaScript — Offset/Limitapp.get('/api/users', async (req, res) => {
  try {
    const page = Math.max(1, parseInt(req.query.page) || 1);
    const limit = Math.min(100, Math.max(1, parseInt(req.query.limit) || 20));
    const offset = (page - 1) * limit;

    const [users, total] = await Promise.all([
      db.users.findMany({
        skip: offset,
        take: limit,
        orderBy: { createdAt: 'desc' }
      }),
      db.users.count()
    ]);

    const totalPages = Math.ceil(total / limit);

    res.json({
      data: users,
      pagination: {
        page,
        limit,
        total,
        totalPages,
        hasNext: page < totalPages,
        hasPrev: page > 1
      }
    });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

// SQL: SELECT * FROM users ORDER BY created_at DESC LIMIT $1 OFFSET $2;

Cursor-based pagination (performant for large datasets):

JavaScript — Cursor-basedapp.get('/api/posts', async (req, res) => {
  try {
    const limit = Math.min(50, parseInt(req.query.limit) || 20);
    let cursor = req.query.cursor;

    let query = db.posts.findMany({
      take: limit + 1,
      orderBy: { id: 'desc' }
    });

    if (cursor) {
      const lastId = parseInt(Buffer.from(cursor, 'base64').toString());
      query = db.posts.findMany({
        take: limit + 1,
        skip: 1,
        cursor: { id: lastId },
        orderBy: { id: 'desc' }
      });
    }

    const posts = await query;
    const hasNext = posts.length > limit;
    if (hasNext) posts.pop();

    let nextCursor = null;
    if (hasNext && posts.length > 0) {
      nextCursor = Buffer.from(posts[posts.length - 1].id.toString()).toString('base64');
    }

    res.json({ data: posts, pagination: { limit, nextCursor, hasNext } });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Keyset pagination (using indexed timestamp):

JavaScript — Keysetapp.get('/api/events', async (req, res) => {
  const limit = 50;
  const { after, before } = req.query;

  let query = db.events.findMany({ take: limit, orderBy: { timestamp: 'desc' } });

  if (after) {
    query = db.events.findMany({
      take: limit,
      where: { timestamp: { gt: new Date(after) } },
      orderBy: { timestamp: 'asc' }
    });
  }

  if (before) {
    query = db.events.findMany({
      take: limit,
      where: { timestamp: { lt: new Date(before) } },
      orderBy: { timestamp: 'desc' }
    });
  }

  const events = await query;
  res.json({
    data: events,
    next: events.length === limit ? events[events.length - 1].timestamp : null
  });
});

Search with pagination:

JavaScriptapp.get('/api/products/search', async (req, res) => {
  const { q, category, minPrice, maxPrice, page = 1, limit = 20 } = req.query;

  const where = {};
  if (q) where.name = { contains: q, mode: 'insensitive' };
  if (category) where.category = category;
  if (minPrice || maxPrice) {
    where.price = {};
    if (minPrice) where.price.gte = parseFloat(minPrice);
    if (maxPrice) where.price.lte = parseFloat(maxPrice);
  }

  const offset = (page - 1) * limit;
  const [products, total] = await Promise.all([
    db.products.findMany({ where, skip: offset, take: limit }),
    db.products.count({ where })
  ]);

  res.json({
    data: products,
    pagination: { page, limit, total, pages: Math.ceil(total / limit) },
    filters: { q, category, minPrice, maxPrice }
  });
});

2. Caching

What it is: Storing frequently accessed data in a fast, temporary storage layer (memory, Redis, CDN) to reduce latency, database load, and costs. Caches serve subsequent requests for the same data without recomputing or re-fetching.

Why it matters

Performance: 10–1000x faster response times
Scalability: Reduces database queries by 80–95%
Cost: Lower compute and bandwidth costs
Availability: Cache survives backend failures

Caching levels

Level	Storage	Speed	Best for	TTL example
Browser	Client	Instant	Static assets, user data	1 year for images
CDN	Edge network	~10ms	Global content	1 hour for API
Application	Memory/Redis	~1ms	Database queries	5–60 minutes
Database	Query cache	~5ms	Expensive queries	1 minute

Cache invalidation strategies

TTL (Time-To-Live): Auto-expire after fixed time
Write-through: Update cache when DB updates
Cache-aside: App manages cache (most common)
Event-based: Invalidate on data changes

In-memory cache with Node.js (node-cache):

JavaScript — node-cacheconst NodeCache = require('node-cache');
const cache = new NodeCache({ stdTTL: 300 });

app.get('/api/users/:id', async (req, res) => {
  const userId = req.params.id;
  let user = cache.get(`user:${userId}`);

  if (!user) {
    user = await db.users.findUnique({ where: { id: userId } });
    if (!user) return res.status(404).json({ error: 'User not found' });
    cache.set(`user:${userId}`, user, 600);
  }

  res.json(user);
});

app.put('/api/users/:id', async (req, res) => {
  const userId = req.params.id;
  const updatedUser = await db.users.update({ where: { id: userId }, data: req.body });
  cache.del(`user:${userId}`);
  cache.set(`user:${userId}`, updatedUser, 600);
  res.json(updatedUser);
});

Redis cache (production-grade):

JavaScript — Redis cache middlewareconst Redis = require('ioredis');
const redis = new Redis({ host: process.env.REDIS_HOST, port: 6379 });

function cacheMiddleware(ttl = 300, keyPrefix = '') {
  return async (req, res, next) => {
    const cacheKey = `${keyPrefix}:${req.user?.id || 'anon'}:${req.originalUrl}`;
    try {
      const cachedData = await redis.get(cacheKey);
      if (cachedData) return res.json(JSON.parse(cachedData));

      const originalSend = res.json;
      res.json = function(data) {
        redis.setex(cacheKey, ttl, JSON.stringify(data));
        originalSend.call(this, data);
      };
      next();
    } catch (err) {
      next();
    }
  };
}

app.get('/api/products', cacheMiddleware(300, 'products'), async (req, res) => {
  const products = await db.products.findMany();
  res.json(products);
});

async function cacheWithTags(key, data, ttl, tags = []) {
  await redis.setex(key, ttl, JSON.stringify(data));
  for (const tag of tags) await redis.sadd(`tag:${tag}`, key);
}

async function invalidateTag(tag) {
  const keys = await redis.smembers(`tag:${tag}`);
  if (keys.length) {
    await redis.del(...keys);
    await redis.del(`tag:${tag}`);
  }
}

HTTP caching (Cache-Control headers):

JavaScriptapp.use('/static', express.static('public', { maxAge: '1y', immutable: true }));

app.get('/api/weather', async (req, res) => {
  const weather = await fetchWeather();
  res.set({
    'Cache-Control': 'public, max-age=300, stale-while-revalidate=60',
    'ETag': generateETag(weather)
  });
  res.json(weather);
});

app.get('/api/report', async (req, res) => {
  const report = await generateReport();
  const etag = `"${Buffer.from(JSON.stringify(report)).toString('base64')}"`;
  if (req.headers['if-none-match'] === etag) return res.status(304).end();
  res.set('ETag', etag);
  res.json(report);
});

Advanced caching patterns:

JavaScript — Cache-aside & rate limitingasync function getCachedUser(userId, forceRefresh = false) {
  const cacheKey = `user:${userId}`;
  if (forceRefresh) await redis.del(cacheKey);

  let user = await redis.get(cacheKey);
  if (user) return JSON.parse(user);

  user = await db.users.findUnique({ where: { id: userId } });
  await redis.setex(cacheKey, 300, JSON.stringify(user));

  setTimeout(async () => {
    const freshUser = await db.users.findUnique({ where: { id: userId } });
    await redis.setex(cacheKey, 300, JSON.stringify(freshUser));
  }, 240000);

  return user;
}

async function rateLimit(userId, action, maxRequests, windowSeconds) {
  const key = `rate:${userId}:${action}`;
  const now = Date.now();
  const windowStart = now - (windowSeconds * 1000);
  await redis.zremrangebyscore(key, 0, windowStart);
  const count = await redis.zcard(key);
  if (count >= maxRequests) return { allowed: false, remaining: 0 };
  await redis.zadd(key, now, `${now}:${Math.random()}`);
  await redis.expire(key, windowSeconds);
  return { allowed: true, remaining: maxRequests - count - 1 };
}

3. API Versioning

What it is: The practice of managing changes to your API without breaking existing clients by maintaining multiple versions simultaneously. Versioning allows you to evolve your API while giving consumers time to migrate.

Why it matters

Backward compatibility: Existing apps keep working
Gradual migration: Clients upgrade on their schedule
Experimentation: Test new features safely
Risk reduction: Rollback problematic versions easily

Versioning strategies

Strategy	Example	Pros	Cons
URL path	/v1/users, /v2/users	Simple, cache-friendly	Clutters URLs
Query param	/users?version=1	Clean URLs	Easy to forget
Custom header	Accept: application/vnd.api.v1+json	Semantically clean	Harder to test
Content negotiation	Accept: application/json; version=1	RESTful standard	Complex parsing

URL path versioning (most common):

JavaScript — URL versioningapp.get('/v1/users', async (req, res) => {
  const users = await db.users.findMany({
    select: { id: true, name: true, email: true }
  });
  res.json({ users });
});

app.get('/v2/users', async (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const limit = parseInt(req.query.limit) || 20;
  const [users, total] = await Promise.all([
    db.users.findMany({
      skip: (page - 1) * limit,
      take: limit,
      select: { id: true, name: true, email: true, avatar: true, createdAt: true }
    }),
    db.users.count()
  ]);
  res.json({ data: users, meta: { page, limit, total, pages: Math.ceil(total / limit) } });
});

app.get('/v3/users', async (req, res) => {
  const { page = 1, limit = 20, includeDeleted = false } = req.query;
  const users = await db.users.findMany({
    where: includeDeleted === 'true' ? {} : { deletedAt: null },
    skip: (page - 1) * limit,
    take: limit
  });
  res.json({
    success: true,
    data: users.map(u => ({
      id: u.id,
      profile: { name: u.name, email: u.email },
      metadata: { created: u.createdAt, updated: u.updatedAt }
    })),
    pagination: { page, limit, hasMore: users.length === limit }
  });
});

Version router organization:

JavaScriptapp.use('/v1/users', require('./routes/v1/users'));
app.use('/v2/users', require('./routes/v2/users'));
app.use('/users', require('./routes/v2/users')); // Default to latest

Header-based versioning:

JavaScriptapp.use('/api', (req, res, next) => {
  const version = req.headers['x-api-version'] || req.headers['accept-version'];
  req.apiVersion = version === '1.0' ? 1 : 2;
  next();
});

app.get('/api/users', async (req, res) => {
  if (req.apiVersion === 1) {
    const users = await db.users.findMany({ take: 100 });
    return res.json({ users });
  }
  const { page = 1, limit = 20 } = req.query;
  const users = await db.users.findMany({ skip: (page - 1) * limit, take: limit });
  res.json({ data: users, page, limit });
});

Accept header versioning (Content negotiation):

JavaScriptapp.get('/api/users', (req, res) => {
  const acceptHeader = req.headers.accept || '';

  if (acceptHeader.includes('application/vnd.myapi.v1+json')) {
    return res.json({ version: 1, users: [] });
  }

  if (acceptHeader.includes('application/vnd.myapi.v2+json')) {
    return res.json({ version: 2, data: { users: [] } });
  }

  res.json({ version: 2, data: { users: [] } });
});

// Client: GET /api/users
// Accept: application/vnd.myapi.v1+json

Version compatibility layer (adapter pattern):

JavaScriptconst adapters = {
  v2toV1: (v2User) => ({
    id: v2User.id,
    name: v2User.name,
    email: v2User.email,
    avatar: v2User.profile.avatar,
    createdAt: v2User.profile.joinedAt
  })
};

app.get('/api/users/:id', async (req, res) => {
  const user = await db.users.findById(req.params.id);
  const requestedVersion = req.query.version || 'v2';
  if (requestedVersion === 'v1') return res.json(adapters.v2toV1(user));
  res.json(user);
});

Deprecation handling:

JavaScriptfunction deprecateVersion(version, deprecationDate, sunsetDate) {
  return (req, res, next) => {
    if (req.apiVersion === version) {
      res.set('Deprecation', `date="${deprecationDate}"`);
      res.set('Sunset', sunsetDate);
      res.set('Link', '<https://docs.myapi.com/v2/migration>; rel="deprecation"');
    }
    next();
  };
}

app.use('/v1', deprecateVersion(1, '2024-01-01', '2024-06-01'));
app.use('/v1', require('./routes/v1'));

4. Logging

What it is: Systematic recording of application events, errors, requests, and business transactions to files or external services. Logging provides visibility into application behavior, performance, and errors for debugging, monitoring, and auditing.

Why it matters

Debugging: Reproduce and fix issues
Monitoring: Track performance and errors
Security: Detect attacks and audit access
Compliance: Meet regulatory requirements (GDPR, HIPAA, PCI)
Analytics: Understand usage patterns

What to log

✅ Request ID, timestamp, user ID, IP address
✅ HTTP method, URL, status code, response time
✅ Database queries (in development)
✅ Authentication attempts (success/failure)
✅ Business events (purchase, account change)
❌ Passwords, tokens, credit cards, PII (unless required)

Log levels (severity)

Level	Use case	Example
ERROR	Application failures	Database connection failed
WARN	Potential issues	Deprecated API used, rate limit nearing
INFO	Important events	User login, order placed
DEBUG	Development details	Query parameters, variable values
TRACE	Very detailed	Function entry/exit, full request body

Basic logging with Winston:

JavaScript — Winston setupconst winston = require('winston');
const { combine, timestamp, printf, colorize, json } = winston.format;

const devFormat = printf(({ level, message, timestamp, ...metadata }) => {
  let log = `${timestamp} [${level}]: ${message}`;
  if (Object.keys(metadata).length) log += ` ${JSON.stringify(metadata)}`;
  return log;
});

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: combine(timestamp(), process.env.NODE_ENV === 'production' ? json() : devFormat),
  transports: [
    new winston.transports.Console({ format: combine(colorize(), devFormat) }),
    new winston.transports.File({ filename: 'logs/error.log', level: 'error' }),
    new winston.transports.File({ filename: 'logs/combined.log' })
  ]
});

logger.info('User logged in', { userId: 123, ip: '192.168.1.1' });
logger.error('Database connection failed', { error: err.message });

Request logging middleware:

JavaScript — Morgan + request IDconst morgan = require('morgan');
const uuid = require('uuid');

app.use(morgan('combined'));

app.use((req, res, next) => {
  req.id = uuid.v4();
  res.setHeader('X-Request-Id', req.id);
  next();
});

app.use((req, res, next) => {
  const start = Date.now();
  logger.info(`${req.method} ${req.url} started`, {
    requestId: req.id,
    userId: req.user?.id,
    ip: req.ip
  });

  res.on('finish', () => {
    const duration = Date.now() - start;
    const level = res.statusCode >= 400 ? 'warn' : 'info';
    logger[level](`${req.method} ${req.url} completed`, {
      requestId: req.id,
      statusCode: res.statusCode,
      duration: `${duration}ms`
    });
  });
  next();
});

Structured logging with context:

JavaScript — AsyncLocalStorageconst { AsyncLocalStorage } = require('async_hooks');
const asyncLocalStorage = new AsyncLocalStorage();

app.use((req, res, next) => {
  const context = {
    requestId: req.id,
    userId: req.user?.id,
    ip: req.ip,
    path: req.path,
    method: req.method
  };
  asyncLocalStorage.run(context, () => next());
});

function logWithContext(level, message, meta = {}) {
  const context = asyncLocalStorage.getStore() || {};
  logger[level](message, { ...context, ...meta });
}

app.post('/api/orders', async (req, res) => {
  logWithContext('info', 'Creating order', { items: req.body.items.length });
  try {
    const order = await createOrder(req.body);
    logWithContext('info', 'Order created', { orderId: order.id, total: order.total });
    res.json(order);
  } catch (error) {
    logWithContext('error', 'Order creation failed', { error: error.message });
    res.status(500).json({ error: 'Order failed' });
  }
});

Database query logging:

JavaScriptconst prisma = new PrismaClient({
  log: process.env.NODE_ENV === 'development'
    ? ['query', 'info', 'warn', 'error']
    : ['error']
});

function logQuery(sql, params, duration) {
  if (duration > 100) {
    logger.warn('Slow query detected', { sql, params, duration: `${duration}ms` });
  } else {
    logger.debug('Query executed', { sql, duration: `${duration}ms` });
  }
}

mongoose.set('debug', (collection, method, query, doc, options) => {
  logger.debug(`MongoDB ${collection}.${method}`, { query, options });
});

Error logging with stack traces:

JavaScriptfunction logError(error, req, additionalInfo = {}) {
  logger.error('Application error', {
    message: error.message,
    stack: error.stack,
    requestId: req?.id,
    userId: req?.user?.id,
    url: req?.url,
    method: req?.method,
    ...additionalInfo
  });
}

app.use((err, req, res, next) => {
  logError(err, req);
  res.status(err.statusCode || 500).json({
    error: process.env.NODE_ENV === 'production' ? 'Internal server error' : err.message,
    requestId: req.id
  });
});

process.on('uncaughtException', (error) => {
  logger.error('Uncaught Exception', { error: error.stack });
  process.exit(1);
});

process.on('unhandledRejection', (reason) => {
  logger.error('Unhandled Rejection', { reason });
});

Production-ready logging with daily rotation:

JavaScriptconst DailyRotateFile = require('winston-daily-rotate-file');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp({ format: 'ISO' }),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: { service: 'my-api', environment: process.env.NODE_ENV },
  transports: [
    new DailyRotateFile({
      filename: 'logs/error-%DATE%.log',
      datePattern: 'YYYY-MM-DD',
      level: 'error',
      maxSize: '20m',
      maxFiles: '14d'
    }),
    new DailyRotateFile({
      filename: 'logs/combined-%DATE%.log',
      datePattern: 'YYYY-MM-DD',
      maxSize: '20m',
      maxFiles: '14d'
    })
  ]
});

Combined Example: Complete API with All Concepts

JavaScriptconst express = require('express');
const Redis = require('ioredis');
const winston = require('winston');
const uuid = require('uuid');

const app = express();
const redis = new Redis();
const logger = winston.createLogger(/* config */);

app.use((req, res, next) => {
  req.id = uuid.v4();
  req.startTime = Date.now();
  res.setHeader('X-Request-Id', req.id);
  next();
});

app.get('/v2/products',
  cacheMiddleware(300, 'products'),
  async (req, res) => {
    const startDb = Date.now();
    const { page = 1, limit = 20, category } = req.query;

    logger.info('Products request', {
      requestId: req.id,
      page, limit, category,
      userId: req.user?.id
    });

    const offset = (page - 1) * limit;
    const where = category ? { category } : {};
    const [products, total] = await Promise.all([
      db.products.findMany({ where, skip: offset, take: limit }),
      db.products.count({ where })
    ]);

    if (Date.now() - startDb > 500) {
      logger.warn('Slow query', { requestId: req.id, duration: Date.now() - startDb });
    }

    res.json({
      success: true,
      version: 'v2',
      data: products,
      pagination: {
        page, limit, total,
        pages: Math.ceil(total / limit),
        hasNext: page * limit < total
      },
      meta: {
        requestId: req.id,
        responseTime: Date.now() - req.startTime,
        cached: false
      }
    });
  }
);

app.listen(3000);

Summary Table

Concept	Purpose	Key Tools	Performance Impact	When to Use
Pagination	Split large datasets	LIMIT/OFFSET, cursor	Reduces load	>100 records
Caching	Speed up repeated requests	Redis, node-cache, CDN	10–1000x faster	Read-heavy data
API Versioning	Maintain compatibility	URL path, headers	Minimal	Breaking changes
Logging	Debugging & monitoring	Winston, Morgan, Pino	1–5% overhead	Always

Pro Tips

Pagination: Use cursor-based for real-time data, offset for admin panels
Caching: Always set TTL, implement cache invalidation strategy
Versioning: Start with URL versioning (easiest), deprecate old versions after 6–12 months
Logging: Use structured JSON logs, include request IDs, never log passwords/PII

Express.js Advanced Topics

Table of Contents

1. Pagination

Why it matters

Pagination strategies

2. Caching

Why it matters

Caching levels

Cache invalidation strategies

3. API Versioning

Why it matters

Versioning strategies

4. Logging

Why it matters

What to log

Log levels (severity)

Combined Example: Complete API with All Concepts

Summary Table

Pro Tips