Interview OS
Back to System Design

System Design

Rate limiter

Senior level · full staged walkthrough

Senior

Architecture

Senior
ClientGatewayCountersConfigfast pathsynclimitsallow / 429RequestGateway middlewarecheck limitLocal cachepre-checkRedis counterstoken bucketConfig serviceper-tier limits
ClientEdgeServiceData

Solution, step by step

  1. 1

    Functional requirements

    • Limit requests per user/API key
    • Configurable limits per route/tier
    • Return clear 429 with Retry-After
  2. 2

    Non-functional requirements

    • Decision latency < 1ms added overhead
    • Accurate across a distributed fleet
    • Fail-open vs fail-closed policy defined
    • Highly available
  3. 3

    Capacity & estimation

    • 1M req/s across the fleet → counter store must handle 1M+ ops/s
    • Per-key counters with TTL; memory ≈ active keys × ~100 bytes
    • Sliding window needs more memory than fixed window
    • Edge nodes: thousands
  4. 4

    Preliminary design

    • Token bucket / sliding-window counter per key
    • Atomic increment + TTL in a shared store
    • Reject with 429 when the limit is exceeded
  5. 5

    Final architecture

    • Gateway middleware checks limits before forwarding
    • Redis (or in-memory + sync) holding token buckets with Lua for atomicity
    • Local pre-check + async reconcile to cut central round-trips
    • Config service pushes per-tier limits; hot-reload
    • Sliding-window-log or token-bucket algorithm by accuracy need

Interview Q&A (8)

Token bucket for smooth bursts with a refill rate; sliding-window-log or counter when you need accuracy across window boundaries at higher memory cost.

Key components

  • Gateway middleware
  • Redis counter store
  • Config service for limits

Bottlenecks & how to address them

  • Central counter round-trip latency → local token cache
  • Hot keys → shard counters
  • Clock skew in sliding windows

Tradeoffs to articulate

  • Fixed window (bursty) vs sliding window (accurate, costlier)
  • Local vs centralized counters
  • Accuracy vs latency under load

Where this content comes from

For full transparency, this content is curated and verified from these sources:

Published architecture case studiesCompany engineering blogsOppZen design rubric library