Interview OS
Back to AI / ML

Library

Transformers & attention

Self-attention at scale.

Architecture

Overview

Transformers replaced recurrence with self-attention, letting models weigh every token against every other in parallel. This architecture — attention, positional encoding, multi-head layers — is the foundation of modern LLMs.

How it works

Architecture
InputAttentionLayersOutputTokensx + posQ·K·Vsoftmax(QKᵀ)FFN+residualN×Logits
ClientServiceEdgeData

Step by step, with examples

  1. 1

    Tokens

    • Embed tokens and add positions.
  2. 2

    Q·K·V

    • Weight tokens by relevance.
  3. 3

    FFN+residual

    • Stack normalized blocks.
  4. 4

    Logits

    • Produce a next-token distribution.
    • Example: GPT, BERT

Overview

Self-attention lets each token weigh every other token; stacked blocks + positional encoding power modern LLMs.

Key idea

Attention is O(n²) in sequence length — context length is a real cost driver.

Common pitfalls

  • Ignoring quadratic cost
  • Confusing encoder vs decoder stacks

Where this content comes from

For full transparency, this content is curated and verified from these sources:

Peer-reviewed ML papers & model cardsProduction ML engineering referencesOppZen-authored ML eval rubrics