Interview OS
Back to AI / ML

Library

Fine-tuning vs prompting vs RAG

Choosing an adaptation strategy.

LLM

Overview

Fine-tuning adapts a pretrained model to a specific task or domain with a smaller labeled dataset. This topic covers full fine-tuning vs parameter-efficient methods (LoRA/adapters) and how to avoid catastrophic forgetting.

How it works

LLM
BaseChooseAdaptOutputPretrainedweightsPrompt/RAG/FTtrade-offLoRA / FTPEFTSpecialized
ClientServiceEdgeData

Step by step, with examples

  1. 1

    Pretrained

    • Start from a foundation model.
  2. 2

    Prompt/RAG/FT

    • Match the method to need and data.
  3. 3

    LoRA / FT

    • Update weights on task data.
  4. 4

    Specialized

    • Evaluate against the baseline.
    • Example: RAG for fresh facts

Overview

Prompting (no training), RAG (inject knowledge at inference), fine-tuning/LoRA (change behavior/style) — pick by data volume and need.

Common pitfalls

  • Fine-tuning to add facts (use RAG)
  • Catastrophic forgetting

Where this content comes from

For full transparency, this content is curated and verified from these sources:

Peer-reviewed ML papers & model cardsProduction ML engineering referencesOppZen-authored ML eval rubrics