Interview OS
Back to AI / ML

Library

LLM evaluation

Beyond accuracy.

Eval

Overview

Evaluating LLMs is hard because outputs are open-ended. This topic covers benchmark suites, LLM-as-judge, human evals, and task-specific rubrics to measure quality, safety, and regressions reliably.

How it works

Eval
TaskRunJudgeOutputTest setdatasetGeneratebatchAuto+humanrubricScores
ClientServiceEdgeData

Step by step, with examples

  1. 1

    Test set

    • Golden prompts and rubrics.
  2. 2

    Generate

    • Collect model outputs.
  3. 3

    Auto+human

    • LLM-judge, exact match, and humans.
  4. 4

    Scores

    • Track and gate regressions.
    • Example: hallucination rate

Overview

Evaluate LLMs with task rubrics, golden sets, LLM-as-judge, and human review — track hallucination, helpfulness, and safety.

Key idea

Build a small, versioned eval set before shipping any prompt change.

Common pitfalls

  • Vibes-based eval
  • Judge model bias
  • No regression set

Where this content comes from

For full transparency, this content is curated and verified from these sources:

Peer-reviewed ML papers & model cardsProduction ML engineering referencesOppZen-authored ML eval rubrics