Evals, safety & limits

Final lesson of module one: how the field keeps score, what the guardrails do and don't do, and the failure modes you must design around.

How models are measured

• Knowledge & reasoning benchmarks (MMLU, GPQA) — multiple-choice exams. Cheap, saturated, and gameable: training on near-duplicates of test items (contamination) inflates scores without improving ability.
• Agentic benchmarks (SWE-bench) — can the model fix real issues in real repos, judged by tests? Far more informative for coding tools — and far more expensive to run.
• Human preference arenas — blind A/B votes on open prompts. Capture taste and usefulness, inherit human biases (verbosity wins more often than it should).

Read any single number with the recipe in mind: what task, what judge, what contamination controls? A 5-point gap means nothing without that context.

Safety: what guardrails are

Refusals, content filters, and safety fine-tuning shape which behaviors surface — they don't remove the underlying capabilities (recall the training lesson: manners, not brain surgery). Jailbreaks — prompts engineered to bypass refusals — work often enough that you should treat any safety property as probabilistic, never as a guarantee your system can rely on.

Never build a control that fails open on model obedience. If harm follows from the model complying, the control belongs outside the model — in code, permissions, and human review.

Limits to design around

• Hallucination is structural. Fluency ≠ truth. Anything load-bearing (prices, doses, commands that delete) gets verified against a source of truth, not a second sampling.
• Long context degrades judgment. Retrieval survives scale better than reasoning does — models find the needle and still fumble the synthesis. Keep the working set small; summarize or retrieve rather than dumping.
• Tool use needs supervision. Agents that can act (shell, payments, merges) need the same controls as junior engineers with production access: scopes, dry-runs, and review gates on irreversible actions.