What you’ll learn
- Build representative evaluation sets with contamination controls.
- Compare models and prompts against meaningful quality and reliability criteria.
- Trace failures while minimizing sensitive-data logging.
Proposed course content
6 sections · 13 illustrative lesson records
Evaluation design3 lessons
- Introduction to LLM Evaluation, Reliability & Observability
Design repeatable LLM evaluations, failure-aware application behavior and privacy-conscious observability with clear quality and resource tradeoffs.
Public introduction · Video not supplied - Define task risks and representative test cases
Define task risks and representative test cases. Module: Evaluation design.
Enroll to access · Video not supplied - Separate development and held-out evaluation data
Separate development and held-out evaluation data. Module: Evaluation design.
Enroll to access · Video not supplied
Metrics and review2 lessons
- Choose task-specific automated and human review criteria
Choose task-specific automated and human review criteria. Module: Metrics and review.
Enroll to access · Video not supplied - Inspect evaluator bias and agreement limits
Inspect evaluator bias and agreement limits. Module: Metrics and review.
Enroll to access · Video not supplied
Reliability engineering2 lessons
- Set timeouts, retries and bounded fallbacks
Set timeouts, retries and bounded fallbacks. Module: Reliability engineering.
Enroll to access · Video not supplied - Test schema failures and provider unavailability
Test schema failures and provider unavailability. Module: Reliability engineering.
Enroll to access · Video not supplied
Tracing and privacy2 lessons
- Capture safe request metadata and tool spans
Capture safe request metadata and tool spans. Module: Tracing and privacy.
Enroll to access · Video not supplied - Redact sensitive content and define retention
Redact sensitive content and define retention. Module: Tracing and privacy.
Enroll to access · Video not supplied
Regression and comparison2 lessons
- Run reproducible model/prompt comparisons
Run reproducible model/prompt comparisons. Module: Regression and comparison.
Enroll to access · Video not supplied - Track quality, latency and resource consumption
Track quality, latency and resource consumption. Module: Regression and comparison.
Enroll to access · Video not supplied
Release criteria2 lessons
- Define evidence-based acceptance and rollback conditions
Define evidence-based acceptance and rollback conditions. Module: Release criteria.
Enroll to access · Video not supplied - Document uncertainty and monitoring blind spots
Document uncertainty and monitoring blind spots. Module: Release criteria.
Enroll to access · Video not supplied
Requirements
- Experience building model-backed applications and automated tests.
- Basic statistics, experiment design and authorized evaluation datasets.
Tools and resources
- A development computer with resources appropriate to the proposed local exercises.
- Python or TypeScript, a test/evaluation runner and permitted model interfaces.
- A local trace store, synthetic datasets and statistical analysis tools.
Description
Design repeatable LLM evaluations, failure-aware application behavior and privacy-conscious observability with clear quality and resource tradeoffs.
Who this course is for
Experienced developers evaluating model-backed systems and their operational limits.
Example practical project
Proposed capstone: an evaluation harness for a synthetic-document assistant with failure fixtures, privacy-aware traces and a regression report.