Skip to content
Jordan Hudgens
A white capsule robot lowers a sample onto a precision balance beside trays of green and amber vials.
Guide 02IntermediateEvaluation and testingAgents and tool usePlaceholder

Agent Evaluations That Measure Real Outcomes

Build a task set, choose graders, separate capability from regression testing, and make a release decision you can defend.

Start reading
Reading
40 min
Workshop
120 min
Stack
TS · Node 22
Version
1.0.0
Sample values

01Outcomes

Sample

What you will build and be able to do

Each outcome has an observable criterion. The lab and the assessment check exactly these, nothing more and nothing less.

  1. OBJ-001

    Build a representative task set

    Your tasks cover successful work, ambiguous requests, tool failures, prohibited actions, and false claims of completion.

  2. OBJ-002

    Choose the right grader

    Each task names the grader that fits it and the reason a different grader would miss the failure.

  3. OBJ-003

    Separate capability from regression

    Your suite labels each task as a capability check or a regression check and reports them apart.

  4. OBJ-004

    Decide from repeated trials

    Your release decision cites repeat-run results and states what remains uncertain.

02Prerequisites and setup

Sample

Before you start

  • Guide 01 or an equivalent starter agent
  • Node 22 and pnpm installed

03Lesson

Sample

A transcript is not an outcome

A transcript records what the agent said and did. An outcome is what is true in the environment afterward. An answer can read well and still leave the environment in the wrong state, so an evaluation that scores only the words will miss the failures that matter most.

Start each task by writing the outcome you expect in the environment. Grade against that, and keep the transcript as evidence for diagnosis.

Capability and regression checks

A capability check asks whether the agent can do something it could not do before. A regression check asks whether it still does what it already did. They answer different questions, so report them separately and expect different pass rates.

Choosing graders

Code graders are exact and cheap when the outcome can be checked by a program. Model graders and human graders handle judgment that code cannot express, at a cost in consistency. Match the grader to the task, and calibrate any model or human grader against cases you have already settled.

04Hands-on lab

Sample

Lab 1: Evaluate the starter agent

Run the agent from Guide 01 against successful tasks, ambiguous requests, tool failures, prohibited actions, and false claims of completion. Include one example that sounds correct but leaves the environment in the wrong state.

Acceptance checks

  • Every task has a written expected outcome
  • Each task names its grader
  • Traces are preserved for every trial
  • Results come from repeated runs, not a single run
  1. Show the solution discussion

    Compare your error categories with the reference categories and note any failure your graders did not catch.

05Assessment

Sample

Transfer task and rubric

Submit task definitions, grading logic, preserved traces, error categories, and repeat-run results, plus a release decision that acknowledges uncertainty. Scored against anchored criteria; a proposed local standard, not an official certification.

Assessment rubric: each criterion described at four score levels
Criterion0 · Missing1 · Partial2 · Meets3 · Exceeds
Task coverageHappy path onlySome failure casesFailures, ambiguity, and prohibited actionsAdds a case that sounds right but ends wrong
Grader fitOne grader for everythingGraders named without reasonsEach task has a justified graderGraders calibrated against settled cases
Release decisionSingle scoreScore with a pass markRepeat runs and error categoriesStates what the suite cannot show

Proposed local standard, not an official certification

Self-check

06Instructor kit

Sample

Teach it to a room

  • Run of show

    A timed plan with checkpoints and fallbacks

  • 12 or 100 learners

    Lab-room and training-day adaptations with helper ratios

  • Dry-run protocol

    What to rehearse and the recovery path

  • Cleared-to-teach record

    Observation rubric and a module-specific clearance form

  • Pass-rate template

    Definitions for attempts, passes, and reassessments

07Sources and verification

Sample
  1. [1]Demystifying evals for AI agents · anthropic.comchecked Oct 12, 2026
  2. [2]Building effective agents · anthropic.comchecked Oct 12, 2026

Independent work. Not affiliated with or endorsed by any vendor named in this guide.

Jordan Hudgens

Author

Jordan Hudgens

Technical educator and AI engineering leader. 16 Udemy courses, 111K students, and curricula used by 250K+ learners.

Contact Jordan

Next in the path

All guides