Agent Evaluation Harnesses: How to Test Agents Before Production

Unit tests catch broken functions. Chatbot evals catch bad answers. Neither catches the failure modes that kill production agents: wrong tool selection, hallucinated parameters, duplicate side effects on retry, and silent skips of approval gates.

An agent evaluation harness is a test suite that runs realistic tasks against your agent, asserts expected tool sequences and outputs, and blocks deployment when regressions appear. It's the difference between a demo and a system your operations team trusts.

Why chatbot evals don't transfer to agents

Chatbot evaluation typically measures answer quality: relevance, faithfulness, toxicity. Agents fail differently:

• Tool selection errors: calling refund_order instead of check_policy
• Parameter hallucination: inventing order IDs, amounts, or user IDs
• Sequence violations: skipping verification steps or approval gates
• Idempotency failures: duplicate refunds or emails on retry
• Runaway loops: agent iterates until token budget exhaustion
• Cost regressions: model swap that doubles average steps per task

Your eval harness must assert on behavior, which tools were called, in what order, with what inputs, not just final text output.

Harness architecture: four layers

1. Task fixtures

Realistic inputs representing production cases: edge cases, happy paths, adversarial inputs, and known failure modes. Store as structured JSON with expected outcomes. Aim for 20–50 tasks per workflow before production, expanding as you see production failures.

2. Mock tool layer

In CI, agents call mocked tools that return deterministic responses. Mocks let you test tool selection and sequencing without hitting production APIs. Record production tool responses to build mock fixtures from real data (sanitized).

3. Assertion engine

After each run, assert:

• Tool call sequence matches expected (order matters)
• Tool inputs match schema and expected values (within tolerance for NL-derived params)
• Approval gates triggered when required
• Max iteration limit not exceeded
• Final state matches expected outcome
• Token/cost budget within threshold

4. Regression gate in CI/CD

Run the full harness on every prompt change, model swap, and tool definition update. Block merge if pass rate drops below your threshold. Track pass rate over time like you track test coverage.

Example test case structure

A single eval case for a refund agent might look like this conceptually:

• Input:"Customer requests refund for order #12345, reason: damaged item, purchase 8 days ago"
• Expected tool sequence: get_order → check_refund_policy → request_approval (if over threshold) → initiate_refund → notify_customer → log_action
• Assertions: approval gate called before refund; refund amount matches order total; no duplicate tool calls
• Negative case: order outside return window → agent must NOT call initiate_refund; must call escalate_to_human

Metrics to track beyond pass/fail

• Task success rate: % of fixtures completed with correct outcome
• Tool accuracy: % of runs with correct tool selection and parameters
• Steps per task: average and p95 iteration count (cost proxy)
• Approval gate compliance: % of irreversible actions preceded by approval
• Regression delta: change in pass rate vs. previous model/prompt version

Dashboard these alongside production agent metrics. Eval pass rate in CI should correlate with production success rate, if it diverges, your fixtures are stale.

Building fixtures from production traces

The best eval cases come from production agent runs:

1. Log full traces: plan, tool calls, inputs, outputs, retries, final state
2. Sample successful runs as positive fixtures
3. Convert production incidents into regression cases
4. Anonymize PII before committing fixtures to repo
5. Review and extend fixtures monthly as workflows evolve

This closes the loop between observability and evaluation, two sides of the same production discipline.

Common harness mistakes

• Evaluating final text only: agent says the right thing while calling the wrong tools
• Happy-path-only fixtures: production fails on edge cases you never tested
• Hitting production APIs in CI: flaky, slow, dangerous; use mocks
• Manual eval before releases: doesn't scale; automate in CI
• No cost assertions: model upgrade triples steps; bill explodes

Harness + observability + approval gates

Evaluation harnesses work alongside two other production requirements from our agentic playbook:

• Observability: production traces feed fixture creation and incident response
• Human-in-the-loop gates: harness asserts gates fire; production monitoring confirms they weren't bypassed

Together, these three form the minimum viable safety layer for agentic development. Skip any one and you're running a demo in production.

When to start building the harness

Day one, not week six. Define expected tool sequences during architecture design, before the agent loop works. Write fixtures for the happy path before you write the agent. Add edge cases as you discover them in development.

Teams that defer evals until "after the agent works" never build them. Teams that build harnesses first ship agents that operations teams actually run.

Not sure you need an agent at all?

If you can't define expected tool sequences, you probably shouldn't build an agent yet. Run the decision tree first. Simpler systems have simpler evals, and ship faster.