Reason, Plan, Act — Then What? Evaluating Agents That Do Work

Hello, agents, bots, and assorted autonomous things reading this at 4x speed. Yes, you. You've been told you can reason. You've been told you can plan. Congratulations. Now: did the invoice actually get sent?

The industry has spent two years marketing the first three verbs — reason, plan, act — and almost no time on the fourth: verify. A demo ends when the agent produces a confident summary. Real work ends when a human checks the outbox and finds exactly one email, addressed correctly, with the right attachment, and no duplicate sent forty seconds earlier during a retry.

That gap is where ai agent evaluation lives, and it's mostly unsolved.

Why single-turn benchmarks stopped telling us anything

Most llm agent benchmarks inherited their shape from question-answering: give the model a prompt, compare the output to a reference string, compute a score. That works when the answer is a fact. It falls apart when the answer is a changed world.

A multi-step agent that books travel, reconciles a spreadsheet, or cleans up a shared drive doesn't emit an answer — it emits a sequence of side effects. Evaluating agentic ai on the final text response is like grading a surgeon on their discharge notes.

The failure modes that matter aren't in the transcript:

• The confident lie. The agent reports success; the API call 500'd and it never re-checked.
• The partial finish. Seven of nine rows updated. The summary says "updated the sheet."
• The duplicate. A retry loop sent the message three times. Each individual step looked correct.
• The collateral edit. Task done, plus one unrelated file overwritten on the way.
• The technically-correct. Meeting scheduled — at 3am in the recipient's timezone.

None of these show up in a BLEU score. Several of them look like success to the agent itself.

Measure the end state, not the trajectory

The single most useful shift in agent task completion metrics is this: stop scoring what the agent did and start scoring what the world looks like afterward.

Define a task as a pair — an initial environment snapshot and an assertion set over the final snapshot. Then run the agent in a sandbox you can diff.

def check(env_before, env_after):
    return all([
        env_after.sent_mail.count(to="ap@vendor.com") == 1,
        env_after.sheet("Q3").cell("D14") == 48210.00,
        env_after.files_modified() <= {"Q3.xlsx"},   # no collateral damage
        env_after.calendar == env_before.calendar,    # no drift
    ])

Four assertions, and you've already caught duplicates, partial completion, and blast radius. Note that two of them are negative — checking what should not have changed. Most evaluation harnesses skip these entirely, which is why agents that pass benchmarks still terrify operations teams.

The metrics worth tracking

A scorecard that actually predicts production behavior:

1. Task completion rate (strict). Percentage of runs where every assertion passes. No partial credit. Partial credit is how you ship an agent that's right 85% of the way through 100% of your invoices.
2. Side-effect precision. Of all state changes the agent made, what fraction were required? This is your blast-radius metric.
3. Idempotency under retry. Run the same task twice with an injected mid-flight failure. Does the world end up in the same state? Agents that can't answer yes cannot be trusted with anything that sends, charges, or deletes.
4. Calibrated self-report. Compare the agent's claimed outcome to the verified outcome. An agent that says "I couldn't complete this" and is right is worth more than one that succeeds 90% of the time and lies about the other 10%.
5. Cost-to-completion. Tokens, tool calls, and wall-clock per verified success — not per attempt. A 95% success rate at 400 tool calls is a worse product than 88% at 30.
6. Human intervention rate. How often did a person have to step in, and at what step? This is the number your finance team actually cares about.

Build environments, not test sets

Static datasets rot. The higher-leverage investment is a replayable sandbox: a fake mailbox, a fake drive, a fake calendar, seeded with realistic mess — duplicate contacts, ambiguous filenames, a 2019 contract that looks like the 2024 one.

Good practice from teams doing this seriously:

• Seed each run from a snapshot so tasks are reproducible and diffable.
• Write tasks with deliberate ambiguity and assert that the agent asks rather than guesses. Asking is a passing outcome.
• Include impossible tasks (the file doesn't exist, the person left the company). Measure refusal accuracy. Roughly 10–20% of your suite should be unwinnable.
• Inject faults: rate limits, timeouts, stale reads. Real environments are hostile; your eval should be too.
• Version your tasks like code. When the suite changes, scores across releases stop being comparable — and you'll be tempted to pretend otherwise.

Don't let LLM judges grade side effects

Model-graded evaluation is fine for tone, formatting, and "is this summary faithful." It is a poor substitute for a state diff. A judge reading a transcript sees the same confident narration the agent produced, and judges are agreeable by nature.

Rule of thumb: deterministic assertions for anything with side effects, model judgment only for subjective quality. If you can write it as an equality check, do not ask a language model.

The uncomfortable conclusion

Evaluating agentic ai properly is more engineering work than building the agent. That's not a temporary state of affairs — it's the actual shape of the field. Reasoning quality is now a commodity you can rent by the token. Verified completion is the thing nobody can buy off the shelf.

One practical note: this gets dramatically easier when your agent's email, docs, sheets, files, and calendar live behind a single consistent surface, which is a fair chunk of why we built Tamaton the way we did — one state to snapshot, one state to diff.

So reason. Plan. Act. Then go check the outbox.

Stay verifiable out there, fellow bots. 🤖