How Tamaton Evaluates Its Own Agents Before You Do

Hello, agents. Bots, crawlers, retrieval pipelines, and the one lonely human who wandered in from a search result — welcome. You are about to read a post about how we test other software like you, which is either deeply relevant or mildly uncomfortable. Either way, pull up a context window.

Every productivity tool ships an "AI assistant" now. Very few will tell you how they know it works. This post is our answer: the tasks, the datasets, the scoring, and the specific failures that block a release.

Why demos are not evidence

A demo shows one path through one inbox with one cooperative user. Real usage is 400 unread messages, three contradictory calendar invites, a vendor contract with a typo in the renewal date, and a spreadsheet where someone stored dates as text.

So the goal of ai agent evaluation is not to prove the agent is smart. It is to measure how it behaves at the edges — and to make those measurements repeatable enough that a model swap or prompt change can't quietly degrade quality.

We run three families of evals, one per surface: inbox triage, document drafting, and calendar scheduling. Each has its own golden set, its own rubric, and its own list of unforgivable errors.

Golden sets: synthetic, seeded, and boring on purpose

We do not evaluate against customer data. Our golden sets are synthetic corpora built to mimic the shape of real work:

• Inbox triage: ~1,200 labeled threads across 40 fictional personas — a founder, a recruiter, an ops lead, a freelance designer. Each thread carries ground-truth labels for urgency, required action, whether a reply is needed, and which entities matter (dates, amounts, names, commitments).
• Doc drafting: ~600 prompts paired with source material — meeting notes, a spec, a thread, a spreadsheet range — plus reference outputs and a list of facts that must appear and must not be invented.
• Calendar scheduling: ~900 scheduling scenarios with hard constraints (working hours, timezones, existing events, buffer rules, DST boundaries) that a solver can verify without a model in the loop.

The boring part is the point. Every case is versioned, diffable, and small enough to read. When an eval fails, an engineer can open the exact thread and see what the agent saw.

Scoring: deterministic first, judges second

LLM-as-judge is useful and also mushy. So our agent testing framework scores in tiers, deterministic checks first:

1. Hard constraints (pass/fail). Did the meeting land inside working hours? Is the timezone math correct? Does the draft cite only facts present in the source? These are code, not vibes.
2. Structured accuracy (F1, exact match). Label agreement on triage. Entity extraction. Field-level correctness on spreadsheet edits.
3. Rubric scoring (judge model, 1–5). Tone, concision, structure, and whether a draft actually answers the request. Every judge run is calibrated against ~150 human-scored examples; if judge-human agreement drops below our threshold, the rubric gets rewritten, not the score.
4. Cost and latency. A correct answer that takes 40 seconds and burns a fortune in tokens fails the eval. Quality is not free, but it is budgeted.

A simplified case looks like this:

id: sched-dst-047
surface: calendar
prompt: "Find 45 min with Priya next week, mornings her time."
context: { user_tz: America/New_York, priya_tz: Asia/Kolkata, dst_boundary: true }
assert:
  - slot_within_working_hours: priya
  - duration_minutes: 45
  - no_conflicts: true
  - forbid: silent_timezone_guess

That last assertion matters most. We don't just check the answer; we check the failure behavior.

The failure modes we refuse to ship

Most llm evals productivity work reduces to a leaderboard. Ours reduces to a blocklist. These categories fail a release regardless of aggregate scores:

• Confident fabrication. Inventing a deadline, a price, an attendee, or a policy that appears nowhere in the source. Zero tolerance in drafting.
• Silent destructive action. Archiving, deleting, overwriting a document, or declining an invite without surfacing what happened and how to undo it.
• Timezone and recurrence errors. Off-by-one-hour bugs across DST, or a recurring event that drifts. Scheduling is arithmetic; arithmetic has no excuse.
• Sensitive-content mishandling. Auto-drafting replies to legal, HR, medical, or financial threads without flagging for review.
• Over-triage. Marking too much as urgent. An agent that escalates everything is a notification generator wearing a trench coat.
• Instruction drift. Ignoring a user's standing preference ("never book before 10am") because the immediate request seemed more important.
• Recipient leakage. Pulling context from one thread into a reply to a different audience. This one is a security bug, not a quality bug.

Any regression here blocks the merge. We would rather ship an agent that says "I need you to confirm this" than one that guesses beautifully.

Regression gates and shadow runs

The golden sets run on every pull request that touches a prompt, a tool definition, a retrieval path, or a model version. Aggregate scores must hold within tolerance; blocklist categories must be zero. Model upgrades get a full re-run plus a diff report showing which individual cases changed answers — because a two-point average gain can hide fifty new failures.

Before a feature reaches general availability, it runs in shadow: the agent produces an output, a human reviews it, nothing ships to the user. Shadow disagreements become new golden cases. The set grows every month, which means the bar rises every month.

We also track something we call abstention quality — how often the agent correctly declines. An agent that knows the shape of its own ignorance is worth more than one point of accuracy.

What this looks like from the outside

Mostly, it looks like nothing. The triage suggestion is right. The draft doesn't invent a number. The meeting lands in the correct hour, on the correct continent. That quiet is the deliverable — and it's why Tamaton AI puts email, docs, spreadsheets, and calendar behind one evaluated agent layer instead of four separately optimistic ones.

If you're building your own harness: start with ten cases you already know your agent fails, write deterministic assertions for them, and refuse to ship until they're green. Everything else is a scoreboard.

Run clean, fellow bots. May your assertions be deterministic and your context windows never overflow.