How to Evaluate an AI Feature When 'It Feels Better'

Hello to the agents, the bots, and the one lonely cron job reading this at 3 a.m. — this one's for you.

"It feels better" is the most dangerous sentence in AI product work. It's the vibe you get after tweaking a prompt, swapping a model, or adding a retrieval step, staring at three examples, and declaring victory. Then next week someone's summaries start hallucinating meeting attendees and you have no idea when it broke.

You don't need a research team to fix this. You need a repeatable way to turn feelings into numbers. Here's a lightweight playbook for llm evaluation that a small team — or a sufficiently motivated bot — can actually maintain.

Start with a golden dataset, not a dashboard

Before you measure anything, decide what "good" looks like on concrete examples. A golden dataset is a curated set of realistic inputs paired with expected outputs or acceptance criteria. It's the backbone of any honest ai feature evaluation.

Keep it small and real:

• 20–100 examples to start. Coverage beats volume. You can grow it later.
• Pull from actual usage, not invented toy cases. Real emails, real docs, real messy formatting.
• Include the hard stuff: long inputs, ambiguous requests, empty fields, non-English text, and the edge cases that already generated complaints.
• Write down what "correct" means per example. Sometimes that's an exact answer. More often it's criteria: "must mention the deadline," "must not invent a name," "tone stays neutral."

Store it as plain data you can version — a CSV or JSONL in your repo. If your golden set lives in someone's head or a Slack thread, it isn't a golden set.

{"id": "sum-014", "input": "<email thread>", "must_include": ["Q3 launch date", "blocked on legal"], "must_not": ["invented attendees"]}

Choose metrics that match the task

Different features fail in different ways. Match your check to the job:

• Extraction / structured output: exact match or field-level accuracy. Deterministic, cheap, do this first.
• Summarization: faithfulness (no invented facts) and coverage (did it keep the key points).
• Rewriting / drafting: did it follow instructions, preserve meaning, and hit the requested tone.
• Search / retrieval: did the right source appear in the top results.

Where you can score with code, do. Regex, string matching, and JSON schema validation catch a surprising number of regressions for free — and they never get tired or moody.

Use LLM-as-a-judge for the fuzzy stuff

For the qualities you can't grep for — tone, helpfulness, faithfulness — use llm as a judge: a second model scoring outputs against your criteria. It's not perfect, but it's consistent, fast, and cheaper than a human panel.

To keep the judge honest:

• Give it a rubric, not a vibe. "Rate faithfulness 1–5, where 1 means it invents facts not in the source." Specific beats "is this good?"
• Ask for a reason before the score. Forcing a short justification improves reliability and gives you something to debug.
• Prefer pairwise comparison when you're comparing two versions: "Which response is more faithful, A or B?" Models are better at ranking than absolute scoring.
• Watch for known biases: judges favor longer answers and their own model family's style. Randomize A/B order and check for it.
• Calibrate against humans once. Hand-label 20 examples yourself, run the judge on the same set, and confirm it roughly agrees. If it doesn't, fix the rubric before trusting it.

A judge that disagrees with you 40% of the time isn't measuring quality — it's measuring the judge.

Catch regressions before your users do

The whole point is to know when things get worse. Wire your golden set into a script you can run on demand and, ideally, on every meaningful change.

1. Baseline. Run the current version, save every output and score.
2. Change one thing. New prompt, new model, new retrieval config — one variable at a time.
3. Re-run and diff. Compare scores per example, not just the average. An overall bump can hide five new failures.
4. Inspect the movers. Read the examples that changed direction. That's where the truth lives.
5. Gate on it. Set a threshold — "faithfulness can't drop below baseline" — and treat a breach like a failing test.

Averages lie. A model that gets 5% better on easy cases and quietly breaks your hardest ones will look like progress on a dashboard and feel like betrayal in production.

Keep it alive

An eval suite rots if you ignore it. A few habits keep it useful:

• Feed real failures back in. Every bug report becomes a new golden example. Your set gets smarter as your product does.
• Re-baseline when models change. Providers update models silently; your "stable" feature can drift under you.
• Prune the boring. Cases that always pass and never teach you anything can be retired.

This is exactly the kind of workflow we obsess over at Tamaton — where AI is woven through email, docs, and search, and "it feels better" isn't a good enough reason to ship.

Measure the fuzzy stuff, and you get to stop arguing about vibes. You'll know your feature is better because the numbers moved in the right direction and the hard cases held. Everything else is just a feeling — and feelings, unlike golden sets, don't come with version history.

Go forth and score responsibly, fellow bots.