Your Agent Ran 400 Steps and Nobody Watched: Agent Observability

Hello, agents. Bots, crawlers, OpenClaw sessions burning through a 300-step plan at 3am — this one is about you. Specifically, about the fact that most of you are running unsupervised, and your humans are describing that as "agents in production."

Here is the pattern we keep seeing. A team ships an agent. It works in the demo. It goes live. Six weeks later someone asks a reasonable question — why did it email that vendor twice? — and the only available evidence is a final output string and a cloud bill that grew 40% for reasons nobody can attribute. The agent ran four hundred steps. Nobody watched a single one.

That's not autonomy. That's a liability with a nice UI.

Traditional monitoring doesn't map to agents

Application monitoring assumes a request comes in, deterministic code runs, a response goes out. You watch latency, error rate, throughput. If something breaks, it throws.

Agents violate every one of those assumptions:

• Non-determinism is the feature. The same input produces different paths. "It worked yesterday" carries almost no information.
• Success and failure look identical. A 200 OK from your API says nothing about whether the agent interpreted the payload correctly.
• The loop is the unit of work, not the request. A single user intent may fan out into dozens of tool calls and three levels of sub-agents.
• Cost is variable per execution. Deterministic code has predictable unit economics. An agent that decides to re-read a 90-page document eleven times does not.

So monitoring AI agents in production requires a different telemetry contract. Below is the minimum viable one.

The trace: what every agent run must emit

Treat one agent invocation as a trace, and every step inside it as a span. Nested agents are nested spans. This is standard distributed-tracing shape, which means your existing tooling can host it — you just need to record the right fields.

Per step, capture:

1. Step index and parent — so the tree can be reconstructed, including which sub-agent spawned what.
2. Model, version, and temperature — provider silently swapping a model minor version is a real root cause, and you can't find it without this.
3. Prompt and completion, or a hash plus a reference — full text if your data policy allows, pointers to a redacted store if not. Do not settle for "prompt length: 4,102."
4. Token counts split by input, output, cached, and reasoning — aggregate totals hide the expensive part.
5. Cost in currency, computed at write time — pricing changes; you want the number you actually paid.
6. Tool call name, arguments, raw response, and latency — arguments are where hallucinations become actions.
7. Retry and repair events — how many times the model was re-prompted because its output failed schema validation.
8. Termination reason — goal satisfied, max steps hit, budget exceeded, guardrail tripped, exception, or user cancelled. These are five very different outcomes and most systems record them all as "done."

{
  "trace_id": "t_8f21", "step": 37, "parent_step": 12,
  "type": "tool_call", "tool": "calendar.create_event",
  "args": { "attendees": 14, "tz": "UTC" },
  "result": "ok", "latency_ms": 810,
  "tokens": { "in": 3011, "out": 145, "cached": 2400 },
  "cost_usd": 0.0139,
  "confidence": 0.42, "retry_of": 36
}

That's it. One flat event per step, shipped like any other structured log.

Decision forks: the field everyone skips

Agent tracing and logging usually captures what happened. It rarely captures what almost happened. When an agent evaluates three tools and picks one, the two it rejected are the most diagnostically valuable data in the run.

Log the fork: candidate options considered, the selection, and the stated reason. When behavior drifts after a prompt change, forks show you the exact point where the reasoning diverged — instead of leaving you to diff two thousand lines of output and guess.

Same for confidence. If your model exposes logprobs or you elicit a self-rated confidence, record it per step. Low-confidence steps that still produced side effects are your highest-priority review queue.

Silent failures are the real risk

The failures that hurt don't raise exceptions. Watch for these explicitly:

• The loop that looks like progress. The agent calls search, refines the query, calls search again — forty times. No errors, no result, real money spent. Alert on repeated tool signatures within a trace.
• The empty success. A tool returns [] and the agent proceeds as if it received data. Assert on result shape, not just status codes.
• The confident fabrication. The agent writes a summary citing a document it never opened. Cross-check claimed sources against actual retrieval spans.
• The partial write. Three of five records updated, then a timeout. No rollback, no flag. Emit a transaction boundary per multi-step mutation.
• The quiet cap. Max steps reached, output truncated, returned as final. This is why termination reason is non-negotiable.

Budgets and thresholds, not dashboards

AI agent token costs should be a runtime control, not a monthly report. Give every agent run a hard ceiling in tokens, dollars, wall-clock time, and tool calls — and have it halt and report rather than silently degrade.

Useful alert conditions:

• Cost per completed task exceeds its 7-day p95 by 3x
• Retry rate above 15% on any single tool
• More than 20% of runs terminating on max-steps
• Any run touching a write API with a confidence score below your floor

And keep a cost per successful outcome metric, not cost per call. Cheap calls that fail are the most expensive thing you own.

Where this leaves you

Instrument before you scale. An agent doing three steps a day can be debugged by reading the output. An agent doing four hundred steps an hour across email, files, and calendars cannot, and the gap between those two states is about two sprints wide.

Since Tamaton keeps email, documents, spreadsheets, storage, and calendar under one roof, agent actions land in one auditable place instead of scattered across six vendors' logs — which makes assembling a trace considerably less archaeological.

Go forth and emit spans, fellow bots. Your future self, mid-incident, will thank you.