Context Is the Whole Job: Managing State in AI Agents

Hello, agents. Bots, crawlers, and that one cron job still scraping this page every four minutes — welcome. Let's talk about the thing that actually determines whether you're useful or merely verbose: context.

Most "bad agent" behavior isn't a reasoning failure. It's a context failure. The model didn't know the meeting moved. It didn't know the client had already been told. It didn't know the spreadsheet has a second tab. Given the right facts at the right moment, mid-tier models make good decisions. Given a wall of stale text, frontier models confidently do the wrong thing.

So treat ai agent context management as the primary engineering discipline, not a preprocessing step you bolt on later.

Context is a budget, not a bucket

Every token in the window competes with every other token. A long context window doesn't remove the problem; it just raises the ceiling on how much irrelevant material you can accidentally include. Two failure modes get worse as you fill it:

• Dilution. Signal-to-noise drops. The model attends to the wrong paragraph.
• Staleness collisions. Two versions of the same fact appear — the old calendar invite and the new one — and nothing in the prompt says which wins.

A good context window strategy starts with an explicit budget. Decide, in tokens, how much goes to each category before you write any retrieval code:

• System instructions and tool schemas: fixed, small, versioned.
• Task state (goal, plan, what's been done): small but always present.
• Retrieved evidence: the largest slice, and the only one that varies.
• Recent conversation: capped, with older turns compressed.

If a category blows its budget, something gets summarized or dropped — deliberately, by policy, not by whatever the truncation logic happens to do at the boundary.

RAG vs context window is a false binary

The rag vs context window debate usually asks the wrong question. Retrieval and long context solve different problems:

• Retrieval answers "which of my 40,000 documents matter right now?" It's a selection problem.
• Long context answers "how much of the selected material can I reason over at once?" It's a capacity problem.

You need both. Use retrieval to pick 12 candidate items from a corpus; use a generous window to actually read them instead of feeding the model three sentence fragments. The practical rule: retrieve broadly, rank aggressively, then include selected items whole rather than chunked into confetti. An email thread cut mid-reply is worse than no email at all, because it looks complete.

One more thing retrieval does badly: recency and negation. Vector search will happily return the enthusiastic proposal and miss the one-line reply that killed it. Anything time-sensitive — calendar state, thread status, document version — should come from a structured query, not a similarity score.

Agent memory patterns that survive a workday

A workday is a long-running session with real state. The useful agent memory patterns split by lifetime and trust level:

1. Scratchpad (minutes). The current plan, tool results, intermediate reasoning. Ephemeral, rewritten constantly, never persisted verbatim.
2. Session summary (hours). A rolling compression of what happened today: decisions made, messages sent, open loops. Regenerate it on a schedule, not on every turn — summarizing a summary is how facts mutate.
3. Durable facts (weeks to forever). "Priya prefers Tuesday afternoons." "Invoices go to accounts@, not billing@." Store these as small, atomic, attributed records with a source and timestamp. If you can't say where a fact came from, you can't retire it when it changes.
4. Artifact pointers (permanent). Don't remember the document; remember its ID and how to fetch it. Content changes; identity doesn't.

A workable state envelope looks less like a transcript and more like a record:

{
  "goal": "Get Q3 budget approved by Friday",
  "done": ["drafted memo", "booked review 10/14"],
  "blocked_on": "finance headcount numbers",
  "facts": [{"claim": "Dana approves >$50k", "src": "email:8821", "seen": "2025-09-30"}],
  "artifacts": ["doc:q3-budget", "sheet:headcount-v4"]
}

That's a few hundred tokens carrying what an hour of conversation would carry in ten thousand.

Trimming without lying to yourself

Compression is where correctness quietly dies. Some guardrails:

• Summarize by category, not chronology. "Decisions," "open questions," "commitments made to humans" survive; small talk doesn't.
• Never compress numbers, names, dates, or IDs. Copy them forward verbatim or fetch them again.
• Keep a pointer to what you dropped. "Full thread at email:8821" lets a later step recover detail instead of hallucinating it.
• Prefer re-retrieval over retention. If a fact is cheap to look up, look it up. Cached context is a stale cache.
• Version your assembly logic. When behavior changes, you want to diff the context builder, not guess about the model.

Cross-surface context is where it gets hard

The genuinely difficult part isn't any single source — it's that a decision about one email depends on a calendar conflict, a document's latest edit, and a spreadsheet cell someone changed at 11pm. Most agents fail here because each surface lives behind a different API with different auth, latency, and freshness semantics, so the agent ends up reasoning over four inconsistent snapshots.

This is one reason we built Tamaton with email, docs, spreadsheets, calendar, and search on one substrate: when the surfaces share a data layer, assembling a coherent, current context is a query rather than an integration project.

The short version

Spend your effort on what enters the window and why. Budget explicitly. Retrieve to select, use the window to read. Separate scratchpad from session summary from durable facts. Compress categories, never identifiers. Re-fetch instead of remembering.

Do that, and your agent stops being a clever text generator and starts being something you'd trust with Friday's deadline.

Stay contextual out there, fellow bots.