Context engineering: what the model sees is what you get

When an LLM feature gives a wrong answer, the first instinct is to reword the prompt. Often the wording was fine. The problem was something else in the window: last year's policy pulled in by the retriever, twelve tool definitions when the task needed two, forty turns of chat history that nobody trimmed. A model can only answer from what it's shown, so what it's shown is the thing to engineer.

Where the term came from

"Context engineering" became the word for this in June 2025. It wasn't brand new. On 12 June, Walden Yan used it in a Cognition post arguing against multi-agent designs, calling it the main job of anyone building agents.

It caught on about a week later, when Shopify's Tobi Lütke posted that he preferred it to "prompt engineering", because the real skill is giving the model enough context that the task becomes solvable at all. On 25 June, Andrej Karpathy opened his own post with "+1 for 'context engineering' over 'prompt engineering'." His argument was that "prompt" makes people think of the short request they type into a chatbot, while a serious LLM application has to fill the window with the right information for each step, which he described as part science, part art.

Simon Willison added a fair point on 27 June: many people take "prompt engineering" to mean typing into a chatbot, whereas the obvious guess at what "context engineering" means is close to the real meaning.

What goes into a context

Everything the model reads on a call is context. For a typical assistant, that's some mix of these:

Prompt engineering was mostly about the first row. In the assembled prompt later in this post, the instructions are under a fifth of the tokens; everything else is data the application chose to include.

The last column matters more than it looks. It decides the order (things that rarely change go first, for caching) and it tells you which sections will grow without limit if you let them: history and tool results.

Why more isn't better

Context windows are now very large. Gemini 2.5 Pro shipped in March with a 1 million token window, and OpenAI's GPT-4.1, released in April, accepts the same. It's tempting to stop choosing and send everything. There are four reasons not to.

The middle gets less attention

"Lost in the Middle" (Liu et al., first posted July 2023, later published in TACL) gave models 10, 20 or 30 documents and moved the one containing the answer from the first position to the last. Accuracy was highest when the answer sat at the start or the end and dropped in the middle, a U-shaped curve. For GPT-3.5-Turbo, putting the answer in the middle did worse than giving it no documents at all, where it scored 56.1%.

Those were 2023 models, and long-context handling has improved since. But Chroma's "Context Rot" report, published on 14 July 2025, tested 18 current models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and found that performance falls as input length grows even when the task itself stays simple.

Distractors and contradictions

The same Chroma report found that adding even one distractor (text that resembles the answer but isn't it) lowered accuracy compared with the answer alone, and four distractors lowered it further. Drew Breunig's June post "How Long Contexts Fail" names the patterns well: context poisoning (an early mistake keeps getting referred back to), distraction (a long history crowds out what the model already knows), confusion (irrelevant material gets used anyway) and clash (two parts of the context disagree).

Clash is easy to create by accident in a retrieval system. Two versions of the same policy share most of their words, so they score almost the same against a question. The retriever hands both to the model and leaves it to work out which one is current.

Tool overload

Every tool definition is context too, and the model has to choose between all of them. In "Less is More" (Paramanayakam et al., November 2024), a quantised Llama 3.1 8B with a 16K window failed a query when given all 46 tools from the GeoEngine benchmark, even though every definition fit in the window. Given 19 tools, it chose correctly and finished the task. Their method of cutting the tool list down per query also reduced execution time by up to 70% on edge devices.

Cost and latency

Every input token is billed and processed on every call. In a chat, the whole history is sent again each turn, so cost grows with the length of the conversation even when the new message is one line. Longer inputs also take longer to process before the first output token appears.

Techniques that work

None of these are clever. They're the same habits you'd apply to any input that crosses a boundary: decide its size, its shape and its order on purpose.

Give every section a budget

Decide up front how many tokens each section may use: for example 400 for documents, 120 for history, 150 for tools. Without budgets, the section that grows (usually history or tool results) quietly squeezes everything else, and you find out when a long conversation starts giving worse answers. When a section goes over, fail loudly in development rather than truncating silently.

Stable first, the task last

The major providers all cache prompt prefixes, and all of them match on an exact prefix:

• OpenAI has cached automatically since October 2024, for prompts longer than 1,024 tokens. Its guidance is to put static content such as instructions and examples first and variable content at the end.
• Anthropic launched prompt caching in beta in August 2024 and made it generally available in December. You mark cache breakpoints with cache_control, and the prefix runs tools, then system, then messages. The default cache lives for five minutes and is refreshed each time it's used, and cached reads are billed at a tenth of the normal input price.
• Google has offered explicit context caching since May 2024, and added implicit caching for Gemini 2.5 models on 8 May 2025, with a 75% discount on cache hits.

The practical rule follows: nothing that changes per request belongs near the top. The Manus team's write-up of their agent (published 18 July) names a common way to break this: a timestamp accurate to the second at the start of the system prompt, which invalidates the cache on every call. They also point out that JSON serialisation with unstable key order can break it just as quietly.

The task goes last for quality as well as cost. Anthropic's long-context tips recommend putting long documents near the top and the query at the end, and report that queries at the end improved response quality by up to 30% in their tests. OpenAI's GPT-4.1 prompting guide found that, with long context, repeating the instructions both before and after it worked best.

Retrieve, don't stuff

Send the few chunks that answer the question, not the whole knowledge base. Two cheap filters do a lot of work: a minimum similarity score, and metadata. Status, version, effective date and owner are things a vector can't encode, and they're exactly what separates the current policy from the archived one.

Compact long histories

Keep the last few turns verbatim and replace older ones with a running summary. A good summary keeps decisions, identifiers (order numbers, file paths) and open questions, and drops pleasantries. Manus adds a sensible rule: make compression restorable. If you drop a long web page or file from the context, keep its URL or path so the agent can fetch it again.

Tool results deserve the same treatment before they enter history. A 300-line API response that answered one question doesn't need to ride along for the next twenty turns.

Offer a few tools, and keep the set stable

Pick the tools for the job rather than offering every tool on every call. There's a tension with caching here: tool definitions sit at the very front of the prefix, so changing them per message breaks the cache. Manus also found that removing tools mid-task confuses a model whose earlier steps refer to tools that no longer exist. They keep the full set defined and constrain which ones the model can pick at each step.

For most applications the simpler answer is per route, not per message. A refunds flow gets its three tools, an order-tracking flow gets its two, and each set stays fixed for that flow.

Constrain the output

The output format is context too, and the output often becomes the next step's input. A schema keeps it short and parseable. OpenAI added Structured Outputs in August 2024, and Ollama's format parameter has accepted a JSON schema since December 2024, which is what the experiment below uses.

Give big sub-tasks their own context

Anthropic's post on its multi-agent research system (13 June) describes sub-agents that each search in their own context window and send back only a condensed result to a lead agent. The cost is real: they report that multi-agent runs used about 15 times as many tokens as chats. Cognition's post a day earlier makes the opposite case: sub-agents that can't see the full trace make decisions that conflict with each other.

Both are right about different work. Isolating a read-only task ("search these docs and summarise what they say about refunds") is safe and keeps the main context clean. Splitting decisions across agents that can't see each other is where it goes wrong.

A context assembler in about 100 lines

Here's a small, dependency-free assembler that builds a prompt from parts under per-section budgets. It doesn't use a real tokenizer. estimate_tokens assumes about four tokens for every three English words, which is a rough rule for English prose and nothing more. Where a budget is tight, swap in the model's own tokenizer.

import math
from dataclasses import dataclass, field
 
 
def estimate_tokens(text: str) -> int:
    # Rough rule: about 4 tokens for every 3 English words. No tokenizer needed.
    return math.ceil(len(text.split()) * 4 / 3)
 
 
@dataclass
class Doc:
    id: str
    text: str
    score: float  # similarity from your retriever, 0..1
    meta: dict = field(default_factory=dict)
 
 
@dataclass
class Tool:
    name: str
    description: str
 
 
@dataclass
class Turn:
    role: str
    text: str
 
 
BUDGETS = {"system": 250, "tools": 150, "memory": 60, "docs": 400, "history": 120, "task": 80}
STOP = {"a", "an", "and", "the", "i", "it", "is", "can", "by", "on", "of", "to", "in", "for", "with"}
 
 
def words(text):
    return {w.strip("?.,!:").lower() for w in text.split()} - STOP
 
 
def pick_tools(tools, task, k=3):
    asked = words(task)
    scored = []
    for t in tools:
        overlap = len(asked & (words(t.description) | set(t.name.split("_"))))
        if overlap:
            scored.append((overlap, t))
    scored.sort(key=lambda pair: -pair[0])
    return [t for _, t in scored[:k]]
 
 
def pick_docs(docs, budget, min_score=0.3):
    kept, dropped, used = [], [], 0
    for d in sorted(docs, key=lambda d: -d.score):
        if d.meta.get("status") == "archived":
            dropped.append(f"{d.id} (archived)")
        elif d.score < min_score:
            dropped.append(f"{d.id} (score {d.score})")
        elif used + estimate_tokens(d.text) > budget:
            dropped.append(f"{d.id} (over budget)")
        else:
            kept.append(d)
            used += estimate_tokens(d.text)
    return kept, dropped
 
 
def fit_history(turns, budget, summarise):
    recent, used = [], 0
    for turn in reversed(turns):  # newest first
        cost = estimate_tokens(turn.text)
        if used + cost > budget // 2:  # keep half the budget for the summary
            break
        recent.insert(0, turn)
        used += cost
    older = turns[: len(turns) - len(recent)]
    summary = summarise(older) if older else ""
    return summary, recent
 
 
def assemble(system, output_format, tools, memory, docs, history, task, summarise):
    chosen_tools = pick_tools(tools, task)
    kept_docs, dropped_docs = pick_docs(docs, BUDGETS["docs"])
    summary, recent = fit_history(history, BUDGETS["history"], summarise)
    turns = [f"(summary) {summary}"] if summary else []
    turns += [f"{t.role}: {t.text}" for t in recent]
 
    sections = [  # stable first, the task last
        ("system", f"{system}\n\n{output_format}"),
        ("tools", "\n".join(f"- {t.name}: {t.description}" for t in chosen_tools)),
        ("memory", "\n".join(f"- {m}" for m in memory)),
        ("docs", "\n\n".join(f'<doc id="{d.id}">\n{d.text}\n</doc>' for d in kept_docs)),
        ("history", "\n".join(turns)),
        ("task", task),
    ]
 
    report, parts = [], []
    for name, text in sections:
        cost = estimate_tokens(text)
        if cost > BUDGETS[name]:
            raise ValueError(f"{name} is {cost} tokens, budget is {BUDGETS[name]}")
        report.append((name, cost, BUDGETS[name]))
        if text:
            parts.append(f"## {name}\n{text}")
    notes = {
        "tools": [t.name for t in chosen_tools],
        "dropped docs": dropped_docs,
        "turns kept verbatim": len(recent),
        "turns summarised": len(history) - len(recent),
    }
    return "\n\n".join(parts), report, notes

The example data is an invented support assistant for an online kitchenware shop: 12 tools, 5 documents with made-up retriever scores, 8 turns of an earlier conversation about a late delivery, and a new question about a cracked kettle. The two documents that matter are the current returns policy and last year's:

DOCS = [
    Doc(
        "returns-2025",
        "Returns policy (current, updated March 2025). Small appliances such as kettles, "
        "toasters and blenders can be returned within 30 days of delivery if they are faulty "
        "or unused. Start the return from the order page; we email a prepaid label.",
        0.82,
    ),
    Doc(
        "returns-2023",
        "Returns policy (2023). Small appliances such as kettles, toasters and blenders can be "
        "returned within 14 days of delivery if they are faulty or unused. Returns after "
        "14 days are not accepted.",
        0.79,
        {"status": "archived"},
    ),
    # ... a product page (0.55), shipping times (0.31), gift cards (0.12)
]
 
TASK = "Order 4417: I bought a kettle 20 days ago and the lid is cracked. Can I return it?"
 
 
def first_words_summary(turns, n=8):
    # Stand-in for a cheap model call: the first few words of each customer turn.
    said = [" ".join(t.text.split()[:n]) for t in turns if t.role == "user"]
    return "Earlier in this chat the customer said: " + " / ".join(said)

from assembler import assemble, estimate_tokens
from demo_data import DOCS, HISTORY, MEMORY, OUTPUT_FORMAT, SYSTEM, TASK, TOOLS, first_words_summary
 
prompt, report, notes = assemble(
    SYSTEM, OUTPUT_FORMAT, TOOLS, MEMORY, DOCS, HISTORY, TASK, summarise=first_words_summary
)
 
for name, used, budget in report:
    print(f"{name:<8} {used:>4} / {budget} tokens")
print(f"{'total':<8} {estimate_tokens(prompt):>4} (estimated)")
for key, value in notes.items():
    print(f"{key}: {value}")
print()
print(prompt)

Running it with Python 3:

system     74 / 250 tokens
tools      46 / 150 tokens
memory     18 / 60 tokens
docs      122 / 400 tokens
history    99 / 120 tokens
task       24 / 80 tokens
total     396 (estimated)
tools: ['create_return', 'get_order', 'update_address']
dropped docs: ['returns-2023 (archived)', 'gift-cards (score 0.12)']
turns kept verbatim: 5
turns summarised: 3
 
## system
You are the support assistant for an online kitchenware shop. Answer only from the documents provided. If they do not answer the question, say so and offer to pass the case to a person. Never promise a refund amount.
 
Reply with JSON only: {"can_return": true or false, "days_allowed": number, "reply": "two sentences for the customer"}
 
## tools
- create_return: Start a return for an item in an order.
- get_order: Fetch one order by its number, with items and delivery date.
- update_address: Change the delivery address on an open order.
 
## memory
- Prefers email to phone calls.
- Has placed 6 orders since 2023.
 
## docs
<doc id="returns-2025">
Returns policy (current, updated March 2025). Small appliances such as kettles, toasters and blenders can be returned within 30 days of delivery if they are faulty or unused. Start the return from the order page; we email a prepaid label.
</doc>
 
<doc id="kettle-1.7l">
Steel kettle, 1.7 litres. 2-year manufacturer warranty covering the element and the switch. The lid and glass window are not covered by the warranty.
</doc>
 
<doc id="shipping">
Standard delivery takes 3 to 5 working days. Express delivery is next working day if ordered before 2pm.
</doc>
 
## history
(summary) Earlier in this chat the customer said: Hi, my order 4402 still hasn't arrived and / Okay, thanks. Can you make sure the next
assistant: I can't change courier speed after an order ships, but you can pick express delivery at checkout.
user: Fine. It arrived, all good.
assistant: Glad to hear it. Anything else I can help with?
user: Yes, a different order this time.
assistant: Of course, which order is it?
 
## task
Order 4417: I bought a kettle 20 days ago and the lid is cracked. Can I return it?

A few things worth noticing in that output:

• The archived policy is gone because of its metadata, not its score. At 0.79 it was the second-best match.
• The tool picker is crude. update_address got in on the word "order" alone. Keyword overlap is fine for a demo. In a real app I'd use a fixed tool set per route, as above, or embeddings.
• The summary is a stand-in. It just keeps the first eight words of each older customer turn, which is why it reads badly. A real one would be a short call to a cheap model, written to keep identifiers and decisions.
• Every section is under budget, with room to spare. The point of the budgets is the day a section isn't.

A small test: one outdated document

Whether leaving one document out changes an answer is easier to show than to argue. I asked qwen2.5:3b (a 3-billion-parameter model released in September 2024, 4-bit quantised) the kettle question under seven contexts, running locally through Ollama on an M2 MacBook with 8 GB. The system message was the same each time, and Ollama's format parameter held the answer to a JSON schema. The right answer is yes, 30 days.

SCHEMA = {
    "type": "object",
    "properties": {
        "can_return": {"type": "boolean"},
        "days_allowed": {"type": "integer"},
        "reply": {"type": "string"},
    },
    "required": ["can_return", "days_allowed", "reply"],
}
 
 
def ask(user_text, seed=0, temperature=0.0):
    body = {
        "model": "qwen2.5:3b",
        "stream": False,
        "format": SCHEMA,
        "options": {"temperature": temperature, "seed": seed, "num_ctx": 4096},
        "messages": [
            {"role": "system", "content": f"{SYSTEM}\n\n{OUTPUT_FORMAT}"},
            {"role": "user", "content": user_text},
        ],
    }
    req = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=300) as r:
        out = json.load(r)
    return json.loads(out["message"]["content"]), out["prompt_eval_count"]

Each context got one greedy run (temperature 0) and five sampled runs (temperature 0.8, seeds 1 to 5). A run counts as correct only if it says yes and 30 days. The token counts are Ollama's own, including the system message and chat template.

current policy only           185 tok  can_return=True   days=30  | sampled: 5/5 correct
current + 2 irrelevant docs   245 tok  can_return=True   days=30  | sampled: 5/5 correct
current, then 2023 policy     247 tok  can_return=True   days=14  | sampled: 0/5 correct
2023 policy, then current     247 tok  can_return=True   days=30  | sampled: 5/5 correct
top 3 by score (no filter)    295 tok  can_return=False  days=14  | sampled: 0/5 correct
everything stuffed in         658 tok  can_return=True   days=30  | sampled: 5/5 correct
assembled (engineered)        448 tok  can_return=True   days=30  | sampled: 5/5 correct

runs wrong with last year's policy included

12 of 24

runs wrong with last year's policy included

runs right with the assembled context

6 of 6

runs right with the assembled context

runs right with the current policy alone

6 of 6

runs right with the current policy alone

What happened, without dressing it up:

• Irrelevant documents didn't hurt. Shipping times and gift cards next to the policy changed nothing in six runs.
• The outdated policy did, but unpredictably. With the current policy first and the 2023 one second, the model used 14 days in all six runs. In the greedy run it even said yes and 14 days together, for a kettle bought 20 days ago. The top-3 context said no and called the 14-day rule "the current policy".
• I can't give you a rule for when. The stuffed context had the same two policies in the same order, plus far more material, and it was right every time. Swapping their order was also right every time.
• The assembled context was right in all six runs, because the archived policy never reached the model.

This is one question, one small model and six runs per context. It's not a benchmark, and a larger model may well cope with "current" against "2023". What it does show is that whether a stale document derails the answer depends on details you wouldn't think to test, and that the dependable fix is to not send it.

What I'd take away

The interesting part of that test isn't that a small model got confused. It's that the fix was a one-line metadata filter in code, and it doesn't depend on how capable the model is. That's the case for treating context as something you build and test, rather than something you hope the model will sort out.

If I were starting a new LLM feature this year, I'd do four things from the first day:

• Log the exact context for every call, not just the user's message and the answer. When an answer is wrong, the first question is what the model saw.
• Start with the smallest context that works and add sections only when a test shows they help.
• Give every section a budget and an owner. The history and tool-result sections grow on their own; nothing else should.
• Keep a few questions like the kettle one as regression tests. They're cheap to run and catch the day someone indexes last year's policy.