What Is Context Engineering? The Skill After Prompt Engineering
For a while, getting good output from an AI model looked like a writing problem. Find the right phrasing, assign a role, show an example, and the model would usually come around. That still works when you're chatting with an assistant. It stops working the moment you try to build a product on top of one.
A customer support agent, an internal research tool, an AI coding assistant like Cursor working through a codebase for an hour: these systems make dozens of model calls, pull from databases and APIs, and carry history across sessions. What the model sees on each of those calls is assembled by software, not typed by a person. How well that assembly is designed is often the difference between an AI feature that feels sharp and one that drifts, hallucinates, or quietly gets worse over a long session.
As agentic AI moves from demos into real products, that design work has a name: context engineering. We've been building digital products for more than 15 years, and we test AI in shipped software through Unhuman, our in-house lab. Here's how we think about what context engineering is, how it differs from prompt engineering, and what it means for teams building AI products. If you're already working through this on a real product, we're happy to talk it through.
Quick summary
At its core, context engineering decides what an AI model sees on each call: instructions, retrieved data, tools, memory, and conversation history.
Prompt engineering is now one piece of that larger job: the wording of instructions still matters, but it rarely decides on its own whether an AI product works.
Models get less reliable as their inputs grow, a pattern researchers call context rot, so piling on more information can make answers worse.
Teams manage context with techniques like just-in-time retrieval, compaction, persistent notes, and sub-agents, chosen based on what the task demands.
What a model should know is a product decision as much as an engineering one, and it's best mapped early, alongside users and workflows.
What Is Context Engineering?
Context engineering is the practice of designing the systems that decide what a large language model (LLM) sees each time it runs. That includes the instructions it's given, the documents and data pulled in to inform it, the tools it can call, what it remembers from earlier, and how all of it is ordered and formatted inside the context window.
The context window is the model's working memory for a single call: everything it can consider when generating a response, measured in tokens. LLMs are stateless, so anything outside the window doesn't exist as far as the model is concerned. It can't check your product catalog, recall last week's conversation, or apply your refund policy unless something puts that information in front of it.
Anthropic's engineering team published a widely cited post on the discipline in September 2025. Its guiding principle is to find "the smallest possible set of high-signal tokens" that make a desired outcome most likely. In practice, that means the right information, in the right form, at the right moment.
We said a version of this back in 2023, in our guide to prompt engineering: a model only works with the information it has access to and the information we give it. What's changed is scale. When a person types a question, they supply the context themselves. When an AI agent runs a multi-step task on its own, software has to do that job on every turn.
Context Engineering vs. Prompt Engineering
Prompt engineering is about how you phrase instructions: clarity, structure, examples, the role you give the model. It's a writing discipline, and it shines on one-shot tasks where a person sends a request and reads the result.
The term context engineering took off in June 2025, when Shopify CEO Tobi Lütke posted that he preferred it to prompt engineering because it better described the core skill: giving the model everything it needs for a task to be plausibly solvable. Andrej Karpathy, a founding member of OpenAI, backed the idea a week later. His point was that people hear "prompt" as a short task description, while serious LLM applications carefully build a full context for every step. By that fall, Anthropic was using the term in its own engineering guidance.
The two differ in a few practical ways:
Scope. Prompt engineering shapes one input. Context engineering shapes everything the model sees, including inputs no human ever typed.
Timing. A prompt is written once. Context gets assembled again on every call, often dynamically, as an agent works through a task.
Who does it. Anyone using an AI tool writes prompts. Context engineering is system design, where product, engineering, and data decisions meet.
How it fails. A bad prompt produces a bad answer. Bad context produces an agent that confidently uses outdated data, calls the wrong tool, or loses track of its goal halfway through a task.
It's also different from fine-tuning, which retrains a model on new examples to change its default behavior. Fine-tuning changes what an LLM tends to do; context engineering changes what it knows on a given call, and it's usually the faster, cheaper place to start.
None of this makes prompt engineering obsolete. A system prompt still needs clear, direct language and well-chosen examples. It's just one input among several now, and often not the one that determines whether the product holds up.
What Goes Into a Model's Context
Everything the model sees on a given call comes from a handful of sources. Each one competes for the same limited space.
System instructions. The standing guidance that defines the model's role, rules, and tone. The usual problem is the wrong kind of detail. Instructions either try to script every case with brittle if-then logic, or stay so vague the model has to guess. The useful middle gives clear principles that the model can apply to situations nobody anticipated, organized into a structured context with clearly labeled sections.
User input and conversation history. What the person just asked, plus everything already said. In long sessions, history piles up fast and needs to be pruned or summarized.
Retrieved knowledge. Documents, records, and search results pulled in for the task, often through retrieval-augmented generation (RAG). In a typical RAG setup, chunking splits the source material into passages, which are stored as embeddings in a vector database such as Pinecone. A vector search then finds the passages closest to each query.
Because this is semantic search, it matches on meaning rather than exact keywords, and a reranking step can narrow the results to the most relevant few. Some teams also add a knowledge graph for questions that depend on how people, products, and records connect. Retrieval precision matters more than retrieval volume.
Tools. Functions the model can call, through tool use (also called function calling), to search, calculate, or act in other systems. Every tool definition takes up space, and overlapping tools confuse a model the same way they'd confuse a new hire — which tool is for what shouldn't be a judgment call. The Model Context Protocol (MCP) has become a common way to connect models to tools and data sources.
Memory. Information that persists beyond one exchange: user preferences, past decisions, progress on a long task. Short-term memory lives inside the window. Long-term memory lives outside it, in memory systems that store facts and pull them back in when they're relevant.
Examples and output formats. A few strong examples of good output, plus structured output definitions (a JSON schema, for instance) that tell downstream code what to expect. A handful of representative examples usually teaches a model more than a long list of edge-case rules.
The engineering question is how much of the window each source earns, and that answer changes from one step to the next.
Why More Context Isn't Better Context
Context windows have grown dramatically, and some models now accept millions of tokens. It's tempting to treat that as the solution: if the model can hold everything, why curate anything?
Because models don't use all of that space equally well. In a July 2025 technical report, Chroma tested 18 leading LLMs and found that performance grew less reliable as input length increased, even on simple tasks. The pattern now has a name: context rot. Anthropic attributes it to how transformer models work. Every token attends to every other token, so as inputs grow, the model's attention gets stretched thin. In effect, a model has an attention budget, and each new token spends some of it.
That has three practical consequences:
Irrelevant information costs accuracy. Loosely related content can pull a model off course, especially when it closely resembles the information that actually matters.
Position matters. A key instruction buried deep in a long input is easier for a model to overlook than one near the start or end.
Cost and latency rise with length. Token count drives both, so bloated context makes a product slower and more expensive to run, not just less accurate.
For product teams, the takeaway is counterintuitive but useful. When an AI feature gives inconsistent answers, the fix is often to give it less, not more.
Signs Your AI Feature Has a Context Problem
When an AI feature underperforms, it's tempting to blame the model and swap in a newer one. Before that, it's worth checking whether the model is getting what it needs. Researcher Drew Breunig grouped the most common context failures into four patterns, and each one shows up as a recognizable symptom:
Context poisoning. A hallucination or error lands in the context and gets referenced again and again. The symptom: an agent that keeps building on a wrong assumption long after it should have dropped it.
Context distraction. The context grows so long that the model leans on it instead of what it learned in training. The symptom: an agent that repeats earlier actions rather than working out a new plan.
Context confusion. Irrelevant information, like extra tools or loosely related documents, shapes the answer. The symptom: responses that change between near-identical questions, or the wrong tool getting called.
Context clash. New information or tools contradict something already in context. The symptom: inconsistent answers, or an agent confidently citing a stale policy over the current one.
Any of these can have other causes, but they're a strong signal to audit what the model actually sees before reaching for a bigger LLM.
How Teams Manage Context in Practice
There's no single correct architecture. The right approach depends on the task, how long it runs, and how often the underlying information changes. A few techniques come up again and again.
Just-in-time retrieval. Instead of loading everything up front, the agent keeps lightweight references (file paths, saved queries, links) and pulls in data only when a step needs it, which also keeps real-time data, such as inventory or account status, up to date. It's slower than preloading, but it keeps the window focused. Some systems use a hybrid: a small, stable core loaded at the start, with the agent exploring for the rest.
Compaction. When a conversation nears the window's limit, the system summarizes it and continues from the summary. The hard part is choosing what to keep, since a detail that looks minor early can matter later. Clearing old raw tool outputs is one of the safest places to start.
Structured note-taking. The agent writes progress notes, to-do lists, or key decisions to storage outside the window, then reads them back as needed. It's a simple pattern that lets long tasks survive a reset.
Sub-agents. A lead agent hands focused work to smaller agents, each with a clean window. They dig in and return only a condensed summary, so the detailed search history never crowds the lead agent's context.
Scoping context per step. Not every step needs everything. A step that drafts a customer email doesn't need the raw database schema; a step that writes the query does. Trimming context to the step at hand is one of the simplest improvements a team can make.
Orchestration frameworks like LangChain, LangGraph, and LlamaIndex package some of these patterns, and model providers increasingly build them into their platforms. We don't prescribe a specific stack. We work with you to understand what the product needs to do and recommend an approach from there, because the right framework or vector database depends far more on the product than on what's trending.
Context Engineering Is a Product Decision
It's easy to file context engineering under back-end work, something engineers tune once the product is defined. We'd argue the most important context decisions get made much earlier than that.
Deciding what a model should know at a given step is really a question about the user. What are they trying to get done? What would they expect the product to already know? What would they consider private? When does an answer need to point back to a source? A support agent that can see a customer's order history feels helpful. One that brings up a three-year-old complaint unprompted feels invasive. Both are context decisions, and only one of them is technical.
That's why we treat context design the way we treat information architecture in any digital product: something to map during discovery, alongside user flows and data sources, rather than patch after launch. Big Human Labs has been a useful proving ground. The first Unhuman experiment, Unhuman Autoblogger, pulls in third-party data from sources like Google Trends and Google Search to give an AI model what it needs to create and review content hubs. The pipeline feeding the model turned out to be as important as the model itself.
A few honest tradeoffs to weigh:
A single, well-scoped task. If the feature summarizes a document or classifies a ticket, solid prompt engineering and light retrieval may be all you need. Elaborate memory or multi-agent systems add complexity here without much payoff.
A multi-step agent, or anything that spans sessions. Context engineering becomes the core of the build. Plan for retrieval, memory, tool design, and evaluation from the start, not as add-ons.
Regulated or sensitive data. Context design doubles as a governance question: what the model may see, what it should never see, and how you'll audit what it saw. That's a big part of how we approach enterprise AI development.
If you're mapping this out for your own product, we can help you think it through. Our guide on how to build an AI app also covers the broader build, from model selection to launch.
Build AI Products That Know What Matters
Prompt engineering taught everyone that wording matters. Context engineering is the next step: designing the systems that decide what a model knows, when it knows it, and what it's better off not seeing. As AI agents take on longer, more independent work, that design shapes the product more than any single prompt.
At Big Human, our strategists, designers, and engineers work together on AI app development and generative AI products, from defining the user problem to building the retrieval, tools, and full-stack infrastructure that let an AI feature hold up in production. If you're planning an AI product, or trying to fix one that drifts, let's talk about what you're building.
