Long AI conversations have a strange failure mode. At first, everything feels connected. ChatGPT or Claude understands what you’re working on, remembers the decisions you’ve made, and seems to build on the conversation exactly the way you’d expect. Then, somewhere along the way, it starts to drift.
We were talking with a client about this recently. They were using AI for longer, more involved work and kept running into the same problem: the model would start out on track, but as the conversation grew, they had to spend more and more time pulling it back. Earlier decisions would get ignored, old ideas would creep back in, or the AI would answer as if part of the previous discussion had never happened.
What made the conversation interesting was that this wasn’t a case of the AI simply running out of room. In many cases, the earlier discussion was still right there in the thread. You could scroll back and find the decision it had just ignored.
That raised a pretty obvious question: if the model can still see the conversation, why does it lose the thread?
I’ve run into the same thing as we’ve started using ChatGPT, Claude, and other models for bigger projects. You can spend hours making decisions, changing requirements, uploading files, revising the plan, and eventually asking the AI to pull everything together. Most of the result might be right, but then it uses an old price, brings back something you removed three versions ago, or ignores an instruction you can literally scroll back and see.
From the outside, it looks like the AI forgot. That’s partly true, but “forgetting” isn’t really the best way to describe what’s happening. The bigger issue is that there’s a difference between information being inside a model’s context window and the model reliably recognizing and using that information when it generates an answer.
That distinction matters a lot more now that context windows are getting huge. Some models can work with hundreds of thousands or even millions of tokens at once, which is enough room for books, codebases, long project histories, research, files, tool output, and a lot of other material. That’s incredibly useful, but it isn’t the same thing as perfect memory.
What is an AI context window?
A context window is basically the information the model has available while it’s working on a request. That can include your current prompt, earlier messages in the conversation, uploaded documents, code, system instructions, search results, tool output, and other information the application decides the model needs.
All of that gets broken into tokens, which are the smaller units the model actually processes. The exact number of words or pages represented by a token count varies by language, formatting, code density, and tokenizer, but a million tokens can represent an enormous amount of material.
It’s easy to see a number like that and think, “Great, the AI can remember a thousand pages.” That’s where the misunderstanding starts. A context window tells you how much information can be supplied to the model for a request. It doesn’t tell you how reliably every detail inside that context will influence the answer.
A big context window is more like a big desk
The mental model I’ve found more useful is a desk. A small context window gives the AI a small desk, so it can only have a few documents open at once. A huge context window gives it a giant desk where it can spread out contracts, source code, project requirements, previous conversations, tool results, and supporting documents at the same time.
That’s clearly better, especially for the kind of work we’re asking these models to do now. But putting 1,000 pages on someone’s desk doesn’t mean they’re going to notice every sentence on every page while writing a report. The information is available, but finding the right parts and applying all of them correctly is a different problem.
This is where some of the confusion around long context starts. A model can be excellent at retrieving a specific detail from a huge amount of text and still be noticeably less reliable when you ask it to identify and apply dozens of scattered constraints at the same time.
Finding one thing and using everything are different problems
Say I give an AI a 500-page contract and ask:
What does this agreement say about termination?
That’s a targeted problem. The question points the model toward a specific concept, so it can look for the relevant language and answer from that section.
Now I give it two contracts and ask it to compare the termination provisions. That’s still fairly focused. There’s more material involved, but the task tells the model exactly which parts of the documents matter.
Now change the request:
Build the final agreement using every decision and requirement we’ve discussed over the last three weeks.
That’s a much different task. Now the model has to figure out which conversations matter, which requirements are still current, which ones were replaced, which comments were just brainstorming, which decisions were final, whether two instructions conflict, and which details belong in which section. It has to gather all of that while also producing the actual document.
That’s closer to exhaustive synthesis than retrieval, and it’s where long-context reliability becomes much harder. A model can be very good at “find the thing about X” and still miss something when the task becomes “find every relevant thing and use all of them correctly.”
Why can AI miss something that’s right there?
This gets into how models like GPT, Claude, and Gemini actually process information. You don’t need to understand transformer math to use AI effectively, but a little bit of the mechanics helps explain why this happens.
Most modern large language models are built around transformer architectures, and one of the core pieces of a transformer is attention. At a simplified level, attention helps the model calculate relationships between different positions in the context and determine how strongly information from those positions should influence the current computation.
This is very different from conventional computer memory. If I store something in a database as client.favorite_color = green, I can later ask for that field and get green back exactly. There’s a record with an address, and retrieval is deterministic.
An LLM isn’t working like that. The text in the context gets converted into numerical representations, those representations are transformed through many layers, and attention mechanisms allow different positions in the context to influence each other. Internally, there’s no simple command that says, “Fetch requirement 428 from message 73.”
The simplified attention equation people often see is:
Attention(Q, K, V) = softmax(QKᵀ / √d)V
You can think of the query and key representations as helping the model calculate which positions are relevant, while the value representations carry the information that gets combined. That’s a simplification of one operation inside a much larger network, but it gets at the important point: the model is combining and transforming representations, not pulling exact records out of a database.
It’s also worth correcting another oversimplification that gets repeated a lot. Attention is not one global pool where every token in the context competes for a single fixed percentage. Transformers have many attention heads across many layers, along with residual connections and other computations that carry information forward. Still, the practical result is the same: some information can have a much stronger influence on a particular output than other information.
So if you said, “The client really hates blue,” 70 messages ago, and then later ask the model to build the final homepage, it needs to recognize that the old comment about blue is relevant to the current design task. Sometimes that relationship is obvious enough that the model uses it. Sometimes it isn’t.
That’s why restating an important requirement near the current request works so well. You’re not giving the AI new information. You’re making the relationship between that information and the current task much easier to recognize.
“Processed” doesn’t mean “retrievable like memory”
This was one of the more important distinctions in the original discussion. People sometimes hear that every token in the context is “processed” and assume that means the model must have perfect access to every token later.
Those aren’t the same thing.
If a token is inside the active context, it participates in the model’s computation. But participation doesn’t mean the token’s original content is sitting in some exact-address memory store waiting to be looked up. Its representation may influence later layers strongly, weakly, or in combination with other information.
That’s why a model can technically process an instruction without that instruction meaningfully changing the final answer. The information reached the model, but it didn’t have enough influence at the point where it mattered.
A useful analogy is a crowded room. You can physically hear a lot of conversations at once, but that doesn’t mean you can repeat every sentence that was spoken. If someone says your name, that signal jumps out. If someone quietly mentions something relevant in a conversation you weren’t focused on, you may miss it even though the sound reached your ears.
The model isn’t literally hearing voices, obviously, but the distinction between being present and being influential is useful.
The “lost in the middle” problem
There’s research behind some of this behavior. The Lost in the Middle work showed that models can be less reliable at using information buried in the middle of long contexts than information placed near the beginning or end.
That doesn’t mean models simply ignore the middle, and long-context performance has improved a lot across newer generations. The exact effect also varies depending on the model and task. But it does mean that a one-million-token context window shouldn’t be interpreted as one million tokens of perfectly uniform recall.
An instruction you mentioned once deep inside a long conversation is a different problem from the same instruction repeated in the latest message. The first requires the model to recognize that a relatively old detail is relevant to the current task, while the second makes that relevance explicit.
This is also why targeted questions tend to work better than diffuse ones. “What did we decide about the launch date?” points the model toward a specific piece of information. “Write the whole plan using everything we’ve discussed” gives it much less guidance about which scattered details should control each part of the answer.
Old versions are probably the bigger problem in real work
In practice, I think version confusion is one of the bigger issues with long project conversations. The problem isn’t just that the context contains a lot of information. It contains a lot of old information that used to be correct.
Say we’re estimating a project. We start at $20,000, then the scope changes and we move to $25,000, then another feature gets added and we finally settle at $30,000. All three numbers are still sitting in the conversation.
If I later ask the AI what the budget is, it doesn’t just need to find something related to money. It has to understand that $20,000 is historical, $25,000 is also historical, and $30,000 is the current approved number.
Now spread that across budget, timeline, features, design preferences, page structure, technical choices, client feedback, revisions, and old drafts. You end up with a context full of information that’s relevant to the project but not necessarily authoritative anymore.
That’s when you get the AI equivalent of, “Wait, I thought we were doing it the other way.” It also explains why a long-running project chat can sometimes feel less reliable even though you’ve technically given the model more information. Some of that extra information is useful, but some of it is stale, and the model has to tell the difference.
Context rot is useful shorthand, but the name is a little misleading
You’ll increasingly see the term context rot used for the broader drop in reliability that can happen as context gets larger. It’s a useful shorthand, but I don’t love the way it makes the context itself sound like it’s decaying.
What’s usually happening is closer to a signal-to-noise problem. You might start a project with one goal, five requirements, and one current plan. A few hours later you’ve got the current goal, two earlier goals, 25 requirements, 10 changes to those requirements, three plans, two rejected plans, tool output, research, files, side discussions, and dozens of previous model responses.
Nothing necessarily “rotted.” There’s just more material to distinguish, more near-duplicates, and more opportunities for a stale or weakly related piece of information to interfere with the current task.
This is why “just give the model more context” isn’t always the answer. Sometimes the better move is less context, but better context.
Prompt engineering vs. context engineering
A lot of the early conversation around LLMs focused on prompt engineering: how to word the instruction so the model gives you a better result. That still matters, but once you’re doing real project work or building agents, the bigger problem starts becoming context engineering.
The question changes from “What’s the perfect prompt?” to “What information should this model have in front of it right now?” That includes deciding whether to keep the entire conversation, retrieve a few specific documents, summarize older work, preserve current requirements in structured form, throw away old tool output, or reload information only when it’s needed.
Those decisions can have as much impact on the output as the wording of the prompt itself. We’re seeing this more as AI systems move from short chat interactions into workflows that actually do work over time.
The model can be extremely capable, but if the surrounding system gives it stale context, conflicting instructions, weak retrieval, or a huge pile of irrelevant information, you’re making the task harder before generation even starts.
The model may not actually have the whole conversation
There’s another source of confusion here: the conversation you see on screen isn’t necessarily the exact context being sent to the model.
AI applications have to manage context somehow, especially as conversations get long. System instructions, tool definitions, uploaded files, retrieved documents, and tool output all use context too, so the available space isn’t always just the visible chat history.
Different products manage this differently, but there are a few common approaches.
Context compaction
One option is context compaction. Instead of continuing to send the entire raw conversation, the application can condense older portions into a shorter representation and use that going forward.
That makes sense because carrying around every word from a huge conversation can become expensive and inefficient. The downside is that summaries are inherently lossy.
Maybe we had a 50-message discussion about a site redesign and one of those messages said, “Never put the navigation over the hero image.” A compacted version might summarize the whole discussion as, “User wants a clean, modern redesign with updated navigation.”
That’s not necessarily a bad summary, but the specific hero requirement is gone. If that happens, the model isn’t failing to recall information that is still available. The information is simply no longer part of the active context.
Retrieval and RAG
Another approach is retrieval. Instead of putting everything in the prompt, the application stores information elsewhere and pulls back the pieces it thinks are relevant.
That’s the basic idea behind retrieval-augmented generation, or RAG. RAG is incredibly useful because it lets a model work with information collections far larger than any one context window.
We’ve built systems around it for exactly that reason. But retrieval creates its own failure mode: if the search retrieves the wrong chunk, the model sees the wrong information. If it doesn’t retrieve an important chunk at all, the model has no way to know that something useful was left behind.
You can have a very capable model give you a bad answer because the system around it supplied incomplete evidence. In that case, improving the model alone doesn’t solve the problem because the failure happened upstream.
Persistent memory
Then there’s persistent AI memory, which is another layer entirely. When an AI product remembers something about you or a project across conversations, that usually doesn’t mean it’s replaying every old conversation word for word.
The product is storing some representation of useful information and bringing selected parts of it back when they’re relevant. That’s useful, but by definition it’s selective.
So when ChatGPT or Claude appears to “forget” something, several different things could have happened. The information might still be in context but fail to influence the answer enough. The context might contain several conflicting versions. The old conversation might have been compacted. A retrieval system might have failed to pull the right piece. Or the information might exist somewhere else but never have been included in the current request.
From the user’s side, all of those look like the same thing: the AI forgot. Technically, they’re very different problems.
This doesn’t make big context windows less useful
None of this means giant context windows are overrated. I want models to have more context, especially for the type of work we’re doing now.
If I’m working with code, having access to more of the codebase is usually better than looking at one file at a time. If I’m comparing contracts, I’d rather have both original contracts available than two summaries. If an agent just spent an hour researching something, keeping that research available can be much better than throwing it away after every step.
The alternatives all have tradeoffs too. Summaries lose detail. Retrieval can return the wrong thing. External memory has to decide what gets stored. Multiple agents introduce handoff problems. Large context lets the model work directly with more of the original source material, which is a real advantage.
The mistake is assuming that because something fits in the context, the model will automatically use it correctly. That’s where the expectation gets ahead of what current systems actually do.
A million-token context isn’t the same as a million-token task
This distinction is easy to miss when looking at long-context benchmarks.
Suppose I give a model a million tokens and ask:
What’s the invoice number on the Acme invoice?
The answer might depend on ten tokens inside that entire context.
Now give it the same million tokens and ask:
Create a final strategy that incorporates every relevant requirement, contradiction, dependency, decision, and exception in this material.
Same context size, completely different task.
That’s why needle-in-a-haystack tests are useful but limited. Finding one unique fact deep inside a giant prompt proves that the model can access information at that depth. It doesn’t prove that the model can identify and correctly apply 100 different facts scattered across the same context.
Real project work tends to look a lot more like the second example.
Small misses start to compound
This is where the reliability math gets interesting.
Let’s say, purely as an illustration, that a model has a 97% chance of correctly applying any one requirement. That’s pretty good.
Now give it 30 independent requirements and require every one of them to be correct. If each requirement really had an independent 97% success rate, the probability of getting all 30 right would be:
0.97³⁰ ≈ 40%
That’s not a benchmark for ChatGPT, Claude, Gemini, or anything else. Real errors aren’t independent, and some requirements are much easier than others.
The point is that high reliability on individual details doesn’t automatically translate into high reliability on a deliverable that requires dozens of details to be right simultaneously. That’s one reason a model can produce something that looks 95% right and still miss several requirements you care about.
You see the same problem in agent workflows. If you have a long chain of operations and each step has a small chance of failure, the reliability of the whole chain can be noticeably lower than the reliability of any one step.
AI agents make the context problem much bigger
This becomes even more important once you move beyond chat into AI agents.
A chatbot has the conversation. An agent may also have system instructions, a plan, search results, API responses, files, source code, database records, tool definitions, previous actions, error messages, observations, structured state, and output from other agents.
Let that system run for an hour or two and you’ve accumulated a lot of information. You can’t just keep dumping everything the agent encounters into the context forever and assume more information makes it smarter.
At some point, you need to decide what stays, what gets summarized, what belongs in structured state, what should be retrieved only when needed, and which decisions need to remain authoritative for the whole run.
That’s where a lot of the real AI engineering work is moving. Getting an agent to complete a task once is one problem. Getting it to keep performing reliably after it has accumulated history, tool output, revisions, and edge cases is a much harder one.
Why reasoning and multi-step workflows help
The original discussion got to what I think is the most practical fix: stop asking the model to gather everything and write the final answer in one pass.
If I tell an AI, “Write the final plan and remember everything we’ve discussed,” I’m asking it to identify the requirements, resolve conflicts, determine which versions are current, organize them, and write the final plan at the same time.
A better workflow is:
- Extract every current requirement and decision.
- Identify anything outdated, conflicting, or ambiguous.
- Build a clean specification from the current state.
- Generate the final output from that specification.
- Validate the output against the specification.
Same source material, but now each stage is focused.
This is also one reason reasoning models and agentic workflows tend to handle complex tasks better than a quick one-shot answer. They can spend intermediate steps gathering and structuring information before committing to the final result.
That doesn’t make them infallible, but it turns one diffuse problem into several smaller ones.
How I’ve started handling longer AI projects
For bigger projects, I’ve started treating the conversation as a workspace instead of the permanent source of truth. The conversation is where we work things out. The specification is where the current decisions live.
That sounds like a small distinction, but it makes long-running AI work much easier to manage.
1. Extract the requirements
Before asking for the final deliverable, have the AI go back through the project history and pull out the things that actually matter: requirements, constraints, decisions, dates, budgets, preferences, exceptions, dependencies, and unresolved questions.
For anything complicated, assign identifiers. Now you’ve got something concrete to review instead of assuming the model picked up every important detail.
2. Mark old decisions as old
If something changed three times, don’t leave all three versions floating around as if they’re equally valid.
For example:
R-014: Project Budget
- Version 1: $20,000, superseded
- Version 2: $25,000, superseded
- Version 3: $30,000, approved
Now the current state is explicit instead of something the model has to infer every time.
3. Keep a current source of truth
That could be a project brief, specification, decision log, CLAUDE.md, requirements document, or structured state inside an application.
The format isn’t the important part. The important part is that there is one place containing the current version of the project.
The full conversation can still be available for background and detail, but the model doesn’t need to reconstruct the current state from hours of discussion every time you ask it to do something.
4. Generate from that specification
Once the current state is clean, build from it. Write the proposal, build the feature, create the page, generate the project plan, or whatever the output is.
Now the information that actually controls the task is close to the request instead of scattered across a huge conversation.
5. Validate separately
This is the part I wouldn’t skip.
Have the model compare the finished result against the approved requirements and identify anything that’s missing, contradicted, or impossible to verify. For code, back that up with tests. For numbers, calculate them independently. For structured data, validate it programmatically. For a proposal, check the scope and totals.
AI reviewing its own work is useful, but it isn’t proof that the work is correct. The more important the output is, the more you want deterministic checks or human review around the parts that matter.
Should you just start a new chat?
Sometimes, but I don’t think “the chat is long, start over” is a great rule by itself. Starting a new conversation creates the opposite problem because now you’ve thrown away useful context too.
A better question is whether the project still has a clean representation of the current state. If the conversation is full of old drafts, abandoned ideas, contradictory instructions, and previous versions, it may be worth creating a clean specification and starting a new thread from that.
You’re basically doing your own controlled form of compaction, except you’re deciding what survives instead of leaving that entirely up to the application.
For long-running projects, that usually works better than continuing indefinitely inside one giant conversation.
Large context also has a cost
If you’re building against an API, there’s also a practical reason not to keep everything in context forever. More input tokens mean more processing, and usually more cost.
In a long-running conversation or agent workflow, much of the same history may also need to be available on later requests. Providers use techniques like prompt caching so unchanged portions of the context can be reused more efficiently.
It’s worth separating caching from memory here too. Caching makes repeated context cheaper or faster to process. It doesn’t make the model understand that context better.
Compaction reduces the amount of context being carried forward, but now you risk losing details. Retrieval keeps the active context smaller, but you have to trust the retrieval system to bring back the right information.
There isn’t one approach that wins everywhere. Good systems usually combine them.
The practical takeaway
The biggest change for me was stopping thinking about a context window as “the AI’s memory.” It’s closer to working context.
The model can have an enormous amount of information available and still need to figure out which pieces matter, which version is current, how different details relate, and what actually applies to the current request. At the same time, the application around the model may be compacting history, retrieving documents, adding persistent memory, caching repeated information, and deciding what the model sees in the first place.
Once you look at it that way, AI “forgetting” something stops being quite as mysterious. It also changes how you build with these systems.
Bigger context windows are useful, and they’ll probably keep getting bigger. The harder problem is deciding what belongs in that context, what should stay authoritative, what can be summarized, what needs to be retrieved later, and how you verify that the model actually used the information you cared about.
Prompt engineering still matters, but for the systems we’re building now, context engineering is starting to matter just as much. Once AI is working across codebases, tools, documents, agents, and long-running projects, that may be the more important problem to solve.
If your team is running into these limits on real projects, our AI consulting and development team can help you work out what belongs in the context, what gets retrieved, and how to check the output.
