Why Your AI Forgets: The Context Window, Explained

Your AI only sees what fits in one window. This 500x Explained draws that window in code: a row of token blocks sliding through a lit frame, the oldest ones dropping off the back, and a dim patch in the middle.

The window

When you chat with an AI, the whole conversation, your messages and its replies, is turned into tokens and sent to the model each turn. The span of tokens the model can look at while it writes is its context window. Anthropic's docs call it the model's "working memory", separate from the data it was trained on.

It has a fixed size

Windows have grown large. Anthropic lists a 1M-token context window for models such as Claude Opus 5.5, while some other Claude models have 200k. Large still means limited: if the input alone is longer than the window, the API rejects it.

When it fills up

In a long chat, the window eventually fills, and something has to give. Anthropic notes that chat interfaces such as claude.ai can manage the window on a rolling "first in, first out" basis, which drops the oldest turns first. Another approach is compaction, which replaces older turns with a summary the model writes. Either way, the earliest details are the first to go, dropped or squeezed into a summary.

The middle gets missed

A fact can sit inside the window and still get missed. In "Lost in the Middle" (Liu et al., 2023), researchers moved the one relevant piece of information around inside long inputs. Performance was often highest when it sat at the beginning or the end, and dropped when the model had to find it in the middle, even for models built for long contexts. The curve in the video is a sketch of that U shape, drawn for this video; the paper's own figures have the measured numbers.

So two things go wrong: old words fall out the back, and the middle goes dim.

Use it

• Long prompt? Put the key instructions at the start or the end. Anthropic suggests placing long documents first and your question after them, and says queries at the end can improve response quality by up to 30 percent in its tests. OpenAI's GPT-4.1 guide suggests putting instructions at both the beginning and the end of long context, and above it if you only state them once.
• Long chat? Restate the key facts, so a fresh copy sits near the end of the window, where the model uses it best.
• New task? Start a fresh chat. A clean window holds only what matters for the new job.

So when your AI forgets, check the window: what's in it, and where.