You know that feeling when you’re reading a 300-page PDF, and someone asks about page 47? You don’t re-read everything. You flip to the right section, skim for relevant bits, and piece together an answer. If you have a really great memory (and, more importantly, great recall), you can reference what you read right off the top of your head.
Current AI models? Not so smart. They try cramming everything into working memory at once. Once that memory fills up (typically around ~100,000 tokens), performance tanks. Facts get jumbled due to what researchers call “context rot,” and facts get lost in the middle.
The fix, however, is deceptively simple: stop trying to remember everything.
MIT’s new Recursive Language Model (RLM) approach flips the script entirely. Instead of forcing everything into the attention window, it treats massive documents like a searchable database that the model can query on demand.
Here’s the core insight
- The text doesn’t get fed directly into the neural network.
- Instead, it becomes an environment the model can programmatically navigate.
- Think of an ordinary large language model (LLM) as someone trying to read an entire encyclopedia before answering your question.
- They get overwhelmed after a few volumes.
- An RLM is like giving that person a searchable library and research assistants who can fetch exactly what’s needed.
The results = RLMs handle inputs 100x larger than a model’s native attention window. We’re talking entire codebases, multi-year document archives, and book-length texts. They beat both base models and common workarounds on complex reasoning benchmarks. And costs remain comparable because the model processes only relevant chunks.
Why this matters
Traditional context window expansion isn’t enough for real-world use cases. Legal teams analyzing entire case histories, engineers searching entire codebases, and researchers synthesizing hundreds of papers: all need fundamentally smarter ways to navigate massive inputs.
The original research from MIT CSAIL’s Alex Zhang, Tim Kraska, and Omar Khattab includes both a full implementation library that supports various sandbox environments and a minimal version for developers to build on.
Also, Prime Intellect is already building production versions.
Instead of asking “how do we make the model remember more?”, researchers asked “how do we make the model search better?” The answer, treating context as an environment to explore rather than data to memorize, might just be how we get AI to handle the truly massive information challenges ahead.
Editor’s note: This content originally ran in the newsletter of our sister publication, The Neuron. To read more from The Neuron, sign up for its newsletter here.