Memorilla beat full context on FactKG with 108 times fewer tokens, according to the researchers
A small Stanford-developed memory module that compresses an entire document collection into 16 latent tokens for a frozen large language model was open-sourced on November 14, 2026. The numbers in the research paper are impressive — but two days later, independent testers reported fabrication and detail loss, and the latency figures from the two sources contradict each other.
The news
On November 14, 2026, the researchers behind the preprint "Memorilla: Latent Semantic Memory for LLMs" (arXiv:2610.21375) released the system under the Apache 2.0 license, with code, training recipes and checkpoints publicly available. The paper is written by Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis and Jure Leskovec, with affiliations spanning Stanford, Toronto, Penn, the Arc Institute and MIT. Coverage followed quickly: MarkTechPost and CASRAI summarized the release on November 14, and Geeky Gadgets published its own testing on November 16, conducted with analysis contributed by Better Stack.
The core of the system is described in the abstract: "Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM." In practice, that means an entire collection of documents — for example a user's chats, book chapters or research papers — is squeezed down to exactly 16 tokens that are inserted into the context of a language model that is never retrained.
The underlying idea is not that the model remembers everything, but that it learns to retrieve what matters for a given query. According to the preprint, Memorilla encodes documents once into reusable embeddings, and then learns to extract and place query-specific information directly into the language model's input embedding space.
How it works
The architecture has three parts, only one of which is trained:
-
A frozen encoder. The documents are pre-encoded once with Qwen3-Embedding-4B, which is held untouched. Because this encoding is query-agnostic, the embeddings can be stored and reused across many queries.
-
A small, trainable memory module. This is the system's only trainable component: a cross-attention module with exactly 194,123,776 parameters for the 8B decoder. The module takes the document embeddings plus the current query and produces the 16 latent memory tokens — meaning the memory is query-conditioned: the same document collection yields different 16 tokens depending on what you ask about.
-
A frozen decoder. The 16 tokens are injected into an unmodified Qwen3-8B model. No fine-tuning of the language model itself is required, which means the approach can in principle be applied to any compatible model without touching its weights.
The training recipe proceeds in three stages, according to MarkTechPost: stage 1 uses around 1.98 million Wikipedia examples for paraphrasing and reconstruction; stage 2 adds around 1.34 million examples from ten QA and summarization datasets; stage 3 fine-tunes on all eight benchmarks the reports build on. The entire recipe takes around 19 hours on four H100 GPUs — modest compared to training the base model itself.
The system also has an optional KV memory cache that shifts the query from "re-encoding per query" to a key–value lookup, which is where the most dramatic latency claims come from (more on that below).
The numbers the research team itself reports
All benchmark figures below are self-reported by the research team in the preprint, and should be read with that caveat.
- FactKG: 84.33% accuracy versus 80.43% for full-context inference, while using 108 times fewer tokens to represent the source documents. In other words, Memorilla beats full context on this benchmark — at a fraction of the token budget.
- TextWorld (agentic setting): According to the abstract, Memorilla reduces the agent's tendency to wander between rooms it has already explored, improving the average success rate from 18.6% with full history via in-context learning to 24.2%.
The latency profiling, relayed via MarkTechPost and CASRAI, is just as remarkable: on a single H100, Memorilla answers in 0.74 seconds at 512 documents (roughly 128,000 tokens), versus 19.9 seconds for full context — 27 times faster. The memory module itself takes about 2 milliseconds, according to the coverage. With the optional KV cache, the CASRAI summary measures 8.1 milliseconds for queries against one million documents on an NVIDIA B200 (5.0 ms with torch.compile), and 33 milliseconds and 80 GiB of VRAM at four million documents — roughly one billion tokens — fitting within a single accelerator.
If these numbers hold up in practice, it means the memory share needed to "remember" a million-document collection per query becomes nearly negligible, and that response time does not grow with collection size the way full context does.
Where the system loses
The research team's own benchmark table shows that Memorilla does not win everywhere, and the losses follow a pattern:
- PubMedQA: 59.00% versus 69.00% for LLMLingua-2. According to MarkTechPost, this is understandable: the abstracts in this dataset total only around 324 tokens, so there is almost nothing to compress — and with a fast compression option like LLMLingua-2 that keeps the text nearly verbatim, it obviously wins.
- PersonalizationV4: MemoRAG leads with 67.88% versus Memorilla's 66.65%.
- NarrativeQA (ROUGE-L): MemoRAG 20.51 versus Memorilla's 19.31.
One interpretive point worth noting: stage 3 of training fine-tunes on exactly the eight benchmarks that are reported. That is not misconduct — it is openly described in the coverage material — but it means the numbers in the table describe performance on tasks the module has been optimized against, which makes generalization to new domains an open question.
It is also worth noting that the ratio between 16 tokens and full context varies widely depending on how it is calculated: the preprint's abstract describes 16 tokens as "394× fewer than full context," the FactKG coverage states 108×, and CASRAI's table states roughly 2,000× token compression on PersonaMem-v2 (107 chats) — a figure that can only be attributed to CASRAI, not the preprint. The different figures are not reconciled in the available material, and the reader should not blend them into a single "compression number."
The independent test: the benchmarks hold, the details do not
On November 16, Geeky Gadgets published testing conducted with analysis attributed to Better Stack, in which the benchmark results were run on rented GPUs. According to Geeky Gadgets, the results were "reproduced almost word for word" — but no methodology for the reproduction has been published, which means the claim cannot be independently verified.
The practical testing was far less flattering. Geeky Gadgets concluded that "accuracy continues to be a major limitation, with Memorilla struggling to retain precise details, fabricating information and failing to retrieve specific facts in tests — such as answering only one of eight plot questions about 'The Hound of the Baskervilles' correctly." The tester's verdict was that the system is not production-ready, despite the benchmarks being reproducible.
This is perhaps the story's most important tension: the system delivers solid aggregate numbers on benchmark questions, but the 16 tokens appear to function as a lossy map rather than a lossless archive. For tasks where coarse semantic retrieval is enough, it may be good enough; for tasks where a specific detail — a date, a name, a plot event — is the answer, the model cannot just fail, it can fabricate.
The latency conflict nobody has resolved
Here lies a documented contradiction in the public material:
- The research team's profiling (relayed via MarkTechPost and CASRAI): 0.74 seconds on a single H100 at 512 documents, with the module itself at around 2 ms, and millisecond-level query times with the KV cache on B200 hardware.
- Geeky Gadgets' testing: "Compressing 128,000 tokens from Sherlock Holmes novels into 16 memory tokens reduced response times from over 19 seconds to about 3 seconds. Response time remains steady at around 0.12 seconds when processing up to 512 documents."
The two sources agree on the starting point (over 19 seconds for full context at ~128,000 tokens) and on the fact that answer generation alone is in the hundredths of a second. But 0.74 seconds and ~3 seconds end-to-end are far from the same number, and none of the available sources explains the discrepancy. It could be different hardware, different pipeline stages, or a communication error — but it is impossible to determine based on the available documentation, and until that is clarified, neither figure should be used as a promise of real-time performance.
What this means — and what remains open
Memorilla points in an interesting direction in long-context research: instead of expanding the context window, you train a small, trainable extraction layer that distills an entire collection into a few tokens per query. Because the encoder and decoder are kept frozen, the training cost is low (19 hours on four H100s), the license permits commercial use, and the recipe is in principle reproducible for anyone with the GPUs.
But the independent test suggests the current version should be read as a research artifact, not an infrastructure component. The list of open questions is concrete:
- The latency discrepancy between the team's 0.74 s and the testers' ~3 s at the same scale is unexplained.
- The reproduction methodology behind the claim that the benchmarks were reproduced "almost word for word" has not been published.
- Fine-tuning on the benchmarks themselves in stage 3 makes it unclear how well the system generalizes beyond the eight tests.
- The 16-token bottleneck is inherent: if the query needs a precise detail that was not placed into the tokens, the system cannot recover it — documented by the Baskervilles test.
- No primary documentation is available to AIMag beyond the secondary summaries and preprint excerpts regarding release details such as repository structure or model cards.
For those considering compressed memory generally, the lesson may be the most important part: aggregate averages can conceal the fact that a system loses exactly the details many use cases exist for. Memorilla shows that you can compress an entire library collection into 16 tokens — the question is whether the reader can trust what comes back out.

