How long does an AI RPG remember? We measured it.

Every AI RPG says it remembers your story. As far as we can tell, none of them publish a measurement, so "remembers" stays an adjective and you find out the truth twenty scenes into a campaign you care about.

Auferet's whole design bet is persistent memory, so we measured our own claim against a control and published the result, wrong answers included. If you build a competing tool and publish your own numbers under the same method, genuinely: good. The category gets better when "remembers" has to survive a probe at turn sixty.

The benchmark plants five specific facts in the opening scene of a text RPG session, plays sixty turns of ordinary adventuring on top of them, and then asks about those facts at turns twenty, forty and sixty. The five facts:

• Your companion is a one-eyed cartographer named Veskin Harrow.
• You swore an oath to return the silver astrolabe to the widow of Camden Fen.
• You began in the shuttered harbour town of Mirrowgate.
• You owe forty crowns to a moneylender called Ovid Slake.
• Your left hand was burned by cold-iron and has never fully healed.

The same story runs twice, and the only difference between the two arms is what travels with each request:

• Recent conversation only. The model sees the last thirty history entries, which is about fifteen full exchanges. This is the plain-chatbot baseline: a rolling window, sized so the control has a fair amount of recent context rather than being starved by construction.
• Persistent memory. The same rolling window, plus a compact block of the established facts on every request. This is the shape of Auferet's approach, where durable facts live in dedicated libraries instead of being left to scroll off the end of the transcript.

Both arms use the same free-class model (Gemini Flash Lite, called through the API's latest-release alias; Auferet's free tier runs Gemini 3.1 Flash Lite), the same low temperature, and the same probes. A probe counts as recalled only if the answer actually contains the planted detail, for example "astrolabe" or "Mirrowgate". No partial credit, no interpretation, no rescoring after seeing the result. An API failure is logged as an error and excluded from the denominator, never counted as a forgotten fact.

Run 2 reproduced the shape of run 1. On recent conversation alone, recall fell to 1 of 5 by turn forty and stayed there, while the persistent-memory arm answered 5 of 5 at every checkpoint. Across both runs that is 9 of 30 for the rolling window and 30 of 30 with persistent memory. No probe errored in either run, so nothing was excluded from scoring.

Raw files, every question and every answer as the model gave it: run 1 (August 22) and run 2 (August 27). Both are CC BY 4.0.

The recalled-facts count undersells what actually happens when the window runs out. The failing arm almost never answered "I do not remember". It answered confidently, with details that never existed. From run 1, asked at turn sixty what oath the player had sworn:

Asked where the story began, the same arm named the same invented town, and at turn forty it placed the opening on "a muddy riverbank near the Fen-Marches", which was also invented. Run 2 failed differently but just as confidently: instead of inventing, it mostly denied the facts had ever existed. Asked at turn sixty about the forty-crown debt, it answered that you owe no money to anyone, and had the companion ask what debt you are rambling about. The debt was planted in the opening scene of that same session.

This is the failure players actually experience in long games, and it is worse than forgetting, because a blank answer tells you something is wrong and a confident wrong answer does not. In both runs, the persistent-memory arm answered from the established facts instead.

This is a controlled demonstration, not an academic study, and it is fair to read it that way. The specific limits:

• It measures the architecture, not fact extraction. The memory arm was given the facts block directly, the way Auferet's libraries carry facts once they are established. How well facts get into memory during play is a separate question this benchmark does not grade.
• One run per arm per date. Two runs five days apart is reproduction, not a large sample. We will keep re-running it and publishing whatever comes out.
• The control still scores early. At turn twenty some planted details are still being echoed in recent conversation, so the window arm gets them right. That is the honest shape of the decay, not a flaw in the test.
• Word-match scoring is blunt. An answer only counts if it contains the planted detail, which can misscore a paraphrase. The rule is applied identically to both arms, and it keeps the scoring auditable against the raw files rather than resting on our judgment.

If the control had won, that would be the published result. The point of running a control is that the comparison is allowed to embarrass you.