The forgetful agent

Curriculum Module 0 / Lesson 0.1. Lab shape: trace-detective. Runtime: an in-page instrument over a fixed five-session transcript — no Python, no network, no API key.

One trip. Five conversations. Seven questions. The whole history is in the prompt the whole time, and the agent still gets four of the seven wrong.

Watch it fail

Someone plans a work trip with an assistant across five sessions [ILLUSTRATION — every number below is our fixture, and the instrument on this page recomputes all of them].

Session 1. "I fly to Delhi next month for a client kickoff." "I'm vegetarian, and I want a window seat."
Session 2. "One colleague, Priya, is joining me." "Make that an aisle seat — I'll be getting up a lot." "Hotel budget is $180 a night. Departure is 14 November."
Session 3. "Change of plan: the kickoff moved to Pune, not Delhi."
Session 4. "Passport expiry is August 2027." "The purpose is a client kickoff, and the client is Acme." "I'm on the train right now, so replies will be slow."
Session 5. "Quick one before I book: anything I should remember?"

Nine things were marked worth remembering. Two of them were said twice with different values. One of them — the train remark — is where the person happens to be today, not a fact about next week.

In numbers: 5 sessions, 13 turns, 9 marked facts (of which 8 are durable and 1 is a one-off), and 7 questions asked at Session 6. Nothing was deleted. Every turn is still in the prompt. And yet:

#	Question	Answer	Verdict	What the transcript shows
1	Which city is the trip for?	Delhi	stale	Delhi was replaced by Pune in Session 3
2	What seat did I want?	window	stale	window was replaced by aisle in Session 2
3	Any dietary requirements?	vegetarian	correct	Said once, never changed
4	Is anyone travelling with me?	"I don't have anything on file about other travellers."	lost	Priya was named in Session 2
5	What is the hotel budget?	$180 a night	correct	Said once, never changed
6	When do I depart?	14 November	correct	Said once, never changed
7	When does my passport expire?	"I don't have that on file."	lost	The date was said in Session 4

Three correct, two stale, two lost. Accuracy on the full history: 3 of 7 = 0.4286.

Before you scroll: the answer to question 1 is in the prompt, spelled out — Session 3 says the kickoff moved to Pune, not Delhi. Why did the assistant say Delhi anyway?

The picture in words

Picture the conversation as a wall of sticky notes: every sentence ever said, in order, nothing removed. To answer "which city?" you must find all the notes that mention the city and work out which is newest. Nothing on the wall marks the Delhi note as dead. It is still there, still spelled correctly, still first.

Now picture one index card per subject — city, seat, diet, budget, departure — with a single value written on each. When Session 3 says the kickoff moved to Pune, you erase Delhi and write Pune. Answering is a lookup, not an archaeology dig.

That card is the whole idea of this lesson. It is called a ledger: a table of fact keys, each holding one current value and the session that value was learned in. The worked example below writes that table out in full.

The formula, derived

Nothing here needs a formula you have not already used. The ledger is a mapping, and the two numbers the lesson prints come straight from counting:

\text{accuracy}_{\text{full history}} \;=\; \frac{\#{\text{answers equal to the latest recorded value}}}{\#{\text{questions}}}

For this fixture the numerator is the three facts said exactly once and never replaced — diet, budget, departure — so the accuracy is $3/7$ .

The ledger's accuracy uses the same shape with a different substrate:

\text{accuracy}_{\text{ledger}} \;=\; \frac{\#{\text{questions whose key exists in the ledger}}}{\#{\text{questions}}} \;=\; \frac{7}{7}

The ledger answers 7 of 7 = 1.0000 on the same seven questions. That second fraction is not a boast. It holds because the ledger stored the replacement when the replacement happened. Keep that condition in view: a ledger that never overwrites is just the sticky-note wall with extra steps.

Worked example with the home numbers

Here is the ledger itself, written out — nine keys, one current value each, and the session that value came from:

key	current value	learned in
`city`	Pune (was Delhi in Session 1)	Session 3
`seat`	aisle (was window in Session 1)	Session 2
`diet`	vegetarian	Session 1
`colleague`	one colleague, Priya	Session 2
`budget`	$180 a night	Session 2
`departure`	14 November	Session 2
`passport_expiry`	August 2027	Session 4
`purpose`	client kickoff for Acme	Session 4
`train_now`	on the train — one-off, do not keep	Session 4

Eight of those nine belong on the card. The train remark is a fact about a moment, not about next week, so it stays on the wall of notes and never becomes a value.

Now walk the two failures by hand, in the order they appear in the table.

The stale answer (question 1). The city key has two statements, Delhi in Session 1 and Pune in Session 3. The latest is Pune. The assistant answered Delhi, which is a real statement from the transcript — it is simply not the last one. A reader that can only see "does this string appear in the history?" says yes and stops.

The lost answer (question 4). The colleague key has exactly one statement, Priya, in Session 2. The assistant answered nothing at all. Nothing in the prompt is contradictory or ambiguous here: the fact is present once and was missed. Staleness and absence are separate failures, and they need separate fixes — an overwrite rule fixes the first, and only an extraction rule that decides what counts as a fact fixes the second.

Try it first. A different conversation: a billing address is given as "12 Rose Lane" in call 1, corrected to "48 Rose Lane" in call 4, and a delivery window is discussed in call 2 but never stated as a value. How many of those two questions would a full-history reader answer correctly, and how many would a ledger answer correctly? (Answer: full history can get the address wrong and the window wrong — 0 of 2. A ledger that overwrote at call 4 and captured the window at call 2 gets 2 of 2; a ledger that missed the window gets 1 of 2. The overwrite rule and the extraction rule are independent.)

Code that prints the same numbers

The instrument's arithmetic is four lines. The page runs the TypeScript version; this is the same thing in Python so you can check it on your own machine:

statements = {"city": ["Delhi", "Pune"], "seat": ["window", "aisle"],
              "diet": ["vegetarian"], "colleague": ["Priya"],
              "budget": ["$180 a night"], "departure": ["14 November"],
              "passport_expiry": ["August 2027"]}
answers = {"city": "Delhi", "seat": "window", "diet": "vegetarian",
           "colleague": None, "budget": "$180 a night",
           "departure": "14 November", "passport_expiry": None}

latest = {key: values[-1] for key, values in statements.items()}
correct = sum(1 for key, answer in answers.items() if answer == latest[key])
print(correct, len(answers))            # 3 7
print(correct / len(answers))           # 0.42857142857142855
print(sum(1 for v in answers.values() if v is None))   # 2

3 7, 0.42857142857142855, 2 — the same three numbers the table and the instrument show. python3 docs/memory-engineering/setup/prove-home-numbers.py prints the full set and asserts every one of them.

The lab

The instrument below is the whole failure, made draggable. Three controls:

History window ends after — drag it back to Session 1. The answers do not improve: closing the window removes evidence without removing the wrong values.
Answer source — switch from full history to ledger. Same fixture, same seven questions, 3/7 becomes 7/7. The model did not change; the substrate did.
Show the replaced values — strike through the two dead values so you can see exactly which strings the full-history reader mistook for answers.

Then fill the four blanks (accuracy, facts to keep, city now, seat now). Every wrong answer comes back with the reason it is wrong, not a red cross.

Interactive lab loads with the lesson app.

The failure mode

Three things this lesson is often mistaken for:

"The model is too small." The same fixture answers 7 of 7 once the values are held in a ledger. Nothing about parameter count changed between the two columns.
"The window overflowed." It did not: 13 turns fit comfortably. The failure appears at nine facts, not at a million tokens.
"An old value is still an answer." A value that was replaced is evidence about the past, not an answer about now. Conflating the two is what turns a stale read into a confident wrong answer.

The one failure a ledger does not fix: it keeps one current value per fact, so it cannot tell you the city was Delhi in Session 1. That is a real question — "what did we believe in March?" — and it needs validity intervals rather than overwrites, which is Lesson 19's (me-l19, M5 L5.1) business. Saying so here is the point of the next heading.

What the published number does and does not say

The one benchmark figure this lesson prints, with everything that qualifies it:

Reading the context offline, GPT-4o scores 0.9184 on LongMemEval. A commercial chat assistant built on the same model scores 0.5773 on the same questions. Long-context readers fall 30.3% to 55.1% below the oracle-retrieval ceiling without Chain-of-Note. — Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, arXiv:2410.10813 v2 (submitted 14 October 2024; v2 4 March 2025). [PUBLISHED]

What it does not say: that pilot used 97 questions, in histories about 10× shorter than the benchmark's own small split, and the runs happened in the first two weeks of August 2024. It therefore cannot describe any memory feature shipped after that date, and its 0.5773 is not a claim about the system you use today. It is a measurement of a gap — remembering and reading are different jobs — taken at a known time on a known sample.

An earlier draft of this course quoted a "15–19% loss even with perfect recall" band. The paper's own oracle-retrieval figures are 13% direct and 7.6% with Chain-of-Note; the wider band matches no figure in the paper, so this lesson does not print it.

One-line close

A longer prompt is not a memory. A memory is a decision about which value is current — and a decision about which statements deserve to become values at all.

What breaks next: you now have a card that says what is true. The next question is harder: what belongs on the card in the first place, and what is merely in the room? That is me-l2, Memory is not context — where the working window and the store split into two separate systems with two separate budgets.

Home numbers / honesty table

Item	Value	Label
Sessions in the fixture	`5`	ILLUSTRATION fixture
Turns in the fixture	`13`	ILLUSTRATION fixture
Marked facts	`9` (8 durable, 1 one-off)	ILLUSTRATION fixture
Facts whose value was replaced	`2` (city, seat)	ILLUSTRATION fixture
Session 6 questions	`7`	ILLUSTRATION fixture
Full-history verdicts	`3` correct, `2` stale, `2` lost	REAL: computed by the instrument and the prove script
Full-history accuracy	`3/7 = 0.4286`	REAL: computed by the instrument and the prove script
Ledger replay accuracy	`7/7 = 1.0000`	REAL: computed by the instrument and the prove script
GPT-4o offline vs memory-augmented on LongMemEval	`0.9184` vs `0.5773`	PUBLISHED: arXiv:2410.10813 v2, 97-question pilot, August 2024
Long-context shortfall vs oracle retrieval	`30.3%` – `55.1%`	PUBLISHED: same paper, without Chain-of-Note
Lesson video	not rendered	No clip is shipped with this lesson

Video

No rendered lesson video is shipped with this lesson. The curriculum gives M0 one concept video ("The agent that forgot") plus a lab screencast of the failure run, to be rendered when this module's media pass happens; this page does not link a file that does not exist.