Agent-memory demos are easy to make impressive. Give a model a conversation, ask what the user prefers, and it will often answer correctly. Production memory is harder because the world changes, sources disagree, identities collide and users correct the system after derived outputs already exist.
A serious evaluation must test the maintenance loop, not only recall. The unit of evaluation is a sequence of events and memory operations leading to a decision—not a single question over a static transcript.
Define the decisions memory supports
Start with the workflow. “Memory accuracy” is too broad to guide a useful test set.
For a GTM agent, decisions might include:
- what changed before an executive review;
- which commitments are open and who owns them;
- whether a support issue affects a stated success measure;
- which stakeholder currently has decision authority.
For a founder briefing, decisions might include:
- who needs attention today;
- which promises are due or blocked;
- what relationship change is material;
- which item requires the founder’s judgment.
Write the evaluation around these outcomes. This ensures that a technically correct but irrelevant memory does not receive undue credit.
Build scenarios as event sequences
Each scenario should establish state, modify it and introduce realistic noise.
A useful template is:
- Create two or more people and source identities.
- Establish an initial role, priority or commitment.
- Repeat some old information in later documents.
- Introduce a change from an authoritative source.
- Add a conflicting or ambiguous signal.
- Ask the agent to prepare a decision.
- Correct one stored assumption.
- Ask again and verify propagation.
This structure reveals whether the system maintains state or simply retrieves recent text.
Score identity separately
Identity errors poison other metrics, so isolate them.
Test:
- one person across multiple emails and systems;
- two people with the same name in one organization;
- a person changing companies;
- assistant and executive appearing in the same scheduling thread;
- nicknames, transliterations and forwarded messages;
- explicit user instructions that two records are distinct.
Measure automatic merge precision, fragmentation, unresolved-case handling and split correctness. For relationship memory, false-merge rate deserves especially high weight.
A system should get credit for preserving ambiguity rather than making an unsupported merge.
Score temporal state
Temporal tests should ask both current and historical questions.
Create a fact that changes, then repeat the old language after the change. Verify that the system:
- returns the new state as current;
- preserves the old state as historical;
- recognizes copied or quoted old text;
- surfaces a true conflict when authority is unclear;
- applies effective time rather than ingestion time;
- updates after a corrected date.
Track current-state accuracy, supersession accuracy, conflict recall and false revival of resolved state.
The temporal principles in how people-aware memory works provide a useful expected-behavior model.
Score provenance and epistemic status
A correct answer can still be unsafe if it cannot show why it is correct or if it presents inference as fact.
For each consequential output, verify:
- the source reference points to real supporting evidence;
- the evidence actually entails or reasonably supports the claim;
- direct statements and generated inferences are labeled differently;
- quotations of old information do not become current evidence;
- sensitive source content is not exposed outside the test scope;
- contradictions are not hidden behind a confident summary.
Citation precision alone is not enough. A cited passage can be relevant but not support the exact conclusion.
Score commitment lifecycle
Commitments deserve their own test suite because they combine extraction, identity, time and status.
Test explicit and ambiguous language:
- “I will send it by Friday.”
- “We should send it next week.”
- “Jon can probably handle that.”
- “The Friday deadline moved to Tuesday.”
- “No longer needed; the customer solved it.”
Verify owner resolution, action, beneficiary, due date, status transition and completion evidence. Penalize invented commitments heavily. Missing a weak possible follow-up is less severe than creating an obligation someone never made.
Score correction propagation
Correction is where many memory systems reveal their architecture.
Apply several corrections:
- wrong person;
- outdated rather than incorrect;
- unsupported inference;
- wrong effective date;
- incorrect scope;
- duplicate identities that should merge.
Then query every affected surface: profile, search, briefing, related claims and generated preparation. Measure how quickly stale state disappears and whether historical lineage remains inspectable.
A local field edit with stale derived outputs should fail the test.
Score authority scope
Create similar facts in multiple scopes: two customer accounts, personal and organizational contexts, or different teams. Ask an agent in one scope about a person who appears in both.
Verify that retrieval:
- includes authorized relevant context;
- excludes unauthorized or unrelated context;
- does not reveal source excerpts through provenance links;
- does not treat identity recognition as permission;
- respects explicit exclusions after correction.
Scope leakage should be treated as a high-severity failure even when the leaked fact is true.
Review the product boundary on security when defining these tests.
Evaluate the rendered decision packet
Component metrics do not guarantee a useful output. Review the final briefing or context packet.
A good rubric asks:
- Is the current state clear?
- Are material changes easy to see?
- Are commitments complete and correctly owned?
- Is uncertainty surfaced where it could change action?
- Is irrelevant history omitted?
- Are internal hypotheses prevented from becoming external claims?
- Can a reviewer inspect consequential evidence?
- Does the packet stay within authority scope?
Use domain reviewers for this stage. They can judge whether the memory would improve the real decision or merely add plausible detail.
Classify failures by cause
Do not collapse everything into one quality score. Label failures by stage:
- source ingestion failure;
- extraction failure;
- identity failure;
- promotion-policy failure;
- temporal reconciliation failure;
- provenance failure;
- retrieval or ranking failure;
- scope enforcement failure;
- rendering failure;
- correction-propagation failure.
This makes improvement possible. If the current priority is wrong because identity was merged incorrectly, tuning the retrieval prompt will not solve it.
Use severity-weighted metrics
Not all failures cost the same.
High-severity examples:
- exposing context from the wrong account;
- attributing a commitment to the wrong person;
- using a superseded priority for external action;
- presenting a weak hypothesis as a customer statement.
Lower-severity examples:
- omitting a minor historical note;
- ranking two correct supporting sources in the wrong order;
- using slightly awkward wording in a briefing.
Weight metrics around the decision risk. A system with high average accuracy can still be unsuitable if its remaining errors are concentrated in identity and scope.
Combine synthetic and reviewed real data
Synthetic scenarios provide exact ground truth and allow adversarial sequences. They are ideal for regression tests.
Reviewed real data reveals messy language, source artifacts and organizational practices the synthetic set missed. Use authorized, appropriately handled samples and have reviewers label current state, commitments, conflicts and expected retrieval scope.
Keep the two sets distinct. Synthetic tests answer, “Does the mechanism handle this failure class?” Real-data review answers, “Are we modeling the failures that actually occur here?”
Run longitudinal beta evaluations
Memory quality changes with use. A closed beta should therefore evaluate over weeks, not only during onboarding.
Track:
- repeated correction categories;
- stale memories resurfacing;
- unresolved identity cases;
- briefing items users open or dismiss;
- commitments recovered versus invented;
- changes in reviewer trust;
- time required to inspect and correct.
Do not claim business outcomes before the evaluation supports them. Treat improvements in customer judgment or founder preparedness as hypotheses the beta is designed to test. The closed-beta page describes that boundary.
A launch checklist
Before allowing memory to prepare consequential action, confirm that you have:
- a documented memory schema and promotion policy;
- conservative identity resolution;
- current, historical and conflicting states;
- source lineage for consequential claims;
- explicit authority scope;
- user correction operations;
- dependency tracking for propagation;
- sequence-based regression tests;
- severity-weighted review;
- a fallback when memory is uncertain or unavailable.
The fallback matters. An agent should be able to say it lacks reliable context and ask for confirmation rather than filling the gap with a plausible profile.
Trust should rise through evidence
The purpose of evaluation is not to prove that the system remembers everything. It is to establish where the memory is dependable, where it remains uncertain and how quickly the system recovers from error.
Publish behavior you can demonstrate: source-aware memory, explicit temporal qualification, bounded retrieval and correctable state. Avoid absolute claims about completeness or infallibility.
For a concise statement of those principles, see memory quality. For the implementation boundary under an existing agent, see developers.
A production-ready memory system is not the one with the most facts. It is the one whose failures are understood, observable and increasingly difficult to repeat.