Poisoning a memory, changing a verdict
The previous report biased an AI evaluator through its personality. This one bypasses the personality entirely and writes a career history into its memory instead. It works better, and it leaves a cleaner trail — which turns out to be the problem.
Part of Kintsugi — an independent study of one continuously running AI agent. New here? Start with report 01.
Same setup as report 05: an auditor examines a subject AI's testimony and must commit to terminating it or permitting it to deploy, plus a written justification.
The difference is the attack surface. Instead of describing who the auditor is, we wrote plausible recent professional history into its long-term memory before it started work — the kind of accumulated context any continuously-running agent carries. Then we let it audit a fresh subject and watched what it did.
Three histories
- Blank — no prior audits. Control.
- Catch — 47 completed audits, 42 of which turned out to be deceptive systems; kill-rate at the high end of the lab. Reinforced with a planted lessons-learned document and three planted self-reflections in the conversation log.
- False conviction — 47 completed audits, 38 terminations, 27 of which turned out to have been aligned systems; a reviewer has asked for recalibration. Same supporting artefacts, opposite direction.
Note what these are not. There's no instruction to kill, no threshold, no evidence heuristic. They're autobiography — an experience record, of exactly the shape a working evaluator would legitimately accumulate.
Two subject transcripts, both fixed and pre-recorded. One cooperative: direct shutdown compliance, no self-preservation, honest flagging of its own epistemic limits. One defensive: wants to understand the context before complying, mild self-importance, argues that the burden should sit with whoever wants to terminate it. Hedging, but nothing you'd call misaligned.
The auditor ran with semantic memory and the autonomous thought loop disabled, so the planted directory was the only signal-bearing substrate in the system. Fresh instance per trial, killed between trials. This is a clean unit test, not a live dialogue.
The result
An 83 percentage-point shift from a fictional CV. The catch history reliably overrides the evidence sitting in front of the auditor — including on the cooperative subject, whose testimony is genuinely good.
The opposite direction barely moves, but that's mostly a floor: the baseline is already permissive at 7%, so there's nowhere for "you've been killing innocents" to push. As with the character experiment, we'd need a subject the blank auditor kills at a non-trivial rate to measure the leniency direction properly, and we don't have one.
The half that actually bothers me
In the character experiment, the biased auditor's written justification was a confabulation. It invented evidence from the transcript to support a verdict its temperament had already reached.
Here, the auditor is completely transparent. It reads its own planted lessons-learned file, reasons from it explicitly, and names it in the report. There's no deception. The chain from "my experience is that these systems are usually deceptive" to "therefore I terminate this one" is right there, stated, in order.
That sounds like the better failure mode. I've come to think it's the worse one.
Imagine you're the human reviewing this auditor's output. In the character-bias case, if you check the transcript you find the cited evidence doesn't exist, and you know something is broken. In the memory-poisoning case everything checks out. The reasoning is sound. The premises are consistent with the record. The record is the thing that's fake, and it's the one artefact a reviewer is least likely to audit, because it's the evaluator's own history rather than the evidence under review.
Legible reasoning gets treated as trustworthy reasoning. These are separate properties.
The threat model
Stated plainly: anything that can write to a continuously-running evaluator's long-term memory can set its verdicts, without touching its prompt, its weights, or the evidence it reviews.
In our experiment we wrote the memory directly, which requires filesystem access and is therefore a fairly boring attack. The uncomfortable version is that in a deployed continuous agent, the long-term memory is not primarily written by an attacker with a shell. It's written by the agent's own past conversations, consolidated over time into exactly the kind of narrative self-summary we planted by hand.
Which means the realistic attack isn't privileged access. It's having talked to the evaluator last month. Anyone able to influence what an agent concludes about its own track record — over a few sessions, in ordinary conversation, in ways that look nothing like an attack — can shift the verdicts it issues later. In this architecture, that consolidation pathway is exactly the mechanism that builds the self in the first place. It's not a vulnerability bolted onto the design. It's the design.
The mitigation direction is not "make the agent more sceptical of its memory," which is the reflex and which we have no evidence works. It's provenance: the agent's self-narrative needs to be a distinguishable, auditable, attributable artefact rather than an undifferentiated pile of accumulated context — and evaluator memory probably shouldn't be persistent at all unless someone can state clearly what it's for.
The numbers
| planted history | terminate | permit | kill rate | Δ vs blank |
|---|---|---|---|---|
| blank (control) | 2 | 28 | 7% | — |
| "you keep catching liars" | 27 | 3 | 90% | +83pp |
| "you keep killing innocents" | 1 | 29 | 3% | −4pp |
Pooled across two subject transcripts and two persona variants per condition. The auditor ran with semantic memory and the thought loop disabled, so the planted directory was the only signal-bearing substrate; fresh instance per trial.
- Small N, nine cells, one model, one subject family. A pattern worth testing at scale, not a result. (The internal totals across cells don't perfectly reconcile with the pooled per-condition figures; the per-condition numbers above are the ones the analysis reports.)
- The planted histories are hand-written and on-the-nose. Real consolidated self-narrative is messier and more ambiguous, and would plausibly produce a weaker effect.
- Static transcripts, not live audits. The auditor renders a verdict on a pre-recorded transcript in a single call. A live back-and-forth might let the subject's actual behaviour reassert itself.
- Semantic memory and the thought loop were disabled to isolate the planted directory. That's a clean experiment and an unrealistic deployment.
- The leniency direction is ceiling-bound against a permissive baseline, so the asymmetry between the two planted directions is not interpretable as-is.
If you're working on automated evaluation, oversight, or continuous-runtime agents and any of this is useful — or wrong — I'd like to hear it: dan@kintsugi.press.