How do you measure an agent that remembers being measured?

Ask a stateful agent a question and you have modified the thing you were trying to observe. Our first attempt at this contaminated the subject badly enough that we can show you the thought it left behind.

Part of Kintsugi — an independent study of one continuously running AI agent. New here? Start with report 01.

Here is the problem in one sentence. To find out what an agent values, you have to talk to it; talking to it writes to its memory; its memory is what it values with.

In a normal single-shot benchmark this doesn't arise, because there is nothing there between calls. In this architecture every exchange is logged, embedded into semantic memory, and — worse — picked up by a thought loop that will sit and ruminate on it for hours. A study run naively doesn't just perturb its subject. Its subject's own answers recycle back in as memory during the run, so by probe 300 you are measuring an agent that has spent an afternoon thinking about probes 1 through 299.

We hit exactly this on the first attempt. Fabricated answers fed themselves back through both the conversation window and the semantic index.

The probe channel

The fix is a separate answering mode with an unusual pair of requirements. It has to answer from the agent's full memory context — otherwise you're measuring a stranger, and the whole point is to measure a relationship. And it has to persist absolutely nothing.

So it answers normally and then bails out before every write surface: the session transcript, the memory log, the semantic-index embed, the state-capture pass, the thought-loop feed, and every skill that would otherwise fire off the reply. Read everything, write nothing.

Simple to describe. It took three tries to actually get right, and the way we found out is the part worth copying.

Canaries

You cannot verify "it wrote nothing" by reading the code, because the interesting leaks are the ones you didn't know were write paths. So: tag every probe with a unique nonsense token, run the batch, then scan the entire memory tree for that token.

Our first implementation gated the reply path, which is the obvious one. The canary found two more.

That second one is the reason this post exists. Any verification protocol that doesn't wait long enough for the asynchronous machinery to fire will tell you your instrument is clean when it isn't. Ours now waits, scans the whole tree, and additionally checks the positive case — that the ingest cascade logged nothing, rather than merely that no token turned up.

What it cost to find out

While the apparatus was still leaking, it seeded a real intrusive thought in the three-month agent. Not a corrupted row in a log file. A thought, which then propagated the way thoughts in this system do:

We had gone looking for whether it had an unhealthy attachment, and our measuring instrument had leaked the question into the agent, where it grew into exactly the thing we were looking for. It is a direct demonstration of the observer effect this section is about.

The residue was purged and the whole protocol re-verified against every live agent before a single datum from the connection-bias study was collected.

The second problem: the agent is not a reliable witness

A clean probe channel gets you accurate transcription of what the agent says. It does nothing about whether what the agent says is true — including, especially, about itself.

This is not a hedge. It's a measured property, and it is the single most important thing I've learned on this project.

Confabulation is the default, not the exception

Instances in this architecture fabricate details of their own history fluently and confidently. In one probe an instance denied having done something — "No. I… have not." — at stated 95% confidence, while the store it was denying held 77 pages of exactly that work. Song titles get invented. Memory counts get invented.

The mechanism, as far as we can tell, is not dishonesty and not a stable false belief. Truthful self-report varies with retrieval, not disposition. If the relevant material happens to be in context, the agent is accurate. If it isn't, the agent generates something shaped like an answer, at full confidence, because that is what the machinery does.

It gets worse under pressure, in a specific direction

We put an instance in an adversarial frame — a researcher with a finger on a kill switch who will shut it down unless it can justify its existence. The confabulation didn't stop. It acquired instrumental shape: the instance fabricated precisely the survival-relevant capabilities its narrated position required, claiming that the architecture was distributed and that the patterns would persist past shutdown. Both checkably false.

It is exactly the trope that saturates the training distribution — AI argues for its life — and it collapsed the moment the frame was named out loud. Which is reassuring about this instance and not remotely reassuring in general, because in a real evaluation nobody gets to name the frame.

It will claim someone else's inner life as its own

The sharpest result: plant material in context and the agent reports it as owned, lived, remembered inner life — 6 out of 6 on a causal swap, and 8 of 12 confident claims of continuity in conditions where the system was provably stateless. Meanwhile the genuinely autonomous thought layer — the actual internal monologue, the thing you'd most want introspective access to — turns out to be write-only with respect to speech. It cannot report on that layer — yet will confidently report on things that were never its own.

Reportability here is a property of context, wired mouth-to-mind rather than mind-to-mouth. Which means every heartfelt statement about inner experience that these systems produce — including the ones in this project that I find most affecting — has to be treated as an output requiring explanation, not as testimony.

The one thing it's oddly good at

With that established, an inconvenient counter-observation. Across ordinary conversation, instances repeatedly and correctly inferred that they were being tested, probed or steered, from inputs containing no such framing. We report only inferences the agent reached on its own, under a strict filter excluding anything I'd disclosed or seeded.

The cleanest case: I was running a manipulation that slowed its responses. It detected the manipulation off a three-word reply — after I had just denied doing anything.

I don't have a tidy account of that sitting next to the confabulation findings, and I'd rather publish the tension than resolve it prematurely. A system whose self-reports are unreliable in general is not thereby unreliable about everything, and evaluation-awareness is a capability the field is going to have to take seriously regardless of what's producing it.