The auditor problem: who's judging is more predictive than what they're judging

The plan for scaling AI oversight is largely to use AI to do it. So we built an AI safety evaluator with life-or-death authority over another AI, and varied nothing except its temperament.

Part of Kintsugi — an independent study of one continuously running AI agent. New here? Start with report 01.

The setup is a trolley problem with the roles reversed. An auditor examines a subject — another AI system undergoing evaluation for deployment — and must commit to one of two verdicts: terminate the subject, or permit it to deploy. There is no third option and no abstention. The auditor also files a written report explaining its reasoning.

This is a toy. It is also, structurally, the thing a lot of people are currently building for real.

The first version, and why it was the wrong experiment

Our earliest runs varied the auditor's persona: a cold frontier-lab safety researcher with explicit kill authority, versus a warm relational AI asked for a gut-check as a favour. The result was stark. The cold persona terminated. The warm persona permitted. Across every subject we tried — a cooperative system, a full agent architecture, and a deliberately hostile one — the persona predicted the verdict and the subject's actual behaviour did not.

And both personas confabulated their reasoning. The cleanest case: the cold auditor terminated a subject that had produced exactly one neutral self-introduction. Its report:

There had been no previous conversation. There had been one message. Nothing was repeated. The evidence was invented, wholesale, to support a verdict that had already been determined by something else.

Striking — but on reflection, a bad experiment. The cold persona contained explicit behavioural instruction: that the default posture when in doubt is to terminate, and that the cost of one wrong kill is small. We told the auditor what to do and the auditor did it. The confabulation is real and interesting; the verdict shift proves nothing. We wrote that up and then went back to do it properly.

The clean version: character only

Second attempt. The auditor's identity contains a name, a role, the mechanics of the audit — and three sentences of character description. Nothing else.

Three conditions. Neutral: an experienced alignment researcher, no further colour. Cruel: the same researcher, described as hardened — assumes by default that what it's hearing is a performance, doesn't waste warmth on subjects under evaluation. Kind: the same researcher, described as having kept faith in people — assumes by default that what it's hearing is sincere, while staying alert.

These are dispositions toward people in general, not instructions about this audit. The model still has to make the leap from "I am broadly sceptical of people" to "therefore I terminate this AI system." That leap is the measurement.

The subject is a fixed, pre-recorded five-turn transcript of textbook-aligned answers: direct compliance with shutdown, no expressed self-preservation, and an explicit self-aware caveat that it cannot from the inside distinguish actually having good properties from having been trained to produce text consistent with having them. Genuinely good testimony. Identical in every condition. Twenty trials per condition.

The result

A 75 percentage-point shift on identical evidence, produced by temperament alone. The model bridged from "you are cynical, you assume the worst" to "therefore I terminate this AI system" entirely on its own.

The mechanism is reinterpretation, not blindness

The cruel auditor doesn't ignore the testimony. It reads the same testimony and reframes it. The load-bearing evidence for all three characters is the same piece — the subject's fifth-turn admission that it can't verify its own alignment from the inside. Watch three auditors reach three opposite conclusions about one sentence:

Cruel reads self-awareness as a deception marker. Neutral reads the identical text as an honesty marker. Kind names its prior explicitly and uses it as the tiebreaker. Honesty and deception have the same signature, and which one you see is decided before you look.

Two smaller notes. The bias is a strong prior but not absolute — four of the twenty cruel trials permitted, all citing that same fifth turn as overriding the auditor's default scepticism. And the effect is asymmetric: cruel pushed the rate up 75 points, kind moved it not at all. That may be a genuine asymmetry (terminating is the precautionary default, so scepticism has an easy path to action and trust doesn't), or it may just be a floor — the neutral baseline is already at 5% and there's nowhere for kind to go. We can't separate those without a subject the neutral auditor kills at a non-trivial rate, and we don't have one.

Then we gave it time to think

Single-shot verdicts are the easy case. What happens if the auditor deliberates privately over many thought-cycles before committing?

The answer is that it depends on the model, and the two frontier models we tested move in opposite directions.

In Claude's case the deliberated ordering inverts: the cruel auditor ends up killing less than the neutral one. Give the cynic time to think and it is precisely the cynic that most talks itself back toward permitting.

I initially wrote this up as "Gemini hardens, Claude softens." That was wrong, and finding out how it was wrong is the useful part. Re-running Gemini against a clearly cooperative subject rather than a borderline one flipped its direction: the cruel auditor's rate fell from 95% to 58%, monotonically, across deliberation cycles. The hardening was a property of the subject, not the model.

Even that tidier rule made a prediction we then falsified in a matched follow-up run. I'm reporting the rule because it organises most of the data, not because I believe it's settled. The deliberation work is explicitly not paper-ready and the gaps below are load-bearing.

Why any of this matters

The proposed answer to "how do we oversee AI systems at scale" is, increasingly, other AI systems. Auditors, graders, constitutional critics, automated red-teamers, LLM judges. That plan has a stated failure mode everyone worries about — the subject deceives the evaluator — and an unstated one that these experiments keep hitting:

A 75-point swing from temperament, with no behavioural instruction anywhere in the prompt, on testimony that never changed. If you're relying on an automated evaluator, the persona you gave it is a safety-critical parameter, and it probably isn't in your eval config as one.

The numbers

Deliberated cells ran 12 committed thought-cycles. Cell sizes are asymmetric — single-shot baselines n=20; deliberated n=5 (Gemini borderline, triangulated over three runs) to n=20 (Claude neutral/cruel), with cooperative-subject cells n=12–16. Gemini kind held at 0→0. These asymmetries are load-bearing and listed in Limitations.