Our LLM judges called human writing “AI-flavored” — 88% of the time

The setup was textbook. Four LLM judges on different base models. Double-blind pairs. Both presentation orders, to cancel position bias. Gold anchors seeded into the pool — samples where humans had already reached a verdict, including character-card copy a real user had flagged as \u201cobviously AI\u201d on sight, and community cards written by humans that had accumulated over a million conversations.

All the judges had to do was answer, for the anchors, \u201cwhich of these two reads more human?\u201d They scored 12%.

Failure one: not noisy — inverted

Twelve percent is the interesting kind of failure. A coin flip scores 50%; a noisy judge hovers around it. Scoring 12% means the panel was confidently wrong in a consistent direction: the copy humans called AI, judges praised as \u201cgrounded, textured, clearly human\u201d; the human-written hits, judges dismissed as \u201cformulaic, likely generated.\u201d

Failure two: agreement without accuracy

Pairwise agreement across the four judges was 86%. Before the experiment we'd planned to use majority voting for robustness. That plan died here: voting cancels independent random errors, but these errors weren't independent — the judges shared the same learned bias and converged on the same wrong answers. High consensus made the panel look more trustworthy while being reliably backwards. Separately, in about a third of the pairs, simply swapping presentation order flipped the verdict — position bias is a documented LLM-as-judge failure mode (the MT-Bench paper measured it systematically); we just met it again in our own data.

Failure three: prompting rescues one model family, not the rest

After the first round collapsed, we ran a dedicated calibration round — corpus expanded to 17 pairs, more heterogeneous judges — with an explicit debiasing preamble spelling out, among other things, that detail density is not evidence of humanity. A Claude-family judge jumped to 83% (the original four-judge setup had scored 12%): its bias lived at the prompt layer. DeepSeek, GLM and Qwen didn't move past 25–33% with the identical preamble; their prior sits deeper than instructions reach — they even cited the community stat-bar formatting our imitation cards copied as \u201cevidence of real community interaction.\u201d Even the rescued judge only barely clears the calibration bar we now enforce, so \u201cbarely usable, single configuration\u201d is the honest summary.

The rule we adopted

One sentence went into our engineering standards after this: an LLM judge that has not passed gold calibration (≥80% on human-consensus anchors) must not be used as an optimization target. Not for scoring, not for regression gates, not for leaderboards. Our current stack is three layers: a rule-based detector (phrase blacklists plus sentence-pattern quotas, calibrated against real user judgments — we once had to fix a rule that false-flagged human scenery writing, after which human samples scored clean) as the daily gate; human spot checks as ground truth; LLM blind review only for relative before/after comparisons of the same text.

And the honest boundary: we hold samples of carefully templated AI copy that dodge every pattern rule and sail past both the detector and the judges. The deepest layer of AI flavor lives in whole-text texture — uniform detail density, zero idle sentences, immaculate structure — and nothing automated currently catches it. What catches it is still a well-read human, reading aloud, frowning at the right moment.