Perplexity-based AI detection is running backwards in 2026

Every AI detector accusation you've seen — the flagged essay, the "92% AI" screenshot, the writer scrubbing their own voice to pass a scan — leans on one assumption, inherited from 2019: machine text is more predictable than human text. We calibrated that assumption against 100 sourced passages this July.

It came back inverted.

Human-written Chinese webfiction: 68.6% of tokens inside the probe model's top-10 guesses. AI novels our team polished under an anti-cliché discipline: 59.7%. Raw, untouched LLM continuations from five current model families: 56.8%. The humans were the easiest to predict. The machines were the hardest.

Where the "too predictable = AI" rule comes from

In 2019, Harvard NLP and the MIT-IBM Watson Lab released GLTR, which colored every word by where it ranked in GPT-2's predictions: top-10 red, beyond rank 1,000 teal. Machine text back then really did glow red wall-to-wall — it was sampled from the model's own distribution. Human prose broke the pattern constantly. GPTZero later built a business on the same family of signals, perplexity plus burstiness, and the assumption hardened into infrastructure: classrooms, hiring filters, editorial pipelines.

The rule was honest about its era. The era ended. Generators moved through five generations of instruction-tuning and preference training; the detectors' reference models mostly stayed put. Nobody re-ran the founding experiment — so we did.

The setup

Five corpora, 20 passages each, every passage with a named source:

The probe is the exact weight file that ships in our browser tool Text X-Ray: a 6-layer distilled Chinese GPT-2 (q8, 73MB) for the Chinese track, DistilGPT-2 (81MB) for English. Offline calibration ran on the same runtime family on 2026-07-25. Two readouts per passage — top-10 hit rate and perplexity — scored independently, reported as percentiles, no weighting.

The inversion

Both languages point the same way: human prose scored as more predictable than AI prose. On the Chinese side the entire human p10–p90 band (64.1%–70.9%) floats above the raw-LLM band (51.7%–60.6%). Perplexity inverts too — by the old rule, a human webnovel at 26.3 reads as machine work while an English AI novel at 147.4 reads as avant-garde literature.

Two honesty notes. The bands overlap at the edges: the least predictable human passage (57.8%) scores below the most predictable crafted-AI passage (66.2%), so this ruler has no verdict for any single passage — the shape only exists at corpus level. And the English classics carry a memorization asterisk: five books this famous are almost certainly in the probe's training data, so part of their "predictability" is recall, not style. The Chinese side has no such contamination, which makes its inversion the cleaner exhibit.

The ruler didn't break. It was never measuring authorship.

A probe model measures one thing: distance from its own training distribution — for our probe, the internet of roughly 2019. Both ends of the scale moved since. Today's generators, after instruction-tuning and preference training, produce text that has drifted well away from that distribution, so an old ruler finds them surprising at every turn. Meanwhile genre fiction written by humans is the densest part of the old distribution — formula is the point of the genre, and the probe grew up reading it.

The tool has a built-in demo that makes this visceral. Hover any word and you see the model's top five bets for that slot. On a stock phrase, the top bet is usually another stock phrase — the probe isn't predicting the author, it's reciting the corpus.

Once you read "predictable" as "close to a 2019 corpus," the casualty list stops looking random. Genre writers. Fluent non-native English students — the Stanford study found detectors flagging over half of TOEFL essays as AI. Anyone who writes inside a well-worn register. They aren't writing like machines; they're writing near a corpus. OpenAI retired its own classifier in July 2023 over exactly this kind of unreliability.

What a perplexity lens is still good for

Strip away the verdict business and three honest uses remain. It exposes formula: we hand-wrote a cliché-saturated English sample and it scans at 74.5% top-10 with perplexity 22.5 — redder than anything in the English calibration corpus — while our crafted English AI novel scans at 51.9% with 12.8% of its tokens landing beyond rank 1,000. Same ruler, and what it separates is template density, not species. For your own drafts, wall-to-wall red is a rut you can see. It teaches: hovering a word and watching a live top-5 distribution is the first time most people actually see what "next-token probability" means. And it calibrates itself in public — the reference bands are drawn right on the result card, so your text lands somewhere visible, not somewhere asserted.

The tool is at /labs/text-xray. Everything runs in your browser: one 73–81MB model download into browser cache, then per-word scoring happens on your device. No upload; works offline once cached; a 400-word passage scans in under a second.

Boundaries, stated plainly: 100 passages, one probe per language, a published method — a yardstick, not a truth. If you re-run this with different corpora and get the opposite ordering, we want to see it. The tool is right there.