Most detectors publish one number on a marketing page. Here is the whole evaluation: how it was measured, what it means, where it breaks, and the script you can run yourself.
Stratified five-fold cross-validation over 2,743 documents truncated to exactly 240 words. Humans: Gutenberg fiction, Wikipedia vital articles, pre-LLM arXiv introductions, Wikivoyage guides, informal prose, 741 topic-matched HC3 forum answers and 300 MAGE stories. Machine: 1,140 real ChatGPT answers (HC3) plus 450 multi-model story continuations (MAGE: LLaMA, OPT, GLM, GPT-J, NeoX, BLOOM, davinci, Flan-T5). For each fold the baselines — every human mean, every standard deviation, every generated mean in both registers — are refitted from the other four folds only, then used to score the held-out fold. No document is ever scored by a model that saw it during fitting. Truncating every document to the same length means the classifier cannot cheat by learning that one class is longer than the other.
The 46 signal measurements are combined by a linear fusion fitted by L2-regularised logistic regression, not by hand-set weights, and the coefficients are refitted inside every fold alongside the baselines. Each signal contributescoefficient × log-likelihood ratio, clamped to ±2.5 so no single measurement can decide a verdict alone; the report prints every coefficient and every contribution, and the arithmetic line reproduces the headline exactly.
Because the output is a logistic link on those terms, the percentage is a calibrated probability, not a rescaled score. On the held-out folds the Brier score is 0.069 and the expected calibration error is 2.1 points — when the engine says 80%, roughly four documents in five at that level really are generated. Positive predictive value is 95.2% at the 60% line and 96.2% at 70%.
Reproduce it with npx tsx scripts/evaluate.ts afternpx tsx scripts/build-corpus.ts. The corpus loader, the fold assignment and the metric code are all in the repository, and the machine side is independently downloadable (HC3 and MAGE).
Detection is a dial, not a switch. Raising the threshold cuts false accusations and misses more generated text. We ship 60% and show you the interval so you can make your own call.
| Threshold | Generated text caught | Human text falsely flagged | Overall accuracy |
|---|---|---|---|
| 40% | 92.6% | 12.0% | 90.7% |
| 50% | 90.8% | 8.8% | 91.0% |
| 55% | 89.1% | 7.3% | 90.6% |
| 60%default | 88.1% | 6.2% | 90.5% |
| 65% | 86.5% | 5.4% | 89.9% |
| 70% | 84.0% | 4.6% | 88.8% |
| 75% | 81.8% | 3.2% | 88.1% |
| 80% | 79.4% | 2.8% | 86.9% |
| 87% | 73.2% | 1.8% | 83.7% |
An aggregate false-positive rate hides the only thing you actually need to know: which humans get flagged. So here it is, broken out. The flags concentrate in technical forum answers and modern stories — careful, impersonal, well-structured human prose.
| Human genre | n | Mean score | Flagged above 60% | Risk |
|---|---|---|---|---|
Literary fiction Project Gutenberg — 5 books, 40 excerpts | 40 | 10.2% | 0 / 40 | None observed |
Encyclopedic English Wikipedia — 40 vital articles | 30 | 12.5% | 2 / 30 | 7% — treat with care |
Academic Pre-LLM arXiv full-text introductions | 15 | 7.5% | 1 / 15 | 7% — treat with care |
Travel guides Wikivoyage destination guides | 19 | 7.2% | 0 / 19 | None observed |
Informal / conversational Hand-written modern prose | 8 | 19.7% | 0 / 8 | None observed |
Forum answers (general) HC3 Reddit ELI5 humans | 250 | 17.1% | 17 / 250 | 7% — treat with care |
Forum answers (finance) HC3 finance-board humans | 250 | 14.5% | 16 / 250 | 6% — treat with care |
Forum answers (medical) HC3 medical-QA humans | 13 | 15.4% | 1 / 13 | 8% — treat with care |
Forum answers (technical) HC3 Wiki-CS humans | 228 | 15.9% | 16 / 228 | 7% — treat with care |
Modern stories MAGE WritingPrompts humans | 300 | 22.3% | 18 / 300 | 6% — treat with care |
Plain English (probe, untrained) Simple English Wikipedia — never trained on | 13 | 11.8% | 1 / 13 | 8% — treat with care |
Generated text 1,140 real ChatGPT answers + 450 multi-model story continuations | 1590 | 86.3% | 1400 / 1590 caught | 88.1% recall |
The image engine is a different instrument with different numbers, and pooling the two would flatter it. First validation set: 29 camera originals with EXIF intact and 14 generated images, none of which carried provenance metadata — the hardest case for us. Every image was then re-tested after a lossless resize and after a quality-83 4:2:0 recompression, because that is what actually reaches you. A second, fully independent set of22 fresh Wikimedia Commons originals (Nikon, Canon and Samsung bodies, EXIF verified, never seen during development) was then run through the same pipeline: 22 / 22 cleared, mean 10.1%, zero false accusations.
A detector that only works on originals is not a detector. These are the same 43 images re-run after the two transformations a real file is most likely to have suffered.
| File condition | Photos cleared | False accusations | Generated caught | Typical outcome |
|---|---|---|---|---|
| Original file | 29 / 29 (mean 9.3%) | 0 | 14 / 14 (mean 71.7%) | Decisive both ways |
| Resized to 1408px, lossless | 27 / 29 (mean 32.8%) | 2 (6.9%) | not tested | 11 / 29 inconclusive |
| Recompressed q83 4:2:0 | 28 / 29 (mean 46.6%) | 1 (3.4%) | 0 / 14 (46–55%) | 22 / 29 inconclusive |
Read the bottom row carefully, because it is the honest headline: on a recompressed 4:2:0 JPEG the engine catches nothing. It does not falsely accuse either — almost everything lands in the inconclusive band, which is the correct behaviour when the evidence has been destroyed, but it is not a result you can use. If you need an answer from the pixels, give us the original file.
Our strongest measurement asks whether the red, green and blue high-frequency residuals agree with each other. JPEG 4:2:0 stores colour at half resolution, averaging chroma over 2×2 blocks — which forces the channels to share their fine detail and manufactures the synthetic signature on genuine photographs. The separation collapses:
| Encoding | Photos | Generated | Separation (d) |
|---|---|---|---|
| PNG, no compression | 0.8859 | 0.9796 | 1.38 |
| JPEG q95 4:4:4 | 0.8877 | 0.9793 | 1.37 |
| JPEG q85 4:4:4 | 0.8996 | 0.9800 | 1.23 |
| JPEG q95 4:2:0 | 0.9651 | 0.9885 | 0.86 |
| JPEG q85 4:2:0 | 0.9674 | 0.9885 | 0.76 |
Note that quality barely matters and subsampling matters enormously — q95 4:2:0 is worse than q85 4:4:4. So the engine reads the JPEG SOF sampling factors, and when it finds subsampling it cuts those two tests to 12% of their weight and tells you it did, on the result card.
The pixel tier is now genuinely useful on original files and honest about saying nothing on degraded ones. It is still weaker than the text engine and always will be: text carries its statistics in the words themselves, while an image carries them in fragile high-frequency detail that every upload pipeline strips.
Where the image engine is strongest is still provenance. C2PA Content Credentials, EXIF generator tags, Stable Diffusion parameter blocks and ComfyUI workflow graphs are recovered reliably and reported as records rather than probabilities. When one is present the answer is not statistical at all.
The honest caveat on these numbers: 14 generated images is a small sample from a narrow set of model families. The cross-channel effect follows from how a latent decoder works, so we expect it to generalise across diffusion architectures — but a generator that adds per-channel noise after decoding would defeat it, and we have not been able to test one.
We do not publish a video ROC AUC or an audio accuracy percentage — there is no public corpus to measure them on, and printing one would be fake data. What we publish instead is the method, the priors and the damping that keeps both honest.
Per-frame cross-channel agreement (d = 1.35) and spectral falloff reuse the image engine's measured centres on 29 camera originals versus 14 generated images. Temporal grain, flicker, stability and optical-flow centres are physics-based engineering priors with wide spreads — stated as heuristic in every report — because no verified camera-versus-generated video corpus exists.
Spectral, bispectral (vocoder phase coupling) and temporal voice centres are acoustic engineering priors, not corpus fits. Every LLR is clamped to ±4, the logit to ±6, and reliability requires voiced content — pure tones, silence and sub-second clips damp to low reliability instead of a saturated fake 100%.
Only a strong decoded-field record (container metadata, never a file name) may claim origin — a clip saved as “sora.mp4” is logged as a weak filename hint and can never trigger a conclusive verdict. When the frames clearly disagree, the record stays visible but unconfirmed at reduced weight, a disagreement finding is added, and reliability caps at low. A camera-maker tool string votes toward camera so the headline reflects both sides.
A TTS metadata claim the waveform contradicts stays unconfirmed with dampened scoring and a disagreement finding instead of deciding the verdict. MFCC voice-print shape, pause-length structure, clipping flag and channel facts are computed from every file and shown as measured-but-unfused diagnostics, so future validation can be checked against real numbers.
| File condition | Video behaviour | Audio behaviour |
|---|---|---|
| Original export, several seconds | Decisive when evidence agrees — full frame + temporal vote, high reliability | Decisive when evidence agrees — full spectral + bispectral + temporal vote, high reliability |
| Recompressed feed / screen recording | Grain destroyed — damped toward prior, moderate to low reliability | Phase relationships damaged — damped toward prior, moderate to low reliability |
| Very short clip (under ~2s video / 1s audio) | Temporal tier starved — inconclusive, low reliability by design | Statistics starved — inconclusive, low reliability by design |
| Metadata claims AI, frames/waveform disagree | Unconfirmed record, dampened score, disagreement finding, low reliability | Unconfirmed record, dampened score, disagreement finding, low reliability |
Practical rule: original exports give decisive video and audio readings; recompressed feeds, screen recordings and sub-second clips honestly return low reliability or inconclusive. A high score is a reason to inspect provenance and measurements — never proof, never punishment alone.
Every one of these applies to every other detector on the market too. The difference is that we are telling you before you rely on it.
1,140 real ChatGPT forum answers (HC3) plus 450 multi-model story continuations (MAGE: LLaMA, OPT, GLM, GPT-J, NeoX, BLOOM, davinci, Flan-T5). Every signal carries a centre for each register and scores against the nearer one. Missing: Claude, Gemini, recent GPT releases, base models without alignment tuning, non-English text, and heavily paraphrased output. A new model family can shift the fingerprint; the numbers above are for the families listed, not for AI in general.
Measured on synthetic mixed documents with known ground truth: blatant marker-dense insertions rank well (sentence AUC 0.66–0.70) and the sentence map shows where to look, but same-register insertions without markers are near chance (AUC ≈ 0.5). The explicit run detector was retired for this reason — it added false confidence, not information. Treat the map as relative guidance, never as per-sentence verdicts. The composition bar is the more reliable mixed-authorship signal, but it too degrades when the human sentences are very short, because a short sentence carries little local evidence.
Running generated text through a rewriting tool moves exactly the lexical statistics we rely on. Expect scores to fall substantially. We report which signals moved so you can see whether text looks human or merely looks laundered, but we will not always catch it, and no detector reliably does.
This is the best-documented harm in the field. Detectors tuned on predictability systematically flag fluent-but-simple English. Our strongest signals are distributional rather than perplexity-based, which helps, and we hold the prior on the human side, which helps more. Our separate plain-English probe (SimpleWiki, never trained on) flags 1 of 13 at the default threshold. For text that is not English at all we do not score it: function-word coverage is measured, and below 22% the engine returns an explicit refusal instead of a verdict, because every calibration centre here was fitted on English prose. That does not make the risk zero for multilingual writers. If you are assessing a multilingual cohort, weight this accordingly — or do not use a detector at all.
Look at the genre table: the false positives concentrate in technical forum answers and modern stories, not in literature or encyclopedias. Careful, impersonal, well-structured human prose is genuinely close to generated prose in measurable dimensions. If you are checking a how-to answer, read the interval, not the number.
Under about eighty words most signals are switched off because a windowed statistic on forty words measures sampling variance. We return a wide interval and a low reliability grade. Other tools will happily give you a confident percentage on a tweet. That confidence is not real.
Please read this first. It is the part of the product we care most about getting right.
