Commit 51a4ed99 by PLN (Algolia)

docs: detector survey — Open Pangram EditLens in scope, APIs banned, cadence detection unclaimed

parent d91b607f
# buvard: survey of AI-feel detectors (2026-11-11)
Companion to `research-litreview.md` (2026-11-07), which covers the detection-unreliability
basics (61.22% FP rate on non-native essays, https://arxiv.org/abs/2304.02819) and the
French coverage gap. This survey answers a different question: what actually exists, what
is open and testable on a laptop, and where the field stands as of November 2026.
Every claim carries its URL; unverifiable details are marked [inferred].
## 1. TL;DR shortlist
| Tool | Open? | French? | How to test | Cost | URL |
|---|---|---|---|---|---|
| Pangram | No (SaaS) | Yes (20+ languages claimed) | 20 free checks/day on the site; API from $0.05/100 words | Free tier daily | https://www.pangram.com/ |
| Binoculars | Yes (BSD-3) | Weak (English-focused, per authors) | `pip install -e .` then `python app.py`; 2x Falcon-7B models needed | Free, local | https://github.com/ahans30/Binoculars |
| Fast-DetectGPT | Yes | English experiments [inferred] | clone repo, run with GPT-Neo-2.7B local checkpoints | Free, local | https://github.com/baoguangsheng/fast-detect-gpt |
| Ghostbuster | Yes | No (English only, per repo) | `pip install -r requirements.txt`, needs an OpenAI API key | Free code, paid API | https://github.com/vivek3141/ghostbuster |
| GLTR | Yes (demo + code) | No (GPT-2 vocabulary) | hosted demo at https://gltr.io, no install | Free | https://gltr.io |
| GPTZero | No (SaaS) | Yes [inferred] | web UI free tier, limited scans | Freemium | https://gptzero.me/ |
| RAID benchmark | Yes (dataset) | English | download from Hugging Face, run any detector against it | Free | see section 3 |
| hc3_french_ood + INRIA FR corpus | Yes (dataset) | Yes | already in buvard's eval memory; streamable | Free | https://gitlab.inria.fr/wantoun/robust-chatgpt-detection |
| DependencyAI (2026) | Paper; code status unknown | English | reimplement dependency-relation classifier from the paper | Free (paper) | https://arxiv.org/abs/2602.15514 |
~~The single best test this week is Pangram against buvard's own French and English
corpora.~~ **Amended 2026-11-11, owner policy: never any online API — no text leaves the
machine, ever.** The Pangram *API* stays out of scope. But Open Pangram exists since
2026-03-24: open-weights EditLens checkpoints (pangram/editlens_roberta-large 355M,
pangram/editlens_Llama-3.2-3B), training data and source code, CC BY-NC-SA 4.0
(non-commercial), English-only training data, runnable on a MacBook
(https://huggingface.co/pangram). Local eval of those checkpoints is in scope — see §5.
## 2. Commercial detectors
**Pangram (pangram.com).** The strongest in class on paper and the most testable.
- Architecture: self-described "traditional language model architecture" classifier
(tokenization, embeddings, neural network), for text, image and video.
https://www.pangram.com/
- Claims: "over 99% accurate", "1 in 10,000 false positive rate". Both are
vendor-published, but unlike competitors they are backed by named third-party studies
(next bullet). https://www.pangram.com/
- Languages: "more than 20 languages" including French, claimed on the front page.
https://www.pangram.com/
- Third-party evaluations, from Pangram's own roundup page
(https://www.pangram.com/blog/third-party-pangram-evals; treat the framing as vendor
curated, follow the links for the originals):
- University of Chicago, Aug 2025: 0.001 FP rate, 0.01 FN rate; "robust to humanizer
tools (e.g., StealthGPT)". https://bfi.uchicago.edu/wp-content/uploads/2025/09/BFI_WP_2025-116.pdf
- University of Maryland: 99.3% TPR, 2.7% FP on humanized text; scored highest for
detecting humanizers and paraphrased text. https://arxiv.org/pdf/2501.15654
- Vrije Universiteit Brussel, June 2026: 97.5% and 95% detection, 0% FP; "only
[Pangram] produced satisfactory results" under editing. https://doi.org/10.1007/s40979-026-00226-w
- Testability: 20 free checks/day on the website, no signup wall reported; API with a
7-day trial ($0 then $20/month), $0.05 per 100 words after. https://www.pangram.com/pricing
Twenty checks a day is enough to walk buvard's French corpus in about a week at
100 texts/day, or to spot-check the Gutenberg pre-2022 FP set.
- Caveats: the eval links above were summarized by Pangram's own blog page; the numbers
should be re-read from the PDFs before quoting them anywhere official. [inferred: the
VUB DOI and Maryland arxiv id were not opened directly in this session]
**GPTZero.** Perplexity and burstiness scores; "99.9% accuracy" marketing beside its own
disclaimer against using scores as a verdict. The 61.22% TOEFL false-positive study
(https://arxiv.org/abs/2304.02819) covered detectors of this class. Web free tier is
scan-limited. https://gptzero.me/
**Originality.ai.** Paid, claims "most accurate in third-party studies", advertises
multilingual support including French [inferred from its languages page, not re-verified
this session]. No visible appeal process. https://originality.ai/
**Sapling.** Detector API and web tool, claims support for ~20 languages including
French [inferred]; the 2304.02819 study found Sapling among the weaker performers on
non-native essays. https://sapling.ai/
**Winston AI.** Claims French support; no independent FP study found in this session.
https://gowinston.ai/ [unverified beyond the product page]
**Copyleaks.** Claims 30+ languages including French; a 2023 study by Stanford
researchers found it among the most inconsistent on human-written text [inferred from
widely reported coverage of https://arxiv.org/abs/2304.02819-era results; not
re-verified this session]. https://copyleaks.com/
## 3. Open-source and academic detectors
Installability summary first, details after: Binoculars and Fast-DetectGPT run locally
with open weights; Ghostbuster needs an OpenAI API key; GLTR is a browser demo plus a
repo. None of them ships French. The benchmarks that measure multilingual performance
(M4, MULTITuDE) also skip French, so the honest answer to "which open detector works in
French" is still none, as of 2026-11.
**Binoculars** (https://github.com/ahans30/Binoculars, paper
https://arxiv.org/abs/2401.12070). Cross-perplexity ratio of Falcon-7B and
Falcon-7B-Instruct. BSD-3-Clause, `pip install -e .`, ships a demo app. The paper claims
over 90% recall on ChatGPT samples at 0.01% FP rate; the repo's own caveat is that it is
"more proficient in detecting English language text" and that failure modes exist.
Two 7B models means roughly 28 GB of weights; it wants a GPU [inferred].
**Fast-DetectGPT** (https://github.com/baoguangsheng/fast-detect-gpt, paper
https://arxiv.org/abs/2310.05130). Conditional probability curvature; "surpasses
DetectGPT by a relative around 75%" and is 340x faster. Open weights (GPT-Neo family in
the paper). English experiments in the paper; no French [inferred]. This is the
academic quality bar for local scoring in 2024-2025.
**Ghostbuster** (https://github.com/vivek3141/ghostbuster, paper
https://arxiv.org/abs/2305.15047). Regression over features from weaker language models.
99.0 F1 claimed across domains. The repo states it focuses on British and American
English and `classify.py` takes `--openai_key`, so testing costs API money and English
only. Deprioritize for buvard.
**GLTR** (https://gltr.io, paper https://arxiv.org/abs/1906.04043). Token-rank
visualization over GPT-2; raised human judge accuracy from 54% to 72%. The hosted demo
is the fastest zero-install way to see token-level signals on English text. Ancient by
2026 standards, but its vocabulary-band idea maps directly onto buvard's lexicon tiers.
**RADAR-Vicuna** (paper https://arxiv.org/abs/2305.13014). Adversarially trained
Vicuna detector with a paraphrase-robust dataset. English; weights and dataset live on
Hugging Face [inferred]. Historical interest: the first widely cited "robust to
paraphrase" open model.
**RAID benchmark** (https://arxiv.org/abs/2410.18405 [id unverified this session; the
abstract fetch returned an unrelated paper], project page https://raid-bench.github.io/).
11 domains, 8 generators, 8 adversarial attack types; it evaluated commercial detectors
including GPTZero and Originality.ai and found both miss many attacks and show
demographic bias in FP rates [inferred from the paper's public summaries; re-verify the
id before citing]. This is the reference adversarial test set if buvard ever wants to
compare itself against commercial tools on English.
**Benchmarks and shared tasks for French.** M4
(https://arxiv.org/abs/2305.14902) is multi-generator, multi-domain, multi-lingual; its
languages are Arabic, Chinese, English, Italian, Portuguese, Russian, Spanish [inferred
from the paper's tables; the repo readme fetch did not list them], French absent.
MULTITuDE (https://arxiv.org/abs/2310.13606) also omits French, as the lit review
already established. The Touché/pan shared-task pages at pan.webis.de returned 404 on
every URL tried in this session (touche.html, clef25, touche.webis.de), so French
coverage there stays [unverified]. Recent shared-task winners are mostly fine-tuned
XLM-RoBERTa-style multilingual encoders (example: https://arxiv.org/abs/2510.20610,
Arabic), a recipe that would transfer to French if a French training corpus existed.
That corpus does not exist publicly; the INRIA French artifacts
(https://gitlab.inria.fr/wantoun/robust-chatgpt-detection) remain the only French set
buvard already has.
## 4. Stylometry and cadence-based approaches
This is the lane closest to buvard: features a regex linter could eventually own.
**What exists.** Classical authorship stylometry (function-word rates, sentence-length
statistics, type-token ratio) transfers to machine-text detection with modest accuracy
on its own; the strong published signals remain frequency shifts of common words
(https://arxiv.org/abs/2403.07183 and https://arxiv.org/abs/2406.07016, both already in
the lit review). A 2026 survey of stylometry for academic integrity
(https://arxiv.org/abs/2601.01225) applies attribution-style methods to human-vs-machine
text but reports no rhetorical-figure features [inferred from the abstract].
**DependencyAI** (https://arxiv.org/abs/2602.15514, Feb 2026) is the most interesting
recent entry: detection from labels of linguistic dependency relations, explicitly
"interpretable" and "non-neural network baseline", competitive across domains and
generators. It needs a parser (spaCy/Stanza would do) rather than model logits. For
buvard it suggests a roadmap rule family: counts of AI-preferred dependency patterns
(parataxis chains, clefts, fronted adverbials) are mechanically countable.
**Rhetorical figures (rule of three, antithesis density).** No published detector was
found that measures these. Wikipedia's signs page lists them as indicators
(https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing) and the lit review
already encodes the density-threshold design. This stays an open lane: nobody has
published a tricolon-frequency study of AI vs human prose, which makes any measurement
buvard does on its Gutenberg FP corpus novel by default [inferred: no such paper found
in the arxiv sweeps run for this survey].
**Burstiness, mechanically.** Sentence-length coefficient of variation is computable
without any model; GPTZero uses a proprietary variant. No public implementation with
published thresholds exists [inferred]. The lit review's recommendation 6 (informational
note, per-genre) is consistent with the literature: low sentence-length variance is
necessary-but-not-sufficient evidence, and punishes trained technical writers.
**Stylometry caveat that matters for buvard.** Function-word and rhythm features are
exactly the features that misfire on non-native writers (61.22% FP study,
https://arxiv.org/abs/2304.02819) and on flatly organized human prose. Every stylometry
rule in buvard should stay at informational severity, never a verdict.
## 5. Testable this week (locally only — owner policy 2026-11-11: no online API, no hosted demo, no text leaves the machine)
1. **Open Pangram EditLens, locally (highest value).** The API is banned (owner policy:
never any online API) but the *open-weights* checkpoints are exactly what the policy
allows. `pangram/editlens_roberta-large` (355M, 512-token input, CC BY-NC-SA 4.0,
English-only training) runs on this MacBook. Plan: sample HC3 EN (human/AI pairs via
the datasets-server rows API — public data downloaded to us, nothing of ours sent
out), score, plus pre-2022 Gutenberg EN for the FP side. Chunk texts to ≤512 tokens.
The 3B checkpoint only if it is not HF-gated. Their own table: roberta-large FPR
0.002 in-domain, 0.099 on non-native English; FPR is the number buvard cares about.
2. **Binoculars on English only** (optional, GPU permitting). `git clone
https://github.com/ahans30/Binoculars && pip install -e . && python app.py`. Skippable
if no GPU: the English numbers are already published and it is documented
English-first.
3. **[STRUCK 2026-11-11 — owner policy: never any online API.]** The GLTR *hosted demo*
sends pasted text to a third-party server. GLTR itself is MIT-licensed and can be
self-hosted; if wanted, self-host it, never use gltr.io.
4. **Fast-DetectGPT spot check** (optional). Clone
https://github.com/baoguangsheng/fast-detect-gpt; runs on open GPT-Neo checkpoints.
Useful only if buvard wants a reference score column for the English corpus.
5. **French cadence measurement in pure Node (the one buvard can ship).** Compute
sentence-length CV and tricolon density over the Gutenberg FR corpus and the
INRIA FR set with buvard's own walker (no new deps). If the two corpora separate,
recommendations 6 and 9 of the lit review graduate from proposal to measured rule.
Skipped deliberately: Ghostbuster (needs an OpenAI key), Originality/Sapling/Winston/
Copyleaks (no free testing surface worth the time this week), any benchmark download
(RAID, M4) until a French split exists.
## 6. Sources
- Pangram front page: https://www.pangram.com/
- Pangram pricing: https://www.pangram.com/pricing
- Pangram third-party eval roundup: https://www.pangram.com/blog/third-party-pangram-evals
- U Chicago eval PDF: https://bfi.uchicago.edu/wp-content/uploads/2025/09/BFI_WP_2025-116.pdf
- U Maryland eval: https://arxiv.org/pdf/2501.15654
- VUB eval: https://doi.org/10.1007/s40979-026-00226-w
- GPTZero: https://gptzero.me/
- Originality.ai: https://originality.ai/
- Sapling: https://sapling.ai/
- Winston AI: https://gowinston.ai/
- Copyleaks: https://copyleaks.com/
- Binoculars: https://github.com/ahans30/Binoculars and https://arxiv.org/abs/2401.12070
- Fast-DetectGPT: https://github.com/baoguangsheng/fast-detect-gpt and https://arxiv.org/abs/2310.05130
- Ghostbuster: https://github.com/vivek3141/ghostbuster and https://arxiv.org/abs/2305.15047
- GLTR: https://gltr.io and https://arxiv.org/abs/1906.04043
- RADAR: https://arxiv.org/abs/2305.13014
- RAID: https://raid-bench.github.io/ (arxiv id unverified this session)
- M4: https://arxiv.org/abs/2305.14902
- MULTITuDE: https://arxiv.org/abs/2310.13606
- INRIA French ChatGPT detection: https://arxiv.org/abs/2306.05871 and https://gitlab.inria.fr/wantoun/robust-chatgpt-detection
- Non-native FP study: https://arxiv.org/abs/2304.02819
- Wikipedia signs of AI writing: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
- DependencyAI: https://arxiv.org/abs/2602.15514
- Stylometry for academic integrity (2026): https://arxiv.org/abs/2601.01225
- Comparative neural framework EN/IT (2026): https://arxiv.org/abs/2603.18750
- Authorship analysis review (2025): https://arxiv.org/abs/2505.15422
- Arabic shared-task example (XLM-R recipe): https://arxiv.org/abs/2510.20610
**Newer than the lit review (2025-2026).** A 2026 comparative framework of neural
detectors against commercial tools on English and Italian
(https://arxiv.org/abs/2603.18750) confirms the pattern: neural fine-tunes lead in
domain, commercial tools are competitive but brittle out of domain [inferred from the
abstract]. A 2026 authorship-analysis review
(https://arxiv.org/abs/2505.15422) lists multilingual adaptation and AI-text detection
as open gaps. Nothing in this sweep changes the lit review's conclusion; French remains
open territory.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment