Commit ecb310bd by PLN (Algolia)

docs: literature review — detection limits, FR gap, market seat, catalog v3 recs

parent 217cabe2
# buvard: literature review
Literature review supporting buvard, a regex + heuristic prose linter for AI-slop
patterns in French and English. Written 2026-11-07. Every non-obvious claim carries a
URL; anything we could not verify is marked [inferred] or [unverified]. This document is
itself written under buvard's own rules: no em-dash clause separators, no contrast
constructions, no rule-of-three defaults.
## 1. Statistical markers: what stylometry actually measures
Formal detection research works at the token level, with probability scores, while
buvard works at the surface level, with regex and counts. The two literatures agree on
one thing that matters for buvard: short texts defeat everything.
The canonical systems and their numbers:
| System / metric | What it measures | Reported result | Source |
|---|---|---|---|
| GLTR (token rank) | position of each token in the model's ranked vocabulary; human text draws from ranks 10-1000 far more than AI text | raised human judges' fake-text accuracy from 54% to 72% on GPT-2 output, with no training | https://arxiv.org/abs/1906.04043 |
| DetectGPT (curvature) | negative curvature of the log-probability surface under random perturbations | AUROC on fake-news detection rose from 0.81 to 0.95; requires logprobs from the source model itself | https://arxiv.org/abs/2301.11305 |
| Binoculars (cross-perplexity) | ratio of perplexities of two closely related models on the same text | over 90% recall on ChatGPT samples at 0.01% false positive rate | https://arxiv.org/abs/2401.12070 |
| Recursive paraphrase attack | paraphrase round-trips push text back toward the human perplexity band | detection rates collapse on ~300-token passages while text quality barely drops | https://arxiv.org/abs/2303.11156 |
Three limits recur across these papers and they define buvard's lane:
1. **Token-level scores need the model's logits.** DetectGPT needs the source model's
own logprobs, Binoculars needs two local models, GLTR needs the vocabulary ranking.
A regex linter cannot have any of these. That is a feature here: the catalog catches
surface evidence, and it says so.
2. **Short texts are unreliable for everyone.** Sadasivan et al. test on ~300-token
passages and show paraphrasing alone destroys detection. Any per-paragraph verdict
from a stylometric score on fewer than ~150 words is noise. buvard's weighted scores
should aggregate per document, per 500-word window, and report counts, never a
probability of AI authorship.
3. **Detectors discriminate against human writers as much as machines.** Seven commercial GPT
detectors misclassified on average 61.22% of 91 TOEFL essays by non-native writers
as AI-generated; 97.80% were flagged by at least one detector. Prompting to sound
"more like a native speaker" cut the false positive rate from 61.22% to 11.77%,
which pins the cause on word choice: common-word-heavy, low-perplexity prose reads
as machine. Source: https://arxiv.org/abs/2304.02819 (published in Patterns:
https://www.cell.com/patterns/fulltext/S2666-3899(23)00130-7). This is the single
strongest argument for buvard to stay a style linter and never an authorship judge.
On the specific surface metrics writers talk about:
- **Perplexity and burstiness.** GPTZero popularized "perplexity" (predictability) and
"burstiness" (sentence-to-sentence variance) as its two scores; the company's own
learning pages define burstiness as variation in sentence structure and length
[inferred URL: https://gptzero.me/learn/, we could not fetch it cleanly]. The
direction is consistent everywhere: AI text has flatter sentence-length variance.
Note the fairness trap: lower variance is also a hallmark of trained technical
writing, so buvard should flag flat rhythm as a structural note, at low weight.
- **Function-word rates and lexical diversity.** Liang et al.'s corpus method
(frequency of common tokens against expert-written reference texts) found a
"decrease in variation of linguistic features" in LLM-modified text and estimated
6.5-16.9% of AI-conference peer-review text as substantially LLM-modified.
Source: https://arxiv.org/abs/2403.07183. Kobak et al.'s excess-vocabulary analysis
of 15M PubMed abstracts (2010-2024) showed an abrupt step increase in specific style
words the month ChatGPT shipped, and a floor of at least 13.5% of 2024 abstracts
LLM-processed, up to 40% in some subcorpora. Source:
https://arxiv.org/abs/2406.07016 (published version in Science Advances, 2025).
For buvard: relative word frequency shifts are the most robust, hardest-to-avoid
signal, and they are exactly what a frequency-banded lexicon encodes.
- **Repetition metrics.** Low type-token ratio and repeated sentence openers are
standard stylometric evidence; the excess-vocabulary paper above is the strongest
published version of the repetition claim (same words, everywhere, all at once).
Sentence-opener repetition is also the top complaint in every practitioner catalog
reviewed in section 4.
## 2. The contrast-construction family and its cousins
**"Not X, but Y" / "Ce n'est pas X, c'est Y".** Wikipedia's "Signs of AI writing"
maintainer page lists the construction as a top indicator, in the forms "not just X,
but also Y" and "serves as / stands as / marks / functions as / operates as /
represents [a]": https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing. The
page's own caveat matters and buvard should quote it in its docs: these are "potential
signs", since human authors use the same styles constantly, especially in blogs and
list-style articles. hardikpandya's stop-slop skill independently catalogs "binary
contrasts" as an AI tell: https://github.com/hardikpandya/stop-slop. On the French
side, both anti-slop-fr skills flag structural contrast and false-agentivity frames
("les chiffres parlent d'eux-mêmes", "plongeons dans"): see section 3.
The craft consensus, reconstructed from these catalogs: the antithesis is classical
rhetoric and legitimate in moderation; the tell is frequency and sameness. LLMs reach
for the construction as a default compression move, so one instance per document is
style and five per 500 words is slop. For buvard this argues a density threshold with
a low per-instance penalty, not a per-hit kill.
**The rising "X, so Y" pattern.** This is the "It's a great tool, so..." hedge-plus-
pivot: a bland claim followed by "so" + grandiose consequence ("so you can focus on
what matters"). We found no published linguistic study of it [unverified]; it appears
in practitioner catalogs, including the 20-pattern creatoreconomy skill buvard already
ingests. Treat the sourcing as folklore for now: flag at low weight until a second
independent source appears.
**Em dash.** Wikipedia's signs page documents "Overuse of em dashes" with an explicit
date range, 2022 to September 2026, tying the spike to LLM output
(https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing). The 2025 press wave on
this is real but unreachable from this environment, so specific articles (Merriam-
Webster wordplay, Guardian comment pieces, New York Magazine's em-dash coverage) and
the reported Microsoft Word autocorrect incident that auto-inserted em dashes and was
reverted after user backlash are marked [unverified] here. buvard's own budget
approach (a cap per 500 words instead of a ban) matches the actual debate: writers
with strong opinions on em dashes are the loudest false-positive victims, and
hand-written English prose legitimately uses them. French differs: French typography
uses the tiret for dialogue and parenthetical incises far less in expository prose, so
the FR budget should be tighter.
**Rule of three.** The tricolon is a classical figure, and Wikipedia's signs page
lists "Rule of three" as an AI indicator without claiming any single tricolon is bad.
The practitioner consensus again is density: a consistent pattern of A, B, and C
triplets in consecutive paragraphs is the tell. buvard should detect the repetition
across a window (three or more tricolons in 500 words) rather than flag each one, and
exempt dialogue, lists, and comedic writing where the triple is the joke.
**Design rule extracted from this whole section.** For every rhetorical device in the
family (contrast, tricolon, dash, so-pivot), the literature and the catalogs converge
on the same shape: legitimate once, slop when uniform. Encode per-instance penalties
near zero and window-density penalties that bite.
## 3. French-specific AI tells: thinner coverage, different typography
**The practitioner layer is where FR is strongest.** The two anti-slop-fr skills are
the main public French catalogs:
- JulienSnsnCode/anti-slop-fr (https://github.com/JulienSnsnCode/anti-slop-fr):
organizes the problem into 13 lexicon categories and 13 structural patterns, flags
over-represented words ("mots surreprésentés"), ambiguous turns, predictable
structures such as "dans un monde en constante évolution", English calques
(« plongeons dans »), and false agentivity (« les chiffres parlent d'eux-mêmes »).
It grades text 1 to 10 on Franchise, Rythme, Confiance, Naturel, Densité and
rewrites under 35/50, which is an LLM-judge pipeline, unlike buvard's deterministic
scoring.
- captain-marketing/anti-slop-fr (https://github.com/captain-marketing/anti-slop-fr):
two main families, "Formules creuses" (empty hyperboles such as « révolutionnaire »)
and "Patterns structurels" (monotone rhythm, vague declaratives). It ships
identity/ voice profiles and examples/, i.e. it is a rewriting skill, again not a
linter.
The letape-dapres.fr writing-unslop source could not be fetched from this environment;
its contribution to buvard's catalog remains [unverified] pending a direct read.
**The research layer barely covers French.** MULTITuDE, the largest multilingual
machine-text benchmark, spans 74,081 texts in 11 languages and French is absent (ar,
ca, cs, de, en, es, nl, pt, ru, uk, zh): https://arxiv.org/abs/2310.13606. The main
French ChatGPT-detection study (INRIA, https://arxiv.org/abs/2306.05871, artifacts at
https://gitlab.inria.fr/wantoun/robust-chatgpt-detection) built its French data by
translating an English dataset, and its own conclusion is caution: in-domain accuracy
held, out-of-domain and adversarial text exposed the detectors. So the only formal FR
work is (a) translated, and (b) out-of-domain fragile. No published French excess-
vocabulary analysis equivalent to Kobak et al. exists as far as we could find
[unverified]. Conclusion: buvard's French catalog is not porting a mature literature,
it is the literature, which is the project's real opportunity.
**French slop vocabulary worth weighting (from the FR skills, cross-checked against
EN patterns by analogy [inferred]):** « plonger dans », « à l'ère de », « en constante
évolution », « il convient de noter », « il est important de noter », « crucial »,
« essentiel », « révolutionnaire », « palpitant », « intrinsèquement », « en
définitive », « à la fin de la journée », « les chiffres parlent d'eux-mêmes », «
permet de », « grâce à » used as a hedge, and the English calques « débloquer » (au
sens figuré), « donner un aperçu de ». These deserve frequency-band treatment like the
EN lexicon rather than flat bans.
**Typography is where FR detection differs structurally.** French orthography requires
a non-breaking (or narrow no-break) space before double punctuation (; : ! ?) and
inside guillemets « » : https://fr.wikipedia.org/wiki/Espace_ins%C3%A9cable (see also
the Lexique des règles typographiques de l'Imprimerie nationale). LLM output is
notoriously inconsistent here: thin or plain spaces slip in, and many LLM pipelines
emit straight quotes instead of « ». buvard can add mechanical FR-only rules with
almost zero false-positive risk: guillemet spacing, insécable before ; : ! ?, and
straight-quote detection. These are the FR equivalent of the em dash, except they are
genuinely wrong under French typographic convention, so they can carry a hard penalty
instead of a stylistic note.
**Coverage gap, quantified as far as the evidence allows:** English has the GLTR /
DetectGPT / Binoculars literature, a commercial detector market, and dozens of
practitioner catalogs. French has one translated-data detector study, one benchmark
that skips it, two practitioner skills, and typography rules. buvard should say this
on its README rather than pretend parity.
## 4. The detector and linter landscape
| Tool | Kind | What it actually does | Accuracy claims and FP profile |
|---|---|---|---|
| stop-slop (hardikpandya) | rules skill, MIT | an LLM agent skill, not a program: flags throat-clearing openers, business jargon, binary contrasts, passive voice; no benchmarks | none published; https://github.com/hardikpandya/stop-slop |
| humanizer (blader) | LLM skill, MIT | a rewrite skill over "26 patterns" (staging instead of stating, rhythm by rule, inflation and borrowed authority, formatting by rule); a blind test had judges prefer its rewrite 16/16 | no detection accuracy claim; it is a rewriter; https://github.com/blader/humanizer |
| deslop / unslop (Cursor ecosystem) | editor rules | could not verify from this environment: plausible slugs exist (cursor.directory returned 429, candidate GitHub repos 404) [unverified]; the pattern family is LLM-agent rules that strip slop from code comments and chat remnants | none |
| proselint | rules linter, BSD | the ancestor of this genre: a spell-checker for style, rules for cliches and errors, explicitly "aggregate knowledge about best practices in writing" | no accuracy claim; it reports rules fired, not authorship; https://github.com/amperser/proselint |
| Vale | rules linter, MIT | "brings code-like linting to prose": YAML-defined styles with regex and extension points, no model, explicitly not a detector | https://docs.vale.sh/ |
| GPTZero | ML service | perplexity and burstiness scores; claims "99.9% Accuracy" while its own text warns no detector is 100% accurate and results "should not be used to punish or as the final verdict" | https://gptzero.me/ ; the 61.22% TOEFL false-positive result (section 1) includes detectors of this class |
| Pangram | ML service | a "traditional language model architecture" classifier for text, image and video; claims "over 99% accurate" with a "1 in 10,000 False positive rate", "verified by researchers" at U Chicago and U Maryland | https://www.pangram.com/ ; the claim is vendor-published [inferred: independent replication details not accessible here] |
| Originality.ai | ML service | "Most Accurate AI Detector in third-party studies"; no specific FP rate on its own front page, no visible appeal process | https://originality.ai/ |
Reading the table as a competitive map for buvard:
1. **The rules side of the market is agent skills, not linters.** stop-slop and
humanizer are prompts for LLMs to rewrite text. Nobody ships a deterministic,
score-weighted, CI-runnable prose linter focused on AI slop. proselint and Vale
are deterministic but generic-style, with no slop catalog and no French. This is
buvard's empty seat: Vale's architecture (YAML rules, regex, severities, exit
codes) pointed at a maintained FR+EN slop catalog.
2. **ML detectors claim 99+ percent and disclaim everything.** GPTZero's own warning
against using results "to punish" is the confession the whole category shares, and
the peer-reviewed floor is the 61.22% false-positive rate on non-native writing
(section 1). Any tool that emits an authorship probability inherits this.
3. **None of the rules tools has French.** proselint is English-only; Vale's official
styles are English-first [inferred]; both FR skills are prompts. buvard's FR rules
(section 3 typography rules in particular) have no direct competitor at all.
4. **False-positive profile to design against.** The documented victims of AI
detection are non-native writers (arxiv 2304.02819) and, anecdotally across the
catalogs, autistic and ADHD writers whose prose is flatly organized or
formality-heavy [unverified, no published study found]. For a rules linter the
equivalent risk is harsh severity on ordinary rhetoric; buvard should default all
contrast-family rules to informational severity.
## 5. Implications for buvard
Ten recommendations, each stated so it can be falsified by a measurement:
1. **Never claim AI detection.** Position buvard as a style linter that reports
surface slop evidence. No authorship probability, no "percent human". Test: the
output schema contains counts and weighted scores only. Grounds: 61.22% FPR on
non-native writers (section 1) and GPTZero's own anti-punishment disclaimer
(section 4).
2. **Score the contrast family by density, not by instance.** Antithesis is
legitimate once and slop when uniform (section 2). Test: a pre-2022 Gutenberg
prose corpus should show the per-instance rule firing on classic essays at rates
indistinguishable from modern human prose, while the window rule separates the
corpora.
3. **Promote FR typography rules to hard penalties.** Insécable before ; : ! ?,
guillemet spacing, straight quotes: mechanically wrong per French typographic law
(fr.wikipedia.org/wiki/Espace_ins%C3%A9cable), near-zero false-positive risk, and
no competitor has them. Test: FP rate on the Gutenberg FR corpus should be ~0
after normalizing Unicode.
4. **Build the FR lexicon as frequency bands, not a flat list.** Tier 1 (always
slop: « il convient de noter », « à l'ère de », « plonger dans ») gets the full
weight; tier 2 (context-dependent: « crucial », « permet de ») fires only on
density. Grounds: the FR skills' two-tier structure (section 3) and the excess-
vocabulary finding that frequency shifts, not single words, are the robust signal
(arxiv 2406.07016).
5. **Cap total lexicon penalty per document.** Common-word-heavy prose is exactly
what detectors punish unfairly (arxiv 2304.02819), so buvard must not reproduce
the failure in lint form. Test: a text with 40 tier-2 hits and nothing else
stays below the "fail" threshold.
6. **Flag flat rhythm as an informational note.** Compute the coefficient of
variation of sentence lengths per 500-word window; flag low CV only in prose
genres where variance is expected (essays, narrative) and never in technical docs.
Thresholds are an open empirical question [inferred]; publish whatever the human
corpus shows.
7. **Keep the "X, so Y" pivot at low weight and dated folklore status.** No
independent source found (section 2). Test for promotion: a second practitioner
catalog or an n-gram frequency study showing post-2022 growth. Until then it
cannot be promoted above -2.
8. **Em-dash budgets per language.** EN 1 per 500 words (as shipped), FR tighter at
1 per 1000, both exempting dialogue lines. Grounds: the documented 2022-2026
em-dash spike is EN-centric (Wikipedia signs page), and FR prose uses tirets
mainly for dialogue and incises.
9. **Rule-of-three fires on the window, never the first tricolon.** Test: flagging
only at three or more tricolons per 500 words should leave the Gutenberg corpus
nearly untouched.
10. **Version the catalog like an API, with provenance on every rule.** Rules get
stable ids (`buvard.fr.lexique.007`), each carries `source` (URL + retrieval
date) and `severity`, the catalog semvers (adding a rule bumps minor, reweighting
bumps major since scores shift), and a machine-readable changelog. Grounds:
catalogs drift as LLMs change their tells; the em-dash range on the Wikipedia
page shows tells have lifetimes. Without provenance, a 2027 user cannot tell a
measured pattern from a 2024 hunch. This review's [unverified] marks are exactly
the metadata such a field would have made mandatory.
**Summary of the position.** The statistical literature says detection itself is
unreliable, biased against non-native writers, and defeated by paraphrasing; the
commercial market sells 99% claims with disclaimers attached; the rules layer of the
market is prompt skills with no determinism, no scores, and no French. A deterministic,
score-weighted, provenance-tracked FR+EN style linter with a public false-positive
test corpus occupies ground that nobody currently holds, and its honest claim ("your
text reads like the slop everyone complains about") is the only claim in this space
that survives contact with the evidence.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment