How AI Detectors Work: Perplexity, Burstiness, Limits

November 5, 2026 · 16 min read

AI text detectors look for patterns associated with generated writing. Some measure perplexity, or how surprising the words are to a reference language model. Some examine variation in the passage, often described as burstiness. Others use classifiers trained on labelled examples. None of these text-only methods checks whether a claim is true or establishes who wrote it.

The premise comes from how large language models work. They learn to predict the most likely next word, then select words using probability distributions. Their output tends to follow familiar patterns. Human writing has more irregular shifts in rhythm, vocabulary, and sentence structure, but the boundary is far from clear. A person following a strict style guide can produce the same statistical pattern a scanner treats as artificial. So can a non-native English speaker using standard phrasing or a technical author using required terminology.

Detection estimates resemblance to examples or patterns. The approaches below use different kinds of evidence. The later style cues can help a person inspect a passage, but they are not a published checklist of inputs used by every scanner.

Understanding Perplexity: How Models Measure Surprise

Perplexity measures how well a probability distribution or language model predicts a sample of text. Its formal definition is the exponentiated average negative log-likelihood of a sequence. Put more simply: how surprised is the model by each word the writer chose?

An autoregressive language model calculates a probability for each possible next token based on the tokens before it. After "the doctor examined the", "patient" is likely to be less surprising than "refrigerator". Decoding methods such as nucleus sampling and top-k filtering limit which tokens can be selected, though settings vary and generated text need not always use the most likely word.

A detector that uses perplexity passes a draft through a reference language model and scores its tokens against that model's predictions. Consistently likely choices keep perplexity low; unexpected choices raise it. Other detectors use different methods.

The Giant Language Model Test Room (GLTR), developed by researchers at Harvard University and the MIT-IBM Watson AI Lab, makes these predictions visible. It compares text with an underlying language model and places each token in one of four categories: - Green tokens: Words within the top 10 most probable choices predicted by the model. - Yellow tokens: Words ranked between 11 and 100 in probability. - Red tokens: Words ranked between 101 and 1,000. - Purple tokens: Rare or unexpected words ranked outside the top 1,000.

GLTR's examples show how token ranks can reveal patterns in generated text from the models it studied. The distribution depends on the generator, decoding settings, reference model, and passage. A human can also choose many common words, so a field of green tokens is a clue to inspect, not a verdict.

A high perplexity score is not a measure of good writing. Random words, broken grammar, and scrambled characters can produce an extremely high score because the model finds them surprising. A careful editor's clear instruction can score low because it uses familiar words. Perplexity alone cannot reliably separate human and machine writing.

Understanding Burstiness: The Cadence of Human Thought

Burstiness measures variation across sentences and paragraphs. In computational linguistics, it describes the uneven distribution and clustering of language features, along with differences in structure throughout a document.

A writer's rhythm changes with their focus, emotional state, and purpose. A long sentence with several clauses and a parenthetical thought might lead into a short one. Three brief statements can create urgency; a longer sentence can then weigh two opposing ideas. These changes give the passage an uneven rhythm.

Some generated passages repeat similar sentence lengths and structures. Others vary considerably, especially when prompted to do so. Uniform rhythm is something a reader may notice, but there is no sentence-length interval that identifies machine writing.

Sentence-length variation can be summarized by the coefficient of variation: the standard deviation divided by the mean. Researchers and tools may also examine punctuation, vocabulary, or syntax. A tool's actual features and decision rule depend on its design, and many commercial systems do not disclose them.

These examples show why the two signals can overlap. They are possible patterns, not predicted classifications by a particular tool:

Three Architectural Approaches to AI Detection

Three approaches help explain the field. A commercial product may combine methods or keep its implementation private.

1. Statistical Scoring Systems (Zero-Shot Detectors)

A zero-shot detector uses an existing, frozen language model without training a separate classifier. It feeds your passage into a model such as GPT-2 or LLaMA, token by token, to calculate each word's probability.

Possible features a statistical detector might use across a passage include: - Perplexity: A value derived from the average negative log-likelihood of the tokens. - Perplexity variance: The degree to which word predictability spikes or stays flat. - Sentence length standard deviation: The spread of word counts across sentences. - Low-rank token ratio: The percentage of words falling within the top 10 most probable choices.

Zero-shot scoring needs no labelled dataset of human and machine texts. Its weakness is the reference model: every prediction is measured against that particular model. Text from a newer model, one with different training weights, or a system using a higher sampling temperature may escape detection.

2. Supervised Deep Learning Classifiers

For a supervised classifier, developers fine-tune a pre-trained transformer such as RoBERTa or DeBERTa on large datasets of paired human and machine texts. The detector learns from those examples instead of relying only on perplexity and burstiness formulas.

During training, the network learns patterns in syntax, word positions, and words that occur together. It maps the input passage into a numerical representation and returns a score used for classification. The meaning of a displayed percentage depends on the product's calibration and interface.

These classifiers can catch patterns that simple formulas miss, but their performance depends on the datasets used to train them. A classifier trained on a narrow set of genres may not generalize well to unfamiliar formats with different stylistic conventions. Editors should keep those boundaries in mind rather than assuming a classifier handles every writing style equally well.

3. Algorithmic Watermarking

Watermarking happens while text is generated. It cannot be added retroactively by a detector. Research by Kirchenbauer and colleagues at the University of Maryland describes how to embed an invisible mathematical signature in a model's token stream.

A cryptographic pseudorandom hash uses the preceding token to divide the vocabulary into a green list and a red list. When the model generates the next word, it adds a small bias to the green tokens' logits. That encourages the sampling engine to choose green words over red ones.

The prose can still read naturally because the favored tokens fit the context. A detector with the key can test whether green-list choices occur more often than chance predicts. Its confidence depends on passage length, watermark strength, and later edits; the result applies to that watermarking scheme.

The model provider has to cooperate. A third-party detector cannot find a watermark that the generator never embedded. Research also shows that substantial manual editing, repeated paraphrasing, or translation into another language and back can weaken or erase watermarks.

Style Cues Are Not Detector Rules

A discussion of generative text markers describes patterns a reader may notice: evenly spaced arguments, steady tone, generic examples, and transitions that make every paragraph feel alike. Those are editorial observations. A vendor does not necessarily measure them, and none establishes authorship.

Look at an argument's priorities. If every point receives the same amount of space, ask whether the important objections need more attention. If the prose gives only a broad consensus, ask what evidence supports its conclusion. These questions improve a draft whether a person or a model wrote it.

Tone and rhythm deserve the same care. A passage may repeat similar clauses or maintain an unwavering formal register. That can fit a legal document, a technical instruction, or a style guide. A conversational aside or a regional expression can make an essay more specific, but their absence is not evidence of automation.

Finally, check concrete knowledge. A passage that sounds authoritative may omit uncertainty, edge cases, and the writer's actual observations. Verify its claims and ask for the relevant records or sources. This is a stronger editorial response than counting stock phrases or assuming polished grammar reveals a generator.

Why Detectors Misfire: The Mechanics of Errors

A false positive occurs when human writing matches the statistical patterns a detector associates with machine text. A false negative occurs when edits disrupt those patterns in an AI draft. Both errors follow from measuring statistical resemblance instead of authorship.

The Non-Native English Speaker Penalty

Bias against writers who use English as a second language is a serious failure of these tools. Liang and colleagues at Stanford University evaluated seven widely used detectors on writing by native and non-native English speakers.

The detectors correctly classified most native English students' essays but repeatedly misclassified essays from non-native students taking the Test of English as a Foreign Language (TOEFL). Across the seven detectors, the average false-positive rate for those TOEFL essays was 61.22 percent. 97.8 percent of the TOEFL essays were flagged by at least one of the seven detectors. Those figures describe the study's sample and tools, not all detectors or all non-native writers.

Perplexity helps explain this bias. Non-native writers tend to use a narrower textbook vocabulary and standard grammatical constructions. Avoiding obscure idioms, elaborate rhetoric, and rare colloquialisms produces low perplexity. The scanner treats that clear, restrained language as evidence of machine generation.

Standardized, Technical, and Legal Writing

Technical and professional writing faces a related challenge. Legal drafting, medical documentation, aviation compliance, and scientific research require clarity and consistency. Attorneys use statutory formulas. Software engineers need exact variable names and architectural terms, while clinical researchers follow rigid reporting structures.

A compliance report or terms of service agreement deliberately uses predictable wording. Ambiguity in a contract or technical manual can cause liability or system failure. Because these fields rely on standardized terms and uniform structures, their statistical profile can naturally exhibit low perplexity and low burstiness. Writers should preserve required terms and maintain drafting records rather than distorting necessary technical language.

Theoretical Limits of Text Classification

Sadasivan and collaborators at the University of Maryland established mathematical bounds on how well AI-generated text can be detected.

Their analysis showed that detection performance is constrained by the statistical distance between human and machine writing. Conditionally, as the two text distributions get closer, limits on detection performance tighten. As total variation distance between the distributions shrinks, the Area Under the Receiver Operating Characteristic curve is restricted, even for an optimal detector.

Under those conditions, reducing false positives increases false negatives. If the distributions become sufficiently close, a detector cannot catch machine text without also flagging an unacceptable share of human writing.

The Impact of Automated Paraphrasing

Edits can change a classification because the detector receives different text. Rewriters may change vocabulary, sentence boundaries, or clause order, and those changes may alter the signals a particular tool uses. Our guide to what an AI humanizer does explains the editing process. A changed score alone does not establish a change in authorship, which is one reason detection scores can mislead.

Worked Examples: Predictability in Action

Compare the word choices and sentence lengths in these versions of a professional project update.

Worked Example 1: High Predictability Draft

Before: "The primary objective of this quarterly project review is to evaluate the technical milestones that our engineering team achieved over the past three months. We have successfully deployed the updated customer authentication service to our staging environment, which significantly reduces login latency across all major platforms. In addition to these performance gains, our development team implemented automated database backups to prevent data loss during scheduled server maintenance periods. Moving forward, the infrastructure group will conduct comprehensive stress testing on the payment gateway to ensure complete operational stability before the public launch."

This 91-word passage has four sentences: 24 words, 22 words, 22 words, and 23 words. Each is within two words of the 23-word average.

The passage has several features a reader might notice. Its sentences are similarly paced, and "primary objective", "successfully deployed", "significantly reduces", and "operational stability" are familiar enterprise phrases. Similarity does not tell us how a detector would score this passage; we have not submitted it to one.

Worked Example 2: Low Predictability Revision

The edit removes padded transitions and varies sentence structure while keeping the facts, deadlines, and technical claims. It adds no details or claims.

After: "Our engineering team hit several key milestones over the past three months. We deployed the updated customer authentication service to staging, which cut login latency across all major platforms, and implemented automated database backups to prevent data loss during scheduled server maintenance. Next, the infrastructure team will stress-test the payment gateway. That is intended to ensure stability before public launch."

The revised passage creates an uneven cadence: - A moderate opening sentence introduces the quarterly milestones. - A longer sentence explains the staging deployment alongside the backup safeguard. - A brief sentence introduces the upcoming stress test. - A concise closing sentence notes the intended stability goal.

The edit cuts filler phrases, including "The primary objective of this quarterly project review is to evaluate" and "In addition to these performance gains". Stock phrasing like this can match expected patterns that could lower a reference model's perplexity. The facts stay the same, while active phrasing and varied sentence lengths could increase calculated burstiness if scored by a detector.

A Step-by-Step Process: Auditing Text for Predictability

Use this five-step audit to find predictable patterns and make a draft easier to read.

Step 1: Map Sentence Lengths and Check Cadence

Count the words in each sentence and write the numbers down in order. - Sentences consistently between 18 and 24 words create a monotonous rhythm. - Split long compound sentences into shorter statements. - Combine choppy sentences with subordinating conjunctions where the ideas belong together. - Vary the cadence, allowing brief five-word statements alongside longer twenty-word explanations.

Step 2: Audit Transitional Crutches and Formulaic Framing

Read the opening of each paragraph and sentence. - Cut filler such as "Moreover," "Furthermore," "In summary," "It is essential to recognize," and "On the other hand." - Delete announcements about what a sentence is about to say. - Start with the subject and active verb.

Step 3: Inject Varied Syntactic Density and Active Verbs

Machine drafts overuse passive constructions and weak uses of "is," "was," and "has been." Look for actions hidden in long phrases. - Replace nominalizations with verbs: "conducted an evaluation of" becomes "evaluated." - Use a parenthetical observation, an appositive, or a brief rhetorical question where it helps break a rigid pattern. - See these techniques for writing clear sentences without flattening your voice for more practical edits.

Step 4: Verify Factual Constraints and Meaning Preservation

Compare the edited passage with the original to check for shifts in meaning. - Keep technical terms rather than replacing them with conversational synonyms to raise perplexity. - Preserve every date, percentage, condition, and exception. - Follow the workflow on humanizing a draft by hand and review common stylistic signs of AI writing for help revising accurately.

Step 5: Test Across Multiple Evaluation Methods

If your organization uses scanners, compare results from more than one tool. - Include both statistical analyzers and supervised classifiers. - If one reports ninety percent AI and another reports zero percent, the disagreement reflects surface patterns rather than reliable forensic evidence. - Use the scores as advice about predictability, not as verdicts on who wrote the text.

Common Mistakes When Dealing With AI Detectors

These five mistakes can turn a questionable score into a poor editing decision.

1. Treating a Probability Output as a Percentage of Synthetic Words

A score of "85%" is often read as a claim that AI wrote 85 percent of the words and a human wrote the remaining 15 percent.

A displayed percentage may be a classification score, a share of text marked by the tool, or another product-specific measure. Without the tool's documentation, an "85%" result cannot be read as an 85 percent probability of machine authorship or as 85 percent of the words being generated. Trying to infer authorship sentence by sentence from that number wastes an editor's time.

2. Falling Into the Synonym Trap

Replacing common words with unusual synonyms may seem sensible once you know how perplexity works. In practice, this synonym trap ruins readable prose.

"Utilize" for "use," "demonstrate" for "show," or "perspicuous" for "clear" might alter raw predictability, but they make writing stiff and harder to follow. Forcing rare words into ordinary sentences rarely improves clarity. Writers are better served by focusing on natural rhythm and clear phrasing rather than swapping words to influence a detector.

3. Stripping Necessary Technical Terminology

Some terms have to stay exact. Replacing "orthogonal frequency-division multiplexing" with informal synonyms would corrupt the engineering meaning.

Rewriting technical documentation in colloquial language to lower a score makes it less useful professionally. Keep the accurate term, even when a scanner flags it.

4. Believing a Clean Score Confirms Factual Accuracy

A low detection score says that the text has statistical characteristics associated with human writing. It says nothing about whether its claims are true or properly cited.

A model prompted to use particular styles can produce high perplexity and burstiness alongside fabricated statistics, nonexistent legal citations, or medical errors. Check the facts before spending time on a detection score.

5. Failing to Retain Version Histories and Drafting Records

A clean score offers professionals and students little protection against a later false accusation. Keep incremental drafts and timestamped version histories, along with research notes, interview recordings, and data logs. A record of how you wrote the text is more persuasive than an argument about detection metrics.

Frequently Asked Questions About AI Detectors

Why do AI detectors give completely different scores for the exact same text?

Tools use different architectures, reference models, training data, and decision thresholds. A detector using token probabilities can disagree with a trained classifier. Products may also display scores with different meanings, so percentages from two tools should not be treated as interchangeable measurements.

Can an AI detector determine which specific LLM model wrote a passage?

Standard commercial detectors cannot reliably identify which model or version wrote a passage. GPT-4, Claude, and Gemini have subtle stylistic preferences, but their output distributions overlap heavily. Without a proprietary watermark and an identifiable cryptographic key, a text-only detector can estimate resemblance to machine text but cannot attribute it to a particular company or engine.

Does using a grammar checker make human writing look like AI to a detector?

It can. Grammar and style tools may standardize phrasing and punctuation, changing the patterns a detector sees. The effect depends on the edits and the detector; a changed score cannot tell you who wrote the original draft.

How does document length affect detector accuracy?

Short passages give a statistical detector less evidence. One unusual word or sentence can have a large effect on a small sample. There is no universal word-count threshold at which detector results become reliable; follow the tested limits stated by the specific tool and still treat the result as a fallible signal.

Can AI watermarks be removed by paraphrasing?

Yes. Watermarks such as those in the Kirchenbauer framework withstand simple deletions and light edits. Aggressive paraphrasing can disrupt the concentration of green-list tokens enough to make the watermark undetectable. Restructuring syntax, replacing a substantial share of words, or translating out of a language and back can have that effect.

What should a writer do if their original writing is falsely flagged as AI?

Request a human review instead of hurriedly rewriting your work to satisfy a scanner. Present the drafting records described above, plus keystroke logs, initial outlines, annotated sources, and dated drafts. Explain that detectors rely on fallible statistical patterns and have documented false-positive rates, as seen in evaluations of non-native writing. Ask the reviewer to consider that evidence together.