LLM Text Watermarking Under Real Edits

LLM text watermarking is a statistical provenance signal, not an authorship oracle. Cropping, substitution, paraphrase, translation, and dilution can weaken detection, while unrelated text can still cross a threshold.

This guide builds a seeded, explicitly simplified detector lab to measure those boundaries. It uses negative controls and a claim ladder so product language remains honest even when a score looks decisive.

Treat LLM text watermarking as uncertain evidence

LLM text watermarking can add a statistical signal during generation, but ordinary editing can weaken that signal and unrelated text can cross a threshold by chance. This laboratory measures crop, substitution, paraphrase proxies, translation, and dilution while keeping false positives visible. The goal is not to identify an author from prose; it is to define product language proportional to detector evidence.

The seeded laboratory is intentionally not an implementation of SynthID-Text. It uses a disclosed green-list toy rule so the score, edits, and false-positive behavior are inspectable; the Nature paper supplies evidence that production watermarking can operate at scale, not permission to clone an undisclosed detector. Keeping that line visible is part of the technical result.

Separate a toy green list from SynthID-Text.

The diagram shows a simplified seeded vocabulary partition that biases selection toward a green subset and later computes a z-like score. It is pedagogical, not SynthID-Text. The Nature SynthID-Text paper documents the production method and large-scale study; our disclosed toy model exists so every arithmetic step can be challenged.

Each edit family gets a controlled severity axis: crop fraction, synonym substitution rate, sentence shuffle count, translation round trip, and unmarked-text dilution. I preserve the original token count and random seed beside every derivative so a survival curve can be reproduced. Calling one hand-edited paragraph robust would confuse an anecdote with a measurement surface.

Runnable artifact: The disclosed toy detector proves edit length and negative controls change evidential strength; it is not SynthID.

Save this proof as toy-watermark-lab.test.mjs and run node toy-watermark-lab.test.mjs. Expected final line: PASS: toy watermark uncertainty.

import assert from "node:assert/strict";
const green=new Set(["fern","glass","orbit","quiet"]); const score=text=>{const words=text.toLowerCase().match(/[a-z]+/g)||[]; const hits=words.filter(word=>green.has(word)).length; return {n:words.length,z:words.length?(hits-words.length*.25)/Math.sqrt(words.length*.1875):0}};
const marked=score("fern glass orbit quiet fern glass orbit quiet"); const cropped=score("fern glass orbit quiet"); const plain=score("the hand made page holds a patient line"); assert.ok(marked.z>plain.z); assert.ok(cropped.n<marked.n); assert.ok(Number.isFinite(plain.z)); console.log("PASS: toy watermark uncertainty");

Build an edit-survival attack ladder

Each source passage produces derivatives at increasing severity, with one edit family changed at a time. Crop removes evidence length, substitution changes selected tokens, sentence movement stresses local structure, translation rewrites vocabulary, and dilution mixes marked with unmarked text. The resulting heat map answers where watermark detection fades instead of compressing heterogeneous attacks into one robustness percentage.

The negative set matters more than a dramatic positive screenshot. Human text, older web text, model text from an unwatermarked system, quoted material, code, and multilingual samples reveal how threshold choices distribute false accusations. I report confidence intervals and subgroup counts because a single aggregate false-positive rate can hide a small cohort with unacceptable harm.

Calibrate watermark detection on negatives.

Thresholds come from a held-out negative corpus representing languages, genres, lengths, quotation, code, and model families the product will see. A confusion matrix reports raw counts beside rates. The DeepMind limitations discussion supports cautious use, while AI confidence UX helps turn an ambiguous score into a reviewable state.

Detector output should be phrased as compatibility with a specific watermark and detector version, never proof of who wrote the document. Watermarks can be absent because the generator did not embed one, the text was edited, or the detector lacks coverage; they can also appear by chance at a threshold. The interface links the score to limitations before offering any review action.

Read forensic evaluations at their tested boundary

A 2026 forensic evaluation can inform attack design only with its detector, model, length, and transformation conditions attached. It does not certify every future watermark. I compare its threat model with the local corpus, note missing languages or edits, and use disagreements to create new fixtures rather than selecting whichever headline supports a preferred launch decision.

The 2026 forensic evaluation is read as one measured setting with named attacks, models, and lengths. I extract those conditions beside results rather than transporting a headline into every product. DeepMind's own limitation language similarly informs the claim ladder: provenance evidence can contribute to an investigation, but no detector resolves authorship alone.

Design a length-aware watermark benchmark. A benchmark begins by stratifying documents into short, medium, and long buckets before applying edits. LLM text watermarking detectors often accumulate evidence across tokens, so comparing a 70-token caption with a 2,000-token essay on one axis would disguise a basic sample-size effect. Each bucket receives equal negative controls, marked controls, and transformation families, while the report publishes raw document counts and token distributions beside every receiver-operating curve.

I would also freeze a multilingual slice and a quotation-heavy slice that is never used to tune the threshold. If those cohorts drift into the indeterminate region, the interface should widen uncertainty rather than borrow confidence from English expository prose. This is where detector labs become product engineering: the evaluation population must resemble actual decisions, and any unsupported genre needs a visible unavailable state instead of an undocumented extrapolation.

Write a claim ladder that cannot accuse

Low evidence becomes no supported signal, the middle region becomes indeterminate, and a high score becomes compatible with this detector version under stated conditions. None of those states says human, machine, fraud, or plagiarism. Pairing the result with C2PA content credentials and multimodal evidence design provides provenance context without converting one channel into a verdict.

A product threshold is selected from an explicit cost ratio between missed marks and false allegations, then tested again at realistic document lengths. Short passages often carry less statistical evidence, so the lab shows an indeterminate region rather than forcing every score into yes or no. Review capacity is part of the threshold because ambiguous cases need a staffed destination.

Connect detector review to provenance recovery. When LLM text watermarking produces a strong compatible signal, the next action is to seek corroborating provenance: generation records under appropriate authorization, Content Credentials, publication history, or source drafts. A reviewer can then distinguish a marked source later quoted by a human from a document wholly generated by the embedding system. The detector score remains attached to its exact analyzed bytes so later edits cannot silently inherit the earlier result.

Appeals need the same evidence package as the original decision, including detector version, threshold, document length, transformation caveats, and negative-calibration date. The product should allow a person to add context and should never imply that failing to disprove a score confirms authorship. LLM text watermarking supports a responsible workflow only when review can lower confidence, correct records, and feed discovered false positives back into calibration.

Publish ROC curves, edits, and detector versions

The artifact emits deterministic scores, but a release lab must publish threshold sweeps, confidence intervals, corpus composition, attack recipes, length buckets, and false-positive review outcomes. Every chart names detector and generator revisions. The same discipline from AI evaluation measurement contracts prevents a favorable benchmark slice from becoming a timeless product promise.

Publishing the experiment includes the seed, vocabulary partition hash, edit recipes, detector formula, corpus licenses, and raw confusion counts. It excludes private user text and avoids optimizing on the final evaluation set. A future detector version receives a fresh calibration split so improved results cannot come from learning the benchmark's accidents.

Set monitoring triggers before detector launch. Post-launch sampling compares score distributions, document lengths, languages, and reviewer reversals with the frozen calibration period. A rise in indeterminate cases may indicate new editing behavior rather than detector failure, while a subgroup false-positive increase demands immediate threshold review. LLM text watermarking is disabled for unsupported inputs instead of returning a cosmetically precise number. The owner publishes a monthly drift receipt and can roll back detector or copy independently, keeping a technical change from silently broadening a consequential product claim.

Use provenance signals without automated punishment

My supported position is that LLM text watermarking can assist provenance workflows when embedding coverage is known and limitations are visible. It should not trigger account action or authorship claims alone. A system earns trust by exposing uncertainty, preserving appeal evidence, and reopening thresholds after distribution drift, new editing tools, or detector changes.

Paraphrase robustness is therefore a measured survival curve, not a permanent property of a detector. AI text provenance remains the broader product question, and a watermark contributes one uncertain signal rather than a complete authorship record.

My release position is conservative: expose watermark detection as one provenance signal, pair it with content credentials and source records, and prohibit punitive automation from the score alone. Reopen that position when attacks, language coverage, or the embedding method changes. A useful system makes uncertainty actionable without laundering it into certainty.

LLM text watermarking can support provenance only while detector version, coverage, edit survival, and false positives remain visible. Treat LLM text watermarking as uncertain evidence and prohibit a score from becoming an automated authorship accusation.