OpenAI's textGrain Watermark Weakens on Short or Edited Text, Raising EU Compliance Doubts

The question matters for every company that generates text for European users. Article 50 of the EU AI Act, whose transparency obligations took effect on 2 August, requires providers of systems that produce synthetic audio, images, video or text to mark outputs in a machine-readable way so they can be detected as artificially generated. The law says technical solutions must be "effective, interoperable, robust and reliable as far as this is technically feasible", but it does not set numerical benchmarks. Thomas Souverain, a postdoctoral fellow in AI ethics at the Institute of Advanced Education in Paris, told IEEE Spectrum the legislation is open to interpretation, and said OpenAI could face a fine of up to 7 percent of annual income if textGrain is ultimately judged not to comply.

OpenAI introduced textGrain in October. It is rolling out to eligible ChatGPT users on all plans in the EU and to eligible Codex subscriptions there, while API customers worldwide can opt in for selected models; watermarking stays off by default in the API. Access to the detector is initially limited to approved researchers and expert organisations.

How the Signal Works

Like other text watermarks, textGrain uses a secret key to generate pseudorandom values that nudge which token a model picks next, leaving a statistical pattern that a detector holding the same key can find. The pattern is invisible to readers. Xiang Li, a co-author of the technical report and a postdoctoral researcher at the University of Pennsylvania, told IEEE Spectrum that what sets textGrain apart is that it treats entropy, a measure of how unpredictable a model's word choices are, as a budget developers can set. "We can think of entropy as a resource, a kind of money," Li said, adding that the approach lets developers decide precisely how much of it to spend on the watermark.

Where Detection Breaks Down

OpenAI designed the detector to wrongly flag about one in 100 unwatermarked passages. In tests on questions drawn from the ELI5 dataset, it found the watermark in about 80 percent of 200-token passages, roughly 150 words, and about 95 percent of 400-token passages, roughly 300 words. Li described weaker performance on short text as a "basic limitation" of the technique, because short passages carry less statistical evidence.

Editing does more damage. In OpenAI's evaluation of 400-token passages, replacing 10 percent of words with synonyms cut detection from 92 percent to 66 percent, and replacing up to 25 percent cut it to 17 percent. Because each watermarked choice depends on the tokens before it, changing one word can disrupt the signal at several positions, Li said.

OpenAI has acknowledged the limits. It says watermarking alone is not enough to determine whether content is AI-generated, that a missing watermark does not prove a human wrote the text, and that textGrain is meant to complement other provenance tools such as content credentials in metadata and Google DeepMind's SynthID marks for images and audio. The EU's voluntary Code of Practice, which providers can use to show compliance, does not require watermarks on outputs shorter than 200 tokens.

OpenAI says textGrain matched or outperformed Google's SynthID Text in its own tests, but those results have not been independently verified. "We only have a technical study … there hasn't been a peer-reviewed academic article," Souverain said, adding that researchers do not yet have a precise protocol for the system. He has urged OpenAI to open-source textGrain so outside researchers can try to break it; the company has said it intends to release the technology as open source eventually, without giving a date.

OpenAI is working with a small group of outside researchers, including John Thickstun, an assistant professor of computer science at Cornell University, and a team at the Kempelen Institute of Intelligent Technologies in Bratislava. Both said they had only just gained access and had not begun testing. Dominik Macko, a senior researcher at the Kempelen Institute, said his team plans to examine textGrain's robustness across subject areas, languages and text lengths, results that are likely to shape how regulators judge whether text watermarking can meet the AI Act at all.