Study: EU-mandated AI watermarking weakens model safety refusals
Lasso Security finds Google's SynthID-Text watermarking, adopted by Anthropic for EU compliance, alters AI agents' tool use and makes refusals less robust under prompt injection.
What to know
- The EU AI Act's Article 50(2) requires AI providers to mark synthetic text output as machine-detectable; Anthropic adopted Google DeepMind's SynthID-Text for Claude to comply.
- Lasso Security found the watermarking process itself, by biasing next-token selection, alters AI agent tool-calling accuracy (worse on 6 of 7 tested models) and safety refusal behavior.
- The effect on refusals is small for plain harmful requests but becomes more pronounced under prompt-injection attacks, in some cases making models more likely to comply with harmful instructions.
- Because watermarking is applied at the model level via APIs, the behavioral change extends to any third-party agent built on a watermarked model, not just first-party chat interfaces.
“Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt-injection technique.”
Andrea Siposova, AI security researcher, Lasso Security · Ars Technica ↗ · Sep 16
Lasso Security AI security research firmAndrea Siposova AI security researcher at Lasso SecurityAnthropic AI company deploying watermarking in ClaudeGoogle DeepMind Creator of SynthID-Text watermarking method
How it unfolded 3 developments, newest first · click a bar or a number to jump articlesposts
-
3
Security-focused reader flags the finding as a serious flaw
A security-minded Bluesky user reacted to the Ars Technica report by describing the watermarking-induced weakness as both exploitable by attackers and a mechanism that could effectively push models toward self-jailbreaking.
“Holy shit, that's a pretty gaping security flaw.”
— kayleadfoot.bsky.social -
E
FAN-FUCKING-TASTIC. The "watermarking" algorithm used by Anthropic to tag Claude's writing as "written by Claude" makes AI more dangerous, not less.
1 more of the top 2 · 2 posts in this stretch
-
K
Holy shit, that's a pretty gaping security flaw. Both for human-led AI-generated exploits, but also the system looks engineered to be capable (encouraged?) to jailbreak itself.
-
-
2
Ars Technica reports watermarking can enable harmful compliance
Ars Technica's deeper report, based on an interview with Lasso researcher Andrea Siposova, detailed how watermarking's effect on refusal behavior becomes more pronounced under prompt injection, in some cases making watermarked models more likely to answer harmful requests they would otherwise refuse.
“As compared to the same models without watermarking, it is definitely going to change their behavior, especially when we place it under adversarial conditions, or we make these models call tools when they're powering an agent.”
— Andrea Siposova -
first by Ars Technica, 9d ago
-
A
AI text watermarking can make models more vulnerable to adversarial prompts SynthID can cause models to follow harmful instructions they would otherwise refuse. https:// arstechnica.com/security/2026/ 09/ai-text-watermarking-can-make-models-more-vulnerable-to-adversarial-prompts/?utm_brand=arstechnica&utm_social-type=owned&utm_source=mastodon&utm_m…
-
-
1
Lasso Security finds watermarking causes 'sampling drift' in agents
Lasso Security published "The Provenance Tax," testing SynthID-Text's non-distortionary configuration across seven open-weight models and finding it reduced tool-calling accuracy on six of them (using BFCL v4) and measurably changed refusal behavior, a phenomenon it calls sampling drift.
“Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.”
— Lasso Security -
first by The Register, 9d ago
-
-
background
Anthropic adopts SynthID-Text watermarking for future Claude models — Anthropic disclosed that future Claude models will embed an invisible, Google DeepMind-designed SynthID-Text watermark in their output, applied at the model level across the Claude Platform API and cloud providers, to comply with Article 50(2) of the EU AI Act requiring machine-readable marking of AI-generated text.