LLM watermarks degrade the quality of medical texts, study finds

Five watermarking schemes, eleven LLMs and seven vision language models evaluated on clinical reasoning tasks. That is the scope of "Marking the Wrong Symptoms", a study published this July 24 on arXiv presenting the first rigorous evaluation of how LLM watermarks affect performance on medical texts. The result is uncomfortable: watermarking text can degrade it in ways that standard metrics fail to detect.

Watermarking works by subtly biasing token selection during generation so that a statistical test can later verify whether a text came from a specific model. It is one of the favorite techniques among regulators and providers for making AI-generated content traceable, and on general-purpose benchmarks its quality cost seems acceptable. The problem, the authors point out, is that in medicine a minimal token-level perturbation can completely change the meaning.

What they found

The study documents several concrete failure modes: lexical corruption, hallucinated terminology and, in multimodal tasks, amplified misattribution or omission of findings in medical images. These are not cosmetic degradations: they affect terminological precision and the quality of clinical reasoning, exactly what cannot fail in a healthcare setting.

The evaluation covers unimodal and multimodal clinical reasoning tasks, from medical exam questions to image interpretation, and compares how each watermarking scheme behaves across every combination of model and task. That results matrix shows the degradation is not uniform: it depends on the chosen scheme and the type of task, with multimodal cases among the most fragile.

To measure it, the authors introduce a human-expert-validated audit pipeline that systematically evaluates three fronts: medical reasoning quality, terminological precision and watermark-induced hallucinations. This is a relevant contribution in its own right, because existing evaluations rely on aggregate metrics that, as the paper shows, miss specifically clinical failures.

Why it matters

LLMs are entering real clinical workflows: discharge notes, history summaries, radiology report drafts. At the same time, regulatory frameworks such as the European AI Act push toward traceability of AI-generated content, and watermarking is the most cited technical mechanism to achieve it. This work puts numbers on a tension that until now was theoretical: the technique that provides traceability may be subtracting clinical safety.

The lesson generalizes beyond medicine. Any domain where small surface changes alter meaning (legal, financial, pharmacovigilance) should evaluate watermarks with domain-specific metrics before enabling them, rather than trusting general-purpose benchmarks.

Who it is useful for

For health tech teams integrating LLMs into clinical products, the paper is required reading before accepting watermarking as a compliance requirement without measuring its cost. For watermarking researchers, it sets a more demanding evaluation standard. And for compliance officers, it provides the technical argument that traceability and quality are not independent variables.

The important nuance is that the study does not conclude watermarking should be discarded, but rather that its cost varies by scheme, model and task, and that it must be measured with domain-sensitive tools before deployment.

Our reading is that this kind of domain-specific audit should be the norm, not the exception, before imposing any traceability mechanism on high-risk systems. A watermark that confuses symptoms is a bigger problem than the one it is trying to solve.