Textbooks or patient records: which data shapes a medical LLM best

A model that answers more medical exam questions correctly does not necessarily reason better when facing a patient. That is the most uncomfortable conclusion of a paper published on arXiv on 23 September, which studies how the type of training data shapes the capabilities of a medical LLM.

The authors start from a common practice: medical models are trained on mixtures of didactic data, such as textbooks, and clinical data, such as patient records. What was unclear is what each type contributes on its own. To find out, they design token-matched experiments, so the only thing that changes between versions is the ratio between didactic and clinical material.

What they found

The main result is an asymmetric transfer across task types. According to the paper's abstract, clinical data improves clinic-oriented tasks and, at the same time, remains competitive on knowledge-intensive tasks. Didactic data, by contrast, mainly improves knowledge tasks, but does not carry that advantage into the clinical domain as effectively.

In other words, training on patient records teaches the model something that also helps with theoretical questions, while training on textbooks does not teach it equally well to work with real cases.

The error analysis points to what the authors call a knowing-doing gap: improvements in knowledge recall do not reliably generalise to clinical reasoning. The model may recall the right fact and still fail to apply it properly when a case is in front of it.

Small amount, large effect

The second finding has direct practical implications. Modest amounts of clinical data are enough to obtain most of the gains on tasks grounded in electronic health records (EHR). A huge clinical corpus is not needed for the model to improve at that kind of work.

Furthermore, the optimal ratio between both data types is not fixed. It depends on how much knowledge and how much clinical reasoning the downstream task demands. That is why the authors conclude that data curation for medical LLMs should be application-driven, with a higher share of clinical data when the goal is clinical use.

Why it matters

The work touches a problem that goes beyond medicine. Many health benchmarks are built from exam-style questions, which reward declarative knowledge. If that knowledge does not translate into reasoning about cases, a model can score very well on leaderboards and perform worse than expected in a hospital.

There is also a cost angle. Clinical data is expensive to obtain, anonymise and manage for legal reasons. The fact that a moderate amount delivers most of the benefit on EHR tasks changes the calculation for anyone considering training or fine-tuning their own model: it may make more sense to invest in a small, well-curated clinical set than to pile up more medical literature.

Who it is useful for

Teams fine-tuning models for healthcare: the study offers a criterion for deciding the data mix based on the use case, instead of a single recipe.

Those evaluating models for hospitals or insurers: it is wise not to rely only on knowledge benchmarks and to add tests with real or simulated clinical cases.

Developers building applications on general-purpose models: even if they train nothing, the knowing-doing gap suggests that the context passed to the model, for example patient record excerpts retrieved through an MCP server, may matter more than adding theoretical documentation to the prompt.

It should be read with caution: it is a freshly published preprint, not yet peer reviewed, and the conclusions depend on the specific tasks and models the authors used.

At ElephantPink we see the same pattern outside medicine: in agent projects, the client's real examples teach more than any internal manual. This study puts numbers to that intuition in a sector where mistakes are especially costly.