Standalone LLM or agent pipeline to explain ICU mortality

On 2,353 ICU stays with 8.1% mortality, an XGBoost model reaches an AUROC of 0.855 (95% CI: 0.796 to 0.906) and an AUPRC of 0.332 (0.217 to 0.494). The model predicts reasonably well. What it does not do is tell anything usable to the person standing next to the patient, and that gap is what a feasibility study published on arXiv on 28 August tries to cover, comparing two ways of producing the clinical explanation: an LLM working alone and a four step agentic pipeline with the tasks separated.

The AUPRC is worth a second of attention. With a prevalence of 8.1%, a classifier ranking at random would sit around 0.081, so 0.332 is roughly four times the baseline. It is not a brilliant figure read cold, it is what you usually get when the event is rare.

What the pipeline separates

The pipeline is pre specified, not improvised by the model, and splits the task into phases: data interpretation, checking against clinical guidelines, and final write up. The authors' hypothesis is that separating responsibilities reduces the model's opportunities to invent the reasoning along the way. The alternative, the standalone LLM, receives the case together with the attribution and writes the explanation in one pass.

The data comes from the eICU Demo, a small public subset of the eICU Collaborative Research Database, which makes the experiment easy to reproduce but limits any conclusion about generalising to other units.

How the strengths split

On a stratified subset of 38 cases, the standalone LLM produced one explanation with explicit outcome leakage: it mentioned the actual outcome of the stay, information that by definition should not be available when explaining a prediction. The four step pipeline produced none.

On the 14 cases that overlapped with the SHAP review, the split partly reverses:

The standalone LLM was better aligned with SHAP attributions (mean Jaccard of 0.171 versus 0.077) and more consistent in the direction of the effect (92.9% versus 78.6%).
The agentic pipeline was far better grounded in clinical guidelines (0.762 versus 0.143).

How to read it without overreaching

38 cases and 14 overlapping ones are very few. The authors themselves frame the work as a feasibility study, and it is a revised version of an earlier text, so the differences point in a direction rather than settling it. What matters is not who wins, because nobody wins outright: these are two distinct error profiles. The direct LLM stays closer to what the model actually weighed; the pipeline says fewer things it cannot back up.

What agent builders take away

The trade off is fairly sharp. By inserting intermediate steps, the explanation drifts away from the model's real attribution and gains external discipline. If the acceptance criterion is «do not claim anything you cannot ground in a guideline», the pipeline is the choice; if the criterion is «describe this model faithfully», the direct LLM was closer. Picking the metric before the architecture saves plenty of arguments later.

In practice, nothing stops you from combining them: an attribution step that queries SHAP as a tool, a guideline verification step and a write up step, each evaluated separately. In Claude Code that translates into subagents with an explicit contract per phase and the statistical computation behind an MCP server, instead of one long prompt asked to do everything at once. The advantage is not elegance, it is that you can measure which step breaks. The public abstract fragment does not go into cost or latency either, and between four calls and one that difference weighs at deployment time.

The lesson we take away is cheap but uncomfortable: the metric you use to define «a good explanation» decides which architecture wins before a single line of code is written. And we would like to see the outcome leakage rate reported in every comparison of this kind, not only when the number comes out favourable.