When an LLM Doesn't Know It Doesn't Know: Epistemic Blind Spots in Clinical Data

An LLM can tell you with apparent certainty that its prediction is correct when it fails in nearly half of cases. That's not a minor nuance: it's a fundamental problem that a new study published this week on arXiv exposes with concrete data from tabular clinical tasks.

The research, available at arXiv:2606.19509, compares the behavior of Qwen 2.5 7B with the classical XGBoost classifier on a prediction task using structured clinical data. The mechanism they use to detect so-called epistemic blind spots, areas where the model doesn't know what it doesn't know, is the analysis of attribution divergence between both models, which the authors call cross-model attribution divergence.

Verbalized Confidence: An Almost Useless Number

The first finding is the most striking and, probably, the most practical for those integrating LLMs into real-world workflows. The confidence the LLM expresses verbally, that percentage the model returns when asked how certain it is, ranges between 0.856 and 0.937 regardless of whether its actual prediction accuracy is 49% or 75.3%. The model doesn't calibrate its certainty downward when it makes more mistakes; it simply adjusts the number to fit the prompt format.

This has direct consequences in clinical settings, where a system that says it is "90% confident" in cases it solves worse than random chance could generate unjustified trust in the clinician or the system orchestrating it.

The Inverse Difficulty Effect

The second finding is counterintuitive: the LLM performs worse precisely on cases that XGBoost solves with the highest certainty. When XGBoost reaches 99% accuracy, that is, on the "easiest" examples according to the classical classifier, the LLM drops to 64.8%. However, in cases where XGBoost is moderately uncertain, both models tie (73.8% versus 73.1%). This suggests that the LLM and XGBoost not only make different errors, but their strengths are to some extent inverted.

For an engineering team working with medical analysis agents, this means that naively combining both models, without accounting for this divergence, can worsen the result compared to using just one.

Few-shot and SHAP Evidence: Interventions That Amplify Each Other

The most constructive part of the study is the third finding. The authors verify that adding few-shot examples and providing the LLM with variable importance scores derived from SHAP (a standard machine learning model interpretability technique) are orthogonal and superadditive interventions: applied together, they reduce the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and raise the LLM's accuracy from 49% to 75.3%, with no retraining required.

This is significant. It means that with well-constructed contextual information—relevant examples plus evidence of which variables matter in the specific case—a 7B-parameter model can recover competitive performance on a task where, without that scaffolding, it failed systematically.

A Cross-Model Calibrator as a Pragmatic Solution

The fourth finding closes the loop: the authors propose a cross-model calibrator that uses the attribution divergence between the LLM and XGBoost as a signal to decide when to trust the LLM. Instead of asking the model if it's confident, which as we've seen is nearly meaningless, the system compares how each model weights the variables and flags those cases where divergence is high as unreliable.

It's a pragmatic approach that requires neither access to model weights nor fine-tuning techniques, making it directly applicable to integrations using the Claude API or any other general-purpose LLM.

Who This Matters For

The research is primarily relevant to teams deploying LLMs on structured data in high-stakes contexts: healthcare, credit, insurance. But its implications extend also to those building agents with data analysis tools, where model confidence is used—sometimes implicitly—to decide whether to escalate a decision to a human or proceed autonomously.

What strikes us most about this work is not the finding itself, poor LLM calibration is well known, but rather the methodological proposal of using divergence between an LLM and a classical model as a reliability signal. It's a way of leveraging what traditional machine learning models do well, rather than discarding them, and that aligns well with the reality of the hybrid systems we see in production today.