A position paper questions how we measure faithfulness in XAI

Ten years of explainability research have not delivered on the promise the field was born with. That is the premise of Faithfulness Without Anchors, a position paper posted on arXiv on October 13 by Marco Zullich, Emily Schiller, Ivan Gentile, David Dembinsky, Adriano Lucieri, Kilian Göller and Steffen Seitz, and presented at the Explainable AI workshop at IJCAI-ECAI 2026.

The authors argue that XAI tools, presented as a way to increase trust in machine learning models, have ended up encouraging overreliance on incorrect predictions, which in turn can lead to algorithmic aversion. Their diagnosis is far from flattering: in practice, XAI works mainly as a debugging tool, useful for gaining anecdotal insights into which features a model has learned.

The faithfulness problem

The work focuses on one specific concept: faithfulness. An explanation is faithful when it genuinely approximates the behaviour of the model it claims to explain. When it oversimplifies, the mismatch the literature calls unfaithfulness or infidelity appears: the explanation tells a coherent story that does not match what the model does internally.

If faithfulness could be measured rigorously, we would have a way to quantify at least one component of trust in an explanation. The problem, according to the authors, is that existing metrics have limitations that make this measurement infeasible or computationally expensive. The paper lists five major shortcomings of current metrics and proposes four research directions to make them more reliable.

Why it is so hard to measure

The title offers a clue: there is no anchor. In most real models there is no ground truth about which features matter, so there is nothing to compare the explanation against. Common metrics rely on indirect approximations. One of the most widespread consists of removing or perturbing the features the method flags as important and observing how much the prediction drops.

That approach carries problems known for years. When pixels or variables are replaced with a baseline value, the model receives inputs unlike anything it saw during training, and the performance drop may be due to that distribution shift rather than the real importance of the feature. Proposals such as ROAR (RemOve And Retrain) tried to fix this by retraining the model after each removal, at the price of multiplying the cost. In addition, different faithfulness metrics applied to the same method can produce contradictory rankings.

This context is ours. The published abstract does not specify exactly what the five shortcomings or the four proposals are, so we recommend reading the full text before citing them.

What it has to do with LLMs

Although the work deals with feature importance methods, the family that includes techniques such as SHAP, LIME or saliency maps, the underlying question resurfaces in language models. In 2025 Anthropic published research showing that reasoning models do not always verbalize in their chain of thought the factors that actually influence their answer. It is another face of the same problem: a plausible explanation is not the same as a faithful one.

Anyone building agents with Claude runs into this often. Asking the model to justify a decision produces reasonable text, but that text is not a verifiable trace of what happened. For auditing behaviour, tool call logs and evaluations on known cases remain more reliable than self explanation.

Who it is for

For machine learning teams in regulated sectors, such as banking, insurance or healthcare, that use explainability dashboards to justify decisions to auditors or clients. If the metric validating those explanations is not sound, the report may convey a confidence it does not have.

Also for anyone evaluating vendors. Asking how the faithfulness of a tool's explanations has been measured, and with which metric, helps separate serious products from cosmetic layers. And for researchers in the field, the paper works as a map of open problems.

At ElephantPink we treat importance attributions as a debugging hint, not as proof. This work does not yet bring a better metric, but it does offer solid reasons not to present a SHAP chart as a guarantee of anything.