The Explainability Illusion:
Must ViTs Exhibit Patch-Decoding Interpretability?

Ronen Schaffer*

*Equal contribution

Faculty of Electrical and Computer Engineering
Technion – Israel Institute of Technology

ArXiv

Code

TL;DR: We show the widespread presumption of patch-decoding explainability is not an inherent property of Vision Transformers, but an outcome that cannot be assumed to hold a-priori.

Abstract

Transformers are widely assumed to exhibit token-decoding explainability. In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human-interpretable outputs. In light of the increasing popularity of this approach, an important question is whether token-decoding explainability an inherent property of transformers that should be expected to hold across architectures, scales, and training objectives, or merely an artifact of some implicit bias? In this paper we systematically study token-decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models. We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics. As we show, these tendencies toward different explainability behaviors are not intrinsic to the architecture and can vary arbitrarily with changes in the optimization landscape. We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns. Our results show that token-decoding explainability is not an inherent property of trained transformers, but rather an outcome contingent on the problem setting and thus should not be a-priori assumed to hold.

Token decoding is assumed to reveal what the model "sees" at each location.

One of the most common ways to interpret the behavior of vision transformers (ViTs) is through Patch-token decoding, where the model's classification head is applied to each patch token, and the resulting scores are visualized as a heatmap over the input image.

It is usually assumed that these heatmaps indicate what the model "sees" at each location in the image. Specifically, if a patch corresponding to an elephant has a high score for the "elephant" class, it is interpreted as the model recognizing the elephant in that patch.

ViT-B/16

ViT-B/16: Golden Retriever — Golden Retriever

DINOv2-B/14

DINOv2-B/14: Golden Retriever — Golden Retriever

Aligned-explainability bias of real-world ViT classifiers. Explainability maps of six image inputs, on ViT-B/16 and DINOv2. All classifiers exhibit an aligned-explainability bias. Warm and cool colors indicate higher or lower token-decoding scores.

Defining explainability

To go beyond a few convincing examples, we first must be able to measure the token-decoding explainability of an image. Using an object's mask, we average the decoded scores over object patches (S_obj) and background patches (S_bg).

An image exhibits aligned-explainability if S_obj > S_bg, inverted-explainability if S_obj < S_bg, and unexplainability if S_obj ≈ S_bg.

An elephant image beside its patch-decoding score map, illustrating how object and background scores are measured. — **Explainability regimes.** For an input image, token-decoding of patches produces explainability maps, where each patch-token receives some score S. The relation between the average score of object tokens, S_obj (tokens inside the black contour), and the average score of background tokens, S_bg (tokens outside the black contour), determine the explainability of a classified input image in a layer: aligned, unexplainable, or inverted. Here, the elephant is aligned and the cat is inverted.

Now, we can just plot the histogram of image scores on a 2D grid, to visualize the distribution of explainability behaviour of a model.

Can we count on explainability across images and models?

Indeed, ViT-based classifiers are usually explainaible:

Aligned-explainability bias across real-world ViT classifiers. 2D histograms of S_obj and S_bg scores for 14 different classifiers. ViT classifiers exhibit a strong bias toward aligned-explainability. More examples can be found in the paper.

However, not all.

Unexplainable and inverted-explainability of real-world ViT classifiers. 2D histograms of S_obj and S_bg scores for 6 different classifiers. Aligned-explainability is not universal among ViT classifiers.

What about contrastive vision-language models?

In contrastive vision-language models like CLIP, the model learns to associate images with their corresponding text descriptions. These models compare an image embedding with a text embedding. To decode an image patch, we apply the image encoder’s projection head and measure cosine similarity to the text embedding of “a photo of a C”, where C is the object’s class.

Contrastive VLMs exhibit a strong inverted-explainability bias, where background tokens score higher than object tokens.

CLIP-L/14

CLIP-L/14: Golden Retriever — Golden Retriever

MetaCLIP-B/32

MetaCLIP-B/32: Golden Retriever — Golden Retriever

Inverted-explainability bias of real-world ViT-based contrastive VLMs. Explainability maps of six image inputs, on CLIP-L/14 and MetaCLIP-B/32. Warm and cool colors indicate higher or lower patch-to-text similarity. VLMs exhibit a strong inverted-explainability bias.

OpenCLIP-B/16 (80M-3B) — OpenCLIP-B/16
(80M-3B)

OpenCLIP-L/14 (80M-3B) — OpenCLIP-L/14
(80M-3B)

OpenCLIP-B/16 (400M-3B) — OpenCLIP-B/16
(400M-3B)

OpenCLIP-B/32 (2B-3B) — OpenCLIP-B/32
(2B-3B)

OpenCLIP-B/16 (80M-34B) — OpenCLIP-B/16
(80M-34B)

OpenCLIP-B/32 (2B-34B) — OpenCLIP-B/32
(2B-34B)

OpenCLIP-L/14 (400M-3B) — OpenCLIP-L/14
(400M-3B)

OpenCLIP-B/16 (2B-3B) — OpenCLIP-B/16
(2B-3B)

OpenCLIP-B/16 (2B-13B) — OpenCLIP-B/16
(2B-13B)

OpenCLIP-L/14 (2B-3B) — OpenCLIP-L/14
(2B-3B)

OpenCLIP-L/14 (2B-34B) — OpenCLIP-L/14
(2B-34B)

OpenCLIP-B/16 (2B-34B) — OpenCLIP-B/16
(2B-34B)

OpenCLIP-g/14 (2B-13B) — OpenCLIP-g/14
(2B-13B)

Inverted-explainability bias across real-world contrastive models. 2D histograms of S_obj and S_bg scores for 21 different contrastive VLMs. All models exhibit an inverted-explainability bias. The scores tend to lie above the diagonal, where background patches score higher than object patches. OpenCLIP's partentheses indicate dataset size and number of samples seen. More examples can be found in the paper.

Should we expect any constant behvaiour at all?

To test the nature of these biases, we perform controlled experiments by training many ViT-based classifiers and contrastive models on synthetic data with varying parameters.

We find that for both classifiers and contrastive VLMs, the training can converge to either aligned or inverted explainability. This demonstrates that beyond the loss, the data distribution also affects the explainability of ViTs.

However, the effect is not deterministic. We show that for a fixed task and data distribution, different training seeds can converge to different explainability regimes. This shows that the training loss and the data distribution do not impose a strict regime, but rather promote different explainability biases.

Observations supported by theory

To support our empirical observation that the loss and data distribution do not necessarily determine the explainability regime of a model, we analyze a classification problem simple enough to solve analytically.

We derive a closed form expression for the optimal classifier, identify a simple attention architecture that precisely implements it, and show that by adjusting its weights, its token-decoding map can be made aligned, unexplainable, or inverted. Importantly, all three versions produce identical logits, making them indistinguishable under any logit-based loss.

Paper

The Explainability Illusion:
Must ViTs Exhibit Patch-Decoding Interpretability?
Ronen Schaffer*, Hila Manor*, Tomer Michaeli

ArXiv

Our official code implementation can be found in the official github repository.

BibTeX

@article{schaffer2026explainability,
    title={The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?},
    author={Schaffer, Ronen and Manor, Hila and Michaeli, Tomer},
    note={Preprint},
    year={2026}
}

Acknowledgements

This webpage was originally made by Matan Kleiner with the help of Hila Manor.
The code for the original template can be found here.
Icons are taken from font awesome or from Academicons.