Taxonomy of Explainable AI (XAI) for Computer Vision
Compare global vs local, gradient vs permutation, and post-hoc vs interpretability-by-design methods

Global vs local explainable AI methods

The first way we can classify XAI methods is by the type of insights they offer. Global interpretations aim to explain overall model behaviour. We want to make general claims about model predictions on different segments of instances or the entire dataset. In comparison, local interpretations aim to explain the predictions for individual instances.

In the context of computer vision, most XAI methods will produce local interpretations. They output a saliency map which shows the pixels in the input image that have contributed the most to the prediction. This is why this book largely revolves around the different mechanisms and properties of various saliency maps. Some of which you can see in Figure 2. These are also commonly referred to as attributions. They attribute a prediction to the features (i.e. pixels) of an image.

To obtain global interpretations, a typical approach is to aggregate saliency maps. For example, we have Grad-CAM heatmaps from a model used to classify X-rays of brain tumours. We could sum the values in different hemispheres or lobes of the brain. Doing this over multiple input images, we can understand which region is more important in general or if the model is biased towards a certain region. Other methods, like weighted context, aggregate the saliency maps of individual pixels in image segmentation models [1]. This is used to quantify how much context is used to classify individual pixels.

Permutation-based vs gradient-based explainable AI methods

The next way we can group XAI methods is by their relationship to the underlying model. Or, in other words, by how they are calculated. The two broad categories are using permutations or looking at the model's gradients.

Permutation-based methods work by systematically or randomly altering pixels in the input and measuring changes in the output. The exact way we do permutations can differ. With permutation channel importance, we will shuffle all pixels in a single channel. With occlusion, we will create saliency maps by systematically replacing regions of an image with neutral values. With SHAP, we will follow a similar process, but by permuting various combinations of regions called superpixels.

Gradient-based methods use the model's gradients acquired by backpropagation. Gradients can be any derivative (or rate of change) of one value w.r.t. another. For most saliency maps, we use the gradients of an output logit w.r.t. the pixels in the input image \(( \frac{\partial y_c}{\partial X} )\). Typically, large absolute gradients indicate that a particular pixel is important to the output. However, as we will see when exploring vanilla gradients, this can produce noisy visualisations.

The remaining methods in the gradient-based section reduce this noise through different mechanisms. For example, Grad-CAM weights the activations in feature maps of a convolutional layer by the average gradients of the map w.r.t. the output. Guided Backpropagation uses ReLU masking to suppress negative gradients, and Integrated Gradients takes an average of gradients computed along a straight-line path from a baseline input to the actual input.

In general, the main benefit of gradient-based methods is that they are significantly less computationally expensive than permutation-based methods. With methods like occlusion, we do multiple (potentially thousands) of forward passes to create saliency maps. For many gradient-based methods, we do only one. On the other hand, permutation-based methods can provide more accurate and less noisy interpretations. As we will discuss in the next section, they can also be used with a larger variety of networks.

Interpretability by design vs post-hoc explainable AI methods

The last way we can group methods is by the timing they are applied. Post-hoc methods are applied after a model is trained. In comparison, interpretability-by-design (IBD) approaches integrate interpretability mechanisms directly into the model’s architecture or decision-making process. In other words, these methods are applied during training.

An example of an IBD method is prototype layers [2]. These work by comparing input features to a learned set of prototypical representations—small, interpretable image patches. The model makes predictions based on the similarity between the input and these prototypes. This makes them easier to understand, as any predictions can be traced back to these human-understandable prototypes.

The downside of IBD methods is that we need to alter the model's architecture. For prototype layers, we need to add new dedicated layers after the convolutional layers. You also need a custom loss function that encourages prototype activation and the separation of prototype layers of different classes. Ultimately, this means that IBD methods are far less flexible and less readily adopted. In comparison, post-hoc methods can be applied to many types of well-established architectures. We explore this trade-off in more detail when comparing Class Activation Maps (CAMs) to Grad-CAM.

Model agnostic methods

We can further distinguish post-hoc methods by whether they are model agnostic. These are methods that can be applied to any architecture as they do not require that we look into their inner workings. So, provided the input and output are the same, we could apply the same method to a feed-forward neural network, CNN or even a transformer.

Permutation-based methods are typically model agnostic. We permute the input image and observe changes in the model's output. At no point do we look at what is happening in the model—it is treated as a black box. Although highly flexible, most gradient-based post hoc methods are not model agnostic. In computer vision, they are typically designed for CNNs only.

Really, when deciding between IBD and post-hoc methods, you should consider your goal. If you want to debug a model before putting it into production, then post-hoc methods are the way to go. They will provide more flexibility when it comes to the architecture, ultimately allowing you to select the one with the best performance.

If you are using deep learning for data exploration, then you can consider IBD methods. In this case, your goal is to learn something new about your dataset or problem and IBD can often provide a unique insight. The lower performance is not necessarily a problem, as you just need a model that captures underlying relationships in the data.

Additional considerations

That covers the main differences in XAI methods. To end, we'll discuss a few other considerations when choosing a method.

Granularity of the output

By granularity, we mean how detailed a saliency map is. In Figure 3, you can see that methods like Guided Backpropagation can show detailed features. In comparison, Grad-CAM can only highlight important regions in an image. This is why saliency maps like those generated by Grad-CAM are referred to as heatmaps. The granularity ultimately determines the kind of conclusion you can make from a method.

Sensitivity-based vs contribution-based

As with many terms in machine learning and XAI, there are often conflicting definitions. [3] provides the following definitions:

• Local attribution methods (sensitivity-based): describe how the output of the network changes for infinitesimally small perturbations around the original input.
• Global attribution methods (contribution-based): describe the marginal effect of a feature on the output with respect to a baseline.

These are given in the context of explanations for individual predictions generated using gradient-based methods. This is not the same as the global/local definitions we gave above. The confusion comes as this definition comes from a mathematical context, and is meant to define a model/function in calculus. So to avoid confusion with the previous definition, we refer to these as sensitivity-based vs contribution-based methods.

We talk a bit more about this definition when applying Input x Gradients and the Integrated Gradients method. To summarise, the researchers argue that looking at gradients or an average of gradients alone provides a local interpretation. This is because we can only understand the sensitivity of a function around a given input. When we multiply by the input (\(X\)) or by the difference between the input and baseline (\(X - X^0\)), we obtain a global attribution. This is because we scale the gradients and obtain a cumulative effect.

Axiomatic foundations of explainable AI methods

When we look at the more academic side of XAI, you will encounter different axioms or mathematical properties. We can derive methods from the axioms or prove that a method satisfies the axioms. They include properties like completeness, dummy, symmetry and linearity.

• Completeness: The total attribution across all features should equal the difference between the model output and the baseline prediction.
• Dummy: Features that are missing or do not contribute to a prediction have zero attribution.
• Symmetry: If two features contribute equally to all predictions, they should receive equal attribution.

Axiom-based methods will be derived from a set of axioms or be shown to satisfy a set of axioms. In comparison, heuristic-based methods will produce saliency maps using a simple rule or heuristic. Although the rule usually has a logical justification for why it will produce reasonable saliency maps, the resulting attribution will not necessarily satisfy any axioms. Ultimately, depending on your application, you may need to consider whether a method satisfies these axioms to make more reliable conclusions.

As you can see, there are a lot of considerations when choosing an XAI method. These include the type of insight they offer (global vs local), how they interact with the model (permutation-based vs gradient-based), and when they are applied (post-hoc vs interpretability-by-design). We also discussed model-agnostic methods, the granularity of saliency maps, and the importance of axiomatic foundations.

By understanding these categories, you can better choose the right method for your goals— whether that’s debugging a production model or exploring data-driven insights. As best practice, it’s often useful to apply multiple XAI methods to gain a more robust understanding of your model’s behaviour. Their complementary strengths and weaknesses can often lead to a more complete view of your model.

Challenge

Read about a method called baseline SHAP (BShap) [4]. Classify this method into the three main categories and give the number of axioms it satisfies.