Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualizations
University of Tübingen
tl;dr: Using human psychophysical experiments, we show that natural images can be significantly more informative for interpreting neural network activations than synthetic feature visualizations.
News
| May '21 | Poster presentation at ICLR 2021. |
| January '21 | The paper was accepted for a poster presentation at ICLR 2021. |
| December '20 | A shorter workshop version of the paper was accepted for a poster presentation at the Shared Visual Representations in Human & Machine Intelligence Workshop at NeurIPS 2020. |
| October '20 | The pre-print is now available on arXiv. |
Abstract
Feature visualizations such as synthetic maximally activating images are a widely used explanation method to better understand the information processing of convolutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs' inner workings. Here, we measure how much extremely activating images help humans to predict CNN activations. Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. 2017 with a simple baseline visualization, namely exemplary natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activation. The experiment is designed to maximize participants' performance, and is the first to probe intermediate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations (82±4% accuracy; chance would be 50%). However, natural images - originally intended to be a baseline - outperform these synthetic images by a wide margin (92±2%). Additionally, participants are faster and more confident for natural images. The higher informativeness of natural images holds across most layers, for both expert and lay participants as well as for hand- and randomly-picked feature visualizations. Even if only a single reference image is given, synthetic images provide less information than natural images (65±5% vs. 73±4%). In summary, synthetic images from a popular feature visualization method are significantly less informative for assessing CNN activations than natural images. We argue that visualization methods should improve over this simple baseline.
Overview: How useful are synthetic compared to natural images for interpreting neural network activations? Given extremely activating reference images (either synthetic or natural), a human participant chooses which out of two query images is also a strongly activating image. Synthetic images were generated via feature visualization (Olah et al. (2017)).
Why
Feature visualizations such as synthetic maximally activating images are a widely used explanation method. They are used to better understand convolutional neural networks (CNNs) as they grant insights into their learned features. At the same time, there are concerns that these visualizations might not accurately represent CNNs’ inner workings. Here, we investigate how much extremely activating images help humans to predict CNN activations.
What we did
Using a well-controlled psychophysical paradigm, we measure the informativeness of synthetic feature visualizations by Olah et al. (2017). Here’s an example trial that participants did in our lab:
Based on minimally and maximally activating feature visualizations for a certain feature map shown at the sides, a participant is asked to select the image from the center that also strongly activates that feature map.
On the right hand side, we’re seeing maximally activating images; while on the left hand side, we’re seeing minimally activating images. The question always concerns the two images at the center of the screen: Which of them is also a strongly activating image? When breaking this down, all we’re really asking is which image at the center is more similar to those at the right side.[1]
As you can imagine, this task will give us some performance value. In order to set this into context besides the chance level of 50% (if you were to randomly guess at the above task, you would on average get 50% correct), we tested another condition that we intended as a baseline: natural images. This time, instead of showing extremely activating synthetic images at the sides, we displayed natural images that elicit either very low or very high activations. The task regarding the images at the center of the screen still remains the same: Which of the two is also a strongly activating image?
Example trial in psychophysical experiments to measure the baseline performance of natural images. What we found
First of all, we found that synthetic feature visualizations are indeed helpful: In 82±4% of the trials (that is many more than the chance proportion of 50%), participants chose the truly strongly activating image (blue bar). Surprisingly, though, participants answered the trials even more often correctly when they were given natural reference images at the sides, namely in 92±2% of the trials.
To see the results of many more conditions, such as performance of lay and expert participants, a comparison between hand-selected and randomly sampled feature maps, check out our paper.
What we take from this
Most importantly, our data show that feature visualizations are indeed helpful for humans - and that natural images can be even more helpful. On a higher level, we take from this that thorough human quantitative evaluations of feature visualizations are needed and that example natural images might provide a surprisingly challenging baseline for understanding CNN activations.
We’re continuing our work and investigating other aspects of the helpfulness of feature visualizations. Stay tuned!
Acknowledgements & Funding
We thank Felix A. Wichmann and Isabel Valera for helpful discussions. We further thank Alexander Böttcher and Stefan Sietzen for support as well as helfpul discussions on technical details. Additionally, we thank Chris Olah for clarifications via slack. Moreover, we thank Leon Sixt for valuable feedback on the introduction and related work.
From our lab, we thank Matthias Kümmerer, Matthias Tangemann, Evgenia Rusak and Ori Press for helping in piloting our experiments, as well as feedback from Evgenia Rusak, Claudio Michaelis, Dylan Paiton and Matthias Kümmerer. And finally, we thank all our participants for taking part in our experiments.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting JB, RZ and RG.
We acknowledge support from the German Federal Ministry of Education and Research (BMBF)
through the Competence Center for Machine
Learning (TUE.AI, FKZ 01IS18039A) and the Bernstein Computational
Neuroscience Program Tübingen (FKZ: 01GQ1002), the Cluster of Excellence Machine Learning: New Perspectives for Sciences (EXC2064/1), and the German Research Foundation (DFG; SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP3, project number 276693517).
BibTeX
Please cite our paper as follows:
author = {
Borowski, Judy and
Zimmermann, Roland S. and
Schepers, Judith and
Geirhos, Robert and
Wallis, Thomas S. A., and
Bethge, Matthias and
Brendel, Wieland
},
title = {
Exemplary Natural Images Explain
CNN Activations Better than
State-of-the-Art Feature Visualization
},
booktitle = {Ninth International Conference on
Learning Representations (ICLR 2021)},
year = {2021},
}