Detecting Shortcuts using Mutual Information

Thank you for the detailed review and insightful feedback. Here is the further explanation/answers to questions raised in the review:

Are there any datasets satisfying $I(X_{test}; Z) \geq I(X_{tr}; Z)$? There lacks empirical evidences in this paper to answer the question and I do not think the algorithm 1 could effectively detect shortcuts.

While $I(X_{test}; Z)$ cannot be greater than $I(X_{tr}; Z)$, however, for $X_{test}$ sampled from same distribution as $X_{tr}$, $I(X_{test}; Z) \approx I(X_{tr}; Z)$, as shown in the different experiments.

Unlike the proposed algorithm 1, the experiments in this paper, on the other hand, mainly compare networks trained on two datasets "with" and "without" shortcuts. This approach also has problems since it requires comparing with a network trained on dataset that is "without" shortcut. By defining "without shortcuts", also involves domain knowledge and human expertise to detect shortcuts.

We use `with' and 'without shortcut' to denote shortcuts in the controlled experiments to demonstrate that mutual information can be used to distinguish between the two datasets. We demonstrated the practical use case in Section 5 where we sample $X_{tr}$ and $X_{test}$ from two different medical imaging datasets. Our method can detect the presence of shortcuts, which is in fact corroborated when the model train on $X_{tr}$ is evaluated on $X_{test}$.

For those datasets without shortcut, are they satisfy $I(X_{test}; Z) = I(X_{tr}; Z)$? Please provide empirical evidences to show that algorithm 1 is feasible.

$I(X_{test}; Z) and I(X_{tr}; Z)$ will similar when sampled from the same distribution. In the experiments reported, MI $I(X_{tr}; Z)$ is calculated on different samples from the training distribution.

I notice that at the begining of the training, before the mutual information starts to be different between "with shortcut" and "without shortcut", the test loss has become different (e.g. 100 epoch in Fig.6 and 1000 epoch in Fig.7). Why is it?

Our intuition in the initial phase network explores the more optimal region of the solution space in the initial training epochs before discovering shortcuts. This is consistent with the findings of Shwartz-Ziv & Tishby (2017) on the behaviour of SGD.

[1] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. 2017

Thanks for the detailed feedback! We will run additional experiments on OOD for future submission.

Thank you for the detailed review and feedback!

The proposed method is limited in the real-world application scenario. The proposed method requires a 'without shortcuts dataset' to detect whether there are shortcuts in the training dataset. I am not sure how many cases we can prepare the 'without shortcuts dataset' before we know whether the training dataset has a shortcut.

We use `with' and 'without shortcut' to denote shortcuts in the controlled experiments to demonstrate that mutual information can be used to distinguish between the two datasets. We demonstrated the practical use case in Section 5 where we sample $X_{tr}$ and $X_{test}$ from two different medical imaging datasets. Our method can detect the presence of shortcuts, which is in fact corroborated when the model trained on $X_{tr}$ is evaluated on $X_{test}$.

(Kirichenko et al., 2022) represented that the model trained on Waterbirds using ERM has the ability to classify the foreground-only and background-only datasets. It conflicts with the main hypothesis that the model trained with the shortcut dataset will have a low $I(X;Z).

Thanks for pointing out this work. One major difference is that we use Neural tangent kernels for training the model and calculating the mutual information value. Kirichenko et al., 2022 used a finite-width neural network in their study.

The experiment graphs show that the mutual information is less discriminative than the losses, which diminishes the necessity of using mutual information to detect the existence of shortcuts in the training dataset.

While using a trained model on new data $X_{test}$, we usually don't have the corresponding label $Y_{test}$. In contrast to the test accuracy, calculating MI $I(X_{test}; Z)$ does not require a labelled dataset and hence our method can be used to detect shortcuts without requiring test labels. We plotted the test accuracy to empirically demonstrate the connection between $I(X_{test}; Z)$ and the test accuracy.

The proposed algorithm and the experiment setting are different.

In our experiments, we calculated both $I(X_{tes}t; Z)$ and $I(X_{tr}; Z)$ using model trained $D_{tr}$

Algorithm 1 step 1) Why $\mathcal{F}$ is initially required?

We need a trained model $\mathcal{F}$ to calculate the mutual information.