NeuronTune: Towards Self-Guided Spurious Bias Mitigation

Thank you for your detailed review and feedback on our submission. We provide our responses to your questions below.

Essential References Not Discussed

• Comparison with NeuroInspect: NeuroInspect identifies neurons responsible for mistakes from the counterfactual explanation perspective and minimizes the influence of identified neurons via regularization. In contrast, our approach identifies biased neurons by probing their activation patterns and directly manipulates identified neurons by suppressing their contributions to the final predictions.
  NeuroInspect emphasizes on interpretable neuron-level debugging, allowing practitioners to further pinpoint the causes of the errors. Our work aims at a self-guided spurious bias detection and mitigation framework without human intervention.
  We see potential in applying the interpretation techniques to our method in future work. We will cite NeuroInspect and add the above discussion to our revised manuscript.

Weaknesses

1. Limited interpretability: We hope to clarify that our work does not focus on interpretation. Instead, we focus on a self-guided framework without human intervention. By definition, biased neurons may encode a certain portion of core features, making the interpretation of neurons nontrivial. Nevertheless, we provide visualizations in Figures 3-6 in Appendix to analyze which spurious concepts are discovered. Please refer to our response to Question 4 for details.

2. Dependence on a separate identification set: One of our major contributions is removing the dependence on group labels in the identification set. This offers great flexibility and freedom for practitioners to select identification data based on their needs to detect and mitigate targeted spurious biases. Hence, obtaining such data is relatively straightforward. In our experiments, we used the validation set for consistency in performance comparison, but in real-world scenarios, any sufficiently representative data can serve this purpose.

Questions:

1. Iterative Tuning: Yes, we performed multiple rounds of neuron identification and suppression (L256-259, right column). Diminishing returns will be observed after a certain number of rounds. In general, running multiple rounds is better than running a single round with improved performance, as new biased neurons will be identified.

2. Partial vs. Full Suppression: Although partial suppression reduces the loss in overall accuracy, its effect is similar to no suppression, as models can continue to adjust their weights on nonzero biased activations, leading to severe spurious bias. For example, on the CelebA dataset, when we multiplied the activations of biased dimensions with the masking values in the table below, we see that only when the masking value is zero (full suppression) did the model achieve improved WGA.

3. Identification Set Size/Composition: The identification set should consist of diverse samples that are not memorized by the model. We find that when we use less than 30% samples of the validation set, the obtained worst-group accuracy is low with high variance, whereas the average accuracy remains high with low variance. When we use around 30% of the validation set, the number of minority group samples (e.g., waterbirds with land backgrounds in Waterbirds or males with blonde hair in CelebA) is approximately 50. Therefore, to ensure the effectiveness of our method, each data group should ideally contain more than 50 samples. In a group-label-free setting, it is suggested to include many diverse samples whenever possible.

4. Interpretability: Yes, we have shown images that highly activate the identified biased neurons along with the heatmaps highlighting regions that contribute to the neuron activations in Figures 3-6 in Appendix. The identified biased neurons mainly represent spurious features. For example, we observe that biased neurons mainly represent blue/green backgrounds for landbirds in Figure 5(b) and represent forest backgrounds for waterbirds in Figure 6(b). Note that since a neuron may represent a mixture of spurious and core features, the visualizations sometimes may not be directly interpretable.

5. Extension to Other Tasks: Yes, NeuronTune can be extended to multi-label prediction tasks. Instead of considering all identified neurons for different labels as biased, as shown in Eq. (6), we can consider biased neurons as label-specific and mask neurons based on the labels of input samples during retraining. This is because in the multi-label setting, a biased neuron for one label may be the core contributor to another label, which is different from the assumption in a standard classification setting (L272-274).

Please let us know if you have further questions.

Thank you for your thoughtful comments and for recognizing both the simplicity and effectiveness of our approach. We appreciate your valuable feedback and are glad that our efforts to explain the model selection strategies were helpful.

Regarding the subpopulation shift interpretation

We appreciate your thorough understanding of our work and your insightful interpretation from the perspective of subpopulation shifts, i.e., changes in the proportion of certain subpopulations between training and testing. We would also like to clarify the distinction between mitigating spurious bias and subpopulation shifts.

• Mitigating subpopulation shifts often relies on reducing spurious bias, such as decreasing the strong correlation between waterbirds and water to improve generalization for waterbirds on land. Tables 1 and 2 demonstrate NeuronTune’s effectiveness in handling subpopulation shifts on the Waterbirds, CelebA, MultiNLI, and CivilComments datasets.

• In contrast, mitigating spurious bias has broader implications beyond subpopulation shifts. For example, Table 3 highlights NeuronTune’s ability to enhance robustness on the ImageNet-A dataset, which consists of samples that are challenging for a pre-trained model but do not conform to well-defined subpopulation shifts, such as images with unique pixels. This evaluation scenario represents a broader and more complex problem than subpopulation shifts.

Therefore, by framing our method as targeting spurious bias, we address a wider scope of challenges beyond subpopulation shifts.

With proper examples, our current presentation aligns well with your proposed subpopulation shift interpretation. We detail the correspondence between your interpretation and our presentation below.

When you train ERM on datasets with imbalanced subpopulations, it maximizes average-case accuracy and performs very poorly on rare subpopulations. E.g. on waterbirds, it focuses on being correct on waterbirds on water, and sacrifices its ability to classify waterbirds on land.

Consider that $a$ in Eq. (2) controls subpopulations in data, e.g., when $a=1$, it may represent a group of waterbirds on water, and when $a=0$, it may represent a group of waterbirds on land. The probability $p$ controls the severity of imbalance in subpopulations. When $p$ is close to one (L159 left), the data is severely imbalanced in subpopulations. After training with ERM, the model minimizes the training loss, i.e., maximizes average-case accuracy, but obtains a large nonzero weight on the spurious feature (Lemma 1) and is away from the optimal model (Corollary 1). For example, the model may focus on being correct on waterbirds on water and sacrifice its ability to classify waterbirds on land.

Then, when you look at the misclassifications on the validation set, the features that you identify are precisely the ones that make you misclassify waterbirds on land. You zero these features out, and retrain the output layer to assign higher weights to the features that would allow you to correctly classify waterbirds on land, i.e., bird features.

Our principle of neuron identification (Proposition 4.1) states that when a spurious correlation breaks, the neurons that still positively contribute to mispredictions will be selected. For example, neurons that lead to misclassification on waterbirds on land will be identified. After retraining, Theorem 4.3 indicates that weights on core features, such as bird features, will be increased (L200-201, right).

By doing this, you slightly sacrifice your ability to classify waterbirds on water, but significantly improve your ability to classify waterbirds on land.

Our findings (L196-201, right) show that our method increases the weights on core features while keeping the weights on spurious features unchanged. This suggests that our approach makes a slight trade-off in average-case accuracy to achieve improved worst-group accuracy. For example, our method may slightly reduce the model’s ability to classify waterbirds on water due to a relative decrease in reliance on the water feature, while significantly enhancing its ability to classify waterbirds on land.

Thanks again for your thoughtful comments. We will incorporate concrete examples to provide readers with better intuition and a clearer understanding of our approach.

Please let us know if you have further questions.

Thank you for your thoughtful review and valuable feedback on our submission. Here are our responses to address your questions.

Evaluation Criteria

• Usefulness of Accuracy Gap: We hope to explicitly provide readers with a direct view on the numeric difference between worst-group accuracy and average accuracy, making it easier to assess how consistently a model performs across different groups of data. When models with very low worst-group accuracy exhibit inflated accuracy gaps, such as the ERM models, this indeed highlights the severe robustness issue of the models. In this paper, we report the average accuracy, worst-group accuracy, and the accuracy gap between them to provide a comprehensive evaluation of each model's overall performance.

Essential References Not Discussed

• Connection to the two related works: Thanks to the reviewer for giving a clear comparison between our work and the works proposed by Kim et al, 2022a and Liu et al., 2021. Indeed, our work builds on idea of these works by extending the focus from identifying bias-conflicting samples to detecting biased neurons within a model, enabling direct intervention in the model’s decision-making process. We will add the above discussion to our revised manuscript to better connect our method to these two works.

Weaknesses

1. Regarding number of trials and clear improvements: Following your suggestion, we ran our method for five additional trials. Overall, the accuracies remained stable with reduced variance. Our previous results in the paper and the new results below clearly demonstrate the advantage of our method in the unsupervised bias mitigation setting, where no group labels are available.

2. Regarding the assumption on the training set: We would like to clarify that our method does not rely on any assumptions about the training set. Since our approach uses an independent identification set for bias detection, it can be applied to models trained on any datasets. Notably, if the training set contains fewer biased samples, the resulting model is less susceptible to spurious bias. In such cases, NeuronTune will identify fewer biased neurons and the model will receive less intervention for bias mitigation.

Other Comments Or Suggestions

We will make revisions at the suggested locations and thoroughly review the rest of our paper.

Questions

1. The error values in Table 2 are quite high. Could this be caused by the low number of trials? Would at least e.g. five trials be possible? Did you conduct a Wilcoxon-rank-sum test? We hope to clarify that the variances of our method are relatively low compared to other methods. After running five additional trials, we observed reduced variance. Please refer to the table in Weakness 1 for the results. Our method demonstrates clear improvements in the unsupervised bias mitigation setting. Since we compare against the best-reported performance of baseline methods, a Wilcoxon rank-sum test is not directly applicable in this context.

2. The decision boundaries are different between training and testing in Figure 2a and 2c: This discrepancy arises because we used two separate K-nearest neighbor classifiers, each fitted to the predictions on the training and test sets separately, to determine the decision boundaries. To ensure consistency, we have now used a shared K-nearest neighbor classifier fitted on both training and test data to determine decision boundaries. The revised figure can be accessed here.

3. Regarding the statement in l.379 left: We hope to clarify that in Tables 1 and 2, we group methods that are directly comparable and boldface the best result within each group of methods (L345 and L398). For example, on MultiNLI (Table 2), AFR belongs to the methods that require group labels, while NeuronTune does not require group labels. NeuronTune achieves the highest WGA within methods that do not require group labels and even remains competitive against methods that require group labels. We will revise L379 to further clarify this in the paper.

4. When using the validation set to find neurons that are sensitive to spurious behavior, is the training set used to tune the last layer? Yes, for "NeuronTune," the validation set is used only to identify biased neurons, while the training set is reused to tune the last layer. "NeuronTune$^\dagger$" is a variant where half of the validation set is used for detection and the other half for tuning.

Please let us know if you have further questions.