Response to Reviewer HCiM

We thank the reviewer for their feedback and are happy to address all concerns raised below.

---

W1 and Q1: Focus on image classification, inclusion of complex, real world data and additional modalities such as text

Thank you for raising these important points. Since they concern multiple aspects of our work, we address each part individually below.

Focus on image classification.

Our main experiments are based on image classification tasks, as saliency concepts—such as foreground and background—are particularly well-defined and interpretable in computer vision. This makes it a natural domain for evaluating H-SPLID’s ability to disentangle salient and non-salient features. To this end, we tested H-SPLID on several challenging CV benchmarks.

Inclusion of complex, real-world data.

As shown in Table 2, we evaluate H-SPLID on datasets that explicitly model real-world feature entanglement. For instance, ImageNet9 [1] and its MixedRand variant (see Figure 4, Appendix F) introduce background shifts by replacing image backgrounds with those from other real-world classes, enabling evaluation under natural distribution changes.

We also include the CounterAnimals dataset [2] (CA-Counter in Table 2), derived from iNaturalist wildlife images, which contrasts typical and atypical contexts (e.g., a polar bear on snow vs. grass; see Figure 3a). This setup provides a realistic test of generalization under spurious correlations.

In addition, and in response to Reviewer 76kv (W1 & 2), we evaluated H-SPLID on the ISIC-2017 skin lesion classification dataset [3], comprising three real-world medical classes. H-SPLID consistently outperformed baselines under multiple real-world perturbations (see our response to Reviewer 76kv for details).

Inclusion of additional modality - text data

In contrast to saliency in images, modalities such as text involve more nuanced, task-specific definitions of saliency, where the notion of foreground versus background is less explicit. As such, further exploration is required to understand how H-SPLID's decomposition applies across modalities and how to appropriately define and measure saliency in those contexts. However, to illustrate the potential of H-SPLID beyond vision, we conducted an experiment on a text classification task using the ERASER Movie Reviews dataset [4]. In this benchmark, each example is annotated with human-provided rationales, which we treat as salient tokens and the task is to predict the reviews' sentiment (positive or negative).

An example from ERASER movie reviews benchmark: "In this movie, the acting is great! The soundtrack is run-of-the-mill, but the action more than makes up for it." The highlighted texts are the rationales for the positive review. These are annotated by human experts. All other tokens we consider as non-salient (background).

We adopted a new backbone—DistilBERT [5]—and evaluated robustness using the HotFlip adversarial attack [6], adapted to perturb non-salient tokens only. This setup tests whether the model relies on core (rationale) features or is vulnerable to noise in peripheral content.

Based on the gradient, a HotFlip-1 attack would change one letter. Using the example above: Attacked text: "In this movie, the acting is great! The sounTtrack is run-of-the-mill, but the action more than makes up for it." --> Changing the sentiment prediction from positive to negative.

Using limited hyperparameter tuning for H-SPLID, it improves adversarial robustness over the DistilBERT baseline, reducing the accuracy drop under both 1-token and 3-token perturbations:

These results, while preliminary, suggest that H-SPLID can improve robustness even in textual domains. We acknowledge, however, that stability and generalization across modalities such as graphs and robotics pipelines remain open challenges. Arguably, an in-depth analysis and application of H-SPLID in domains such as text would be an independent contribution in its own right, perhaps even a separate paper.

---

W2: Sensitivity study of introduced hyperparameters

We conducted a detailed hyperparameter sensitivity analysis on the COCO-Animals dataset by varying one hyperparameter at a time, starting from the best-performing configuration. The tables below report the model’s performance in terms of Clean Accuracy (%), PGD adversarial robustness across different attack strengths, and the dimensionality of the salient space. Each table shows how changes to a single hyperparameter affect these metrics while keeping all others fixed.

The hyperparameter sensitivity study on the COCO-Animals dataset highlights that $\lambda_n$ and $\rho_n$ are key to balancing adversarial robustness and the dimensionality of the salient space. Notably, the best model exhibits a salient dimensionality of just 14 (out of 512), indicating highly effective compression of salient features.

In general, increasing $\lambda_n$ and $\rho_n$ improved robustness and reduced the salient dimension, although very high values led to performance collapse. For $\lambda_s$, robustness initially improved before slightly declining, while increasing $\rho_s$ yielded modest but consistent gains in robustness—both also contributing to a reduction in salient dimensionality. We will include the remaining analysis on mask update rates and scheduling strategies in the camera-ready version of the paper.

W3: Scaling of HSIC computation to high-dimensional embeddings

Empirical scalability. Our method is practically feasible, as demonstrated by competitive training times on large-scale datasets such as ImageNet-1K with a ResNet50 backbone (embedding dimension 2048), as shown in Tables 9 and 10. This suggests that H-SPLID is viable for real-world applications. Moreover, further approximations can extend its scalability while preserving statistical reliability and interpretability.

Approximation strategies. While we adopt the HSIC implementation from HBaR [7], various approximation techniques can further reduce its cost. Block-based HSIC [8], for example, partitions each minibatch into blocks of size $B$, computes HSIC within each block, and averages the results. This reduces complexity to $\mathcal{O}(Bn)$, making it suitable for large batches. In addition, general kernel approximation methods such as random Fourier features and Nyström approximations [8,9] offer promising directions for future extensions.

---

Overall we want to thank the reviewer for the detailed feedback, it helped to further improve our paper.

---

References

[1] Xiao, et al: Noise or Signal: The Role of Image Backgrounds in Object Recognition. ICLR 2021

[2] Wang, et al. A Sober Look at the Robustness of CLIPs to Spurious Features. NeurIPS 2024

[3] Codella, et al. Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI). arXiv: 1710.05006

[4] DeYoung, et al. ERASER: A Benchmark to Evaluate Rationalized NLP Models. ACL, 2020.

[5] Sanh, et al. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108, 2019.

[6] Ebrahimi, et al. HotFlip: White-Box Adversarial Examples for Text Classification. ACL, 2018.

[7] Wang, et al. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. NeurIPS 2021.

[8] Zhang,et al. Large-scale kernel methods for independence testing. Statistics and Computing, 28(1), 2018.

[9] Ninh and Pagh. Fast and scalable polynomial kernels via explicit feature maps. KDD 2013.

Response to Reviewer P27a

We appreciate the reviewer’s positive feedback on our work, particularly their recognition of the novelty of our method and the thoroughness of our empirical and theoretical analysis. We address all remaining concerns and questions below.

---

W2: Streamlining of methodology section (e.g. Sec. 3.1)

We appreciate the suggestion to improve the clarity of the methodology section. In the camera-ready version, we will streamline the presentation of Section 3 by reorganizing the conceptual flow and presenting the loss terms in a more concise and intuitive manner. We will also polish the overall writing and notation to improve readability and precision.

---

Q1: Selection criteria for the COCO subset

We selected a COCO subset in which the object of interest is correlated with its background. We focused on images of animals in their natural habitats—e.g., giraffes in the savannah—where background and object are contextually linked. We also chose non-overlapping classes (e.g., excluding dogs and humans, which frequently co-occur) and prioritized images where the background occupies a large portion of the image, like for images of animals in the wild, to have enough room for attacks on the background. This setup provides a challenging test-bed for our approach, which learns salient object-specific features and non-salient background features. We will clarify this selection criterion in the paper.

---

W1 and Q2: Additional experiments on different classifiers (alternative backbones)

We would like to clarify that our experiments already incorporate a variety of backbone architectures. Specifically, we use two ResNet variants of different capacities—ResNet18 for COCO and ResNet50 for ImageNet—as well as LeNet-3 [1] for C-MNIST (see Figure 1 and Appendix F.4 Hyperparameters).

In response to Reviewer HCiM’s suggestion to include a text-based dataset, we further evaluated H-SPLID on the ERASER Movie Reviews dataset [2], a benchmark with human-annotated rationales.

We adopted the DistilBERT language model [3] as a new backbone and assessed robustness using the HotFlip adversarial attack [4], which we adapted to adversarially perturb $l$ letters of non-salient "background" words. Here, we define the salient words by the rationale masks provided by human annotators. For example, given the text: "In this movie, the acting is great! The soundtrack is run-of-the-mill, but the action more than makes up for it.", the highlighted part was the rationale by a human annotator and the remaining words serve as "background" words that will be attacked.

Thus, the sentiment analysis setup probes whether the model relies primarily on core (rationale) features or is sensitive to noise in peripheral content.

For HotFlip attacks with $l=1$ and $l=3$ H-SPLID outperforms the DistilBERT baseline. These preliminary results suggest that H-SPLID can transfer to different backbones and even to different modalities, such as text. Based on this result we see the in-depth study of H-SPLID across modalities as promising future work.

---

We thank the reviewer for their thoughtful feedback and will revise the final version to further improve the presentation and highlight the generality of H-SPLID.

---

References

[1] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.381 Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):382 541–551, 1989.

[2] DeYoung, Jay, et al. "ERASER: A Benchmark to Evaluate Rationalized NLP Models." Transactions of the Association for Computational Linguistics, 2020.

[3] Sanh, Victor, et al. "DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter." arXiv preprint arXiv:1910.01108, 2019.

[4] Ebrahimi, Javid, et al. "HotFlip: White-Box Adversarial Examples for Text Classification." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018.

Response to Reviewer XsTp

We thank the reviewer for their thoughtful and detailed assessment of our work. We are particularly grateful for their recognition of the clarity and reproducibility of our experimental design, and the effort invested in the theoretical analysis. We also appreciate the positive remarks on the structure and accessibility of the proof provided in Appendix C. We address the remaining questions and concerns below.

---

W2: Clarification on the role and benefit of learning non-salient features

Thank you for the thoughtful question. While non-salient features do not directly affect the final prediction, they play an important role during training, as salient features are defined in contrast to them. Their presence is therefore essential. By capturing task-irrelevant variations—such as background noise—non-salient features help isolate predictive factors in the salient space, enhancing robustness and disentanglement. These features are not expected to become salient over time; rather, the model learns a stable partition of latent dimensions based on the H-SPLID loss. We will clarify this in the paper.

More broadly, our approach serves as a novel form of regularization. Although deep neural networks are highly expressive, their performance can degrade due to entanglement with irrelevant features. By jointly learning the classification task and the decomposition into salient and non-salient components, our method improves robustness, as shown in our experiments. In essence, H-SPLID encourages the network to focus on the most relevant information during training.

---

Q1: Typo in line 115 (identity matrix)

Thank you for pointing out the typo. We will fix it.

---

Q2a: Difference between $L_{ST}$ and Equation 5?

We apologize for the typo and confusion. They are the same thing. We will unify the notation.

Q2b: Are $L_s$ and $L_n$ used in both steps of H-SPLID or just the mask update?

$L_s$ and $L_n$ are used in both steps. As they will allow backpropagation to update the encoder $f_\psi$.

---

Q3: Definition of β and range of its values

Thank you for pointing out this ambiguity—we apologize for the omission in the original text. To clarify, we apply a hard threshold to the continuous solution for $\beta$, mapping it to $\lbrace 0, 1 \rbrace^d$. While the optimization initially produces a continuous vector $\beta \in [0,1]^d$, we threshold each component at 0.5 to obtain a binary mask. We first experimented with using the continuous solution directly (i.e., without thresholding), and found that the learned values typically concentrate near 0 or 1. As a result, both variants—the soft (continuous) and hard (thresholded) masks—perform similarly in practice. We will revise the manuscript to explicitly define $\beta$ as a vector in $\lbrace 0, 1 \rbrace^d$, clarifying its role as the diagonal of the mask matrix, and to explain the thresholding procedure it entails.

---

Q4: What variable is the expectation in Equation 9 taken with respect to?

The expectation in Equation (9) is taken with respect to $x$, i.e.:

$\mathbb{E}_x\lbrack \cdot \rbrack$,

where $x$ is sampled from the data distribution. The perturbation $\delta$ is an arbitrary vector constrained in norm by $r$, and not a random variable. This distinction is explicitly stated in the theorem and elaborated in Appendix C, but we acknowledge that the notation in Equation (9) could have been clearer. To avoid any confusion, we will revise the equation to use $\mathbb{E}_{x}$ explicitly in the camera-ready version.

---

We appreciate the reviewer’s careful reading and the opportunity to clarify the content of our paper. We have already incorporated the reviewer's suggestion into our manuscript.

Response to Reviewer to 76kv

Thank you for your thoughtful feedback. We appreciate the recognition of the problem’s significance—learning representations robust to irrelevant input is indeed essential for real-world deployment of deep neural networks. We address the open questions and concerns below.

---

W1 and W2: Additional results on medical imaging dataset and real world perturbations

We would like to clarify that while we do explore the effect of adversarial attacks on the COCO dataset (Table 1), our evaluation already incorporates datasets offering additional perspectives on robustness. For instance, the ImageNet9 [1] dataset (see Figure 4) introduces background modifications, with its MixedRand version occluding the background of the ImageNet images with background taken from other natural images (see Figure 4 in Appendix F for an example). Furthermore, we already include experiments with natural, real-world perturbations using the Counter animals [2] dataset (abbreviated as CA-Counter in Table 2). The counter animals dataset is derived from iNaturalist wildlife photos and is split into a common subset (featuring typical backgrounds) and a counter subset (with atypical yet plausible backgrounds). For example, as shown in Figure 3a, a common background for an ice bear is a winter landscape, whereas an atypical background for an ice bear would be green grass.

In response to Reviewer HCiM’s suggestion to include a text-based dataset, we further evaluated H-SPLID on the ERASER Movie Reviews dataset [3], a benchmark for sentiment analysis that provides human annotated rationales.

We fully agree that the inclusion of a medical imaging dataset would further strengthen the paper's demonstration of broader applicability and robustness. To this end, we have conducted new experiments on the ISIC-2017 skin lesion classification dataset [4], which comprises three classes: nevus, melanoma, and seborrheic keratosis. For this, we used a ResNet-50 pre-trained on ImageNet for feature extraction and trained a three-class classification head for 50 epochs, followed by 50 epochs of method-specific training. Model tuning and selection were conducted based on the procedure outlined in Appendix F.4.

To evaluate robustness against the reviewer-suggested real-world perturbations: lighting, blur, and occlusion - we utilized the implementations of brightness, defocus blur, and occlusions$^1$ from the corruptions benchmark [5]. Following the analysis in the paper, we perturbed only the non-salient regions, such as non-lesion pixels. The results averaged over 10 random seeds are presented below:

Our method, H-SPLID, outperforms the competitors across all three perturbation types on the ISIC-2017 dataset, further demonstrating its robustness to more natural, real-world perturbations and its applicability in specialized domains like medical imaging. We thank the reviewer for the suggestion and we will incorporate the medical imaging dataset analysis into the revised paper.

---

$^1$ The snow perturbation is effectively occluding small patches of the image with white snow, thus we refer to it as occlusion. An example is given in Figure 1 of the original paper [5].

---

Q1: Does the learning rate influence the final outcome, e.g. salience subspace versus the non-salient subspace?

Yes, the learning rate is an important hyperparameter for deep learning methods in general, and our H-SPLID approach is no exception to that. We conducted additional experiments on the COCO-Animals dataset w.r.t. the learning rate, where we take the model with the best hyperparameters and vary the learning rate by powers of ten to test the learning rates influence.

Our experiments demonstrate that the learning rate significantly influences both: the model’s performance, as well as the learned salient dimension. Specifically, a learning rate of 0.0005 yielded the best overall performance, achieving the highest clean accuracy and robustness across different strengths of PGD and AutoAttacks (AA). At this learning rate, the salient dimension was found to be 14 out of 512. Lower learning rates (e.g., 0.000005 and 0.00005) resulted in progressively lower accuracy and robustness, indicating that the model did not learn an informative salient representation. Notably, a learning rate of 0.000005 resulted in a significantly higher salient dimension (450/512), suggesting that a very low learning rate might lead to a less compact salient space. Conversely, a higher learning rate (0.005) led to a complete collapse of performance, with the model failing to learn any meaningful representation (salient dimension of 0/512). We will include these results in to the paper.

---

Q2: Why PGD over any other attack?

We would like to draw the reviewer's attention to Table 1, where we not only compare robustness w.r.t. PGD, but also to AutoAttack. AutoAttack serves as a strong attack that uses an ensemble of diverse parameter-free adversarial attacks.

---

We thank the reviewer for the extensive feedback and will revise the final version by including the suggested experiments and clarifications. These will further strengthen our work.

---

References

[1] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, Aleksander Madry: Noise or Signal: The Role of Image Backgrounds in Object Recognition. ICLR 2021

[2] Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, Tong Zhang: A Sober Look at the Robustness of CLIPs to Spurious Features. NeurIPS 2024

[3] DeYoung, Jay, et al. "ERASER: A Benchmark to Evaluate Rationalized NLP Models." Transactions of the Association for Computational Linguistics, 2020.

[4] Codella N, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza S, Kalloo A, Liopyris K, Mishra N, Kittler H, Halpern A. "Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)". arXiv: 1710.05006

[5] Dan Hendrycks and Thomas Dietterich: Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019.