SCISSOR: Mitigating Semantic Bias through Cluster-Aware Siamese Networks for Robust Classification

1. On Clustering Method (with Larger Models)

We conducted an ablation experiment using DBSCAN with LLaMA3, the largest model we used. The results are as follows:

We observed that the improvement of DBSCAN and MCL on LLaMA3 is similar, both being approximately 1%. This indicates that SCISSOR exhibits robustness to variations in clustering algorithms.

2. On Semantic Bias and Initial Embeddings

Since most contemporary debiasing works rely on pre-trained language models, we followed their works. We will consider debiasing methods in future work that do not rely on initial information.

3. On Background Colors in the Table in Step2 of Figure 2

We are sorry for making the confusion. The background colors are used to distinguish samples (i.e., Anchor, Positive, Intermediate, Negative). Their function is same to the color of the "Sun" and "Moon" markers. We will improve the readability of Figure 2 in the camera-ready version.

1. On the Lack of Novelty

We respectfully clarify that, to our knowledge, shortcut learning stemming from a semantic imbalance in pre-trained models (which requires unsupervised analysis) is not widely studied. The theoretical reference [1] you mentioned deals with label quantity bias, rather than semantic distribution. In fact, our experiments (lines 192-196) use balanced label distributions. Likewise, GroupDRO also focuses on imbalances in categories, not the distribution of semantic features. The debiasing in our paper refers to mitigating shortcuts which universally across all groups in the training set, rather than focusing on the worst group. Meanwhile, [2] requires prior knowledge of shortcuts and access to an unbiased dataset, both of which can be impractical in real-world scenarios [3][4][5].

Consequently, our assertion that “an imbalance in semantic distribution can lead to shortcuts” remains valid and distinct from the biases explored in [1], GroupDRO, and [2].

2. On Baseline Comparisons

As noted in our Introduction, existing shortcut-mitigation research typically addresses label imbalance or focuses on token/pixel-level biases. Please note that direct reweighting methods, for example, are not applicable here because our training samples are already balanced by label. This underscores our key statement: “Shortcuts can persist even when labels are balanced.”

To address concerns about baselines, we have added a comparative experiment with IRM; please see our “to reviewer gyYJ” response for details. We also compare our approach with RAZOR, a leading method that already surpasses several simpler baselines. Because our method outperforms RAZOR, we did not repeat comparisons with those baselines, especially since their focus is on token-level biases, whereas our work targets semantic-level biases.

3. On the Usefulness of Tables 2 and 3

In response to concerns about the value of Tables 2 and 3, our main motivation arises from the imbalance in semantic distribution, which the Hopkins Statistic measures effectively. By comparing Tables 4 and 5, we illustrate that our method’s effectiveness increases as semantic imbalance grows, demonstrating that our semantic debiasing approach genuinely shifts how samples are represented.

4. Clarity Concerns

4.1 Choosing Intermediate Samples

In lines 214-215, we explain that “Intermediate samples share the same label and cluster as the anchor,” and this process is also illustrated in Figure 2. Nevertheless, we will try to better highlight this aspect in our paper.

4.2 Multi-Label Classification

Our “Problem Formulation” section does not restrict us to binary classification. Figure 1 uses a binary example solely for illustrative purposes. In fact, we have already evaluated multi-label scenarios in the paper, such as the Not-MNIST dataset, which has ten labels (A-J) as noted in lines 291-293. Finally, we emphasize that shortcut learning is multifaceted; recent works on style-based shortcuts [6] and in-context learning shortcuts [7] highlight various dimensions of this issue. We believe their contributions are valuable, as they address different forms of shortcuts that are similar to our notion of semantic shortcuts.

Finally, we emphasize that our work tackles semantic shortcuts, a fundamentally different challenge than label-based or count-based bias mitigation. Addressing these subtler cues demands deeper scrutiny than simply balancing label distributions or removing obvious spurious correlations. Consequently, our approach necessitates unsupervised techniques to effectively uncover and mitigate these hidden biases.

[1] Chaudhuri, K., Ahuja, K., Arjovsky, M. & Lopez-Paz, D. (2023). Why does Throwing Away Data Improve Worst-Group Error? Proceedings of the 40th International Conference on Machine Learning

[2] Kirichenko, P., Izmailov, P., & Wilson, A. G. (2022). Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937.

[3] Weizhi Xu, Qiang Liu, Shu Wu, Liang Wang (2023), Counterfactual Debiasing for Fact Verification

[4] Shuo Yang, Bardh Prenkaj, Gjergji Kasneci. (2025). RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting

[5] Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson. (2024). DISCO: Distilling Counterfactuals with Large Language Models

[6] Yuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei Zhu. (2024). Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models

[7] Joonwon Jang, Sanghwan Jang, Wonbin Kweon, Minjin Jeon, Hwanjo Yu. (2024). Rectifying Demonstration Shortcut in In-Context Learning

Thank you for your positive feedback on our work. We sincerely appreciate your time and support. We'll address any further questions you should have during the discussion period to improve our paper, and make it through the finish line.

1. We added a comparative experiment with Invariant Risk Minimization (IRM) [1] and will include the results as well the corresponding references in the Related Work section.

We observe that although IRM does mitigate shortcuts in many cases, our method still significantly outperforms it across all tests. Moreover, IRM performs worse than the baseline on small datasets, such as Chest-XRay (w/SWIN) and GYAFC (w/LLaMA). We attribute this to IRM assigning excessive training weight to features that remain invariant, which prevents other useful features from being accurately identified and utilized.

2. Adding a Visual Example of Semantic Bias

We used PCA to reduce the BERT embeddings of the Yelp dataset to two dimensions and visualized three clusters, as shown in the Anonymous github we attached in the paper. In the image (https://anonymous.4open.science/r/SCISSOR-3F55/unb.svg), the colors represent cluster labels, while in the image (https://anonymous.4open.science/r/SCISSOR-3F55/unb_l.svg), colors indicate task labels (positive/negative).

We observe that different clusters are located in distinct regions. Furthermore, the task labels within each cluster are imbalanced: in two of the three clusters, positive samples are more prevalent, while in the remaining cluster, negative samples dominate. Nevertheless, the total number of positive and negative samples across all three clusters is equal.

In this scenario, negative samples belonging to clusters with more positive samples are theoretically more likely to be misclassified as positive by the models (as suggested by the Lemma 1 and Lemma 2), leading to the semantic shortcut we discuss.

Furthermore, in the image (https://anonymous.4open.science/r/SCISSOR-3F55/pca_SCISSOR.svg), we show the semantic representations of samples after being remapped by our debiasing module. Through using SCISSOR, we observe that the task-irrelevant cluster information has been successfully removed. In contrast, classification information is preserved, naturally leading to a distinct classification boundary among the samples. To better illustrate the separability of the samples here, we visualize the three-dimensional PCA results of the output of our debiasing module (https://anonymous.4open.science/r/SCISSOR-3F55/pca_SCISSOR_3d.svg).

3. Experiments on Large-Scale benchmarks

Thank you for these valuable pointers. We are running experiments on the WILD and DomainBed benchmarks. However, due to the large scale of these experiments, we may not be able to obtain results within the rebuttal period (we might make it within the discussion period). Nevertheless, we emphasize that our current experiments already include 4 commonly used datasets, which we believe provide a solid basis for evaluating our approach and demonstrating its reliability.

[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.