Preserving Task-Relevant Information Under Linear Concept Removal

We thank reviewer hbbQ for their thorough examination of the paper and the suggested points of improvement. We also thank them for pointing out that the paper contains 'Extensive experimental results' and our approach is 'theoretically grounded'. Below, we address their feedback point-by-point.

The paper addresses the linear concept erasure problem and assumes that task-relevant information can be represented through covariance. However, these assumptions may not hold in complex representations within large models, where concepts are often encoded in a highly entangled manner. Experimental results further reveal that the proposed method performs less effectively when applied to intermediate layers.

We thank the reviewer for highlighting this point and giving us the opportunity to clarify what we mean by "task‑relevant information". Our work fits within a broad line of research on linear representation engineering—that is, using linear algebra to manipulate neural network representations and control the information they encode. This area now includes dozens of studies, mostly on large language models but also on other neural architectures, and it has proved useful for applications ranging from bias mitigation to causal analysis. In particular, the "linear representation hypothesis" — the claim that many concepts are, to a large extent, linearly encoded — has become central to mechanistic interpretability [1, 2, 3]. Within this context, our contribution can be viewed as a natural extension of erasure methods: we show that it is possible not only to remove unwanted information, but also to preserve information that we wish to keep.

At the same time, we acknowledge that many features are also non‑linearly encoded; hence, the linear representation hypothesis holds only approximately. As the reviewer points out, this limitation may explain the weaker performance we observe in middle layers. Nonetheless, we believe that providing a closed‑form solution for the linear case is valuable (the experimental section lists several concrete use cases). In future work we plan to extend our objective to the non‑linear setting via, for example, kernelization — an approach that has already proven effective in adapting other linear techniques [4].

[1] Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. T. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29.

[2] Vargas, F. and Cotterell, R. Exploring the linear subspace hypothesis in gender bias mitigation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2902–2913, Online, November 2020.

[3] Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 1605, 39643–39666.

[4] Ravfogel, Shauli, et al. "Adversarial concept erasure in kernel space." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

Some relevant works are missing in the comparison and discussion. In particular, methods that also aim to preserve task-relevant information during concept erasure, such as [1] and [2] ([2] is potentially considered concurrent work under the NeurIPS policy).

We thank the reviewer for raising these relevant works. We will include both works in the related work section of an updated version of the manuscript.

Our reasoning for not including methods such as [1] and [2] in our manuscript is that we focus on a) linear transformations that b) guarantee linear guardedness. We believe this focus is appropriate as methods that fulfill these criteria are similar in their fairness guarantees and computational resources. If we expand the comparison to methods beyond these criteria, we would have to include a whole range of concept-removal methods in addition to [1, 2] - see for instance several methods mentioned in the related work section - and significantly increase the scope of the paper.

[1] Robust Concept Erasure via Kernelized Rate-Distortion Maximization, NeurIPS 2023

[2] Fundamental Limits of Perfect Concept Erasure, AISTATS 2025

Q1: SPLICE demonstrates superior performance compared to baseline methods in two out of three tested Llama models in Fig. 3. Could the authors provide insight into the potential causes of this performance variation across different Llama model versions? To strengthen the generalizability of these findings, would the authors consider expanding the evaluation to include additional LLM architectures?

It is true that there is no formal guarantee that SPLICE always significantly outperforms baseline methods, although the paper tries to make clear heuristically and empirically that it often does. The fact that SPLICE demonstrates superior performance compared to baseline methods in two out of three tested Llama models, should be seen in this light. We do not have a detailed explanation as to why there is variation in performance between the Llama 2 and 3.

That being said, we agree with the suggestion of the reviewer to strengthen the generalizability of our findings. We are currently working on repeating the experiments in section 4.2 for other open-source LLMs, namely Mistral 7B and phi-2. We will include these experiments in an updated version of the manuscript.

Q2: In the CelebA experiment, the difference maps for SPLICE show notable changes around the glasses regions. Intuitively, since the smile concept is being removed, changes should be concentrated primarily in the smile regions rather than the glasses regions. Could the authors clarify this observation?

We thank the reviewer for allowing us to clarify the results in section 4.3 of the paper. Figure 4 and Figure 8 (the latter is in Appendix B.3) show that SPLICE leads to a transformation of the image around both the eyes and the mouth. This is as expected. The change around the eyes is to preserve $\mathrm{Cov}(\boldsymbol{x}, y_{\mathrm{glasses}})$. The alterations around the mouth are in order to ensure linear guardedness with respect to the concept $z_{\mathrm{glasses}}.$

We are grateful to reviewer z2Ty for carefully examining the paper and their feedback. We also thank them for their remarks that the paper is 'extremely well-written', and that both theorems in the paper are 'relevant results'. Below, we address the feedback point-by-point.

The most obvious ... such as UrbanCars. See [1].

We agree with reviewer z2Ty that this is a limitation of our approach. It requires access to both a concept label $z$ and a task of interest $y$. We also agree that this is a limitation of the current literature on concept-removal - virtually all existing approaches (see for instance [4, 5, 6, 7]) require a concept label $z$. Extending these approaches (including SPLICE) to a scenario where no (or limited) labels for $z$ are available is an interesting area of future research.

While our paper does not include multi-shortcuts as suggested in [1], our suggested projection can be easily extended to this case. The $z$ will then include both the labels of the first spurious attribute (e.g., background) as well as the second spurious attribute (e.g., another co-occurring object).

There are also ... adding Waterbirds.

We thank the reviewer for this suggestion. We emphasize that the goal of our proposed projection (or concept-removal methods in general) is not to have a state-of-the-art performance in terms of worst-group accuracy. Instead, it is to achieve certain fairness guarantees (via linear guardedness) and give the user control over features used by the deep neural network (DNN).

Per suggestion of the reviewer, we have evaluated SPLICE, as well as the baselines in the paper (SAL, LEACE) on the Waterbirds dataset. We have also evaluated each projection in a similar manner on the black-white version of the CelebA dataset in Section 4.3. In the original version of the manuscript we did not consider this evaluation for this dataset, since it was only used to illustrate which features are removed by each projection.

For Waterbirds, SPLICE performs similar to LEACE and SAL, although it is worth noting that all concept-removal methods perform poorly on this dataset (in comparison to standard group robustness methods - see our reply to reviewer 5q88). For CelebA, SPLICE outperforms LEACE and SAL.

(Note that we only report here WGA and not average accuracy, whose results are in line with WGA, due to length constraints of the rebuttal. We will gladly provide the average accuracy tables during the discussion phase, if the reviewer wishes us to.)

=== Waterbirds Dataset === Worst-Group Accuracy (%)

=== CelebA Dataset === Worst-Group Accuracy (%)

Based on these results, it appears that SPLICE (as well as other concept-removal methods) performs much better for classification tasks in the language domain (Bias in Bios, Multilingual detoxification dataset) than in vision (Waterbirds, CelebA). We believe this limitation is an interesting direction for future research. We will include this limitation and the results for Waterbirds and CelebA in an updated version of the manuscript.

An out-of-distribution evaluation ... interpretation of $\mathbf{P}$ in this case?

As said in our previous reply, the primary goal of SPLICE is not OOD generalization. Instead, its aim is linearly guarded concept removal while maintaining main-task performance. The experimental setup of OOD generalization is used to test to what extend SPLICE (and other existing methods) achieves this goal.

That being said, the default setup of OOD generalization experiments is a training set with a spurious correlation and a test set without this spurious correlation. In the case of Waterbirds, this amounts to a test set with four equal-size subgroups. The evaluation metrics are typically the average accuracy and the worst-group accuracy. The reviewer suggests to work with a test set consisting of only two of the four subgroups: landbirds on water and waterbirds on land.

It does not seem directly clear what this would add to the current experiment. The average accuracy now becomes less informative, as it neglects performance on the majority subgroups. Moreover, it is unlikely that this change will affect the worst-group accuracy, as the models typically perform the worst on one of the minority subgroups. Finally, the projection $\mathbf{P}$ is defined independently from the choice of the test set and its interpretation therefore is unaltered: it removes (linear) information about the spurious feature from the embeddings, while maintaining the covariance with the main-task label.

I am not convinced ... example, see Theorem 2 in [2].

In reply to the first concern of the reviewer, it is true that SPLICE assumes knowledge of the labels associated with the spurious correlation (as acknowledged earlier in our reply). And this, of course, also assumes that the shortcut itself is known. We point out that this also holds for competing concept-removal methods like LEACE and SAL.

Regarding the location of the spurious features, it is typically not true that the span/range of the learned $\mathbf{P}$ belongs to the span of the orthogonal complement of the spurious features. The kernel of $\mathbf{P}$ is spanned by the spurious features, but $\mathbf{P}$ is typically an oblique projection and not an orthogonal projection. This was already the case for LEACE, due to the whitening transformation which is shared with SPLICE. Following the framework of Thm. 2 in [2], this suggests a bias in the predictive model after projection of the last-layer embeddings. We agree with the reviewer that it would be very interesting to investigate this connection, possibly in the context of a bias-variance trade-off. We thank the reviewer for highlighting the connection with [2] and will include this in the final version of the paper.

Minor comment first: This ... a different name.

We thank the reviewer for highlighting this namesake method and will consider to update the name from SPLICE to SPLINCE (Simultaneous Projection for LINear concept removal and Covariance prEservation) for the final version of the paper to avoid confusion.

The U and V matrices ... such spurious features.

The fact that the matrices U and V in Theorem 1 have orthonormal columns is just for technical convenience. This choice of bases for U and V, respectively, is always possible and keeps Equation 4 simple. Furthermore, the matrices U and V act on the whitened features (see W in Eq. 4), whose covariance matrix has a trivial spectral decomposition. We therefore do not think that SVD (or other factorization techniques) applied to the matrices U and V would lead to new insights.

Since the authors adopt ... the term "concepts"?

We adopt the term to be line with the literature on concept-removal methods. In this literature, a concept refers to a representation in the data of a human-defined object or phenomenon (in line with the original paper introducing concept activation vectors [8]). Concepts are usually represented via binary, one-hot encoded, or continuous variables, and often align with protected attributes (e.g., gender).

What also interests me ... and vice versa.

We thank the reviewer for raising the question as to how our projection would function with CLIP. Suppose that for a text-image pair, we have the text features $\boldsymbol{t}$, and the image features $\boldsymbol{v}$. If for the text-image pair the concept labels $z$ and $y$ are available, one can simply apply our projection to both $\boldsymbol{t}$ and $\boldsymbol{v}$. This approach appears to address the risk raised by the reviewer.

The reviewer also raises an interesting point as to the interpretation of $\mathbf{P}$ if it is applied to a vector that represents alignment between modalities. In this case, SPLICE would work in a similar fashion, only now operating on the features where the text features $\boldsymbol{t}$ and vision features $\boldsymbol{v}$ align.

[1] Li, Zhiheng, et al. "A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others." 2023

[2] Ruiz Luyten, Max, and Mihaela van der Schaar. "A theoretical design of concept sets: improving the predictability of concept bottleneck models." 2024

[3] Fel, Thomas, et al. "A holistic approach to unifying automatic concept extraction and concept importance estimation." 2023

[4] Ravfogel, S., et al. Linear adversarial concept erasure. 2022

[5] Belrose, N., et al. Perfect linear concept erasure in closed form. 2023

[6] Ravfogel, et al. Null it out: Guarding protected attributes by iterative nullspace projection. 2020

[7] Shun Shao, et al. Gold doesn’t always glitter: Spectral removal of linear and nonlinear guarded attribute information. 2023

[8] Kim, B., et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). 2018

1. Thank you for the follow-up! We want to be sure we have correctly understand your multimodal concern. Are you suggesting that a multimodal encoder might assign distinct latent directions to the same high-level concept across modalities -- for instance, CLIP allocating one direction to the visual expression of positive sentiment (a smile) and another to its textual expression (words such as happy)? We agree that this cross-modal gap is intriguing. We remark that (i) This seems like a limitation of the underlying encoder. If the encoder itself does not align the modalities, our debiasing procedure can only operate on the representations it receives. (ii) The influence of such gap depends on the supervision. In our setting each training example carries a ground-truth label (e.g., “positive”), so the learned manipulation should follow whichever modality aligns with that label. When the modalities disagree, the method will naturally adjust the modality correlated with the labels and leave the other largely unchanged. We believe it is possible to test this hypothesis empirically -- for example, by selectively perturbing features in one modality-- and will discuss this issue in the final version.

2. We will add these results to the final version of this work. As our method is not causal in nature, we fully agree that debiasing without carefully considering confounds can even introduce new spurious correlations. We will improve the discussion of this point in the final version.

3. We fully agree with this observation and will discuss the disparity between representation-notions of debiasing and fairness outcomes. While we intuitively hope (and empirically test) that our interventions would improve certain notions of fairness, we acknowledge that different fairness notions exist (and are sometimes mutually exclusive), and that our objective does not directly translate into these fairness conditions.

We thank reviewer ynPj for their delicate consideration of the paper and feedback. We also thank them for stating that the paper is 'well-written and clearly presented', and 'makes a valuable theoretical contribution'. Below, we address their feedback point-by-point.

The concept of "task-relevant information" is not clearly defined in the paper and appears to be conflated with the classification target variable

Please define “task-relevant information” and clarify how to choose a suitable proxy for it in practice.

We thank the reviewer for pointing this out and allowing us to clarify the notion of "task-relevant information". Ideally, we would want to preserve the features that are relevant for recovering the performance of the main task. In the LEACE paper, it is proven that erasing the covariance with the sensitive information variable prevents a linear model from recovering this information. Extending this link between covariance and information, here we want to make sure that the covariance with the classification target variable, $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{y})$, is preserved. We agree that alternative notions of ``task-relevant information'' are possible, but we believe that instantiation of this information by the covariance with the target variable is a natural extension of LEACE. Moreover, this notion admits a closed-form solution and is empirically useful, as we show in the paper. Note that under isotropic features, it is easy to prove that preserving $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{y})$ is sufficient to preserve performance for classifying $\boldsymbol{y}$. We will include a proof for this in an updated version of the appendix.

There is no discussion of potential bias in the task-relevant information, which could itself be a result of biased data-generating processes. For instance, in the case of debiasing a classifier screening job applications (lines 27–32), the dataset may inherently reflect gender bias. Preserving covariance with such a task variable might inadvertently retain the very bias the method aims to remove, especially when applying SPLICE in earlier layers.

After the SPLICE (or LEACE, or SAL) projection the embeddings are linearly guarded for the spurious concept label $\boldsymbol{z}.$ This means that, restricted to linear models, the projected embeddings cannot be meaningfully used to predict the spurious label. This is also the case if the spurious label is predicted indirectly, via a linear model that predicts the main-task label $\boldsymbol{y}.$ The logic is that the projection also inevitably affects any linear prediction model for $\boldsymbol{y},$ in such a way that the correlation between the prediction of $\boldsymbol{y}$ and the spurious label $\boldsymbol{z}$ is broken.

We emphasize that information about the spurious concept might still be present in the projected embeddings, but only in a non-linear fashion. This information can indeed be uncovered by a DNN if SPLICE is applied only to some earlier layers of the network, as the reviewer suggests. Therefore, if SPLICE is applied to earlier layers of the network, it should also be applied to all subsequent layers. This is done in the LEACE paper, where it is called scrubbing [1].

In a revised version of the manuscript, we intend to communicate these points more clearly.

The paper lacks a discussion of which applications might benefit more from preserving main task performance (as done by SPLICE) versus preserving internal representations (as in methods like LEACE)

Please discuss which applications call for preserving main task performance (as in SPLICE) versus internal feature representations (as in LEACE).

We thank the reviewer for allowing us to make the comparison between SPLICE and LEACE more explicit. The key difference between the two methods is that SPLICE fulfills the constraint of preserving $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{y})$, at the cost of minimizing $\mathbb{E}[||\boldsymbol{x} - \mathbf{P}\boldsymbol{x}||_2^2]$. This can be particularly useful if there is some sort of correlation between $\boldsymbol{y}$ and $\boldsymbol{z},$ which there typically is, leading LEACE to remove $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{y})$. One typically expects a bigger advantage of SPLICE compared to LEACE when this correlation is stronger. This is illustrated empirically in Appendix B.2. The additional constraint of SPLICE comes at a cost - it minimizes $\mathbb{E}[||\boldsymbol{x} - \mathbf{P}\boldsymbol{x}||_2^2]$ to a lesser extent. Our results in Appendix B.4 indicate that this can potentially pose a problem when applying SPLICE at middle layers.

The assumption (line 133) is quite strong and may not hold in general

For each main assumption of SPLICE (e.g., ), please provide theoretical or empirical justification for its realism.

We believe that the assumption $U^\perp \cap \mathrm{colsp}(W \Sigma_{\boldsymbol{x},\boldsymbol{y}}) = \mathbf{0}$ is not very strong. It only says that the linear subspaces of the covariance directions $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{y})$ and $\mathrm{Cov}(\boldsymbol{x}, \boldsymbol{z})$ do not perfectly overlap. For the case of binary labels, this means that the covariance vectors are not the very same vector (up to a scaling factor). To be more concrete, for the Waterbirds dataset this would mean there are only landbirds on a land background and waterbirds on a water background. This is typically not the case in realistic datasets and would make any attempt to distinguish main-task and spurious features based on the labels futile. The assumption still allows for vectors with a high cosine similarity, i.e., we do not assume no overlap between the underlying features that encode the two concepts. We intend to clarify these subtleties around the assumption in the final version of the paper.

The theoretical justification and empirical performance of the method seem to apply primarily to the later or final layers of a model (line 323, Appendix B.4).

We broadly agree with this limitation as raised by the reviewer. There is indeed a risk that SPLICE performs worse for middle layers, as it deprioritizes minimizing $\mathbb{E}[||\boldsymbol{x} - \mathbf{P}\boldsymbol{x}||_2^2]$ in favor of fulfilling an additional constraint. However, the results in Appendix B.4 also indicate that there are cases where SPLICE applied intermediate layers can be beneficial (such as in the Bias in Bios dataset, or the experiment with the Profession dataset).

The methods discussed in lines 73–74 may still satisfy linear guardedness, even if they do not explicitly mention it, unless a theoretical proof is provided to the contrary.

We thank the reviewer for pointing this out. We will adapt the writing in our related work section, as well as our writing in line 214-217 to reflect this point.

Typo: “an projection” → “a projection” (line 167)

We thank the reviewer for raising this typo and will adapt it in an updated version of the manuscript.

[1] Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. Leace: Perfect linear concept erasure in closed form. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 66044–66063.

We are grateful to reviewer 5q88 for the careful and considerate review of our paper, as well as for the suggestions for improvement, which we find extremely valuable. We also thank them for commenting that our approach is 'conceptually appealing'. Below, we address the feedback point-by-point.

The contribution is incremental, primarily extending LEACE by introducing an additional constraint

We agree with the reviewer that the method we introduce extends LEACE by introducing an additional constraint. However, by doing so, we are (to the best of our knowledge) the first to consider preservation of main-task information in the context of linearly guarded concept removal. This is a well-known problem in the general field of post-hoc (linear) concept removal methods [see, e.g., references in line 26 of the manuscript], but our paper makes it clear that it is possible to address this problem in conjunction with the demand of linear guardedness.

Furthermore, as a by-product our paper also advances the more general understanding of linear concept erasure methods. Theorem 2 of the manuscript shows that the range of the projection does not impact the predictions of linear models (which includes a logistic regression on last-layer embeddings) that are re-trained after the projection without regularization. Focusing on linearly guarded projections, this means that in this setting, methods like SAL, LEACE and SPLICE only improve on a basic orthogonal projection because of regularization. We consider these theoretical advances of our paper as valuable contributions to the existing literature about (linear) concept erasure.

The implementation details regarding last-layer retraining and language modeling are unclear, especially the calculation of features for language modeling

Can you clarify the closed-form solutions and practical computation of SPLICE in language model experiments?

We thank the reviewer for allowing us to clarify how the projections (SAL, LEACE, SPLICE) are applied to an LLM. We discuss the two cases from the paper.

When applied to the last layer: we apply the projections to the embeddings $\boldsymbol{x} \in \mathbb{R}^d$ before they are mapped to the logits of the LLM over the vocabulary. Let $\mathcal{V}$ be the vocabulary of the LLM, and $\mathbf{A} \in \mathbb{R}^{d \times |\mathcal{V}|}$ the weights of the last layer of the LLM. The logits at the last layer are now given by $h(\boldsymbol{x}) = \boldsymbol{x} \mathbf{P} \mathbf{A}$, where $\mathbf{P} \in \mathbb{R}^{d\times d}$ is the respective projection matrix (SAL, LEACE, or SPLICE).

When applied to the middle layers: suppose we apply the projection to the last $k$ layers out of a total of $L$. Let $\boldsymbol{x}^{(L - k)} \in \mathbb{R}^{d_{L-k}}$ denote the output of layer $L - k$. We then apply the projections to these embeddings as $h(\boldsymbol{x}^{(L - k)}) = \boldsymbol{x}^{(L - k)}\mathbf{P}$. These transformed embeddings $h(\boldsymbol{x}^{(L - k)})$ are then processed by layer $L - k + 1$, after which we again gather the embeddings and fit a projection. This is repeated until we get to the last layer.

Note that the description above applies to the embeddings for a single token. As noted in the appendix, when determining the projection on our training data, we use the embeddings of the last token of a prompt. During test time, we apply the projection to each token. If the embeddings are normalized via the RMSNorm operation (as is the case for the Llama models considered), we apply the projection after the normalization.

We will clarify our approach in an updated version of the manuscript.

How exactly are the embeddings computed for language modeling tasks without retraining LLM parameters?

There is no need to re-train the LLM to calculate the embeddings. We can simply gather the embeddings during a forward pass for a prompt.

Experimental validation is limited, with no worst-group accuracy reported on the widely-used CelebA dataset.

We thank the reviewer for raising this issue. In the original manuscript, we conducted the experiment on the black & white images from the CelebA dataset in section 4.3 in order to visually illustrate the differences between the different projections. Per the suggestion of the reviewer, we have conducted an experiment where we alter the correlation between smiling and whether or not someone is wearing glasses in these images. See our reply to reviewer z2Ty for the results of this experiment, which we will add to the final version of the paper.

Lacks comparison with established group robustness methods like DFR, AFR, and SELF Why were existing robust methods such as DFR, AFR, and SELF not compared in your experiments?

We thank the reviewer for this suggestion. As stated in response to reviewer z2Ty, we emphasize that the goal of our suggested projection (or concept-removal methods in general) is not to have a state-of-the-art performance in terms of worst-group accuracy. It is to achieve fairness guarantees (via linear guardedness) and give control over features used by DNNs.

We have added a comparison with two popular group robustness methods. First, we use deep feature reweighting (DFR, see [1]), using the same training data available to SPLICE. This amounts to the $\mathrm{DFR}_{\mathrm{tr}}$ method introduced in [1]. Second, we apply group-distributional robust optimization (GDRO) to the last layer of the trained DNN.

(Note that we only report here worst-group accuracy and not average accuracy, whose results are in line with worst-group accuracy, due to length constraints of the rebuttal. We will gladly provide the average accuracy tables during the discussion phase, if the reviewer wishes us to.)

=== Waterbirds dataset === Worst-Group Accuracy (%)

=== CelebA dataset === Worst-Group Accuracy (%)

=== Bias in Bios dataset === Worst-Group Accuracy (%)

=== Multilingual dataset === Worst-Group Accuracy (%)

The results show that SPLICE is strongly outperformed by DFR and GDRO for the Waterbirds and CelebA datasets. We emphasize that this is also true for the other concept-removal methods in our baseline, namely LEACE and SAL (see our reply to reviewer z2Ty for experimental verification). For the Bias in Bios dataset, SPLICE performs similar to both DFR and GDRO. For the Multilingual dataset, it outperforms both methods when there is a strong relationship between language and toxicity (0.9). As noted in our reply to reviewer z2Ty, we believe this discrepancy in performance between vision and language classification tasks is an interesting area of future research.

[1] Kirichenko, P., Izmailov, P., and Wilson, A. G. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, ICLR 2023.

What does the comma in Table 1 and following tables mean?

The comma's are used to denote decimal points. We apologize for any confusion created by this convention.