Confounder-Free Continual Learning via Recursive Feature Normalization

Thank you for your thorough review and thoughtful feedback—we respond below to the concerns you have raised.

---

The paper only reports results in medical datasets. It seems that the method should cover various datasets (at least from the title and abstract). The reviewer concerns that the proposed method cannot handle large-scale problems, e.g., ImageNet-1000.

Limitations of the methods on dataset size are definitely an important consideration. In this paper, we work within the framework of medical datasets for real-world setups (wherein issues with confounded learning is very prominent), and non-medical datasets for synthetic setups. We build on prior works from Zhao et al. (2020), Adeli et al. (2020a), Lu et al. (2021), and Vento et al. (2022), who have proposed methods within this framework that we can directly compare to. Several of these datasets are medium-sized, wherein our method successfully learns confounder-free representations, as demonstrated empirically. Larger problems manifest through either larger dataset sizes, longer pre-training, or both. We evaluate the effect of the number of training epochs on confounder-free learning in Suppl. K, where R-MDN’s quick convergence property becomes an advantage. For larger dataset sizes, catastrophic forgetting becomes an issue with task learning. We make preliminary advances in this direction in Exp. 4.2 by integrating R-MDN with existing continual learning frameworks such as LwF and EWC that deal with catastrophic forgetting. Memory replay is another technique that our method could be integrated with. Nevertheless, we acknowledge this issue, and implementing and analyzing plausible solutions is an exciting direction for future work.

---

Please state the difference between your method and the analytic continual learning techniques. Could it be a trivial implementation? If not, please do make a thorough comparison.

Thank you for referencing the ACIL paper. ACIL works within the domain of class-incremental continual learning by breaking down training into two stages: (a) base training via backpropagation, which allows learning of model parameters for the task; and (b) analytic re-alignment, which maps intermediate model features onto the label matrix recursively. This recursive construction permits absolute memorization and effectively mitigates catastrophic forgetting. Our approach is different since we map intermediate features not onto the label space but onto the confounder space so that the residual component of the features can be extracted and passed to downstream layers. This effectively removes the influence of such confounding variables from learned feature representations. Additionally, everything happens in a single-stage end-to-end framework for R-MDN: no network parameters are frozen for the recursive updates. Thus, the network has to learn task-relevant features that are also confounder-free during base training. Overall, the implementation nuance (single-stage learning), the target for the recursive least squares estimator (confounding variables), as well as the input to downstream layers (the residual) differ between our method and ACIL. We will include these details in the final paper.

Thank you for your thorough review and thoughtful feedback—we respond below to the concerns you have raised.

---

No theoretical analysis is provided to justify the proposed method.

A more extensive exploration of the theoretical framework would surely be valuable. However, we would like to highlight that our work follows the definitions of MDN, which provides a closed-form solution to the generalized linear model removing the effect of the confounders (this is a common practice in statistics for controlling for confounders). Since the closed-form solution cannot be applied to newer architectures or continual learning settings (see the section on Related Works) because of its dependence on batch statistics and knowing all confounding values (even for future time points), we proposed a recursive reformulation of MDN. Theoretically, these two formulations are equivalent under specific conditions that we mention in Section 3.1 and Suppl. A. Additionally, we explore synthetic setups based on carefully controlled environments for various variables to identify theoretical maximum accuracies achievable that methods can be validated against. As an addition, we are now constructing a second, slightly more complex, synthetic setup to further justify the effectiveness of the proposed method---specifically, we vary both the position as well as the intensity of the confounding variable over various stages of continual learning. Please refer to our response to Reviewer Shhj above (first paragraph) for more details about the setup and results. Results are presented in Figure 1 and Table 2 at https://docs.google.com/document/d/1VPSuH5B_XvGlGoSlg_jCikB8OPLfyI65oSgACuWxzco/edit?usp=sharing, which we will add to the final paper. We hope that this addition can further support our findings.

---

The performance improvement of the proposed method is not always significant.

Please see our response to Reviewer s7kX (first paragraph) about this point. Across multiple experiments, we observe that our method achieves substantial improvements over prior works when results are analyzed over multiple different metrics, since baseline models might perform well on specific individual metrics.

---

The adopted baselines are pretty old, and recent baselines are not included. Discuss why baselines proposed in recent years (2023, 2024) are not included in experiments.

Baselines for our work are of two different kinds: one that introduces methods for continual learning and two that introduces methods for confounder-free learning. Supervised continual learning is broadly partitioned into three categories: replay-based methods, regularization-based methods, and architecture-based methods. We include a baseline from each of these categories. While this might be non-exhaustive, these methods do not enforce any constraints on learning confounder-free features, so we think that any other recent baseline will perform poorly on our chosen metrics, specifically dcor$^2$. On the other hand, we test our method against the current state-of-the-art confounder-free learning methods such as BR-Net, MDN, and P-MDN, which encompass adversarial-, statistical regression-based, and gradient-based learning. We hope that this provides a comprehensive picture of the landscape.

---

…whether the proposed method can be applied to structured data and integrated with models like graph neural networks?

While the focus of our work in this paper has been on learning over Euclidean data (e.g., images), there are no reasons why the proposed method could not be applied to structured data. We used images because the prior work on confounder-free learning that we discussed in the paper also deals with image input, so this allowed for effective comparison. If the input data is of tabular or structured graph forms, as opposed to images, similar layers of neural network (e.g., fully connected for tabular data or graph convolution for graphs) could be applied to the input. This results in mid-level feature representations. Hence, the R-MDN (and all other prior work) could be simply applied on top of the feature representations for de-confounding.

It is noteworthy that online continual graph learning is an emerging direction (Donghi et al., 2025). This is particularly interesting because we can construct causal graphs that capture the interactions of various observed variables (confounders, mediators, etc.) and use them for learning bias-free representations. Exploring the application of our proposed method, as well as other methods for confounder-free learning in static or continual graph learning frameworks, is an exciting direction for future work.

Thank you for your thorough review and thoughtful feedback—we respond below to the concerns you have raised.

---

Without the theory, the experiments should have explored more synthetic examples…

We agree that additional synthetic setups would strengthen the empirical results we observe. We selected the synthetic setup in the paper, which plays with the distribution of the intensity of main effects and the confounder across different training stages, as it is used extensively in prior work such as by Adeli et al. (2020a), Lu et al. (2021), and Vento et al. (2022). But complex setups seem important. We are now constructing a second synthetic setup which explores the influence of both the position of the confounder as well as the intensities of the main effects and the confounder. More specifically, we generate 1024 32$\times$32 images that are implicitly broken down into 16 8$\times$8 grids. The top left and bottom right grids contain Gaussian kernels of intensity $\sigma_A$, denoting the main effects. The confounder is represented by a Gaussian kernel of intensity $\sigma_B$, whose position varies from the bottom left to top right of the image over 4 different training stages. Both $\sigma_A$ and $\sigma_B$ are sampled from the distribution $\mathcal{U}(3, 5)$ for group 1, and $\mathcal{U}(4, 6)$ for group 2. A fair classifier should remain unaffected by confounder information, irrespective of its position within the image. Such a setup also allows us to compute theoretical maximum accuracy achievable for each training stage that we can validate methods against. Results are presented in Figure 1 and Table 2 at https://docs.google.com/document/d/1VPSuH5B_XvGlGoSlg_jCikB8OPLfyI65oSgACuWxzco/edit?usp=sharing. We will include the same in the final paper.

---

The work does not discuss recent works like https://arxiv.org/pdf/2404.19132

Unsupervised continual learning is an interesting framework to work in. We apologize for not discussing related works in the paper, but rest assured we will cite them in the final paper. In our work, we focus on supervised continual learning, where we train classification networks on labeled data. In UCL, the primary challenge is the lack of labeled confounders. Such a scenario is similar to when certain factors that bias learning are unobserved or hidden. Effectively ensuring plasticity, stability, and cross-task consolidation through the objective function, while making sure that intermediate features can be normalized by removing the influence of confounders (after identification) is an exciting opportunity for future work. In medical contexts, labeled information is often present as metadata with collected samples. Thus, techniques from UCL can be coupled with R-MDN to learn confounder-free representations that additionally exhibit minimized forgetting.

---

It would be useful to understand why P-MDN works better in some settings.

P-MDN minimizes a proxy to the bi-level optimization problem that MDN tries to minimize with a closed-form solution. P-MDN replaces the MDN’s closed-form solution, which requires a batch-level operation (not friendly for many architects and deep learning), with a proxy objective. Successful minimization of the P-MDN proxy objective depends on the careful tuning of the hyperparameter that controls the loss function computed over the features and the metadata. The success of P-MDN is then affected by this choice of hyperparameter as well as how close the proxy objective represents the true objective to be minimized. In our experiments, we extensively tuned this hyperparameter and analyzed the influence from the learning rate and optimizer and found that gradient updates due to this proxy objective led to large variance in performance across runs (different model seeds), showing improvements in some settings and weaker performance in others. In contrast, our approach is better since it is a recursive reformulation of MDN that theoretically approaches the solution from MDN under specific conditions we analyze in Section 3.1.

---

I am not sure that I fully appreciate the reasons behind the linear de-confounding… Can the authors intuitively explain the type of relationships between confounder and input (as a thinking tool for representations) that the proposed method handles?

Similar to the original MDN formulation, R-MDN does not assume that confounders have to linearly influence the input image, because the method does not directly operate on images, but on the feature embeddings, which are non-linear abstractions of the input. Moreover, R-MDN can be applied to feature embeddings at multiple layers, so overall it can effectively remove non-linear confounding between the input and the confounders. However, we admit that neither R-MDN nor MDN has a theoretical guarantee to remove all the non-linear confounding relationships. We will make this discussion more explicit in the final paper.

Thank you for your thorough review and thoughtful feedback - we respond below to the concerns you have raised.

---

… marginal improvements or I was not able to see clear improvement separation

In this work, we introduce a method that allows DNNs to learn confounder-free features during continual learning. A consequence is that the model promotes equitable predictions across class categories. Thus, we must analyze results across multiple metrics, as a baseline model might perform well on specific individual metrics.

For example, in Exp. 4.1, a baseline CNN scores the best on the BWTd metric, but comparably on ACCd and FWTd. Here, lower is better all three metrics. Similarly, a stage-specific MDN model does poorly on ACCd. P-MDN, on the contrary, does well on ACCd and FWTd, but not on BWTd. With our approach, we observe that the method achieves competitive performance across all metrics, significantly outperforming other methods in metrics they perform poorly at (such as 1% improvement on FWTd but 10% on ACCd when compared to stage-specific MDN). In Exp. 4.3, while our approach leads to a modest 2% improvement on the dcor^2 (CN) metric, it has a significant 15% improvement on dcor^2 (AD) when compared to P-MDN. P-MDN is biased towards the AD category, while our approach has similar ranges of dcor^2 across all categories. Furthermore, significant improvements can be seen in Figure 3 (as evidenced by the five straight lines) and Figure 6 (as evidenced by the identification of the cerebellum).

We believe that these, along with other results from the paper, provide clear evidence of substantial improvement over prior works for the claims we make, as analyzed through a combination of qualitative and quantitative evaluations.

---

I would have expected use of the methodology at 2 or three different NN architectures …

Please note that we tested our method on both CNN and ViT with a mix of datasets and different architectural setups; though, we agree that evaluating the method on more DNN architectures would add more value to the paper. We selected a CNN and ViT model backbone because they lend themselves well to the task of image categorization, which is what we focus on in this paper. Synthetic datasets (as presented in Exp. 4.1 and Suppl. C) are evaluated using a CNN backbone to enable direct comparison to prior de-biasing methods in this space such as MDN, P-MDN, and BR-Net, all of which adopt a similar backbone and model size. This helps us understand how well our method performs. Additionally, we include an experiment on the ABCD dataset (Exp. 4.4) using a 3D CNN model to extend its application to a real-world setup. Finally, we evaluate a 2D ViT (Exp. 4.2) and a 3D ViT (Exp. 4.3) on different real-world setups to analyze their performance. To the best of our knowledge, this is the first time transformers have been used for confounder-free feature learning in this context. We hope that the demonstration of both 2D and 3D CNNs and ViTs will add value to our claim. To complete the loop and further strengthen our claim, we are now running an experiment with a ViT as a model backbone on the synthetic dataset. Results are presented in Table 1 at https://docs.google.com/document/d/1VPSuH5B_XvGlGoSlg_jCikB8OPLfyI65oSgACuWxzco/edit?usp=sharing.

---

On the metrics that they introduce it looks like the proposed approach does not have favorable scores.

Across the different metrics, our method favorably improves scores on dcor^2, which quantifies the influence of the confounder on learned features. Improvement is seen through a decrease in dcor^2 in Tables 2, 3, and 4, often across all class categories to reduce inequitable predictions. On the accuracy metric, scores sometimes modestly drop to reflect the model not learning using shortcuts by looking at confounder information for prediction. Only for Exp. 4.2 we see that our approach leads to a slightly weaker score on FWT, possibly due to the corresponding influence from a slightly higher accuracy due to utilizing confounder information, which we seek to avoid. Since we can design carefully controlled environments through synthetic datasets, we can analyze the true theoretical accuracy, BWT, and FWT a fair model should get, and we see favorable scores across each in Exp. 4.1.

---

Have the authors checked the performance over different NN architectures under one setup…

Yes, the placement of the R-MDN layer is an important hyperparameter that must be tuned to the specific DNN architecture being used. We analyze various placements across different architectures through Exp. 4.2 and Suppl. H. Prior work (Lu et al., 2021) has shown that applying MDN after both convolutional and fully connected layers results in the best performance in CNNs. We see similar performance improvement in CNNs with R-MDN in Suppl. H. We construct a similar setup for ViTs in Exp. 4.2 and find that adding R-MDN only after the pre-logits layer provides the best performance.