Learn and Ensemble Bridge Adapters for Multi-domain Task Incremental Learning

Thank you for your positive comments (novel formulation of the knowledge transfer problem, well-designed). All your questions and weeknesses have been answered below.

Q1: Could the authors elaborate on the trade-off between the flexibility of generative replay and the potential performance impact from distributional bias in the generated samples? How sensitive is the framework to the quality of the generator?

A1: For the question you asked, it should refer to some noises when using generated samples, which might result into potential performance impact although generative replay is flexible. For this, we mitigate this risk from two below aspects: 1) the use of a high-quality pretrained diffusion model (DDPM) [1], which achieves strong fidelity and diversity (e.g., FID 3.17, IS 9.46 on CIFAR), and 2) our LEBA focuses on semantic-level alignment using adapter responses to high-level features, rather than relying on low-level pixel fidelity. This design choice enhances robustness and ensures that minor generation artifacts do not adversely impact the alignment or learning. To evaluate the sensitivity of our LEBA to the quality of the generator, we conducted experiments using different versions of the diffusion model, as shown in below table. This indicates that our LEBA exhibits low sensitivity to the choice of generator.

Performance with different diffusion generator

[1] Denoising Diffusion Probabilistic Models NIPS 2020.

Q2: If historical adapters are suboptimal, the PKE module mitigates this risk by adaptively weighting and integrating multiple prior adapters, reducing reliance on any single flawed representation and enhancing overall robustness.

A2: It is indeed common in incremental learning that historical adapters may be suboptimal, as no approach can fully preserve all prior knowledge perfectly over time. To address this, our framework incorporates the PKE module, which adaptively integrates $K$ historical adapters for each replay sample using learnable weights. This mechanism enables the model to prioritize more robust and relevant representations, rather than relying on any single adapter. In Table 3 of the manuscript, incorporating the PKE module which includes the use of these generated replay samples—results in a good performance gain across all evaluation metrics, demonstrating its effectiveness in mitigating the impact of imperfect historical adapters through selective and weighted knowledge integration.

Q3: While KL-divergence looks a reasonable choice, could the authors justify that this single metric is sufficient to capture the multi-faceted concept of an adapter's "usefulness"? Have alternative, more sophisticated selection mechanisms (e.g., based on diversity) been considered?

A3: We chose KL-divergence as our primary metric because it effectively captures the task-specific similarity between adapter outputs, aligning well with our goal of preserving semantically relevant knowledge. To further validate this choice, we also experimented with alternative metrics such as cosine distance and mutual information for adapter selection. As shown in the below table, these methods yield comparable performance across all four evaluation metrics, suggesting that KL-divergence is sufficiently effective in capturing the relevant aspects of an adapter’s “usefulness” in our setting.

Comparison of adapter selection strategies on four evaluation metrics

Q4: The fixed-size adapter pool effectively ensures stability as the number of total tasks grows. However, how does this stability mechanism interact with the replay batch size? Have the authors tested the framework's robustness with larger replay batches, and does a larger number of simultaneous alignment targets introduce new optimization challenges? A key hyperparameter, the number of replayed samples per-training step, is not explicitly mentioned, which hinders exact reproducibility.

A4: Firstly, we confirm that the fixed-size adapter pool provides stability as tasks accumulate, and we have evaluated the framework under varying replay batch sizes. To evaluate the interaction between this mechanism and replay batch size, we conducted experiments under different replay settings: 500$\times$16, 500$\times$32, and 500$\times$64 (i.e., generating 16, 32, or 64 replay samples per iteration over 500 iterations for each task). The results are shown in below table. As shown, performance gains saturate beyond the 500$\times$32 setting, indicating that LEBA is robust to variations in replay size. To balance performance and computational efficiency, we adopt the 500$\times$32 configuration as the default throughout our experiments.

Secondly, in our LEBA, increasing the number of simultaneous alignment targets does not introduce notable optimization challenges, as the alignment operates at the semantic feature level with lightweight adapter outputs rather than raw high-dimensional data. As shown in below table, performance remains stable even when the number of replay samples increases, suggesting that the optimization process is not adversely affected.

Performance on varying numbers of replayed samples

Q5: The authors state that the code will be released upon acceptance. Could they confirm if the release will include the necessary scripts to faithfully reproduce the main results?

A5: We commit to releasing the corresponding code and script resources that can reproduce the main results if the manuscript is accepted.

Q6: The framework relies on samples from a pre-trained diffusion model. These models are known to have their own biases and do not perfectly replicate the true data distribution, which could potentially mislead the alignment process.

A6: We would like to clarify that the replay samples used in the PKE module are generated by a pretrained diffusion model (DDPM) [1], which has demonstrated strong generative quality. For example, on the CIFAR dataset, DDPM achieves FID 3.17 and IS 9.46, indicating high fidelity and discriminability relative to the original data distribution. While generative models may introduce certain biases, our LEBA relies more on the semantic alignment between adapter representations than on pixel-level realism. The consistent performance gains observed in Table 3 of the manuscript suggest that the generated samples are sufficiently informative for stable knowledge alignment and replay. As part of future work, we plan to explore post-processing strategies, or employ more advanced generative models to further enhance the fidelity and diversity of replay data.

[1] Denoising Diffusion Probabilistic Models NIPS 2020.

Q7: The framework lacks proactive guidance to generate samples optimal for mitigating forgetting.

A7: We have carefully reviewed the LEBA framework and concur that, as you noted, this aspect presents an opportunity for further enhancement. We appreciate your insightful suggestion and regard it as a direction for future improvements.

Dear reviewer SAmD,

Thank you sincerely for the time and effort dedicated to reviewing our manuscript and providing constructive feedback. We have detailed our responses to your queries and trust they adequately address your points. We remain fully available to discuss any further questions or clarifications you may have. We would greatly appreciate it if you could update your rating by synthesizing both the reviewers' comments and our corresponding replies.

Dear Reviewer,

This is a friendly reminder to check the authors' rebuttal and adjust your rating if necessary. Thanks for your contributions to the NeurIPS reviewing process.

Thanks,

Your AC

Dear reviewer SAmD,

We sincerely appreciate your recognition of our rebuttal efforts and the time taken to re-evaluate our work. We are delighted to learn that most of your concerns have been addressed, and you are inclined to raise your score.

Regarding the concern about the Gaussian Bridge Path assumption, while we acknowledge its approximate nature, it is theoretically grounded and has been widely adopted in generative model method [1,2].

Thank you for your time and proposing these valuable comments/suggestions again!

Authors

[1] Diffusion schrödinger bridge matching NIPS 2023.

[2] I$^2$SB: Image-to-Image Schrödinger Bridge ICML 2023.

Thanks for your positive comments on the well-organized structure, the extent of novelty, and the comprehensive experiments. The point-to-point answers are provided below.

Q1: The claim of being “independent of task order” is overstated, as replay relies on accumulated class labels, which are inherently tied to task sequence, making the process still temporally dependent.

A1: Thank you for raising this point. We would like to clarify our intended meaning behind the claim that our method is "independent of task-learning sequence". This statement refers specifically to the replay process in LEBA, which differs from standard replay-based methods that typically require explicit task identifiers or must associate each replayed sample with its original task or dataset. In contrast, LEBA generates replay samples based on an accumulated semantic set—i.e., the union of previously encountered class labels—without requiring knowledge of which specific task or dataset the sample originally belonged to. This task-agnostic design is the motivation for constructing a semantic concept set, and it enables flexible and unified replay across all previously learned knowledge, thus reducing the system’s reliance on task sequence during knowledge consolidation. We will revise this in the final version to better reflect this distinction and avoid overstating the claim.

Q2: Relying solely on class labels to condition the diffusion model assumes they suffice to recover representative samples, yet no evidence is provided to validate their fidelity to original distributions.

A2: Thank you for raising this point. We would like to clarify that the generated samples used in the PKE module are produced by a diffusion-based model (DDPM) [1], which has demonstrated strong generative quality—achieving FID 3.17 and IS 9.46 on CIFAR dataset, indicating sufficient fidelity and discriminability relative to the original data distribution. In the Table 3 of the manuscript, incorporating the PKE module which includes the use of these generated replay samples—results in a good performance gain across all evaluation metrics, confirming that the generated samples are informative and beneficial to the continual learning process. As part of future work, we plan to explore post-processing strategies, or employ more advanced generative models to further enhance the fidelity and diversity of replay data.

[1] Denoising Diffusion Probabilistic Models NIPS 2020.

Q3: The adapter pool size is fixed and increasing it shows negligible gain, suggesting that either the replayed information is not diverse, or the adapter selection mechanism is too coarse.

A3: We would like to clarify that the negligible gain from increasing the adapter pool size $K$ is not attributed to a lack of diversity in the replayed information or a coarse adapter selection mechanism. First, our replay samples are generated via a diffusion-based generator conditioned on the accumulated semantic set, ensuring high semantic diversity across tasks without relying on raw data storage. Second, adapter selection is guided by KL divergence, which replaces only semantically redundant adapters and preserves a compact yet diverse pool. As illustrated in Figure 3 of the manuscript, the model performance tends to saturate when the adapter pool size reaches $K = 2$. As a result, expanding the adapter pool offers negligible performance gains. Specifically, LEBA employs learnable weights to assign importance scores to adapters based on their relevance to current replay samples. Consequently, larger values of $K$ lead to diminishing returns in both representational diversity and performance gains. Considering the minimal improvement beyond $K=2$ and the trade-off between memory overhead and model complexity, we adopt $K=2$ as the default setting to ensure efficient memory utilization.

Q4: The CBA module, while well-formulated, does not introduce novel theoretical contributions beyond adapting existing SB-based score matching to the adapter alignment setting.

A4: While the CBA module builds on established principles of Schrödinger Bridge (SB)-based score matching, its novelty lies in adapting and extending this framework to the unique setting of continual adapter alignment in multi-domain task incremental learning (MTIL). Unlike prior work that applies SB to generative modeling, we recontextualize it for cross-domain feature alignment across sequential tasks, using replayed samples and adapter responses as the distributions to be bridged. To our knowledge, this is the first work to apply SB-based methods for aligning modular representations in continual learning, offering a new perspective on stable knowledge transfer under task shifts.

Q5: The writing quality in the Related Work section could be improved for clarity and logical flow.

A5: Thank you for the comment. We will refine the Related Work in the next version to improve its writing quality.

Dear reviewer VGXi,

Thanks for your timely reply! We would like to know whether our responses have addressed your concerns. We hope our rebuttal could address your concerns. We would appreciate it if you could re-evaluate our submission and we are looking forward to discussions if you have any other concerns. Thank you so much again!

Authors

Thanks for your positive comments (an interesting and important research, a new way of replaying experiences). All your questions have been answered below.

Q1: Why does increasing the adapter pool size beyond two not yield further performance gains?

A1: To evaluate the effect of the adapter pool size, we conduct ablation studies varying $K$. As illustrated in Figure 3 of the manuscript, the model performance tends to saturate when the adapter pool size reaches $K = 2$. Consequently, adding more adapters introduces fluctuations.Our LEBA uses learnable weights to assign importance to each adapter based on its relevance to the current replayed sample. As a result, a larger $K$ provides diminishing returns in terms of both representational diversity and performance gains. Given the negligible performance improvement beyond $K=2$, and considering the trade-off between memory overhead and model complexity, we select $K=2$ as the default configuration to ensure efficient memory usage.

Q2: How many replay samples do you generate?

A2: We generate 32 replay samples per iteration over 500 iterations for each task. To further investigate the impact of replay quantity, we conduct additional experiments by generating training samples over 500 iterations with either 16 or 64 samples per iteration. The results are shown in the below table.To balance performance and computational cost, our LEBA adopts the 500$\times$32 replay setting, which achieves competitive results while maintaining efficiency.

Performance on varying numbers of replayed samples

Q3: How is the "semantic set" actually computed? Is it based solely on class labels?

A3: The semantic set refers to the collection of all classes encountered throughout the incremental learning process, and is defined solely based on the class labels. In the manuscript, we elaborate on the definition of the semantic set as " The semantic set $\mathcal{C}^t \coloneqq \{c_j^t\}_{j=1}^{M_t}$ describes certain semantic information (e.g., class information $y_i^t$), with $M_t$ denoting the number of distinct class names."

Q4: In line 193, what metric is used to identify the "least representative" sample? Under what conditions do you replace an existing adapter?

A4: Please allow us to offer a further clarification. In line 193, the entity being evaluated as the "least representative" is not a training sample, but rather a task-specific adapter from the dynamic adapter pool. To determine whether a newly learned adapter should replace an existing one, we compute the KL-divergence between the current adapter and each adapter in the pool, based on their responses to the current task samples. A fixed threshold is employed to guide the replacement decision: if the divergence is below the threshold, the adapter pool remains unchanged; otherwise, the newly learned adapter replaces the most divergent one, thereby updating the pool accordingly.

Q5: How were the hyperparameters selected?

A5: We present the hyperparameter studies and corresponding rationale in the supplementary material (Lines #519–#523). The same hyperparameter settings are used for the other benchmarks.

Sensitivity to balance factor $\gamma$

Sensitivity to balance factor $\beta$

The best performance is achieved when $\gamma = 0.1$ and $\beta = 0.4$. Setting $\gamma$ too high (e.g., $\gamma > 0.1$) causes the CBA loss to dominate training, hindering the acquisition of new knowledge. Likewise, a large $\beta$ (e.g., $\beta > 0.4$) leads to overly smoothed integration weights, which diminish task-specific distinctions and negatively impact performance.

Q6: Do you use the task ID at test time? Is the method applicable to multi-domain class-incremental learning?

A6: Unlike class-incremental settings, task-incremental learning in MTIL typically assumes that the task ID is known at test time that follows the previous work [1,2]. Thus we also strictly follow the previous protocol in task-incremental learning [3]. Currently, our LEBA is not directly applicable to multi-domain class-incremental learning, as it is tailored for multi-domain task-incremental settings where task identities are known during training and inference, with each adapter aligned to a task-level distribution. In contrast, class-incremental learning requires unified classification without task labels, fundamentally altering the problem. Adapting our LEBA would require merging or aligning adapters across domains and classes, posing challenges in representation overlap and compatibility. We will explore such adaptations in future work.

Q7: How many random seeds were used for the experiments, and what is the standard deviation of the results?

A7: Following your suggestion, we have conducted experiments with five different random seeds and computed the standard deviations. The results are shown in the below table. The low standard deviations across all metrics demonstrate the robustness and stability of LEBA under randomized conditions.

Performance and standard deviation of LEBA across four metrics

Q8: Why does PKE with generative replay yield only modest gains in Table 3?

A8: We would like to clarify that the PKE module actually includes the replay of generated data (as described in Line #201). While the numerical improvements in Table3 may appear modest, the contribution of PKE lies not in delivering large standalone gains, but in enhancing robustness and consistency. Specifically, PKE adaptively integrates previously learned adapters during replay, facilitating stable knowledge reuse and sample-level alignment. These benefits are reflected in the consistent improvements observed across all four evaluation metrics.

Q9: What are the exact benefits that PKE offers over standard replay for addressing task-order sensitivity?

A9: Compared to standard data replay, the PKE module alleviates task-order sensitivity through two strategies.
First, it introduces task-order independence by leveraging a dynamic adapter pool with learnable weights, enabling the adaptive integration of prior knowledge regardless of task sequence. Second, it supports semantic-level replay, where generated samples are conditioned on accumulated semantic concepts rather than stored per-task instances, resulting in more balanced and representative replay across tasks.

Q10: For heterogeneous learning, is it better to learn domain-specific knowledge or enforce cross-domain alignment?

A10: The issue you mentioned remains an open research challenge. Different methodological paradigms offer distinct advantages: learning domain-specific knowledge emphasizes specialization, while enforcing cross-domain alignment promotes generalization. Our work falls into the latter category—focusing on task alignment across domains via a shared adapter integration framework. Although this may introduce some suboptimality when domain gaps are large, we mitigate this risk through adaptive adapter selection and semantic-aware replay, which promote stable transfer while preserving task-relevant knowledge.

Q11: The robustness claim would be stronger with results on more than two task orders.

A11: According to your suggestion, we randomized the task order and conducted two independent experiments based on the resulting orders. The corresponding results are presented in blow table, with the task orders specified as: (a) [CIFAR100, DTD, Aircraft, Flowers, Food, StanfordCars, MNIST, EuroSAT, SUN397, OxfordPet, Caltech101] and (b) [EuroSAT, OxfordPet, SUN397, DTD, CIFAR100, Food, StanfordCars, MNIST, Caltech101, Flowers, Aircraft].

Comparison of methods across four evaluation metrics for task order (a)

Comparison of methods across four evaluation metrics for task order (b)

Experimental results demonstrate that the proposed LEBA consistently achieves superior performance compared to state-of-the-art methods across different randomized task orders, indicating its robustness to task sequence variations.

Q12: In line 309, the paragraph title is "Computational Cost," but it discusses memory costs instead.

A12: Sorry for the typo Figure 4 of our manuscript. It should be corrected to "GPU Memory (MiB)". Figure 4 on the manuscript reports GPU memory consumption, and we will revise this error accordingly in the next version. Furthermore, Table 8 of the supplementary material presents a comparison of training time, as shown in the table below, where our LEBA demonstrates slightly better efficiency compared to current state-of-the-art methods.

Comparison of memory usage and training time

[1] Preventing zero-shot transfer degradation in continual learning of vision-language model ICCV 2023.

[2] Boosting continual learning of vision-language models via mixture-of-experts adapters CVPR 2024.

[3] Enhancing knowledge transfer for task incremental learning with data-free subnetwork NIPS 2023.

Dear reviewer wSj1,

We thank you sincerely for your time and effort in reviewing our manuscript and providing valuable suggestions. We have provided detailed responses to your questions and hope that they adequately address your concerns. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you. We would greatly appreciate it if you would update the rating by synthesizing other reviewers' comments as well as our responses.

Thank you for your positive feedback on the novelty and writing quality of LEBA. For your concerns, we make the response below.

Q1: The main weakness that I observe is in the experiments, where two possible task orders are used: alphabetically and random. Although I appreciate the consideration of multiple orders, I think these are insufficient to mitigate the effect that a particular task order may have on the results. I would expect to see at least a few (possibly random) orders, and average results of the metrics across these.

A1: In accordance with your suggestion, we randomized the task order and conducted two independent experiments based on the resulting orders. The corresponding results are presented in blow table, with the task orders specified as: (a) [CIFAR100, DTD, Aircraft, Flowers, Food, StanfordCars, MNIST, EuroSAT, SUN397, OxfordPet, Caltech101] and (b) [EuroSAT, OxfordPet, SUN397, DTD, CIFAR100, Food, StanfordCars, MNIST, Caltech101, Flowers, Aircraft]. Experimental results demonstrate that the proposed LEBA consistently achieves superior performance compared to state-of-the-art methods across different randomized task orders, indicating its robustness to task sequence variations.

Comparison of methods across four evaluation metrics for task order (a)

Comparison of methods across four evaluation metrics for task order (b)

Furthermore, by combining these results with those obtained under the randomized task order (Order II) presented in our manuscript, we compute the average performance across all four experimental settings, as shown below table. Experimental results demonstrate that our proposed LEBA consistently outperforms current state-of-the-art methods under randomized task orders.

Average performance across three randomized task orders

[1] Preventing zero-shot transfer degradation in continual learning of vision-language model ICCV 2023.

[2] Boosting continual learning of vision-language models via mixture-of-experts adapters CVPR 2024.

Q2: Another weakness that I observe is in the description of the pool of adapters, where the word “ensemble” is repeatedly used. Based on the design proposed for this method as described in section 3.4, it does not entail an “ensemble” as known, but rather it’s merely a pool that gets updated depending on the similarity of new adapters with previous ones. What do you actually mean when you use the word "ensemble" in the context of the pool of adapters Pk?

A2: We appreciate your comment and would like to provide additional clarification regarding the ensemble mechanism. Ensemble learning typically refers to the process of combining multiple models to improve robustness and generalization. In our LEBA, the ensembling process doesn't happen in the construction of the adapter pool itself, but through the adaptive integration of prior knowledge as defined in Eqn.(9) and Eqn.(10) of the manuscript. Specifically, we introduce a set of learnable parameters that assign sample-specific weights to each adapter in the pool. These weights are used to dynamically combine the outputs of historical adapters based on their relevance to the current task. This mechanism allows the model to selectively integrate previously learned knowledge in a principled and flexible manner, which aligns with the essence of ensemble learning. We will also make the distinction clearer between the adapter pool construction and the ensemble operation performed over it.

Dear Reviewer,

This is a friendly reminder to check the authors' rebuttal and adjust your rating if necessary. Thanks for your contributions to the NeurIPS reviewing process.

Thanks,

Your AC