Thank you for your review and affirmation of our work. We'll answer your questions one by one below.

Response to Weakness 1: About the "Approximation Error"

Regarding your concern about our method relying on task vectors instead of actual data or gradients, we would like to provide a detailed explanation as follows:

First, why don't we use real data or true gradients? The continual model merging (CMM) setting [1], like traditional model merging [3,4], assumes that access to the original training data is unavailable. Without access to the data, it is impossible to compute true gradients, so we must approximate them using task vectors to enable effective model merging.

Second, regarding the ability of task vectors to approximate true gradients and the data subspace, we provide a detailed theoretical analysis in Section 3.2.2. Specifically, the approximation error of task vectors to the input data in DOP can be theoretically bounded as follows: Let the task vector for a linear layer be $\tau_l$, and the input samples be $\{x_n\}\_{n=1}^{N}$, with each sample undergoing $T$ optimization iterations. Suppose the gradient norm of the loss function with respect to $\tau_l \cdot x_n$ is upper-bounded by $C_l$. Then, there exists a constant $\Phi_l$ such that: $\| \tau_l - \sum_{n=1}^{N} \alpha_{n,l} (x\_n^\top) \|\_2 \leq \Phi\_l \cdot \left( \sum_{t=1}^{T} \sum_{i=t}^{T} \eta\_t \eta_i \right)$, where $\eta_t$ denotes the learning rate at iteration $t$, $\alpha_{n,l}$ are reconstruction coefficients, and $\| \cdot \|_2$ is the Euclidean norm. Due to space limitations, we will provide the detailed proof of this approximation bound in the revised version.

Furthermore, from the perspective of empirical error, our experiments cover four different complex architectures—including three ViT models of varying sizes for vision tasks. As summarized in the table below, DOP consistently achieves the best or near-best performance across different architectures and task scales, which strongly validates the effectiveness of using task vectors as gradient approximations in our method.

In summary, although task vectors are an approximation of the true gradients, in scenarios where training data is inaccessible, they provide a reasonable and efficient solution for model merging.

Response to Weakness 2: About the "Nonlinear Modules"

We agree with your concerns and have supplemented our analysis and experiments on both linear and nonlinear layers to address them.

First, while modern neural network architectures do contain certain nonlinear components, in mainstream architectures such as Transformers, linear layers (e.g., self-attention and feed-forward networks) overwhelmingly dominate the parameter space. As shown in Table 1, the proportion of linear layer parameters exceeds 90% in Flan-T5-Base and all three ViT architectures, with ViT-B/16 reaching as high as 99%. This indicates that, although nonlinear modules exist, linear layers constitute the vast majority of parameters in these architectures.

Furthermore, to directly address your concerns about the impact of nonlinear layers (such as bias parameters and parameters of normalization layers), we conducted experiments on various nonlinear layer merging strategies and evaluated their effects on the final model performance. Specifically, we explored several common approaches, including directly using the nonlinear weights from the pretrained model, applying weight averaging, and employing task arithmetic to merge the nonlinear weights of expert models. As shown in Table 2, the performance differences among these three strategies are minimal (79.2, 78.3, and 78.9, respectively). This demonstrates that, in mainstream real-world architectures, the impact of nonlinear layers on the final merged model performance is limited, primarily because nonlinear layers account for only a small fraction of the total parameters, while the overall model performance is mainly determined by the linear layers.

In summary, although nonlinear modules play an important role in modern neural networks, the dominance of linear layers in the continual model merging setting means that the merging of nonlinear modules has a relatively limited effect on overall performance.

Table 1

Table 2

Response to Weakness 3: About the "Continual Model Merging Setup"

Regarding your concern about the storage overhead of task vectors, we acknowledge that in certain cases (e.g., when model size is small or the number of tasks is limited), the cost of storing task vectors may not be significant. However, the goal of this work is to provide a continual model merging (CMM) solution for memory-constrained environments where access to data is unavailable. This CMM setting is not introduced for the first time in our paper; prior studies have also explored similar scenarios [1,2].

Furthermore, traditional model merging aims to combine multiple expert models into a single unified model, thereby reducing storage overhead and enabling knowledge transfer across tasks—a trend that has gained significant traction in recent research [3,4,5,6]. Compared to conventional model merging, the CMM Settings in this work place greater emphasis on scenarios where the model is gradually available in chronological order. Specifically, we introduce additional constraints to make this setting more realistic:

• (i) Traditional model merging assumes all models are available simultaneously, whereas in practical scenarios, models may become available sequentially. Thus, CMM is more aligned with real-world applications.
• (ii) Traditional model merging requires loading all models into memory, while CMM only needs to load the pretrained model, the new model, and the merged model at each step, further reducing resource requirements during merging and enhancing practicality.

Overall, we believe that avoiding the need to store all historical task vectors will greatly improve the scalability and efficiency of the model, especially as the number of tasks or model parameters increases substantially. In addition, given the broad applicability of model merging, further optimizing its use in continual scenarios is of significant practical value.

Response to Weakness 4: About "Novelty of the Method"

Thank you for your attention to the novelty of our work. At first glance, our method shares some similarities with OPCM, as both leverage projection operations of task vectors to address the continual model merging (CMM) problem. However, the contribution of our proposed DOP method goes far beyond simply extending OPCM's unidirectional projection to a bidirectional one. Specifically, our approach offers several significant advantages:

• Theoretical Analysis: DOP provides a deeper theoretical framework by analyzing the relationships among data, gradients, and task vectors, and elucidates their interactions in CMM. Through this analysis, DOP explicitly defines two key objectives—stability and plasticity—from the perspective of orthogonal projection. In contrast, OPCM does not formally define these objectives nor offer a systematic theoretical analysis.
• Stability-Plasticity Trade-off: DOP simultaneously optimizes both stability and plasticity by formulating the problem as a multi-objective optimization, thus achieving a better balance between adapting to new tasks (plasticity) and retaining knowledge from previous tasks (stability). OPCM, on the other hand, focuses solely on stability—aiming to minimize interference from new tasks to old knowledge—while overlooking the crucial aspect of plasticity. Notably, OPCM merely performs a matrix projection operation without truly balancing stability and plasticity through an explicit optimization objective, whereas our method defines a loss function to jointly optimize both goals.
• Empirical Performance: Experimental results across multiple vision and language tasks, four model architectures, and three task scales consistently demonstrate that DOP significantly outperforms OPCM, achieving higher average performance, less forgetting, and stronger task adaptability.

In summary, the DOP method surpasses OPCM in terms of theoretical depth, methodological soundness, and empirical performance.

Reference

[1] Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging. Arxiv, 2025.

[2] How to Merge Your Multimodal Models Over Time?. CVPR, 2025.

[3] Editing Models with Task Arithmetic. ICLR, 2023.

[4] TIES-Merging: Resolving Interference When Merging Models. NeurIPS, 2023.

[5] Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025.

[6] Learning from models beyond fine-tuning. Nature Machine Intelligence, 2025.

Thank you for your detailed clarifications. However, I still find it difficult to fully understand the practical motivation and significance of the Continual Model Merging (CMM) setting proposed in your work.

You mention that CMM is designed for memory-constrained environments where access to data is unavailable. However, this justification remains unconvincing to me for two reasons. First, the merging process itself can often be performed offline, where memory resources are generally not a strict limitation. Even if the final model is to be deployed in a memory-constrained environment, the merging step can still happen separately in a server or cloud infrastructure with sufficient resources. Second, while I acknowledge there are cases where task data might not be stored or shared, it seems entirely feasible—and in practice inexpensive—to store the task vectors themselves, which are small, model-agnostic, and generally non-sensitive. As a result, the necessity and impact of avoiding historical task vectors, which is central to your method, still feels insufficiently justified. I believe this is a critical issue: it fundamentally affects whether the proposed work is realistic and valuable, and thus whether it supports a higher score or should lead to a lower one.

Regarding novelty (Weakness 4), I appreciate the effort to clarify the technical contributions beyond OPCM. However, your reply still appears to focus more on the motivation and experimental improvements rather than on concrete algorithmic or theoretical innovation. The main technical difference described is still seems incremental rather than substantially novel.

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

Response to Weakness 1 & Question 2: About the "More Baselines"

Thank you for your suggestion. We have added MagMax[1] as a new baseline and evaluated it across three architectures (ViT-B/32, ViT-B/16, and ViT-L/14) and three task scales (8, 14, and 20 tasks). As shown in the table below, we observe that MagMax outperforms the three baselines—SWA, C. Task Arithmetic, and C. Ties-Merging—but still lags behind OPCM, which is specifically designed for continual model merging, and our proposed DOP. Overall, our DOP consistently achieves the best performance.

[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV 2024.

Response to Weakness 2: About the "Why Traditional Model Merging Methods Fail"

Thank you very much for your insightful question. We explain why traditional model merging methods fail in the continual model merging (CMM) setting from two perspectives:

First, traditional methods (such as Ties-Merging) assume simultaneous access to all models and rely on consistent parameter sign alignment across models. However, these approaches face significant limitations in the CMM setting, where tasks arrive sequentially and, at each merging step, only the current model and the previously merged model are accessible. As a result, methods like Ties-Merging cannot effectively handle scenarios with only local information available at each step, leading to inaccurate sign alignment and degraded merged model quality.

Another critical challenge is that continual model merging does not satisfy the commutative property [1]:

where $\mathrm{CMM}(\cdot)$ denotes the continual merging algorithm. This formula highlights that, in continual model merging, the order of merging affects the final result, which explains the performance degradation of traditional model merging methods in this scenario.

[1] "Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging", Arxiv, 2025.

Response to Weakness 3: About the "Training of DOP Algorithm"

Regarding the concern that DOP requires gradient-based optimization, we would like to clarify the following points. Although DOP does involve gradient computation during optimization, it offers several key advantages that keep the extra training cost low and ensure high computational efficiency:

• No Forward Pass Required: DOP does not rely on the original task data. Instead, it directly optimizes based on the parameter differences between models. This means that, during merging, there is no need to perform traditional forward passes to compute losses or update gradients.
• No Backpropagation Through the Whole Network: DOP performs optimization layer by layer, i.e., it independently optimizes each linear layer rather than jointly optimizing the entire model. This approach eliminates the need for complex backpropagation through the entire network, resulting in significant savings in both time and GPU memory.

In summary, while DOP requires gradient computation, it is highly efficient, does not depend on original data, and avoids the computational burden of full backpropagation by optimizing each layer independently. As shown in the table below, the optimization time for merging each linear layer is very short, typically only 1 to 3 seconds.

Response to Question 1: About the "Projection in Scenarios Where the Old Model Helps the New Model"

In our DOP method, although we minimize the parameter differences between the old and new models to reduce interference from previous tasks, we do not completely disregard the beneficial knowledge contained in the old model (in practice, $\Delta W_{n}^{*}X_n$ does not converge perfectly to zero, but only approaches this extreme value). More importantly, when computing gradients and performing optimization, DOP adjusts the parameter differences via orthogonal projection, thereby avoiding excessive interference with the new task. In this process, if some knowledge from the old model aligns with the objectives of the new task, DOP will retain this information, as it resides in a shared subspace and is thus unaffected by the projection.

Response to Question 3: About the "Analysis of Variants of Projection Space"

Thank you very much for your suggestion. We analyzed the performance of the following two variants on the ViT-B/32 architecture:

• (i) DOP (w/o $\Sigma$) refers to directly removing the singular values $\Sigma$. We observed that its performance drops significantly (although it still outperforms the baseline method SWA), as it discards the projection penalty along the principal directions (those corresponding to larger singular values).
• (ii) DOP ($\sqrt{\Sigma}$) follows your suggestion of applying a square root to the singular values, which still preserves the relative importance across different directions. As shown in the table below, we observe that the results using $\sqrt{\Sigma}$ are essentially the same as those using $\Sigma$ (78.2% vs. 78.3%).

In summary, retaining the magnitude of the singular values as the relative penalty strength for each projection direction is important.

Response to Question 4: About the "Variation of Coefficient Alpha"

In our method, the coefficients $\alpha = \{\alpha_{s,k}, \alpha_{p,k}\}$ are adaptively updated across different layers, parameter modules, and merging stages. To further analyze the dynamic evolution of $\alpha$ during optimization, we visualize the values of $\{\alpha_{s,k}, \alpha_{p,k}\}$ at various training steps.

For illustration, we randomly select two linear modules from two different stages, and record the values of $\alpha$ at different training steps ($k=0,25,\ldots,200$). As shown in the following tables, we consistently observe that the stability coefficient $\alpha_{s,k}$ gradually decreases from the initial value of 0.8, while the plasticity coefficient $\alpha_{p,k}$ increases from 0.2 to around 0.3. This trend reflects that, in the early stages, the model tends to retain more knowledge from the merged model, and as training progresses, it gradually incorporates new knowledge, resulting in a decrease in $\alpha_{s,k}$ and an increase in $\alpha_{p,k}$.

Stage 2 - vision_model.encoder.layers.11.self_attn.k_proj:

Stage 8 - vision_model.encoder.layers.11.self_attn.v_proj:

Response to Question 5: About "More Metrics on Computational Efficiency"

Thank you for your suggestion. To provide a more comprehensive evaluation, we have included the peak GPU memory usage (Peak GPU Memory Allocated) during both the model merging and inference stages on the ViT-B/32 architecture. As shown in the table below, during the merging stage, our method requires slightly more GPU memory than OPCM due to the optimization performed on each linear layer. However, the overall GPU consumption remains very low, at only 247.3 MB. In the inference stage, DOP and OPCM exhibit comparable GPU memory usage. While previous results have demonstrated the time efficiency of our method during optimization, these findings further confirm that our approach is also highly efficient in terms of memory usage.

Response to Q6: About the "Typo Error in Eq(3)"

We have carefully reviewed and corrected it in the revised version.

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

Response to Weakness 1 & Question 2: About the "Analysis under Complex Network Architectures"

We understand your concerns regarding the linearity assumption, especially given that neural networks typically contain nonlinear activation functions and multiple stacked layers. We would like to clarify that, although modern neural network architectures generally include both linear and nonlinear components, mainstream architectures such as Transformers are in fact dominated by a series of linear layers (e.g., embedding layers, self-attention layers, and feed-forward networks). These linear layers constitute the majority of the parameters in the overall architecture. As shown in the table below, the proportion of linear layer parameters exceeds 90% in both Flan-T5 and ViT architectures, reaching as high as 99% in some cases.

Moreover, all experiments in this paper are conducted on real-world, complex network architectures (including ViT-B/32, ViT-B/16, ViT-L/14, and Flan-T5, which are widely used vision and language models). These architectures contain multiple stacked linear layers, nonlinear activation functions, and other nonlinear components. Our method consistently outperforms baseline methods across all these real architectures, demonstrating its applicability to complex, practical network settings.

On the other hand, to address your concerns regarding the treatment of nonlinear layers, we have implemented and compared different merging strategies for nonlinear layers to observe their impact on final performance. Specifically, we experimented with several approaches, such as directly using the nonlinear weights from the pretrained model, applying weight averaging, and employing task arithmetic to merge the nonlinear weights of expert models. As shown in the table below, the performance differences among these strategies are minimal (79.2% vs. 78.3% vs. 78.9%), and all consistently outperform baselines such as SWA and OPCM. This indicates that the impact of nonlinear layers on the merged model's performance is limited, as they account for only a small proportion of the total parameters.

Response to Weakness 2: About the "More Baselines"

Thank you for your suggestion. We have further included MagMax[1] as a new baseline and evaluated it across three architectures (ViT-B/32, ViT-B/16, and ViT-L/14) and three task scales (8, 14, and 20 tasks). As shown in the table below, we observe that MagMax outperforms the three baselines—SWA, C. Task Arithmetic, and C. Ties-Merging—but still lags behind OPCM, which is specifically designed for continual model merging, and our proposed DOP. Overall, our DOP consistently achieves the best performance.

[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV 2024.

Response to Question 1: About the "Dynamic Model Merging Library"

We agree with your perspective that maintaining a model library and leveraging model composition to cover more tasks or categories is indeed a promising direction. This approach is closely related to ensemble learning, which aims to collaboratively solve complex tasks by integrating the capabilities of multiple experts. However, the continual model merging (CMM) setting discussed in this paper is more aligned with multi-task learning or continual learning, where only a single set of model parameters is maintained to accomplish all tasks, resulting in higher efficiency. In summary, dynamic model libraries and unified models have different application scenarios:

• Expert Library or Ensemble Learning: The expert library approach is suitable for scenarios with ample memory resources, where each model specializes in different tasks or categories, and model composition is used to enhance overall task coverage. The advantage of this method lies in its ability to handle more complex tasks through the collaboration of multiple expert models. However, it requires significant memory to store multiple models and may lead to inefficiency, especially in resource-constrained environments.
• Continual Model Merging: In contrast, continual model merging is more appropriate for resource-limited scenarios, particularly when memory or computational capacity is restricted. In the continual model merging setting, the goal is to build a single unified model that incorporates the capabilities of all tasks by sequentially merging models for new tasks as they arrive. This approach avoids the need to store a large number of expert models, thereby reducing memory consumption while maintaining adaptability to new tasks and retention of previous knowledge.

Therefore, expert libraries and continual model merging serve different application needs. We believe that each has its own advantages and limitations, and the appropriate strategy can be chosen based on actual resource constraints and task requirements.

Response to Question 3: About the "Extended to Networks with Different Structures"

In our experiments, we include four different architectures: three ViT models of varying sizes for vision tasks (ViT-B/32, ViT-B/16, and ViT-L/14) and the Flan-T5-Base model for language tasks. Additionally, our experiments cover different numbers of tasks (8, 14, and 20). As summarized in the table below, DOP consistently achieves the best or near-best performance across different architectures and task scales, which strongly demonstrates the effectiveness of our method under various complex architectures.

For the rebuttal, I have a question that while linear layers constitute the dominant component of most existing neural network architectures, the non-linear activation functions play a crucial role in determining the model's overall capability. Given this, can the current theory be extended to incorporate both the single linear layer and non-linear activation function, such as ReLU?

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

Response to Weakness 1: About "Nonlinear Layer Processing Methods"

Thank you for your suggestion regarding the treatment of nonlinear layers. In the main text, we have by default adopted simple weight averaging for handling nonlinear layers. Following your advice, we further explored additional strategies for merging nonlinear layers, including Weight Averaging and Task Arithmetic[1]. As shown in the table below, we observe that the performance differences between these two approaches are minor, and both significantly outperform baselines such as SWA and OPCM. This is mainly because the parameters of nonlinear layers constitute only a small proportion of the entire model.

[1] Editing Models with Task Arithmetic. ICLR, 2023.

Response to Question 1: About "Advantages of our DOP Compared to OPCM"

Compared to OPCM, the proposed DOP method demonstrates clear advantages in several aspects:

• Theoretical Analysis: DOP provides a more comprehensive theoretical framework by analyzing the relationships among data, gradients, and task vectors in the context of continual model merging (CMM). Through this analysis, DOP explicitly defines two key objectives—stability and plasticity—from the perspective of orthogonal projection. In contrast, OPCM does not formally define these objectives nor provide such theoretical analysis.
• Stability-Plasticity Trade-off: DOP simultaneously optimizes both stability and plasticity by formulating the problem as a multi-objective optimization, thus achieving a better balance between adapting to new tasks (plasticity) and retaining knowledge from previous tasks (stability). OPCM, however, focuses only on stability, aiming to minimize interference from new tasks to old knowledge, and overlooks the important aspect of plasticity.
• Empirical Performance: Experimental results across multiple vision and language tasks, four model architectures, and three task scales consistently show that DOP outperforms OPCM, achieving higher average performance and less forgetting.

In summary, DOP not only offers a more rigorous theoretical foundation and methodological soundness for continual model merging, but also demonstrates superior empirical performance in practical applications.

Response to Question 2: About "Differences in Projection Space Construction"

Thank you very much for your insightful question. We believe that the row space and column space possess distinct geometric properties in capturing the structure and variations of task data. Specifically, the row space primarily reflects the variation of task vectors in the input data (feature space), while the column space directly reveals the variation of task vectors in the model parameters (output space). As discussed in our main text and supported by previous works on orthogonal projection, traditional orthogonal projection operations in continual learning are typically performed in the input space [1,2], which aligns with our results. Therefore, compared to the output space, projection in the input space may be more effective, as it better captures critical information from the input data and thus impacts the overall model performance. Furthermore, simultaneously capturing the characteristics of both input and output spaces is superior to considering either alone.

[1] Gradient projection memory for continual learning. ICLR, 2021.

[2] TRGP: Trust region gradient projection for continual learning. ICLR, 2022.

Response to Question 3: About "Visualization of Coefficient Alpha"

In our method, the coefficients $\alpha = \{\alpha_{s,k}, \alpha_{p,k}\}$ are adaptively updated across different layers, parameter modules, and merging stages. To further analyze the dynamic evolution of $\alpha$ during optimization, we visualize the values of $\{\alpha_{s,k}, \alpha_{p,k}\}$ at various training steps.

For illustration, we randomly select three linear modules from three different stages (Stage 2 - vision_model.encoder.layers.11.self_attn.k_proj, Stage 4 - vision_model.encoder.layers.0.self_attn.k_proj, and Stage 8 - vision_model.encoder.layers.11.self_attn.v_proj), and record the values of $\alpha$ at different training steps ($k=0,25,50,75,100,125,150,175,200$). As shown in the following tables, we consistently observe that the stability coefficient $\alpha_{s,k}$ gradually decreases from the initial value of 0.8, while the plasticity coefficient $\alpha_{p,k}$ increases from 0.2 to around 0.3. This trend reflects that, in the early stages, the model tends to retain more knowledge from the merged model, and as training progresses, it gradually incorporates new knowledge, resulting in a decrease in $\alpha_{s,k}$ and an increase in $\alpha_{p,k}$.

We will provide detailed plots of the evolution of $\alpha = \{\alpha_{s,k}, \alpha_{p,k}\}$ at each iteration step in the revised version.

Stage 2 - vision_model.encoder.layers.11.self_attn.k_proj:

Stage 4 - vision_model.encoder.layers.0.self_attn.k_proj:

Stage 8 - vision_model.encoder.layers.11.self_attn.v_proj:

Response to Question 4: About "Improvements of DOP in Different Linear Layers"

Thank you very much for raising this insightful question. We agree that analyzing whether DOP consistently outperforms OPCM across different linear submodules is highly valuable. To this end, we conducted the following ablation experiments:

• (i) None: Neither OPCM nor DOP is applied to any linear layer.
• (ii) Q/K/V Projection: OPCM and DOP are applied only to the Q/K/V projection layers.
• (iii) FFN: OPCM and DOP are applied only to the FFN layers.
• (iv) ALL: OPCM and DOP are applied to all linear layers.

We observe that DOP consistently outperforms OPCM. Notably, on the Q/K/V Projection layers, DOP achieves an accuracy of 72.9, compared to 66.6 for OPCM. Furthermore, when optimizing all linear layers, DOP achieves an accuracy of 78.3, surpassing OPCM's 75.5. These results demonstrate that DOP consistently outperforms OPCM across different linear layers.