W1 – Comparison of Efficiency and Storage Overhead with Baselines

Table Z.1: Comparison of inference efficiency and parameter overhead on CLIP ViT-B/32 model after eight tasks merging

We sincerely thank the reviewer for raising this point—efficiency and overhead are indeed critical factors for real-world deployments, and we greatly appreciate the suggestion to expand this comparison.

In response, we have included a consolidated table (Table Z.1) comparing inference speed and storage overhead across all baselines, which demonstrates the practical efficiency of MINGLE. Despite employing MoE-style routing, MINGLE incurs minimal additional inference overhead, with only a small number of extra parameters used for the gating mechanism (i.e., the MoE routing gates). The inference throughput is comparable to static merging methods, only slightly lower due to the additional MoE parameters, while providing substantial accuracy improvements.

W2 – Discussion to Rehearsal-Free Continual Learning (RFCL)

We thank the reviewer for the insightful question, and for prompting a more precise comparison between TTCMM and rehearsal-free continual learning (RFCL)—a widely studied paradigm.

Indeed, TTCMM is inspired by RFCL and shares important constraints:

• both do not retain past training data, and
• both avoid storing previous task models.

However, the core difference lies in what is received at each stage:

• RFCL assumes access to the training data of the current task, and incrementally fine-tunes a single model over time.
• TTCMM, by contrast, assumes access to an already independently fine-tuned model for each new task—thus focusing on merging existing expert models rather than learning each from scratch. Additionally, TTCMM requires a small unlabeled test-time buffer (e.g., 5 samples per class) to guide the merge.

On privacy benefits: While both RFCL and TTCMM operate without historical data, TTCMM offers improved privacy guarantees for the following reasons:

1. TTCMM does not require access to full training sets. Instead, it only uses a tiny buffer (e.g., 5 unlabeled samples per class) from the test or calibration set.
2. These small unlabeled samples can be:

• incoming test-time data (e.g., recent user queries or model inputs),
• held-out validation inputs or small training subsets (if available),
• user-provided samples without privacy risks (e.g., few-shot examples),
• synthetically generated,
• or manually curated from public data.

In contrast, RFCL relies on large-scale labeled training data for each task, which raises significant privacy, storage, and legal concerns (e.g., medical images, user data, copyrighted materials). In comparison, small unlabeled samples are easier to obtain while maintaining privacy protection.

W3 – On Flexibility and Test-Time Adaptation in TTCMM

We sincerely thank the reviewer for their insightful reflection. Your interpretation of TTCMM as offering high flexibility through the ability to merge independently trained task-specific models in arbitrary orders is absolutely correct—and we deeply appreciate your articulation of this strength.

You also raise an important point: that independently applying TTA to each downstream task might yield better performance in some cases, as it avoids reconciling conflicts across tasks. We agree that this is indeed a powerful strategy—but only under the assumption that all task-specific models are available at inference time.

However, we respectfully emphasize that this assumption does not hold in the TTCMM setting, which imposes strict continual learning constraints:

• Previous expert models are not retained—once merged, earlier models are discarded.
• The protocol is strictly incremental, with one expert model arriving at a time; merging must maintain knowledge across tasks without re-accessing past models or data.

Thus, the setting envisioned in your comment—while valid and practically useful—requires simultaneous access to all expert models, and is closer in nature to multi-task model merging with task-specific adaptation, which lies outside the scope of TTCMM.

We fully agree that this contrast is important and worth clarifying more explicitly. In the revision, we will revise Sec. 1 and Sec. 3.2 to:

• More clearly distinguish TTCMM from settings that assume full model access.
• Emphasize that TTCMM’s flexibility stems from supporting arbitrary task sequences, but only when models are provided incrementally, not jointly.

Q2 – On Forward Transfer

We sincerely thank the reviewer for raising this insightful question. Forward transfer is indeed a critical capability in continual learning, measuring how well prior knowledge aids the learning of future tasks. Your suggestion meaningfully strengthens the completeness of our evaluation.

To quantify forward transfer (FWT), we adopt the standard definition:

$$\text{FWT} = \frac{1}{T-1} \sum_{t=2}^{T} \left[ a_t(\theta_t^{merged}) - \bar{a}_t \right]$$

where $a_t(\theta_t^{merged})$ denotes the test accuracy on task $t$ using the model merged after task $t$, and $\bar{a}_t$ is the test accuracy of task $t$ individual finetuned model. Positive FWT indicates beneficial transfer, while negative values suggest interference.

We report the following results for the 8-task continual merging setup with CLIP ViT-B/16:

These results show that MINGLE achieves both zero forgetting (BWT ≈ 0) and the highest forward transfer, indicating that our adaptive gating and merging pipeline not only preserves knowledge but also improves representation utility for downstream tasks.

We will include these results and the FWT definition in the final version for completeness.

We sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation.

W1– Clarifying MoE as a Form of Model Merging

We thank the reviewer for this insightful concern. Your point about the potential advantages of MoE over static merging is valid and important.

We clarify that MoE is increasingly adopted in recent model merging works, such as WEMOE (ICML’24) and TwinMerging (NeurIPS’24), which use MoE-style routing to merge fine-tuned experts. These methods treat MoE as a scalable and valid alternative to static merging.

We fully acknowledge that MoE introduces dynamic parameter selection, which may appear stronger than static approaches at first glance. However, our method operates under strict continual learning constraints that make the router tuning significantly more challenging.

To ensure fairness and transparency, we have taken several steps:

• We add comparison with WEMOE in Table 1. Results are shown in Table Y.1.
• We include a static MINGLE baseline (Table 4, Row 1), where LoRA modules are combined with fixed coefficients, without any adaptation or routing. Results are shown in Table Y.1.
• We will clearly separate static and adaptive methods in Table 1, and explicitly compare all baselines' assumptions regarding data access, activation storage, and test-time compute to ensure transparency.(see Table Y.2).

In summary, although our method uses a MoE-style mechanism, it remains fully aligned with the goals of model merging: combining multiple experts into a single model. Crucially, we address continual merging, which is not covered in prior MoE-based merging works. We will clarify this distinction more explicitly in Sec. 2 and Sec. 4.2 of the main paper.

Table Y.1: More comparative results (ACC%) for continual merging performance.

Table Y.2: Comparison of baseline assumptions and requirements

W2 – On MTIL Results and Comparison to RAIL / MoE-based Baselines

We thank the reviewer for the thoughtful comment.

We would like to clarify that MINGLE was not specifically designed for the conventional continual learning setup used in the MTIL benchmark (Table 3). Instead, it was developed for the continual model merging scenario.

Nonetheless, we applied MINGLE directly to the MTIL benchmark in order to demonstrate its generality and robustness beyond its original design scope. We adapted it to this benchmark by merging models obtained via Sequential Finetuning (Sequential FT). This yields MINGLE-SEQ, a direct application of our merging strategy on sequentially trained models.

As shown in Table Y.3, MINGLE-SEQ outperforming several CL baselines on early and mid-stage tasks (e.g., Aircraft, Caltech101, CIFAR100), but underperforms on the last two tasks (Cars, SUN397). Importantly, this drop is not due to MINGLE itself, but rather a consequence of the declining quality of the sequential FT models used as inputs.

To verify this, we examined the Sequential FT (Immediate Eval) baseline, which evaluates each task immediately after learning it. It exhibits the same degradation on the final tasks. We attribute this to the loss of general pretraining knowledge caused by continued finetuning, which results in suboptimal task-specific performance on later tasks, even before merging. Since MINGLE operates on these models, this limits its upper-bound performance in this setting.

This insight suggests that MINGLE-SEQ could benefit from upstream continual learning techniques. For instance, using regularization-based methods like EWC during sequential finetuning may better preserve pretraining knowledge and improve the quality of task models fed into MINGLE, potentially leading to stronger results.

Table Y.3: Comparison of last accuracy (%) with conventional CL approaches on MTIL benchmark.

W3– On Under-activation and Gating Specialization

We sincerely thank the reviewer for the insightful observation regarding the under-activation behavior in Fig. 3.

To investigate this, we provide a new comparison (Table Y.4) between MINGLE and individual fine-tuned models across all tasks. The results show that our method achieves per-task performance close to that of individually fine-tuned models, indicating that the gate indeed selects meaningful experts as intended. This empirical evidence helps to refute concern (a)—that the gate fails to specialize.

Table Y.4: Accuracy (%) of continual merging performance using CLIP ViT-B/16 on the 8-task Order-1 sequence.

Regarding (b)—that null-space constraints may overly suppress task signals—we refer to the w/o Null-Space variant already shown in Fig. 3 (blue curve). While this variant does exhibit higher gate activations, most values still do not approach 1.0, suggesting that under-activation is not solely caused by the null-space constraint. In fact, this variant also shows worse performance in accuracy, indicating that the constraint is helpful overall, rather than overly suppressive.

As to why gate activations rarely reach 1.0, we believe this may reflect the complementary nature of experts rather than a failure of specialization. In many cases, multiple LoRA experts may capture overlapping yet distinct subspaces of knowledge. As a result, softly combining several related experts—rather than activating only a single one—can lead to equal or even better performance. This behavior is especially reasonable in settings like TTCMM, where task boundaries may be fuzzy and information from adjacent tasks remains useful.

We sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation.

W1 – Limited Architectural Scope

We thank the reviewer for this point. While our main experiments use CLIP-ViT, MINGLE is not vision-specific. We applied it to Flan-T5 (T5-base) on 7 GLUE tasks (merged alphabetically) using only 100 unlabeled test samples, confirming strong performance on encoder-decoder models and demonstrating generality.

Table X.1: Accuracy (%) / Spearman’s ρ of continual merging performance using Flan-T5 (T5-base) on 7 GLUE benchmark tasks.

W2 – Efficiency

(i) Wall-clock adaptation time

We report the wall-clock adaptation time in Table X.2. Adaptation runs once per task (~10s on RTX 4090, Table X.2). Afterward, the router is fixed and inference is feedforward with no TTA, including prior tasks—supporting low-latency deployment. (see Table Z.1 for Reviewer np8P for inference efficiency).

Table X.2: Wall-clock adaptation time for ViT-B/32

(ii) Ablation on TTA optimization steps

We performed an ablation by varying the number of adaptation steps. The table below shows the trade-off between accuracy and forgetting:

Table X.3: Ablation on number of adaptation steps for ViT-B/32 across 8, 14, and 20 tasks

• Increasing the number of steps consistently improves accuracy across all task counts.
• While longer schedules introduce slightly more forgetting, the degradation is minor (e.g., BWT drop of only ~2% in 20-task setting).

These results confirm that MINGLE achieves strong performance even under tight compute budgets—for example, using only 20 steps, it already outperforms all baseline methods in Table 1. Longer adaptation improves accuracy with only minor forgetting

(iii) Discussion: Step-freezing and fast first-order updates

We thank the reviewer for the insightful suggestion. Both step-freezing and first-order adaptation are promising directions to improve TTA efficiency.

• Step-freezing: Our current method updates all router parameters across layers during adaptation. Step-freezing (e.g., freezing early layers once stabilized) may reduce compute. Preliminary results show early gates converge quickly (<10 steps).
• Fast first-order updates can benefit from first-order strategies such as Reptile, FOMAML, or linearized single-step updates, which avoid full optimization but enable quick convergence. These methods are especially attractive in low-latency or edge settings,

We see both directions as highly complementary to our framework. We will include this discussion in the final version to highlight efficient variants as a promising area for future work.

W3 – Hyper-parameter transparenty

We thank the reviewer for the suggestion. As requested, we have added a consolidated table (see Table X.4) listing the key hyperparameter settings for all baseline methods and task configurations.

We also clarify that none of the baselines were tuned per task order. For our method, we emphasize that we used a single fixed hyperparameter configuration across all experiments, including:

• all backbones (ViT-B/32, B/16, L/14),
• all task counts (8, 14, 20), and
• all 10 task orders.

This highlights the robustness and generality of our method, and ensures that performance gains are not due to fine-grained tuning.

Table X.4: Hyperparameter settings for all baselines across different task configurations.

W4 – On the use of unlabeled adaptation samples

(i) How is the buffer obtained in practice? We thank the reviewer for raising this important question. In our experiments, we simulate a realistic scenario by randomly sampling five unlabeled examples per class from the test split—serving as a proxy for real-world deployment data.

In practical settings, such small unlabeled buffers are easily obtainable from a wide range of sources, including:

• incoming test-time data (e.g., recent user queries or model inputs),
• held-out validation inputs or small training subsets (if available),
• user-provided samples without privacy risks (e.g., few-shot examples),
• synthetically generated,
• or manually curated from public data.

We will include this discussion in the revision, making it practical for real-world deployment.

(ii) Zero-sample condition (“pure parameter-space merge”)

We thank the reviewer for this suggestion. As shown in Table 4 (Row 1), we already include a zero-sample baseline where LoRA adapters are merged with fixed weights (e.g., 0.3) without any adaptation data. (refer to MINGLE-Static in Table S.2 for Reviewer chHs).

Surprisingly, this simple strategy still outperforms classical baselines like Task Arithmetic and Ties-Merging. We attribute this to the low-rank, near-orthogonal structure of LoRA modules, which reduces interference and preserves task-specific features in disjoint subspaces.

To improve clarity, we will (i) note in the main text that Table 4’s first row reflects the zero-sample case, and (ii) add it to Fig. 5 to better visualize the trade-off between adaptation-free and adaptive merging across buffer sizes.

We sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation.

W1 — Problem setting motivation

We thank the reviewer for prompting us to better position our setting within established paradigms.

Our proposed setting (TTCMM) is directly inspired by the well-known rehearsal-free continual learning (RFCL) framework [1,2], sharing key constraints:

• No access to prior task data
• No storage of past task-specific models

This setup is practical, motivated by two real-world constraints:

• Storage: Retaining full models—or even task vectors, which are equally large—leads to linear memory growth, which is impractical for edge or long-horizon deployments
• Privacy: Even unlabeled data (e.g., medical scans, GPS logs) can be sensitive and must be discarded post-training due to laws like HIPAA or GDPR

Thus, TTCMM offers a natural extension of RFCL into model merging. We will clarify this link in Sec. 1 & 2 and include examples (e.g., mobile agents, medical federations) to highlight practical relevance.

[1] Dualprompt: Complementary prompting for rehearsal-free continual learning (ECCV2022)

[2] Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning (CVPR2022)

W2 – Potentially Unfair Baseline Comparisons

We acknowledge that our setup is not identical to every baseline. To promote transparency, we include the following table comparing each method on their data access, extra parameters, and test-time compute:

Table S.1: Comparison of baseline assumptions and requirements

*¹: Our method does not store activations. Only a fixed-size covariance matrix is maintained (Line 195), resulting in constant memory regardless of test set size.

*²: While LoRA experts are stored, the router merges them into a single model per input. Thus, the effective inference size matches a standard individual model.

Table S.2: More comparative results (ACC%) for continual merging performance.

On TTA & MoE: TTA and MoE are not entirely new assumptions in the model merging literature (e.g., WEMOE (TTA+MoE) [ICML 2024], AdaMerging (TTA) [ICLR 2024], Twin-Merging (TTA+MoE) [NeurIPS 2024]).

To ensure fair comparisons, we have added results for WEMOE and AdaMerging in Table 1 (also shown in Table S.2). Additionally, we report a MINGLE-Static baseline (Table 4, Row 1), where LoRA modules are combined using fixed coefficients (e.g., 0.3) without any adaptation or routing. We will also include Table S.1 to summarize the assumptions and requirements of all baselines. These additions ensure clarity and transparency.

Why the gap vs. WEMOE?

To clarify: both our method and WEMOE rely on the same MoE and TTA assumptions. In our experiments, we carefully re-implemented WEMOE under TTCMM setting. This ensures that any observed performance gap arises not from differences in assumptions, but purely from methodology.

WEMOE uses full MLP blocks as experts, which are parameter-heavy and hard to route with limited seed data. Our method uses lightweight LoRA adapters, which are easier to gate and effectively regularized via a null-space constraint.

To isolate this factor, we introduced a "WEMOE-LoRA" variant, replacing WEMOE’s MLP experts with LoRA. As Table S.2 shows, this closes the gap and supports that:

• heavy experts degrade routing in low-data regimes, and
• our gains primarily stem from constrained gating + LoRA, not from relaxed assumptions.

We deeply appreciate the reviewer’s question. We will clarify this explicitly in revision.

W3 – High number of hyperparameters

We sincerely thank the reviewer for this thoughtful and valid concern. While our method introduces five scalar hyperparameters, each plays a minimal but necessary role.

Importantly, our method requires no tuning. We apply a single fixed configuration across all experiments: all task counts (8/14/20), all 10 task orders, and all model backbones (ViT-B/32, B/16, L/14)—without any modification. This deliberate design ensures strong generalization and ease of deployment.

We’ll clarify this global configuration strategy in the main text.

W4 – Limited Architectural Scope

We thank the reviewer for raising this point. While our main experiments focus on CLIP model, our method is not restricted to vision models.

To show generality, we applied MINGLE to Flan-T5 (T5-base) on 7 GLUE tasks (merged alphabetically), using only 100 unlabeled test samples for adaptation. The results below show that our method remains effective in non-visual, sequence-based encoder-decoder architectures, confirming that MINGLE generalizes beyond CLIP-style backbones.

Table S.3: Accuracy (%) / Spearman’s ρ of continual merging performance using Flan-T5 (T5-base) on 7 GLUE benchmark tasks.

W5 – Minor Writing Issues

Equation 3 — Clarifying $\alpha$: We will clarify that $\alpha$ denotes the effective rank of the merged task vector, computed as:

$$\alpha = \left\lfloor \exp\left( -\sum_i p_i \log p_i \right) \right\rfloor, \quad \text{where } p_i = \frac{s_i}{\sum_j s_j}$$

and ${s_i}$ are singular values from SVD. This will be added near Equation 3.

Missing [a] in benchmarks: Thanks for catching this. We’ve reproduced baseline [a] (see Table S.2), will include it in Table 1, and cite it in Sec. 2 for completeness.

Static vs. Continual baselines: We’ll revise the Introduction and Related Work to clarify that TA and WiSE-FT are static model merging methods—not MoE-based or continual—and ensure they are properly grouped.

Q1 – On the task-specific routing error $\varepsilon_t$

We thank the reviewer for raising this point and will clarify the definition in the main text.

We define the task-specific routing error as:

$$\varepsilon_t = \Pr\big(i^*(x) \ne t \mid x \sim D_t\big),$$

where $t$ is the ground-truth task and $i^*(x)$ is the expert selected by the router. Thus, $1 - \varepsilon_t$ reflects correct routing probability. A lower $\varepsilon_t$ indicates better expert-task alignment, with $\varepsilon_t = 0$ meaning perfect routing. This is defined in Appendix (Line 602) and will be referenced in the main text.

Regarding Lines 143–144, the critique targets static baselines, which use fixed or implicit routing. In contrast, our method actively minimizes $\varepsilon_t$ via TTA, enabling expert specialization.

We sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation.