CALM: Consensus-Aware Localized Merging for Multi-Task Learning

Question 1: Does the Efficient-Aware Framework apply to existing model merging methods?

Answer: Thanks for the inspiring question.

The Efficient-Aware Framework (EAF) introduces a novel serialized merging approach for model merging, which impacts existing methods as follows:

• For task quantity/order-insensitive methods: EAF does not directly affect merging results and can be naturally applied to methods like Task Arithmetic and Localize-and-Stitch. These approaches typically disregard global task information during merging.

• For task quantity/order-sensitive methods: Existing techniques (e.g., Ties-Merging, Adamerging) can adopt EAF through pairwise merging substitution. However, their performance will significantly degrade due to EAF's inability to comprehensively integrate multi-task information. Experimental validation appears in Reviewer oEok Question 2. This limitation further constrains their scalability for new tasks.

---

Question 2: How is global task consensus captured through Mask Optimization in practice?

Answer: Thank you for the valuable question. We would like to explain it as follows:

• Theoretical perspective: The Mask Optimization process extracts gradient directions beneficial for global tasks, aligning with the global task consensus principle. Detailed theoretical analysis is provided in Reviewer y6gA Question 1.

• Empirical perspective: Mask Optimization effectively compresses task-relevant information while eliminating inter-task interference. Experimental results are available in Reviewer Rsjn Question 3.

---

Question 3: Additional experiments on extensive visual tasks.

Answer: Thanks for the constructive comments.

We supplement six new datasets (CIFAR100, Flowers102, OxfordIIITPet, STL10, KMNIST, FashionMNIST), expanding the vision tasks to 14 in total, to validate CALM's stability under increased task diversity. Results are as follows.

Key Findings from Experiments on 14 Visual Tasks:

• Scalability: CALM achieves robust performance (average accuracy ≈87% for the original 8 tasks) through sequential merging of a minimal task subset, demonstrating sustained effectiveness under task scaling.
• Flexibility: Superiority over existing methods is attained even with limited merging steps, highlighting its efficiency in resource-constrained scenarios.
• Robustness: The ordering of sequential merging tasks shows negligible impact on final performance, confirming task-agnostic stability.

---

Question 4: Additional analysis of the Consensus-Aware Mask Optimization module.

Answer: Thanks for the practical comments.

In the review, we provided a more detailed analysis of the Consensus-Aware Mask Optimization module, including:

• The definition of global consensus: refer to Reviewer Rsjn Question 1;
• Experimental analysis of the binary mask: refer to Reviewer Rsjn Question 3;
• Ablation Study of Consensus-Aware Mask Optimization: refer to Reviewer oEok Question 3.

---

Question 5: Essential References Discussion.

Answer: Thank you for your constructive advice.

Comparison between CALM and Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks:

• Commonality: Both methods leverage task vectors and acknowledge the significance of local parameters for multi-task model merging.
• Divergence: Model Breadcrumbs filters anomalous parameter values based on local prior characteristics (e.g., magnitude distribution thresholds), whereas CALM dynamically extracts task-agnostic knowledge to identify optimal local parameters that align with global task consensus.
• Advantage of CALM: CALM more effectively resolves conflicts among local parameters by integrating cross-task knowledge. The consensus-aware optimization ensures that selected parameters exhibit enhanced reliability and generalizability across diverse tasks, thereby offering a more principled and robust solution.

Question 1: The sampling rates of CB-EMS in experiments appear inconsistent; how would this be resolved in practical applications?

Answer: Thanks for the practical comments. We would like to explain it as follows:

• CB-EMS Sampling Rates Show Cross-Task Generalizability.
  Experimental results demonstrate that CALM exhibits consistent performance trends across language and visual benchmarks: accuracy first increases then decreases with higher sampling rates, peaking at 0.8 (language) and 0.9 (visual). This validates CALM's cross-domain adaptability and task-agnostic performance patterns.

• Optimal Sampling Rates Depend on Unsupervised Data Properties.
  The choice of optimal sampling rates is dynamically determined by the scale and quality of unsupervised data. For small datasets, higher rates (0.8-0.9) help mitigate data scarcity through increased sample diversity. When individual task models show high prediction confidence on the unsupervised data, high sampling rates remain viable despite potential noise. Conversely, lower rates (0.4-0.5) are recommended for unreliable predictions to prioritize data credibility.

• Supervised Data Guides Sampling Rate Optimization.
  Supervised data enables adaptive sampling: set sampling rate equal to task models' accuracy on the supervised set. This balances data trustworthiness and utilization by filtering noise while keeping useful data. Below we demonstrate the performance improvements enabled by this method. Setting the sampling rate to match the accuracy of the supervised dataset significantly enhances data credibility. ||SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD| |-|-|-|-|-|-|-|-|-| |Individual(Full Data)|75.3|77.7|96.1|99.7|97.5|98.7|99.7|79.4| |Sampling Rate|75.3|77.7|96.1|99.7|97.5|98.7|99.7|79.4| |Individual(Sampled Data)|88.3|87.3|97.4|99.9|98.5|99.2|99.8|87.5|

---

Question 2: A detailed ablation study on the individual components of CALM.

Answer: Thanks for the very constructive comments.

• Ablation Study of CB-EMS.
  As shown in Figure 5, we present ablation study results of CB-EMS. The experimental results demonstrate that adding the CB-EMS method achieves performance comparable to supervised learning results and is significantly better than unsupervised EM methods using entropy.
  Below, we provide the "Class-balanced" ablation results. Incorporating the Class-balanced component effectively avoids class imbalance and shows significant performance improvement. |Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|Avg Acc| |-|-|-|-|-|-|-|-|-|-| |CB-EMS|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|87.7| |EMS|70.6|73.0|88.2|93.3|88.1|75.9|93.7|70.0|81.5|

• Ablation Study of Efficient-Aware Framework.
  We conduct additional ablation studies by adding the Efficient-Aware Framework (EAF) to existing model merging methods and removing it from CALM. EAF has no effect on order-insensitive methods (e.g., Task Arithmetic) but effects order-sensitive methods (AdaMerging). Without EAF, CALM reduces to randomly selecting a task for mask learning, with results as follows.
  Existing methods exhibit significant performance degradation under the Efficient-Aware Framework since they require simultaneous consideration of all task vectors.
  Without this framework, CALM still maintains robust performance. Notably, CALM still requires EAF for large-scale task merging. |Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|Avg Acc| |-|-|-|-|-|-|-|-|-|-| |AdaMerging (+EAF)|70.7|55.3|61.3|66.5|74.0|58.1|99.1|36.2|65.2| |AdaMerging (-EAF)|64.5|68.1|79.2|93.8|87.0|91.9|97.5|59.1|80.1| |CALM (+EAF)|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|87.7| |CALM (-EAF)|71.9|74.1|91.6|98.7|95.2|96.2|98.9|71.2|87.3|

• Ablation Study of Consensus-Aware Mask Optimization.
  Consensus-Aware Mask Optimization contributes most performance gains as our core component and cannot be fully ablated. We instead conduct ablation studies on our core insights and key components.

  • Ablation Study of Localized Information and Global Consenus
    As core insights, CALM degenerates into Localize-and-Stitch (extracting localized task-specific information) when focusing solely on localized patterns, and degenerates into TW AdaMerging (applying identical masks to all parameters) when only global consensus is considered.
  • Ablation Study of Regularization
    Evaluating the impact of regularization in the optimization equation.
  • Ablation Study of Binary Mask
    Please refer to Reviewer Rsjn Question 2.
    |Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|Avg Acc| |-|-|-|-|-|-|-|-|-|-| |Local-only|67.2|68.3|81.8|89.4|87.9|86.6|94.8|62.9|79.9| |Global-only|58.0|53.2|68.8|85.7|81.1|84.4|92.4|44.8|71.1| |W/o Regularization|70.1|72.4|90.8|97.6|94.2|95.3|98.2|70.5|86.3| |CALM|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|87.7|

Question 1: The authors need to provide a clearer definition and explanation of global task consensus to establish the core idea of this work.

Answer: Thanks for your valuable question. We clarify the concept of global task consensus as follows:

• Definition: Global task consensus refers to shared and generalizable knowledge patterns or parameter adaptation directions among multiple independently fine-tuned models during model merging.
• Key Properties:
  • Cross-Task Validity: Consistent parameter adaptation directions across tasks during fine-tuning.
  • Task-Specific Feature Preservation: Avoidance of significant performance degradation on any task after model merging.
• Validation: Due to length limitations, we provide theoretical error analysis to validate its effectiveness. For details, please refer to Reviewer y6gA Question 1.

---

Question 2: Can CALM adapt to existing frameworks without an efficiency-aware framework?

Answer: Thank you for your valuable question.

The efficiency-aware framework enables CALM to achieve efficient model merging with strong scalability. Our method can also adapt to scenarios without this framework, maintaining compatibility with existing approaches. Below we present two feasible solutions:

• Single-Task Merging: Randomly select one task for model merging. As shown in Figure 6, the results demonstrate that applying CALM to a single task achieves comparable or even better performance than sequential multi-task merging. Thus, CALM can be applied to a single task without requiring sequential merging.
• Multi-Task Simultaneous Merging: Utilize multiple binary masks to merge all tasks simultaneously. We can slightly modify the CALM method to align with existing model merging frameworks, with the following optimization objective. Within a single optimization equation, we optimize a binary mask for each task and enforce that the sum of binary masks equals an all-ones matrix. This allows simultaneous extraction of information from all tasks while maintaining global consensus. During optimization, we still employ real-valued masks $R$ and classify each parameter to tasks via softmax.

$$\min_{\{M\}} \ \sum_{t=1}^T\mathcal{L}(\theta_{mtl})+\alpha\sum_{t=1}^T||M_t||_1$$ $$s.t. \ \theta_{mtl}=\theta_{pre}+\sum_{t=1}^TM_t\odot\tau_t,\ \sum_{t=1}^TM_t=**1**_{n \times n}.$$

---

Question 3: Does the binary mask capture global consensus?

Answer: Thanks for the practical comment.

In Appendix D, we provide visualizations and analysis of the binary mask properties. Here, we present a concise experiment to demonstrate the effectiveness of binary mask.

For each task, we apply CALM with the following definitions: $\theta_{pre}$ denotes the pre-trained model, $\tau_{1}$ is the task vector from the remaining seven tasks via task arithmetic, $\tau_{2}$ is the target task vector, and $M$ is the optimized binary mask. We evaluate the target task performance under five configurations:

CALM extracts task vectors with approximately 5% parameters. We have the following conclusions:

• Interference suppression (Rows 2-3): Removing 5% parameters from $\tau_{1}$ via $(1-M)\odot\tau_{1}$ significantly improves target task adaptation, demonstrating effective identification of interference parameters.
• Task-specific preservation (Rows 4-5): Confining $\tau_{2}$ to 5% masked parameters $M\odot\tau_{2}$ maintains strong performance, indicating precise capture of essential transfer signals.
• Conflict-free merging (Rows 3,5,6): Simultaneous application of $(1-M)\odot\tau_{1}$ and $M\odot\tau_{2}$ achieves performance gains without parameter conflicts, verifying CALM's effective multi-task merging capability.

---

Question 4: Additional experiments of language multi-task benchmarks.

Answer: Thanks for the very constructive comments.

CALM demonstrates consistent behavior across language and visual benchmarks. We give an example, shown in the table below. Both domains exhibit nearly identical trends in average accuracy versus CB-EMS sampling rates - initially increasing then decreasing with higher sampling. Additional experiments confirm this cross-domain consistency.

Question 1: A more formal theoretical analysis of the effectiveness of localized information with global consensus

Answer 1: Thanks for the inspiring question. We perform a theoretical analysis of the following three aspects based on error.

• Localized information reduces interference in model merging. Such interference arises from intra-task noise and conflicts across task vectors (e.g., cancellation due to opposing parameter values).
  Denote the interference by $\epsilon$, so that task vector$\tau_j = \Delta\tau_j+\epsilon,$ where $\Delta\tau_j$ is the effective information for the global optimal update (nonzero in dimensions $I$) and $\epsilon$ is concentrated in the complementary set $J$. Define the binary mask $M_L$ such that $(M_L)_i=1$ if $i \in I$ and $(M_L)_i=0$ if $i \in J$. Thus, the merged task vector becomes $\Delta \tau_L = M_L\odot\tau_j = \Delta\tau_j+M_L\odot\epsilon,$ reducing the interference energy from $||\epsilon||^2$ to $E_L=||M_L\odot\epsilon||^2.$ Although methods like Ties-Merging and Consensus TA aim to lower interference, their effectiveness is limited by the absence of global task information.

• Global consensus enables the update to closely align with the optimal direction common to all tasks, reducing the overall loss.
  For all visible tasks $S_v$, we utilize global consensus to optimize a mask $M_G$ by minimizing

$$\min_{M_G} \sum_{t_v \in S_v} L_{t_v}(\theta_{pre} + \tau_{seq}^{(j-1)} + M_G \odot \tau_j).$$

Assuming a quadratic approximation near $\theta_{pre} + \tau_{seq}^{(j-1)}$, we have

$$L_{t_v}(\theta_{pre} + \tau_{seq}^{(j-1)} + M_G \odot \tau_j) \approx L_{t_v}(\theta_{pre} + \tau_{seq}^{(j-1)}) + \langle g_{t_v}, M_G \odot \tau_j \rangle + \frac{1}{2}(M_G \odot \tau_j)^T H_{t_v} (M_G \odot \tau_j),$$

where $H_{t_v}$ is the Hessian. Assume $H_{t_v}$ is the identity matrix and $\Delta\tau_j$ denote the global optimal update. The update error is

$$E_{G} = ||M_G \odot \tau_j - \Delta\tau_j||^2.$$

The gain from global consensus is defined as

$$\Delta_{G} = ||\tau_j - \Delta\tau_j||^2 - ||M_G \odot \tau_j - \Delta\tau_j||^2.$$

The optimization process shows that global consensus effectively reduces the overall loss.

• Our CALM method integrates localized information and global consensus to effectively mitigate local parameter interference while capturing information beneficial for global tasks.
  Let the obtained binary mask be $M^*$; then the update error is

$$E_{CALM} = ||M^* \odot \tau_j - \Delta \tau_j||^2.$$

Based on the optimization objective, the error can also be expressed as

$$E_{CALM} = \min(E_{loc}, E_{glob}) - \Delta_{syn},$$

where $\Delta_{syn} > 0$ represents the additional error reduction from joint optimization.

Thus, CALM achieves both local and global gains, effectively reducing the overall error.

---

Question 2: Additional ablation studies on the individual components of CALM.

Answer: Thank you for your valuable question.

Due to length limitations, we will present additional ablation study results in Reviewer oEok Question 2.

---

Question 3: The sensitivity of CALM to hyperparameters (the regularization parameter $\lambda$ and the sampling rate for CB-EMS).

Anwser: Many thanks for the insightful suggestions.

Below are the model merging results on eight vision datasets under different regularization parameters $\lambda$ and sampling rates.

CALM is insensitive to $\lambda$, with merging performance remaining stable.

Thus, $\lambda$ between 0.5 and 2 is acceptable.

Performance first increases then decreases with the sampling rate, peaking at 0.9. This aligns with our theory.

In practice, the optimal sampling rate depends on data quantity and quality; for high-quality data, a higher rate (e.g., 0.8–0.9) is preferred, otherwise a lower rate is recommended.

---

Question 4: efficient-aware-> efficiency-aware.

Answer: Thank you for your constructive advice.

We will seriously consider making changes in the final version.