Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging

Thank you for your valuable suggestions. Below, we provide a detailed response to each point.

---

R1: Direct incorporation of preference to Task Arithmetic and AdaMerging In Section 3, we frame model merging as a MOO problem and propose both straightforward methods (Section 3.1) and our Pareto Merging (PM) approach (Sections 3.2-3.3). While straightforward methods incorporate preferences, they have significant limitations. Our PM method is more parameter-efficient and computationally effective, achieving better performance trade-offs.

Following your suggestion, we tested Task Arithmetic + Preference and AdaMerging + Preference. Results (available at https://anonymous.4open.science/r/ICML-0464/) show:

• Task Arithmetic + Preference has 113.4M parameter overhead and some extreme solutions obtained are typically undesirable for users. Our Task Arithmetic + PM approach with the default configuration (low-rank tensor applied only to attention layers) achieves reasonable trade-offs with 0.6M parameter overhead. Extending low-rank tensors to both attention and MLP layers shows better performance with only 2.1M parameters. Users can adjust settings to balance solution diversity.

• AdaMerging + PM (0.6M parameter overhead) achieves better trade-offs than AdaMerging + Preference (113.4M parameter overhead). Moreover, our approach requires only a single run (0.28 GPU hours), whereas AdaMerging + Preference demands 11 runs (1.65 GPU hours).

We will include the above disscussion in the final version.

R2: Task arithmetic and task arithmetic+PM(equal) perform almost identical, while AdaMerging+PM(equal) significantly outperform AdaMerging Task arithmetic is data-free merging, so task arithmetic+PM(equal) has no additional information, resulting in similar performance. AdaMerging+PM leverages unlabeled data, allowing PM to better resolve task conflicts. This pattern of improvement with data incorporation appears consistently across various compression techniques, such as merging, pruning, and quantization.

R3: How much training is needed We apologize for the omission of detail. We use 2,000 gradient steps. As shown in Table 1, the additional training time is small.

R3: ViT-L/14 results The improvement needs to be considered with respect to traditional MTL, which serves as a performance upper bound. As can be seen from Table 6, with ViT-L/14, AdaMerging already achieves 90.8% (vs. 80.1% with ViT-B/32), naturally limiting improvement margins. Still, AdaMerging+PM (priority) achieves a significant 1.3% improvement over AdaMerging. This smaller margin reflects approaching optimal performance rather than method limitations.

R4: Computational overhead As shown in Table 1, our computational overhead during training is small. for data-free merging (with task arithmetic), the overhead is 0.04 GPU hours; whereas for merging based on unlabeled data (with AdaMerging), the overhead is 0.13 GPU hours.

R5: Priorities in Image Classification and Extension to LLM There seems to be a misunderstanding. Setting priorities does not mean that only one task is important while all other tasks are unimportant. Rather, it means that the model should excel at a specific task while still achieving acceptable performance on others. Maintaining separate models for different preferences is inefficient in terms of parameters, as it requires storing and managing multiple full models.

Our Pareto Merging enables a single model to excel at prioritized tasks while maintaining reasonable performance across all tasks, with minimal parameter overhead. Additionally, our method supports flexible adjustment of prioritized tasks. We recognize the exciting potential of this approach for LLMs and are actively working on extending our algorithm to these models. However, due to time constraints, we can only include this extension in the final version of the paper.

R5: Preference Vector Sampling Thanks for your question. We optimize the expected loss across the entire preference space to account for all possible user preferences. Different preferences weight the parameters in the low-rank tensor differently, enabling the model to capture diverse characteristics. While our method already demonstrates strong performance, we acknowledge that developing more effective preference vector sampling techniques is an interesting future work that could further enhance efficiency.

---

We hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.

Thank you for your valuable suggestions. Below, we provide a detailed response to each point.

---

Response to Questions For Authors

R1: Parameter count in Table 1 The "parameter count" refers to the number of parameters that need to be stored after merging. Note that for each merging method, there are two options on what to store. For instance, with two models in the soup and 100 user preferences, one can either store the 100 merged models corresponding to these 100 preferences, or store only the two original models and merge them on-the-fly based on the user preference. Obviously, the latter is more memory-efficient, and is we adopt when calculating the parameter count. Hence, for MAP and the Rewarded Soups, the parameter counts are doubled in the two-model case.

R2: Number of solutions in Table 1 Thank you for pointing this out. In the paper, we mentioned 100 as the original NSGA-III algorithm enforces a fixed population size by removing dominated or overcrowded solutions during evolution. Upon double checking the MAP implementation, we observed that unlike the original NSGA-III, it retains all the non-dominated solutions, resulting in around 500 solutions at the end. We will correct this in the final version. However, please note that since we used the official implementation from the authors' github in the experiments, this does not affect any experimental results or discussions in the paper.

---

We hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.

Thank you for your valuable suggestions. Below, we provide a detailed response to each point.

---

Response to Claims And Evidence

R1: Comparison with [1] Note that [1] is indeed a concurrent work of ours but with different goal. [1] aims to improve the efficiency of general Pareto set learning algorithm. Instead of training from scratch, [1] employs task arithmetic to merge specific modules while keeping others (e.g., MLP layers or both the MLP and attention layers) separate. They then train a MOE router using labeled data to weight these unmerged components. Note that when both MLP and attention layers are not merged, [1] needs to keep the $K$ models after training.

On the other hand, our method focuses on improving model merging and formulates it as a MOO problem. This extension has not been studied before (including [1]). Unlike [1], our method merges all modules while incurring a small low-rank tensor (with only 0.5% parameters of the pretrained model).

Note that formulating model merging as a MOO problem requires first identifying the MOO objectives for model merging, which is not trivial. For instance, in data-free merging, we observe that Task Arithmetic can be viewed as optimizing the distances between models in the weight space. This then allows us to define the MOO objectives based on these distances. While traditional model merging considers finding a single solution based on a specific preference, our MOO formulation transforms the problem to the finding of a continuous space of solutions, which is novel.

To further compare with MoE-based fusion in [1], below we compare its performance on merging eight ViT-B/32 models using the same setup as in [1]. As reported in [1], with the use of labeled data, MoE-based fusion (with a final model size of 567M) achieves an average accuracy of 77.2% when only the MLP is unmerged, and 83.5% when all modules are unmerged (with final model size 1.02B). On the other hand, our method (with only 114M parameters and NOT requiring labeled data) achieves a higher accuracy at 85.2%. We will add the above discussion to the final version.

Response to Methods And Evaluation Criteria

R2: Comparison with LoRA-based Pareto Manifold leanring [2, 3] Thanks for your comment but that is not the key difference. As mentioned earlier, our primary objective is to improve model merging, rather than using model merging for efficient Pareto set learning. Our first key contribution is identifying the limitations of current model merging techniques and reformulating it as a MOO problem to effectively incorporate user preferences. Furthermore, we propose an efficient preference-aware tensor structure, which is different from the weighted sum of LoRAs used in [2, 3].

To further illustrate our advantages, we adapt [2,3] for use in our model merging setting. Using the setup in Section 4.2.2 and Table 2, the results on merging eight ViT-B/32 models are:

As can be seen, the proposed tensor structure has better performance while significantly reducing the number of parameters. We will add the above discussion to the final version.

R3: Real-time trade-off adjustment & Merge for each preference independently The proposed method can be used in both scenarios: (1) As you mentioned, it is promising for real-time trade-off adjustment. (2) When different users have different preferences, the straightforward preference incorporation method, as discussed in Section 3.1, has two limitations: (i) It requires storing all $K$ original models to address $n$ different user preferences. In contrast, our approach only requires a single model and a small low-rank tensor. (ii) For methods like AdaMerging, they require $n$ separate runs for $n$ preferences. For instance, handling 100 preferences with straightforward method would require $0.15 \times 100 = 15$ GPU hours, whereas our method requires only $0.28$ GPU hours.

Response to Experimental Designs Or Analyses

For comparison with [1], please refer to R1.

R4: Comparison with other scalarization methods Thanks for your question. Following your suggestion, we add experiment with different scalarization methods on merging eight ViT-B/32 models (as in Section 4.2.2 and Table 2). The average accuracies obtained are: (i) linear scalarization: 84.3%; (ii) Tchebycheff scalarization: 82.8%; and (iii) smooth Tchebycheff scalarization: 85.2%. These indicate that smooth Tchebycheff scalarization outperforms the other methods for our model merging task. We will add the discussion to the final version.

---

We hope the above responses address your concerns. If you have any additional suggestions, we are more than happy to discuss them further.

Thank you for your valuable suggestions. Below, we provide a detailed response to each point.

---

Response to Essential References Not Discussed

R1: MGDA Reference Thanks. We'll include it in our final version.

R2: Relation with preference-based MOO works [1,2,3] They differ fundamentally from ours. They aim to generate a single solution per run based on a given preference. To handle different preferences, they require multiple runs and obtain multiple models. On the other hand, our method produces a continuous Pareto set of solutions. Once trained, a MOO solution can be generated from the given preference without optimization. Currently, we use smooth Tchebycheff scalarization for its simplicity and effectiveness. But algorithms in [1,2,3] can be integrated into our framework by modifying Equation (10). Specifically, instead of directly using $\gamma_k$ to weight $S_k$, we can use algorithms in [1, 2, 3] to compute a weighting for $S_k$.

R3: Using low-rank embedding for the preference There might be some misunderstanding. We are not using low-rank embedding for preference. Instead, we employ a low-rank tensor to integrate the preference into a single model, which is significantly more parameter- and compute-efficient compared to generating a separate model for each preference (using methods such as [1, 2, 3]) or use a full-rank tensor structure.

R4: Difference with Pareto set learning works [4, 5] We would like to emphasize that our contributions are:

1. Problem formulation: We address the challenges of model merging by reformulating it as a MOO problem. This requires identifying the MOO objectives for model merging, and is not trivial. In data-free merging, we observe that Task Arithmetic can be viewed as optimizing the distances between models in the weight space. This then allows us to define the MOO objectives based on these distances. Moreover, while traditional model merging considers finding a single solution for a specific preference, our formulation transforms the problem to the finding of a continuous space of solutions, which is novel.

2. Scalable Pareto Set Learning: While works [4, 5] use hypernetworks (typically 100× larger than base networks), our low-rank tensor approach dramatically reduces the computational costs, making Pareto exploration feasible for large-scale models.

R5: Comparison with LLM alignment work [6] Note that [6] is indeed a concurrent work with 3 key differences:

1. Problem Formulation: [6] is used for LLM alignment, while this paper is the first to apply MOO to model merging (both data-free and data-based). As mentioned in R4, it is non-trivial. As can be seen from empirical results, this leads to the ability to generate different models for different preferences and improved performance compared to the baseline algorithms.
2. Efficient Low-Rank Tensor Structure: [6] uses a SVD-LoRA-based approach, while we propose a low-rank tensor structure. In the following, we adapt the approach in [6] for use in our model merging setting. We set the rank in their approach to 16, so that its number of parameters is comparable with ours. Using the setup in Section 4.2.2 and Table 2, the results on merging eight ViT-B/32 models are: |Method| Model|Test accuracy| |---|---|---| | AdaMerging+PM (equal)|ours | 84.9| | AdaMerging+PM (equal) | structure in [6]|84.1| | AdaMerging+PM (priority)| ours | 85.5| | AdaMerging+PM (priority) | structure in [6]|84.4|

As can be seen, our low-rank tensor structure outperforms [6], particularly in the priority preference setting. This is because while [6] assigns a fixed singular matrix row to each objective, our tensor $G$ adaptively learns the relationships between objectives and so our structure is more flexible.

3. Optimization: [6] focuses on LLM alignment using labeled data, whereas we focus on model merging with either no data or unlabeled data. This distinction presents overfitting challenges, which we address using an efficient structure and tensor regularization.

We will include the above discussions in the final version.

Response to Weakness 1

R6: Training and inference times Table 1 is on training time. As can be seen, our method has small computational overhead compared to the baselines. Specifically, during training, for a layer, AdaMerging optimization has a per-iteration complexity of $O(Kcd)$, while ours is $O(Kcd + (c+d)r + Kr^2)$ (where $K$ is the number of models, $c \times d$ the shape of the layer parameter, and $r$ is the rank.). As $r$ is much smaller than $c$ and $d$, the computational overhead is small.

For inference, once the preference is fixed, the low-rank tensor can be merged into the base model, resulting in no inference overhead.

Response to Weakness 2 and Questions For Authors

Please refer to R5

---

We hope the above responses address your concerns. If you have any additional questions or suggestions, we are more than happy to discuss them further.