Model Fusion through Bayesian Optimization in Language Model Fine-Tuning

We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.

[W1, Q3, Q4] Empirical observations of discrepancy between metric and loss landscape, and hyperparameter alignment are performed on the RoBERTa model. Thus it requires results on Llama models. Also how does the accuracy computed on RoBERTa?

First, we want to clarify that we use a classifier head for the classification task because our task involves multiple-choice questions, allowing us to directly output an answer and measure accuracy. However, we want to emphasize that BOMF is not limited to this specific scenario. It can be applied to any model as long as the loss and metrics can be calculated. Since fusion does not require backpropagation, BOMF is equally applicable in situations where there is no classifier head. And in our paper, we demonstrated that both full model fine-tuning and LoRA fine-tuning exhibit 1) metric and loss misalignment and 2) optimal hyperparameters alignment even when freeze layers or ranks change. However, we agree that extending these experiments to larger-scale models would strengthen our findings. Therefore, we conducted additional experiments on the Llama-2 model to verify if 1) and 2) still hold. Firstly, regarding point 2), the alignment of hyperparameters can be easily observed in Figures A.1. For point 1), the misalignment between loss and metrics is evident in Figure A.2. To further show the misalignment between loss and metrics, as well as among different metrics, we measured Spearman’s correlation using the loss and metrics of 20 sampled models. Also, we empirically prove that BOMF significantly improves this correlation by considering both loss and metrics simultaneously. Refer to Tables 9, 10, and 11 in Appendix C for full experimental results. We have included a portion of these results here. According to Table R.2 (c), (d), and (e), language tasks involving the Llama model show significantly lower correlation compared to vision tasks (a) and (b). Then, Table R.3 shows that BOMF successfully increases the correlation between loss and metrics compared to other baseline methods.

[W2, Q1, Q2] How much extra time is required for BOMF.

Please refer to our global response for answers.

[W3] Performance improvements of BOMF are very small

The improvement of BOMF is not negligible compared to other baselines. First, we would like to emphasize that our method consistently outperforms various baselines across a wide range of task types (i.e., classification, question answering, and the medical domain) and models (i.e., T5, LLaMA3). While the degree of improvement might appear marginal in some cases, it is essential to recognize that the extent of improvement, whether small or large, is relative. The fact that our method consistently achieves superior performance in nearly every case serves as an absolute metric that is universally recognizable. Our method consistently outperforms the baselines in these diverse scenarios. Specifically, BOMF achieved a 13%, 14%, 10% error improvement in the GLUE tasks, SQuAD task and KorMedQA tasks, respectively.

Furthermore, the recent fusion research [1,2,3] in various scenarios has shown similar tendency to our results, showing that even if the improvement in specific tasks seems marginal, they exhibit overall superior performance compared to the baseline, proving the necessity of such methods. In particular, even when compared to these methods, our method demonstrates improved performance in almost all experimental results, indicating its effectiveness.

[Q5] Can you provide an algorithm description of MOBO in the main paper? Specifically, how is qNEHVI done?

We have already described the details of our algorithm in Sec A.1.3, B, and Algorithm 1 of the appendices. Due to the limited space of the NeurIPS submission, they should be located in the appendices. We will reorganize our contents in the final version.

As described in our manuscript, qNEHVI sequentially samples optimum candidates by maximizing the expected hypervolume improvement, which can be considered as an acquisition function in Bayesian optimization. At every iteration of MOBO, it chooses the maximizer of the alternative objective rather than one of multiple unknown target objectives. We will clarify it in the final version.

[Q6, Q7] Where is freeze in Table 1 and rank 4 in Table 2?

Thank you for pointing out the confusion. The dagger symbol ($\dagger$) in Tables 1 and 2 indicates whether low rank or freeze layers were used. Specifically, "BOMF with dagger symbol ($\dagger$)" refers to cases where the best hyperparameters were determined using full rank or full layer settings, and these hyperparameters were then used for training before performing fusion. On the other hand, "BOMF without a dagger symbol" refers to cases where low rank or freeze layers were used to find the best hyperparameters. This information is clarified in the footnote on Page 8 of the paper.

[Q8] Why are there no grid fine-tune results in Table 2?

The models used in Table 2 are LLMs, so even when tuning with LoRA, the forward path takes a considerable amount of time. This made it time-prohibitive to perform grid search, which requires a relatively large number of iterations to find the best-performing hyperparameters. However, to strengthen our argument in line with your suggestion, we are running the grid search in the remaining time and will present the results as soon as they are available.

References

[1] Wortsman, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022.

[2] Weng, R., et al. G-tuning: Improving generalization of pre-trained language models with generative adversarial network. ACL, 2023.

[3] Malladi, S., et al. Fine-tuning language models with just forward passes. NeurIPS 2023.

We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.

[W1] There are no discussions on computational efficiency: More detailed complexity analysis or comparisons could help readers understand the practicality of applying BOMF at scale

Please refer to the general response for the analysis and additional experiments regarding the computational cost.

[W2] Hyperparameter sensitivity analysis: The paper could provide a more in-depth analysis of how sensitive BOMF is to the choice of hyperparameters for the Bayesian optimization process itself

Since the test suites used in this work require huge compute resources, it is difficult to provide hyperparameter sensitive analysis on Bayesian optimization processes. It is noteworthy that the hyperparameters of Bayesian optimization can be thought of as meta-hyperparameters, which should be tuned with multiple tasks and multiple domains. It implies that it necessitates utilizing more compute resources. In addition, from the standpoint of black-box optimization, the sensitive analysis assumes that the specifics of objective functions are known, which is generally infeasible in black-box optimization. Therefore, instead of analyzing the sensitivity of the hyperparameters of Bayesian optimization, we followed their conventional settings. All hyperparameters related to Gaussian processes are optimized via model selection. Matern 5/2 kernel for Gaussian processes and the logarithmic form of expected improvement are used.

We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.

[W1, W2] It feels as though there are two distinct contributions that are not necessarily related. Because of this disconnect, one of these contributions feels less important than the other, resulting in the MOBO for MF feeling under-explored as the “main contribution”.

To simplify the notations, we will refer to BO-based hyperparameter selection as outer BO and BO-based coefficient search as inner BO. The bottom line is that the outer BO process and the inner BO process are coupled with each other, so they were not conducted for different purposes.

As outlined in Section 1, BOMF aims to construct the best-performing single model through model fusion within a parameter space. As detailed in Section 4.2, different combinations of fine-tuning hyperparameters (such as learning rate and batch size) yield varied generalization performances after the fine-tuning process. Figure 3 demonstrates that the generalization performances of models before and after fusion align well. Therefore, to identify the best-performing single model after the model fusion, we must first determine the optimal hyperparameters that yield the best-performing single model before the fusion.

For a simpler explanation, let us consider two fine-tuning hyperparameter configurations, H1 and H2. Figure 3 illustrates that if the performance of the fine-tuned solution from the training trajectory with H1 surpasses that of H2, then the performance after the model fusion with fusion members S1, sampled from the trajectory with H1, remains higher than that of the fusion members S2, sampled from the trajectory with H2. This finding, coupled with the non-differentiable nature of metric functions, underscores the necessity of identifying the best hyperparameter configurations through outer BO to achieve BOMF's ultimate objective with inner BO.

[W3] Both the misalignment between metric and loss surface, and the alignment of optimal hyperparameters over capacity are only shown over a single dataset and model setting.

In our paper, we demonstrated that both full model fine-tuning and LoRA fine-tuning exhibit 1) metric and loss misalignment and 2) optimal hyperparameters alignment even when freeze layers or ranks change. However, we agree that extending these experiments to larger-scale models would strengthen our findings. Therefore, we conducted additional experiments on the Llama-2 model to verify if 1) and 2) still hold.

Firstly, regarding point 2), the alignment of hyperparameters can be easily observed in Figures A.1. For point 1), the misalignment between loss and metrics is evident in Figure A.2. To further demonstrate the misalignment between loss and metrics, as well as among different metrics, we measured Spearman’s correlation using the loss and metrics of 20 sampled models. Also, we empirically demonstrate that BOMF significantly improves this correlation by considering both loss and metrics simultaneously. Please refer to Tables 9, 10, and 11 in Appendix C for full experimental results. We have included a portion of these results here. According to Table R.2 (c), (d), and (e), language tasks involving the Llama model show significantly lower correlation compared to vision tasks (a) and (b). Then, Table R.3 shows that BOMF successfully increases the correlation between loss and metrics compared to other baseline methods.

[W4] All results are taken from a single run.

Thank you for your insightful comment regarding the use of random seeds. We agree that running multiple experiments with different seeds is crucial for ensuring the reliability and completeness of the results. Accordingly, we will provide experimental results obtained using multiple seeds for each experiment. Due to computational limitations during the rebuttal period, we were only able to add results for the medium-sized language models in Table 1 in the main paper, using five seeds per method and dataset. This result can be found in Table R.6 and Table R.7.

Additionally, we plan to conduct further experiments on large language models and update all tables with results from multiple seeds in the camera-ready version of this paper.

We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.

[W1] The observed improvements from the proposed BOMF method are relatively small

The improvement of BOMF is not negligible compared to other baselines. First, we would like to emphasize that our method consistently outperforms various baselines across a wide range of task types (i.e., classification, question answering, and the medical domain) and models (i.e., T5, LLaMA3). While the degree of improvement might appear marginal in some cases, it is essential to recognize that the extent of improvement, whether small or large, is relative. The fact that our method consistently achieves superior performance in nearly every case serves as an absolute metric that is universally recognizable. Our method consistently outperforms the baselines in these diverse scenarios. Specifically, BOMF achieved a 13%, 14%, 10% error improvement in the GLUE tasks, SQuAD task and KorMedQA tasks, respectively.

Furthermore, the recent fusion research [1,2,3] in various scenarios has shown similar tendency to our results, showing that even if the improvement in specific tasks seems marginal, they exhibit overall superior performance compared to the baseline, proving the necessity of such methods. In particular, even when compared to these methods, our method demonstrates improved performance in almost all experimental results, indicating its effectiveness.

[W2, Q2] The study relies on older base models (RoBERTa and T5) and simple datasets. How did the method perform in recent models such as the Llama family?

We have already conducted experiments using Llama-2 and Llama-3 in Sec 6.2 of the main paper, and included the results for summarization, Korean multi-choice medical question answering, and dialogue generation. In most cases of these three experimental settings, our method outperforms other baseline methods. For detailed experimental setting and results, refer to Table 2 in the main text and Tables 15 and 16 in the appendices.

[W3, Q1] The evaluation of generative tasks lacks the use of LLM-based evaluation or human evaluation. How to deal with complex evaluation criteria commonly seen in LLM evaluation?

Thank you for suggesting additional evaluations on generative tasks. We would like to address this question from two major perspectives:

1. When a large language model (LLM) is used to evaluate the outputs of various models, BOMF demonstrates superior performance compared to the existing baselines.

2. BOMF can utilize more complex evaluation criteria such as human evaluation or LLM assessment in the process of model fusion.

Regarding the first one, following your suggestion, we conducted an evaluation with a ChatGPT-turbo-3.5-based approach. This evaluation method involves generating scores by asking the LLM to assess the similarity between the generated responses and the ground-truth answers. Using this, we compared BOMF's performance with other baselines. Here, BOMF refers to the model optimized with the R1, R2, and RL metrics. As shown in Table R.1, even when optimized with this specific metric, BOMF outperformed other baselines. This indicates that our approach not only excels with traditional metrics but also adapts well to the latest evaluation methods.

Additionally, the consideration of both loss and multiple metrics helps the model remain robust across various unseen metrics. Furthermore, since BOMF uses BO to optimize combination coefficients, it requires only the evaluation metric values corresponding to each set of combination coefficients to update them. This means that the optimization process does not rely on a backward process through the metric; as long as evaluation values are available, BOMF can optimize regardless of the complexity of the evaluation procedure. To illustrate this, we conducted optimization using metrics evaluated by LLMs, denoted as the column of ChatGPT BOMF in Table R.1. The result shows that BOMF can optimize the coefficients and achieve performance improvements even with these complex metrics.

[W4] RLHF and DPO can directly optimize towards desired metrics, potentially bypassing the need for fusion altogether.

We agree that methods like RLHF can be used to optimize desired metrics. However, we want to emphasize that our method complements rather than contradicts methods like RLHF or DPO. For instance, the hyperparameters used in RLHF or DPO can be optimized more efficiently using BO with a lower rank or fewer layers. Also, when performing RLHF or DPO, multiple checkpoints can be generated during training procedure. Typically, the best validation performance checkpoint is selected from these. However, with BOMF, these checkpoints can be fused to produce better results. By combining multiple checkpoints through BOMF, better generalization performance can be achieved compared to selecting a single checkpoint. This capability is well illustrated in our experiments, as shown in Tables 2 and 4. Another important point is that BOMF can consider multiple metrics simultaneously. While methods like RLHF or DPO require detailed preference settings for each metric in different scenarios, BOMF automatically balances various metrics and loss functions to find the optimal combination. To demonstrate that RLHF and BOMF can indeed be combined, we conducted additional experiments. According to Table R.2, performing various fusion methods on RLHF training checkpoints showed that BOMF outperformed other methods.

References

[1] Wortsman, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022.

[2] Weng, R., et al. G-tuning: Improving generalization of pre-trained language models with generative adversarial network. ACL, 2023.

[3] Malladi, S., et al. Fine-tuning language models with just forward passes. NeurIPS 2023.