Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence

We would like to thank the reviewer for their thoughtful comments and feedback.

It would be beneficial to include a convergence analysis of the proposed update mechanism to better understand the theoretical properties and stability of the optimization process.

We investigate convergence and stability empirically in several studies in the appendix.

In line 872 and Figure 6, we visualize the search trajectory of models. Starting as diverse model checkpoints, they gradually converge to an area in the weight space that optimizes utility function f.

In line 860 on the right, we identify the three randomness factors in Model Swarms: while the algorithm is not 100% stable and deterministic, these stochastic factors actually help exploration and performance in Table 5.

In line 1028 on the right and Figure 15, we demonstrate the convergence of model performance across tasks and optimization iterations. The stability of the search is task-dependent, with some per-model performance plots smooth and some turbulent. Nevertheless, the global best performance steadily increases across tasks.

There are theoretical analyses of PSO convergence in evolutionary algorithm research such as [1], while the assumptions made in the classic optimization problems are not applicable for the LLM setting. For example, the initial particles in classic PSO are often initialized randomly or in a grid, while the initial LMs in the swarm cannot be random/arbitrary (taking a random point in the 7-billion-dimensional space would almost certainly be a failed language model). Instead they are seeded with curated model checkpoints with different training and fine-tuning data mixtures. Given these differences, we decided to take an empirical route to analyze convergence and stability properties.

[1] Van den Bergh, Frans, and Andries P. Engelbrecht. "A study of particle swarm optimization particle trajectories." Information sciences 176.8 (2006): 937-971.

Could the authors clarify the computational cost associated with training the proposed method and provide a detailed comparison of training efficiency relative to the baseline approaches?

The main computational cost comes from model inference and evaluating the LM checkpoints on the utility function, most simply performance on a small validation set. Trivial and static baselines in Table 1 are less expensive since they don’t require this evaluation. Model Swarms is on par with the dynamic baselines such as EvolMerge and Lorahub, evaluating models on the validation set at each iteration.

We refer the reviewer to lines 980 in the appendix for a more detailed discussion on the computational cost. To recap, Model Swarms has linear complexity to the number of models and to the cost of one model inference, while it takes about 10-15 iterations for each run on average. Empirically, with 5 40GB GPUs you could run a Model Swarms search under an hour, with 2 GPUs you need about 3 hours. We propose further acceleration with dropout-k and dropout-n in Figure 7.

We would like to thank the reviewer for their thoughtful comments and feedback.

Besides, the proposed method does not provide the alignment method for LLMs with different sizes, which may limit the further application of Model Swarms.

Usually the initial expert LLMs for different fields have different model architectures and model sizes, since the proposed method could only be applied to a set of LLMs with the same architecture and size, the contribution of the proposed method might be limited.

We propose token swarms on line 372, an extended version to compose models with different sizes/architectures, by conducting PSO on the token probabilities space.

Instead of defining the search space as model parameters, in token swarms the PSO search determines how token probability distributions from heterogeneous models should be aggregated. We show that this works for a swarm of 4 Gemma and 4 Mistral models in Figure 5, with performance improvement regardless of the architecture.

The token probability variant takes inspiration from [1] and it provides great context about aggregation in the token probability space, reconciling tokenizer differences, etc.

[1] Liu, Alisa, et al. "Tuning Language Models by Proxy." First Conference on Language Modeling.

However, this demonstration doesn’t show the efficiency of the searched LLM over state-of-the-art LLMs on these tasks.

It makes sense that the searched LLM could surpass the initial models, but the absolute performance is not evaluated.

Besides, the state-of-the-art LLMs for these tested tasks and the GEMMA-7B models fine-tuned on these tested tasks are not included. Since fine-tuning GEMMA-7B for 5*10 epochs (the resource used for training the initial models) on the target tasks may obtain better performance.

The reviewer asks for two comparisons: Model Swarms against fine-tuning Gemma-7B on the target task, and Model Swarms against “state-of-the-art LLMs”.

The former is a fair comparison. In line 1065 and Table 9 we compare Model Swarms against direct fine-tuning on the task data of the utility function. We see consistent improvement. The main issue here is generalization: when the available data is small, direct fine-tuning tends to memorize with very large validation-test set gaps while Model Swarms leads to better skill learning and composition. We fine-tuned for 5 epochs in Table 9 and empirically see that fine-tuning for 5*10 epochs exacerbate memorization and harm generalization.

We are unsure if the latter is a fair ask. We don’t think taking 7B models and applying adaptation would outperform “state-of-the-art” models that are much larger and much more extensively trained. We don’t see the goal of this work as establishing a new state-of-the-art on the evaluated tasks, but rather demonstrating the adaptation and composition of diverse LMs could be achieved through collaborative search and swarm intelligence. Thus we believe our selection of 12 model composition baselines is adequate for this purpose.

However, the initial models are fine-tuned on data in Tulu-v2 which are significantly different from the tested tasks, which may lead to biased evaluation.

This difference between SFT data and evaluation tasks is actually a good thing, meaning that the initial swarm was not overfitted for the evaluation tasks and there is genuine adaptation/generalization to these tasks on-the-fly in Model Swarms searches.

There are various advanced and adaptive PSO variants in EC community, I wonder why authors use a (relatively) simple PSO method, using advanced PSO might obtain better performance?

We agree that there is broad literature on evolutionary algorithms (EAs) about PSO and beyond, many of them could be adapted for today’s LLM research. Model Swarms take one of the first stabs at this interdisciplinary research direction so we decided to keep the EA part simple and straightforward, hoping to get the ball rolling on future evolutionary designs of LLMs.

We would like to thank the reviewer for their thoughtful comments and feedback.

Will the code be released publicly upon publication?

Yes, along with the best-found model checkpoints for all tasks.

We would like to thank the reviewer for their thoughtful comments and feedback.

How does the computational cost of ModelSwarms compare with baselines?

The main computational cost comes from model inference and evaluating the LM checkpoints on the utility function, most simply performance on a small validation set. Trivial and static baselines in Table 1 are less expensive since they don’t require this evaluation. Model Swarms is on par with the dynamic baselines such as EvolMerge and Lorahub, evaluating models on the validation set at each iteration.

We refer the reviewer to lines 980 in the appendix for a more detailed discussion on the computational cost. To recap, Model Swarms has linear complexity to the number of models and to the cost of one model inference, while it takes about 10-15 iterations for each run on average. Empirically, with 5 40GB GPUs you could run a Model Swarms search under an hour, with 2 GPUs you need about 3 hours. We propose further acceleration with dropout-k and dropout-n in Figure 7.