Fusion-$\mathcal{X}$: Advancing LLM Ability with Adaptive Heterogeneous Model Integration

The main concerns I have this paper are the lack of novelty and motivation. In terms of novelty, it seems a somewhat simple idea to "learn" the fusion in some way. The terminology used in the paper makes their scheme look a lot more complex than it really is. For instance, Dynamic Candidate Selection is simply thresholded softmax selection using a hardcoded (magic) threshold. The adaptive selection network is a bunch of FFN layers and dynamic weighted fusion is simply selection based on argmax after some softmax selection. The authors also use a loss function to prevent selection collapse and this loss is similar to the loss proposed in [1].

My other main concern is the motivation behind this fusion. If your target LLM is not powerful, it is not going to become as great as the base LLMs. If a base LLM is already pretty good, then it is perhaps better to use it. The authors motivate their work by saying that organizations find it hard to train models given limited data and training resources. But then they do not demonstrate the case where this problem is solved because their results show slightly better performance on tasks which the base models are already pretty good at. For example, Table 3, last line, StarCoder is better on average than all the fused models, so why would we not simply use that model?

I have one more concern about the paper. This is not a weakness but it would be interesting to know the computational overhead introduced by your proposed scheme. If I understand correctly, the input to the adaptive network is a concatenation of probability distributions over model vocabs for a sequence. This is a really huge vector as it is (Seq_Len * Vocab_size) so what exactly are the details for the architecture of the MLPs used?

The notation in the paper is a bit confusing. Firstly, $i$ is used to index multiple terms, the LLM and the $k$th token being referred to on line 192. Secondly, the refer to $P_{t}^{\theta}$ as a matrix but denote it as a rows of distributions concatenated together in Line 183.

The citations include author names but not putting brackets around them makes the text look quite confusing.

Typo and Grammar checks Line 24 - from a flexible number model candidates Line 82 - , where existing methods > which existing methods

[1] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." arXiv preprint arXiv:1701.06538 (2017).

1. The paper highlights the benefits of model fusion but could better specify scenarios where it is essential, such as in domains with limited data or the need for rapid deployment, where integrating specialized small models with large general models could be transformative.
2. Despite the potential of fusing models, the paper does not investigate whether this approach effectively bridges the specific capability gap between these disparate models.
3. The effectiveness of the feedback-driven loss function may be compromised when there is a significant disparity in model capabilities. The paper should address how this method performs under such conditions.
4. Although comparisons are made with LLM-Blender, OAssistRM, UltraRM, and FuseLLM, the paper lacks evaluations against traditional ensemble and weight merging methods, which would provide a more comprehensive assessment of FUSION-X's performance. FUSECHAT was not selected as a baseline for comparison.
5. The discussion on model scaling does not adequately address the integration of newer models like Llama 3.1 8B and Qwen 2.5 into the fusion process.
6. What specific benefits does the dynamic fusion of these smaller models offer over the larger model in terms of performance, efficiency, or adaptability? For example, the advantages of using four 2B models compared to a single 8B model are not clearly articulated.

1. The main challenge is not clear and therefore the contribution is vague, i.e., not clear what problem is addressed. The authors discussed many challenges in the 2nd paragraph of the introduction, but none of them have a clear definition, including "simple training", or "interference". What kind of training is simple and why is it important for the fusion method? What is interference? The performance degradation under two different tasks? No citation or definition is ever provided in the introduction. The so-called reduction in interference is also confusing at the end of the introduction. What kind of measure is used? The goal of the work seems to address multiple limitations as shown in Figure 1. But in the contribution, the authors only mention interference and performance improvement by improving interference.
2. The whole method lacks intuition and therefore the paper is hard to follow. What is the intuition behind the proposed method to reduce interference?
3. The proposed method is very incremental, probably because it does not bring in any new insights on the fusion problem itself. The method combines the fusion method, model selection, and feedback-driven loss. Yet, in the ablation study (Sec 6, Table 3), the components are not ablated individually. Thus, it is not even clear how this method work? Which component is the key contributor? Note there is a typo on Sec 6, it should be Table 3 instead of Fig 3.
4. In the ablation study (Table 3), the proposed method only marginally outperforms the FuseLLM. In many tasks, the new method is even worse than the FuseLLM. Without standard deviations, it is hard to justify if the proposed method is better than FuseLLM.

1. The presentation is problematic. Section 4 (methodology) first introduces the adaptive selection network in Section 4.1, but its training pipeline is deferred to Section 4.3. The potential methods explored during the design process in Table 1 are hard to interpret without formally introducing the pipeline. This presentation issue makes the thresholding and weighting operations in Sections 4.2 - 4.3 hard to follow.
2. The design choice of the adaptive selection network is unclear. The weighted model merging problem aims to identify the optimal weight for each candidate model. Here, the input to the adaptive selection network is fixed. The adaptive selection network only takes one input for a given set of candidate models. I'm not sure whether we need to train a network work to make predictions over a single input data point.
3. Identifying the optimal weight is a hyper-parameter optimization problem (HPO). In auto-ML literature, the strong baselines for HPO include Bayesian optimization and bandit. However, these baselines are missing.