Sparta Alignment: Collectively Aligning Multiple Language Models through Combat

We would like to thank the reviewer for their thoughtful comments and feedback.

Although the authors mention bias multiple times in their motivation, they do not report any corresponding results, for example, a bias evaluation comparing BEST INIT and SPARTA.

We evaluate the judgment bias of BEST INIT and SPARTA-ed model via JudgeBench [1]. From the following table, SPARTA demonstrates a clear advantage in reducing judge bias, achieving the highest accuracy with no invalid responses. This supports our hypothesis that employing collective intelligence of diverse models can mitigate the self-bias [2]. We leverage JudgeBench [5] to evaluate the judgment bias of models. Specifically, each of the ten models used in Sparta is tasked with independently scoring a set of prompt–answer pairs. Based on the provided preference labels, we report the best accuracy achieved by any single initial model. To test our hypothesis regarding collective bias mitigation, we compute the numerical average of the scores from all ten models (excluding invalid scores), and use the resulting aggregate preference to calculate accuracy. This result allows us to assess whether ensembling judgments reduces bias and improves alignment with ground-truth preferences:

While the results are extensive and span a range of tasks, they are limited to the Qwen2.5 model. Including at least one additional model—whether open-source or closed-source—would strengthen the findings.

We extend our experiments on another set of models based on gemma-7b-it, and report our results below:

These results confirm that Sparta maintains strong performance even when applied to a different base model. In particular, Sparta achieves the highest scores on GSM8K and alpaca, and remains competitive on normad_country. This supports the generality of our method across both mathematical reasoning and instruction-following tasks, beyond a single model family.

Did the authors explore alternative models for the LLM-as-a-judge component, and if so, did they observe similar trends across their experiments?

In this work, our primary goal is to explore self-improvement within a multi-model system, where models learn from one another through competitive alignment. As such, we intentionally avoid relying on an external LLM-as-a-judge and instead design a fully internal mechanism, letting models judge each other based on preference signals that emerge from their interactions. And for evaluation on open-domained tasks, we try QwQ-32B as an alternative reward model, and find Sparta still outperforms other methods on the alpaca task.

[1] Tan, Sijun, et al. “JUDGEBENCH: A BENCHMARK FOR EVALUATING LLM-BASED JUDGES”, ICLR 2025.

[2] Xu, Wenda, et al. "Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement." ACL 2024.

We would like to thank the reviewer for their thoughtful comments and feedback.

I recommend adjusting the size of some of the figures/tables to make them more visually appealing. For example, reducing the size of Tables 6 and 7.

We agree that visual clarity is important and have carefully considered the sizing of Tables 6 and 7 to ensure readability and consistency with the surrounding text. That said, we are happy to adjust their sizes in the camera-ready version to enhance visual appeal.

Is this research the first to propose the idea of aligning multiple large models simultaneously and leveraging their differentiated capabilities? I believe this is an important question. If so, the authors could emphasize this point in the paper. If not, they should introduce the existing methods.

To the best of our knowledge, our work is the first to align multiple large models simultaneously and explicitly leverage their differentiated capabilities to generate training pairs for preference learning, enabling self-improvement without relying on gold labels or human-written demonstrations.

Prior works such as [1] and [2] have explored multi-model settings, but their focus remains on improving supervised fine-tuning (SFT) or enhancing human-annotated examples. In contrast, our method harnesses the disagreement and diversity among multiple models to construct preference pairs without external supervision which are then used to improve the models collectively via alignment and competition.

[1] Subramaniam, Vighnesh, et al. "Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains."ICLR 2025.

[2] Wanjia, Zhao, et al. “SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning”, arxiv.

We would like to thank the reviewer for their thoughtful comments and feedback.

The method of generating preference samples to improve model is intuitive, but the method lack theoretical backing to explain multi-LLM alignment and self-improvement especially a theoretical justification of saturation or optima performance is necessary.

While there is theoretical research on LLM alignment [1,2,3,4], their assumptions might not apply to the nuances/complexities in our case.

1. We are jointly aligning multiple models, while existing theory work focuses on the behavior of a single model. This breaks independence and introduces aspects such as mutual information among models.

2. Our reward signals (using other LMs as evaluators) are evolving, since all models are iteratively changing in weights and subsequently reward modeling behavior.

While theoretical implications of multi-LLM and self-evolving alignment remains an open research question, we instead pursue empirical evidence to justify the steps in generating preference samples. We conduct a comprehensive ablation study across four diverse datasets (Alpaca, COM2, MedQA, and GSM8K) in Table 3, to briefly recap the results:

The results consistently show that removing any individual component of SPARTA ALIGNMENT leads to measurable performance degradation, confirming that the system’s effectiveness arises from the synergy of its modules.

It tests on a single model trained to generate different models, however this cannot be compared to multi-LLM system. In reality multi-LLMs are very diverse. Even though they may have similar performance, different LLM architecture have different format of answer generation and may favor different format of answers.

We additionally test out Sparta Alignment with a pool of heterogeneous model architectures, specifically 3 models fine-tuned from gemma-7b-it (code alpaca, cot and flan_v2) and 3 models fine-tuned from qwen2.5-7b-instruct (gemini_alpaca, lima, oasst1). We report the performance of best initial gemma-based models and Sparta-ed gemma-based models on three tasks:

Results show that Sparta Alignment is compatible with heterogeneous model pools. These results demonstrate that Sparta Alignment is compatible with heterogeneous model architectures, maintaining strong performance across both reasoning and instruction-following tasks. This supports the generality of our framework and its applicability in settings where model diversity is the norm rather than the exception. As this is a quick test with limited compute, we expect more models, architectures, and self-alignment iterations would further boost performance.

SPIN and other baselines mentioned in the paper are not multi-LLM baseline. Multi-LLM baseline should be uses

While inference-time techniques such as LLM debate and self-consistency may show gains during multi-LLM inference, our work targets a training-time approach with self-improvement setting, where multiple models interact to generate preference data and update model parameters through training. In contrast, methods like LLM debate and self-consistency operate purely at inference time and do not involve any model updates.

Therefore, while methods like SPIN are more comparable in terms of their training-based objective, LLM debate and self-consistency differ fundamentally in that they do not perform any model updates, and thus address a different goal. That said, we view these methods as complementary—SPARTA ALIGNMENT can potentially benefit from inference-time enhancements as well, and integrating the two is an exciting direction for future work.

Importantly, to the best of our knowledge, this is the first work that applies preference learning in a multi-model self-improvement framework. As such, existing multi-LLM baselines—whether inference-only or alignment-focused—do not fully align with our setup. This makes direct baseline selection nontrivial, and we have chosen the most relevant training-based methods currently available.

Nevertheless, we additionally compare with two multi-LLM non-training baselines, self-consistency and multi-model debate.

For self-consistency, we report best majority voting accuracy over 10 samples (maj@10) from the best init model:

Sparta outperforms Self-consistency on gsm8k and normad_value tasks.

Due to context length limits, we use 9 models to participate in the debate and 1 model to summarize the content. All 10 models are identical to those used in the initial Sparta setup. For fair comparison, we run 8 rounds of debate and report the best performance:

Across all three tasks, our proposed Sparta method consistently outperforms the debate baseline. This demonstrates the effectiveness of Sparta in collaborative self-improvement, especially under subjective and open-ended evaluation settings.

What is the main motivation of this work: self improvement or multi-LLM alignment. I feel the baselines corresponds to self-improvement and are for single LLM. This work is also trying to improve the same model architecture.

The main motivation is to improve self-alignment, while multi-LLM is only a means to the end, specifically by leveraging a competitive interaction strategy among multiple models. Rather than improving a single LLM in isolation, our method facilitates collective self-improvement within a multi-agent system, where models co-evolve by learning from and challenging each other. As such, much of the baselines and evaluation focus on self-alignment methods and evaluation.

Firstly this work is using 10 instances of the same models in the experiments. However methods such as SPIN, SPPO do not need training of 10 different models. Hence a direct comparison with this work should also take into account the extra computational load.

We would like to clarify that we have explicitly discussed the computational implications in both the main text and the limitations section. Specifically, in lines 159–161, we compare inference costs and show that SPARTA requires equal or even lower inference time than other baselines under our experimental setting. Additionally, in lines 774–775, we clearly acknowledge that SPARTA ALIGNMENT involves greater computational resources due to multi-model collaboration, which we frame as a trade-off for enhanced alignment dynamics and substantial weak-to-strong generalization, that is to say, weak models at first can become the most powerful models finally.

I also wonder if more simpler approaches like train the model first on the complete training data and then use best-of-N, SPIN or SPPO to make pairs to improve it further.

Our work targets at a realistic setting where high quality ground truth or preference labels are hard to get at scale. We would like to clarify that our method does not rely on external supervision and do not assume access to external reward models, ground truths, etc. Instead, all preference data is generated and judged dynamically by the pool of models that are collectively evolving. So we can’t “train the model first” as there is no training data available.

Our approach enables emergent self-improvement through multi-model interaction, where models co-evolve via competitive alignment and collective feedback. This dynamic interaction helps mitigate biases (e.g., from a fixed judge) and improves robustness without relying on predefined external metrics.

We believe this shift from single-model fine-tuning to multi-agent alignment-based self-improvement is a key contribution of our work, and we view it as complementary rather than redundant to existing simpler methods.

this work should be tested on different and diverse models.

We extend our experiments on another set of diverse SFT models based on gemma-7b-it, and report our results below:

These results confirm that Sparta maintains strong performance even when applied to a different model architecture.

[1] Rafailov, Rafael, et al “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, NeurIPS 2023.

[2] Wu, Yue, et al “Self-Play Preference Optimization for Language Model Alignment”, ICLR 2025

[3] Meng, Yu, et al “SimPO: Simple Preference Optimization with a Reference-Free Reward”, NeurIPS 2024.

[4] Shao, Zhihong et al “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”, arxiv.

We would like to thank the reviewer for their thoughtful comments and feedback.

The approach requires maintaining and fine-tuning multiple large models in parallel, which may be infeasible for many labs or practical deployment scenarios.

While our default setup uses parallelization to accelerate the inference and alignment process on 5 GPUs, Sparta Alignment can also run on a single 40GB GPU by serializing the computation. Our implementation supports any (# model, # GPU) combinations.

Empirically, in our main experiments, SPARTA runs on 5 40GB GPUs for around 6 hours for training 10 models with 8 iterations. We personally would’t call 5 GPU for 6 hours “infeasible” amount of compute.

The proposed method can be further verified in a real-world test.

While our current experiments are conducted in controlled settings, we have carefully designed them to closely reflect practical usage scenarios and challenges. We agree that deployment in real-world environments would be a valuable next step and plan to pursue this direction in future work. If you have a specific type of real-world test in mind, we would be grateful for further clarification to better align with those expectations.

Using peer models as judges might risk reinforcing collective biases, especially if all initial models share certain weaknesses or cultural biases. Given that LLMs act as judges for each other, is there a risk of reinforcing shared biases? Is it possible to foresee and avoid this?

Thank you for the insightful comment. We agree that using peer models as judges may risk reinforcing shared biases, a challenge that is not unique to our method but common across all existing self-improvement frameworks that rely on LLM-as-a-judge. To mitigate this issue, we deliberately employ ten diverse LLM instances with differentiated capabilities. This diversity enables a richer spectrum of perspectives in the evaluation process, reducing the likelihood that a single dominant bias will propagate unchecked.This design draws on the Language Model Council framework [1], which shows that a pluralistic set of judges leads to more robust, human-aligned, and bias-resistant evaluations.

This idea also resonates with the philosophy behind Modular Pluralism [2] which advocates for representational diversity in alignment by incorporating community-specific language models. While Modular Pluralism focuses on pluralistic human values, our approach similarly embraces model-level pluralism to foster more balanced judgment dynamics within a self-improving system.

Could you provide detailed compute cost estimates (e.g., GPU hours, memory usage) compared to self-alignment methods?

For parallel structure, we use 5 40GB GPU to run about 7 hours to finish the whole process with 10 models with transformers to roll out. For SPIN and SPPO, the whole process lasts for 8 hours: 1 40GB GPU for rolling out responses and 4 40 GB GPU for training models; for self-rewarding, we use a single 40 GB GPU to run about 8 hours; and for meta-rewarding, we use a single 40 GB GPU to run about 20 hours.

How might this method be used in practice?

In real-world deployment settings where human evaluation is costly or infeasible at scale, our method allows a system of models to self-improve through structured multi-agent interaction. By enabling models to critique, compete, and align with one another, we effectively offload the supervisory burden from humans to an automated process—offering a path toward scalable, low-cost, and high-quality oversight. Furthermore, while self-alignment based methods tend to converge early and then plateau, SPARTA provides more consistent improvement over iterations.

Can this method effectively scale to larger models? Since larger models might generate more diverse and less biased outputs, would this affect the system's dynamics?

Yes, our method can effectively scale to larger models. In principle, the alignment and self-improvement dynamics are model-agnostic and can be applied to stronger or larger base models. Practically, we can develop more efficient multi-GPU parallelization strategies to support larger model sizes without compromising interaction throughput if needed.

[1] Zhao, Justin, et al. “Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks”, NAACL 2025.

[2] Feng, Shangbin, et al. "Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration" EMNLP 2024.