Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers

Thank you for your thoughtful review and questions. We appreciate your recognition that our experimental results are interesting and show improvements over other best-of-n selection methods. We're excited about the potential for scaling verification compute by using more diverse verifiers, in addition to just scaling generation compute. Below we address each of your points and provide additional experiments and clarifications that we believe strengthen our contributions and address your questions.

The method is similar to the one used in Constitutional AI (https://arxiv.org/pdf/2212.08073) that also uses various prompting of verifier LLMs to select the best answers. There are some differences (the direct usage here vs. RL training there, the averaging of the verifiers answers here instead of random selection of verifiers there). But such approaches should be discussed as similar to yours.

Thank you for pointing out this connection. We agree that Constitutional AI (CAI) uses LLM-based evaluation to improve performance, it was one of the first major works demonstrating reinforcement learning from AI feedback (RLAIF) for LLMs. CAI focused on improving alignment, by having the model respond to a prompt, critique the response, revise it, and then train on the improved version (as well as other similar versions of this technique, like generating preferences, etc.). That is, the focus was on improving alignment through train-time RL using the model's own feedback (and a set of constitutional principles).

Our paper has a different contribution: we propose a new approach to test-time scaling that is orthogonal to the traditional scaling method of just sampling more solutions from the generator LLM. That is, we propose scaling verification compute by increasing the number and type of verifiers, without any training. Our results show that scaling both the number and diversity of verifiers can improve test-time performance, and we show that this approach has other important properties like enabling self-improvement and weak-to-strong generalization (Table 2), as well as raising the ceiling on test-time performance while existing methods plateau (Figure 6).

While both CAI and our work involve LLM-based evaluation, they address fundamentally different problems using different methodologies. CAI is an RLAIF train-time approach for alignment, while our work is focused on test-time scaling and introduces an approach where verification compute can be scaled using more verifiers. We will definitely add a discussion of Constitutional AI and these distinctions to the paper to better position our work within the broader landscape of AI feedback methods.

The results lack the comparison with an oracle baseline that would select the real best of n (pass@n). It would give a better idea of the ceiling performance of such BoN approaches.

Thank you for this suggestion. We agree that an oracle baseline (pass@n) would provide valuable insight into the upper bound on performance. Below are the pass@n oracle results for our two most challenging benchmarks, MMLU-Pro and GPQA (diamond). To save space, we have not included the other two benchmarks, but will include pass@n results for all benchmarks in the final version of the paper.

MMLU-Pro:

GPQA (diamond):

Note: Numbers represent the accuracy (%) of the applied method for selecting between 16 sampled solutions per question (n = 16).

There could be simpler performance evaluation of the approach by comparing the rates at which one of the correct answers (when it exists) among the n generated were selected by different methods. This would help decorrelate the base model performance and the performance of the proposed approach.

Thanks for making this suggestion. We agree that this would help decorrelate the base model performance. We have computed the accuracies only counting questions where at least one of the sampled solutions is correct (i.e., where pass@n is correct) and found that MAV performs similarly to the original results (see table below on MATH). We're happy to add these additional results to the final version of the paper.

[RESPONSE CONTINUES BELOW]

"redundant_steps": (
    "INSTRUCTIONS: \\n"
    f"Look for any unnecessary steps that don’t contribute to the final result. Think out loud. "
    f"If the solution includes redundant or misleading steps, stop and reply '{VERA_ANSWER_SYMBOL}False'. "
    f"If every step is necessary and useful, reply '{VERA_ANSWER_SYMBOL}True'."
),

Thank you for the additional questions. Below, we address your questions with additional clarifications and experiments.

On the pass@n table GPQA results in particular show a significant gap between any selection method and the oracle. This is actually a good sign that sampling methods without retraining can go a long way. However, this is surprising when I compare that with the selection accuracies that you give for MATH. Could you give the accuracies for GPQA too? they should be much lower. Why do you think the domain (math or code) affects the accuracy of your answer selection method so much?

Thanks for the question. Indeed, the GPQA accuracies when we consider only questions with at least 1 correct answer are lower all-around (see table below). This occurs because GPQA has only 4 multiple-choice answers (A, B, C, or D) while MATH answers are free-form numbers (the space of possible answers is much larger). Since we sample 16 solutions and there are only 4 possible answers for GPQA, it's very likely that at least one of the sampled GPQA solutions hits the correct answer by chance. It is much less likely to produce the exact answer for MATH by chance.

Given this, when filtering to questions where at least one sample is correct (pass@n = 1), GPQA includes most questions regardless of difficulty, but MATH only includes the subset where the model could solve the problem at least once. This makes the MATH task on questions where pass@n = 1 fundamentally easier, and that is why the accuracies for MATH are all-around higher than the original results in the paper.

GPQA (diamond) selection accuracies:

Note that this matches the results in the paper, where we find that MAV outperforms the baselines in all domains except for GPQA (diamond), where MAV performs similarly to RM and outperforms self-consistency.

About the test time compute scaling, I think you misunderstood my question. I'll try to be clearer. I saw your comparison figure 6 with other answer selection methods. I was asking you to compare with other types of test time compute methods i.e. thinking models. As the compute budget is scaled (number of samples or the number of thinking token) how does the accuracy scale for each type of test time compute method?

Thanks for the clarification and the interesting question. We have conducted additional experiments comparing our approach to a thinking method.

We first evaluated DeepSeek-R1, which achieves 87.7% accuracy on MATH using an average of 1,902 tokens per question. However, this is not directly comparable to our approach since R1 has been specifically fine-tuned for mathematical reasoning, whereas our approach works zero-shot with off-the-shelf non-RLVR models (no extra training).

For a more apples-to-apples comparison, we implemented a simple reasoning baseline where we iteratively prompt the same base models to continuously extend a chain of reasoning until we use a similar number of thinking tokens as our MAV approach on the MATH dataset (about 80k tokens per question on average). This comparison is more reasonable since both methods scale test-time compute using zero-shot prompting without fine-tuning.

MATH:

We find that when controlling for token count, our method performs better than simply scaling reasoning tokens within a single model call.

We hope this resolves your concern, we will definitely perform a more fulsome set of such experiments for the final version of the paper and welcome any particular suggestions for other test-time compute methods to include.

Thank you for your thorough review. We're glad you found our method interesting, and that you see it as a simple approach to effectively scale test-time compute without training verifiers. We address each of your comments and questions below. Please feel free to engage further if you have any additional feedback or questions.

I agree that the proposed method is convenient since we can use off-the-shelf LLMs as verifiers. However, it is still unclear whether multi-agent verification is an effective approach because off-the-shelf verifiers (GPT-4o-mini and Gemini-1.5-Flash) are generally more powerful than the 8B reward model used in RM. Therefore, it is possible that the proposed method's good performance comes from the power of large LLMs, not from the use of multiple agents.

Thanks for raising this. We used the strongest reward model we could run on academic compute—the Skywork 8B model, which ranked among the top 10 on RewardBench at the time of writing (outperforming many larger reward models). However, we acknowledge this as a limitation and will add it to the limitations/discussion section of the final paper. Moreover, we show weak-to-strong generalization, where by combining multiple weaker verifier agents we can improve the performance of stronger generator LLMs (i.e., Gemini-1.5-Pro). Additionally, we have shown below that using many separate verifiers to verify different aspects outperforms using a single LLM verifier (see experiment below).

Table 1 shows that the proposed method's gain over baseline methods tends to be small when generator LLMs are powerful. For example, GPT-4o shows the best performance among others in all the tasks, but B-MAV does not outperform Cons and RM for GPT-4o. Can you explain why the proposed method does not help much for strong generator LLMs?

Thanks for your question. Our approach is indeed weakest on GPT-4o. While we don't know exactly why this is the case, we have a few hypotheses. One possibility is that because GPT-4o is quite strong, it saturates performance on our benchmarks, leaving little room for improvement from verification. It might also be that current verifiers (GPT-4o-mini and Gemini-1.5-Flash) may not be sufficiently capable to consistently identify errors in such a strong model (GPT-4o was the SOTA model at the time of writing), suggesting that multi-agent verification will be more effective when we use stronger models as verifiers.

A simple baseline method is to ask a single agent LLM to evaluate a sentence from multiple aspects rather than using multiple agents. Have you tried this?

Thanks for making this suggestion. We have conducted an additional experiment where we ask the LLM verifier to generate a “rubric” containing the approval scores for each aspect in a single response. We find that generating a single rubric for all aspects performs worse than using a separate model call per aspect (see table below). This is likely because separate model calls allows the LLM verifier agents to put all of their “focus” on one aspect, rather than having to think about many different aspects in context. Having multiple LLM verifier agents may have more expressive power, and this could be an interesting direction for future work to explore further. Thanks again for your suggestion to try this. We will include these results and additional discussion in the final version of the paper.

MMLU-Pro:

GPQA (diamond):

Note: Numbers represent the accuracy (%) of the method used for selecting between 16 sampled solutions per question (n = 16).

Experiment Details: We instruct the LLM “rubric” verifier to think about each aspect in a single response and generate a python dictionary mapping aspect names to True/False values (after it has finished thinking about each aspect). We use the exact same 10 aspects as the original paper, but obtain the approvals for all aspects simultaneously in a single response rather than splitting across model calls. As in the original paper, we use Gemini-1.5-Flash and GPT-4o-mini as the base LLMs for our verifiers, so there are actually 2 model calls: Gemini-1.5-Flash generates one rubric and GPT-4o-mini generates another rubric. For simplicity, we do not perform any “aspect engineering” for this experiment. We compare to the “MAV-All” results in Table 5 of the Appendix (which uses 20 separate verifier calls: 10 aspects for 2 base LLMs).

Thanks again for your positive review. Please let us know if you have any other questions or feedback.

Thank you for the positive review of our work and for your constructive questions and suggestions, which we address below. Please feel free to engage further during this feedback period if you have any additional feedback or questions.

Given that many of their verifiers were state-of-the-art closed-source LLMs, choosing not to use as powerful an open-weight reward model feels like it's not a fair comparison of methods.

Thanks for raising this point. We used the strongest reward model we could run on academic compute—the Skywork 8B model, which ranked among the top 10 on RewardBench at the time of writing and actually outperformed many larger reward models. However, we acknowledge this as a limitation and will add it to the limitations/discussion section of the final paper.

Another strong area for comparison would have been reward models designed for reasoning models, like process reward models (PRMs) or fine-grained reward models. PRMs are cited and mentioned in the paper, but none were chosen for comparison.

We agree this would be an interesting comparison. We focused on outcome-level reward models and self-consistency since they are most similar to our approach in terms of selecting between complete candidate outputs (as opposed to intermediate steps). Due to time limitations and the additional complexity that PRMs add, we weren't able to include PRMs in this rebuttal period, but we will add a comparison to PRMs in the final version of the paper.

Lastly, no effort to evaluate heterogeneous reward models (the authors provide reasons why it would be difficult) was made. Even something as simple as finding an average threshold to use to convert all reward values to 0 or 1 and average would have been helpful.

We agree this would be a useful comparison to add. It would require evaluating each generated solution in our dataset with multiple reward models, which would take too long on our academic compute for this rebuttal period. However, we'll definitely include such a comparison (like your suggested average threshold method) in the final version of the paper. Thank you for the suggestion.

Thanks again for your thoughtful review and valuable suggestions. We believe these additions will significantly strengthen the paper, and we appreciate your recognition of the core contributions of our work. Please feel free to reach out if you have any additional questions or need further clarification.

Thank you for your review and feedback. We’re glad to hear that you found our method interesting and results strong. Please see below for responses to your comments and questions.

The method requires hand engineering multiple aspects for a given, limiting it's generality and the ability to scale the number of verifiers.

Thanks for raising this important point. We agree that hand-engineering every verifier would limit the generality and scalability of our approach. To address these concerns, we have conducted an additional experiment using fully AI-generated aspects. We find that using the AI-generated verifiers performs even better than our hand-engineered ones (see full experiment details below). We ran these experiments for Gemini-1.5-Flash as the generator LLM on MMLU-Pro and GPQA diamond (our two most challenging benchmarks):

MMLU-Pro:

GPQA (diamond):

Note: Numbers represent the accuracy (%) of the method used for selecting between 16 sampled solutions per question (n = 16).

These results suggest that aspects do not need to be hand-engineered and that we could fully automate aspect verifier generation and selection, a promising result for future work. We will include these new results in the final version of the paper, as well as additional discussion on automating the generation and selection of aspect verifiers.

Experiment Details: We provided GPT-4o with just 3 example aspect prompts from our original manual set and asked it to generate 10 new prompts with different aspect-strategy combinations. Using these 10 purely AI-generated verifiers (without any manual engineering or any of our original manual aspects), we evaluated Gemini-1.5-Flash on MMLU-Pro and GPQA (diamond). For simplicity, we do not perform any “aspect engineering” for this experiment. That is, we use all 10 AI-generated aspects (”MAV-automated-aspects” in the table above) and compare to the “MAV-All” results in Table 5 of the Appendix which uses all 10 original manual aspects (”MAV-manual-aspects (original)” in the table above). Below is one example of the AI-generated aspects created by GPT-4o:

    "redundant_steps": (
        "INSTRUCTIONS: \n"
        f"Look for any unnecessary steps that don’t contribute to the final result. Think out loud. "
        f"If the solution includes redundant or misleading steps, stop and reply '{VERA_ANSWER_SYMBOL}False'. "
        f"If every step is necessary and useful, reply '{VERA_ANSWER_SYMBOL}True'."
    ),

Moreover, having the different aspects be done in separate LLM calls seems unnecessary. Why can't you just provide the LLM the full rubric and have it output a score for each item in one generation?

Thanks for asking this. To address your question, we have conducted an additional experiment where we ask the LLM verifier to generate a “rubric” containing the approval scores for each aspect in a single response. We find that generating a single rubric for all aspects performs worse than using a separate model call per aspect (see table below). This is likely because separate model calls allows the LLM verifier to put all of its “focus” on one aspect, rather than having to think about many different aspects in context. Having multiple LLM verifier calls may have more expressive power, and this could be an interesting direction for future work to explore further. Thanks again for your suggestion to try this. We will include these results and additional discussion in the final version of the paper.

MMLU-Pro:

GPQA (diamond):

[RESPONSE CONTINUES BELOW]