ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

We really appreciate the reviewer for their time and for giving a detailed review and interesting direction to calculate test time compute.

§ Weakness 1

I appreciate that the authors compute a Cost column in Table 2, but this is only the AC cost. The authors justify this column as the monetary cost of querying an OpenAI API. However, in practice, this is not the only cost we have in evaluation: test-time compute (number of FLOPs processed) would be a more valid metric.

We thank the reviewer for pointing out the need for another cost metric like inference time compute (number of FLOPs processed). The suggestion of the reviewer to visualize the performance-compute tradeoff through the scatter plot with performance and compute as axes is a very good way to compare the effectiveness of the methods. We follow the following formula for the approximate measurement of FLOPs computation for all the models:

$FLOPs = layers \times [4 \times {sequence \space length} \times d_{model}^2 + 8 \times d_{model} \times ffn_{dim}]$

To know all parameter counts, we use an open source Qwen 2.5-72B as the AC with the same peer models as in our framework. We also perform the experiments with G-Eval and Analyze-rate using the AC model as Qwen 2.5-72B. We modulate ‘n’ for each method to see whether our method lies above the baselines in the visualization of performance vs compute. Although ‘n’ parameter doesn’t exist in any other model than the openai family, we simulated it by making n calls. We use spearman correlation as the performance metric for this visualization. Below is a table summarizing various models parameters for calculating their FLOPs.

ReFeR’s FLOPs is AC’s FLOPS (n=20 for Turbo and n=1 for Lite) + Sum of FLOPs of Llama, Mistral-nemo and Gemma.

(After all, in reality the authors are paying Together AI for inference of the open weight models as well.)

We have used togetherai for open source models for the sake of easiness to use and to save time while running different experiments. However the compute is same whether we deploy the models on our GPU or use a 3rd party API.

I suspect that if we looked at test-time compute, ReFeR becomes significantly more heavyweight than other methods: after all, we need to inference 4 large language models!

We agree that overall test-time compute is more than other baselines for ReFeR-Turbo, but we also get superior performance justifying the compute. Our ReFeR-Lite is significantly lesser in test-time compute compared to G-Eval or Analyze-Rate.

We have added the Simple peer average and the AC mode(GPT-4o-mini) with peer setup, as additional baselines in peer agents section of the table above.

We observe that the simple average of scores of peers as baseline performs just below G-Eval and Analyze-Rate, while ReFeR comfortably outperforms it. When we find the base performance using peer’s setup on GPT-4o-mini we get an average spearman correlation of 0.566 which is slightly better than G-Eval and Analyze-Rate on average. This shows our optimized peer setup by utilizing the eval_guidelines_prompt structure.

§ Question 1

I had some questions about how the G-Eval and Analyze-Rate baselines were configured; it would be good to move this information into the main text. Is the backbone model matched to the AC model used in ReFeR? (GPT-4o-mini for NLG/Reasoning, GPT-4o for multimodal)? Was n=20 for these?

Yes the backbone of all the baselines is always the same as the AC in the ReFeR. ‘n’ was always configured as 20 for both the baselines. And we used the same hyperparameters given in their respective papers. We set temperature to 1 to ensure uniformity in all the hyperparams of all the baselines and ReFeR. We did mention about this in our discussion that we utilized n=20, we will make it more explicit in the edited version.

§ Question 2

Similarly, was the single-agent CoT model matched to the AC model? Which agents were used in the multi-agent baseline?

Yes, the single-agent CoT was matched to the AC model. All multi-agent baselines have the same model as our AC model with their respective hyperparams mentioned in the codebase. All multi-agent baselines were essentially a single model acting as multi-agents so they have multiple instances of the AC model.

§ Question 3

Figure 2 was a bit challenging to parse: which model is the AC model for the "Llama + Gemma + Nemo" point, and which models represent the "3 Peers + Mixtral" point?

We mentioned the details in figure-2 caption, all the experiments in the ablation utilized GPT-4o-mini as the AC. -GPT-4o-mini is AC model for the "Llama + Gemma + Nemo" -"3 Peers + Mixtral" refers to the previous point in the graph which is Llama+Gemma+Nemo

We hope that our responses have answered your questions and satisfied your concerns. We are happy to provide any additional details or clarifications that you need. We look forward to your response.

We appreciate the reviewer for their genuine review and time.

§ Weakness 1

Even with the ReFeR-Lite variant, the framework still demands quite significant computational resources, making it still expensive as an evaluation tool.

We do agree that ReFeR-Lite is also costlier than G-Eval and that is because G-eval only produces scores and doesn't give any meaningful reasoning or comments of why it gave the score and this makes G-eval cheaper but also has lesser correlation with humans. Although ReFeR-Lite is costlier than G-Eval, it is slightly cheaper than Analyze-Rate which also gives feedback like ReFeR, but ReFeR-Lite is better than Analyze-Rate (which generates 20 responses like G-eval whereas ReFeR-Lite generates only single response) in terms of correlation. Purpose of ReFeR is to enhance the evaluation of NLG outputs using LLM agents and we do agree that this is more computationally expensive than non LLM based methods.

§ Weakness 2

Another concern is that this framework can be considered as a hierarchical version of LLM-as-a-Judge with more carefully designed prompting, which may limit the novelty of this work.

Although it may seem like LLM-as-a-judge and ReFeR are similar, our purposes are completely different. LLM-as-a-judge is primarily designed to evaluate chatbots using another LLM like GPT-4, similar to chatbotArena(which is crowd sourced). It is a pair-wise evaluation of chatbot responses. Whereas ReFeR is primarily designed to evaluate any generative output and score it on various metrics using Multiple LLMs or VLMs while increasing the overall reasoning of the framework. Another important difference between LLM-as-a-judge and ReFeR is that the former uses the LLM judge to evaluate the assistant by assessing its output, whereas the latter (ReFeR) utilizes the assistants evaluation to evaluate the given NLG output.

§ Question 1

Maybe I missed something, but how are the costs in Table 2-4 calculated?

We apologize for any confusion in the cost calculation. Tables 2-4 in the paper show relative costs in comparison to the most costly part of the table. For eg: in Table-2, we have ReFeR-Turbo as the costly one which costs around $1.5 so we take this as the base cost (1.00) and then calculate every other cost relative to this cost. For Analyze-Rate we have the total cost as$1.16 so its relative cost would essentially become 1.16/1.50 ~ 0.77. This way we calculate all the costs of each row relatively to understand the impact of cost beyond the dataset that is tested.

We hope this helps to clear any confusion and bring more clarity. We are open for discussing any other concerns or comments you have.

Dear Reviewer So4X,

We hope this message finds you well. We kindly want to check if you had a chance to review our rebuttal as we have addressed all of your concerns, and if you have any further questions or comments we can address to help with your evaluation. Thanks again for your review and questions.

Sincerely,

Authors

We thank the reviewer for their time and valuable review.

§ Weakness 1

The paper's main weakness is its unclear primary contribution, as it's difficult to distinguish how the ReFeR framework meaningfully differs from other multi-LLM evaluation systems beyond borrowing concepts from academic peer review (MoA).

We only borrow the idea of Academic Peer Review from the real world seeing its effectiveness in discussing the originality and relevance of a research paper. Apart from the idea we show how utilizing various LLMs or VLMs as agents in peer review can be applied to evaluate NLG outputs. We show how to utilize multiple and different LLMs as a consensus effectively while answering important questions on how to select models, how to communicate etc. We also give a fresh thought to the prompting structure that is different and better from G-Eval’s prompting structure. Finally we show how this improved framework can help us in better reasoning when compared to other multi-LLM agent systems. Most of the existing multi agent systems for better reasoning are primarily focused on using a single model as multiple agents whereas our system utilizes different models effectively.

§ Weakness 2

Additionally, the paper's organization is scattered across too many topics, with important elements like the instruction tuning dataset (shown in Figure 1) being buried in the appendix rather than properly discussed in the main text.

We agree that the instruction tuning dataset contribution could’ve been highlighted in the main paper. However, to make space for other important aspects (like testing multimodal ability of the framework, Reasoning ability of the framework, detailed error analysis and ablation etc) we had to keep the instruction tuning part in the appendix.

§ Question 1

Isn't the concept of using multiple LLMs to achieve better results in this context just derived from Mixture of Agents? Seems like an extension of the work; yet no reference.

Although mixture of agents also discuss the idea of how we can use multiple LLMs at each layer and then pass all the responses to the LLM in the next layer and so on is similar to ReFeR, it is yet different in the following ways:

• We have a hierarchical structure where the peer reviewers count is always greater than the AC count.
• We utilize different LLMs or VLMs as peers and AC
• And our AC is different from the aggregator LLM in the MoA as AC here does another evaluation utilizing the evaluations of peers and not just compiling the peer evaluations.

ReFeR and MoA are contemporary works and we did not take any inspiration from MoA, but we have included MoA as another recent work in our related works section.

§ Question 2

How is the coverage of the instruction tuning dataset, does it scale?

The instruction tuning dataset is the output of the final evaluation done by Area Chair for all the samples in the given dataset. So for a given datapoint the dataset has a score given by Area Chair (final score) and the final comment (i.e randomly chosen comment from n whose score is nearest to final score). This dataset was later utilized on a smaller model (mistral-7B) to improve its evaluation ability. This experiment was done to show that the final dataset is good enough for improving the evaluation ability of smaller models like mistral-7B.

§ Question 3

Is ReFeR's main purpose to be a framework to create instruction tuning sets? Or an evaluation system?

We apologize if it is not clear in the paper, but ReFeR’s primary purpose is to act as an Multi-Agent framework that can do NLG evaluation and give enhanced reasoning. This should be treated as the multimodal, multi-agent version of G-eval which is limited to single LLM and limited to text. We also provide an enhanced evaluation and comment at the end of evaluation which when collected can give rise to the instruction tuning dataset which can be used by smaller models to improve their performance, although this is not the primary focus of the framework.

We hope our responses resolve your concerns and questions. We are happy to discuss further. Please let us know if you have any additional questions.

Dear Reviewer M8bC,

We hope this message finds you well. We kindly want to check if you had a chance to review our rebuttal as we have addressed all of your concerns, and if you have any further questions or comments we can address to help with your evaluation. Thanks again for your review and questions.

Sincerely,

Authors

What makes the judge LLM trustworthy especially given the hallucination problem in LLMs.

Regarding the problem of hallucination, we don't clearly know a way to mitigate the hallucination problem in LLMs and hence we are not claiming we are 100% aligning our evaluation with human evaluation and 0% of hallucination. All our experiments show that the current state of the LLMs gives us around 60% (correlation nearly 0.6) alignment with human evaluation. Until the fundamental problem of hallucination is solved we don’t see a way out of this problem, which is applicable for all LLM based works.

It would be useful to see where the peer and AC explanations differ, and quantitative metrics to support the claim of explainability.

Although we are not claiming the explainability part, our figure-3 explains where the framework is lacking and where it is working properly. For eg: Figure-3 b shows that even though the majority of the peers are incorrect, the AC is able to identify and give a correct answer. We have added examples in appendix N showing how the AC benefits from the peer explanations although they are wrong at times, in the appendix of the updated draft.

§ Question 1 (robustness) and Question 2

Robustness: It is not clear how the framework promotes robustness? Can perturbations in prompts not affect agentic systems? Is the claim based on the fact that a perturbation for some LLM will not affect another, if so do we have a measure of what part of perturbation space agentic systems are more robust to? Are certain combinations of LLMs more robust than the others? Similarly to the previous point, the conclusion says the framework is robust, but I see no experiments to validate this claim. This could be supported to some extent by showing the standard deviations in final ratings of each peer/AC, and the complete framework. However, in order to call the framework robust, there have to be evaluations of this system on common adversarial attacks to LLMs.

We do not claim that the framework is robust for all changes. Agentic frameworks are well known to be susceptible for prompt changes which even we showed and mentioned in the limitations. When we mention robustness we did not mean robustness in general. What we intended to say is that our framework is robust/consistent across runs, as we show the std. deviation across runs in Table-4. We have reworded this accordingly in the revised version (line 53-54). We did not explore the directions of adversarial attacks on the framework as we never claimed the framework has any measures to tackle adversarial attacks. We leave this direction open for the future works to explore.

§ Question 3

The work says on line 99-100 that the area chair should be a larger or more capable LM, but it is not clear why? Is using smaller peers justified because of efficiency? Given the ablation where a weaker area chair(section 5.2) also works, why use a larger one?

Section 5.2 is dedicated to explain why a larger model (relatively better model than the peers) is required. Table-5 shows that average spearman correlation of Qwen is 0.492 but when used in the framework as the AC we get a correlation of 0.555 which clearly shows a better performance for Qwen because of the framework. But we see that one of the peer’s (Gemma2-9B) has a correlation of 0.568 hence we deduce that if the AC model is relatively stronger than most of the peers then we get improved performance but to get the best results out of the framework we see that we need a larger or better model as AC to better utilize the evaluations done by the peers and incorporate them in it’s own evaluation. We clarified this point clearly in Section 5.2 again in the updated submission.

We thank the reviewer for his detailed feedback and review.

§ Weakness 1

The paper highlights on lines 36-47 that there are not many works on using multiple-LLMs for evaluation, but there do exist some methods using multiple LLMs, for example: https://openreview.net/forum?id=FQepisCUWu and https://arxiv.org/html/2401.16788v1. The authors highlight very early and strongly the novelty of the work of using multiple-LLMs for evaluation but existence of the above works do bring in questions. The paper would benefit much more from better literature review, and inclusion of these other methods in the results section for a fair and faithful comparison.

We don’t claim that there weren’t any works that involve multiple LLMs for evaluation, rather we said “there has been limited research on how to align evaluations from multiple VLMs or LLMs with human judgments.” (Line 43-44). We also mentioned ChatEval in the related works section (Line 485). The works mentioned (ChatEval [https://openreview.net/forum?id=FQepisCUWu] and ScaleEval [https://arxiv.org/html/2401.16788v1] ) are primarily focused on pair-wise evaluation of NLG responses whose primary purpose is to identify which LLM is better at generation on various topics or which of the given 2 responses are better. For Eg: ScaleEval clearly mentions the motivation and usage in Section 2 of their paper that they want to utilize LLMs to make a more scalable and reliable evaluation system for a diverse set of tasks. ScaleEval work primarily falls in the category of Chatbot Arena but using LLMs (https://arxiv.org/abs/2306.05685) hence it is not a baseline in our direction. Regarding ChatEval, it is a multi-agent framework that utilizes same model as multiple agents with varied personas in a debating setting hence it is different from our idea of different heterogeneous models as peers and ACs in a hierarchical format enabling composing a modular framework to leverage different model’s strengths. As ChatEval’s focus is pairwise evaluation of NLG responses with a different goal, this cannot be utilized as a relevant baseline for all datasets. Hence, ChatEval and Scale-Eval are not relevant baselines for us since they are designed for a different type of evaluation tasks.

§ Weakness 2

The work claims of developing a novel prompting schema by introducing "guidelines"(that can be auto-generated) and lists it as a main contribution. However, I think creating detailed prompts for specialized tasks is a well known prompt engineering practice, thus I do not see it as a substantial contribution.

We think evaluation guidelines is an important step in the framework and is different from regular prompt engineering practice or prompt optimization. It leverages the idea of procedure-cloning to enable models to understand (in-context) the fine-grained details of evaluation similar to human evaluation of datasets like topicalchat where the human annotators were generally given the guidelines and details on how to score on a given metric. We conducted detailed prompt optimization experiments where we used different prompt optimization methods and none of the methods gave prompts which have the content of evaluation guidelines or gave similar performance as this prompt structure.

§ Weakness 3

Several key claims require stronger support, especially on explainability and robustness, see questions below.

We haven’t claimed that the framework is very robust for all changes and that we explored explainability as they were not the primary goals of the paper and we have removed this part of the statement (line 53-54 “promoting model self-improvement, explainability, and robustness in complex scenarios”) which might imply that we are claiming explainability and robustness. When mentioning explainability, we mean the explainability on the reasoning/evaluation process of the LLM which might not be present in methods like G-Eval and robustness/consistency of framework across runs. We explained the reasons below to give more clarity on what we meant by promoting explainability and robustness.

§ Question 1

The paper claims that on line 51-52 the framework promotes explainability and robustness.

We agree that we have not explored explainability and robustness in detail in this work. We thank the reviewer for pointing this out and we have now reworded the statement to imply this meaning clearly in the paper.

Explainabiltiy: It is not clear how this happens? Why is trying to use another LLM to explain another LLM's output sound?

Just like in multi-agent peer-review/multi-agent debate where models are provided with evaluations of other agents to reassess/defend their own evaluation, we believe that providing the peer evaluations to the AC would improve the AC’s evaluation. And in our framework the AC model is not explaining the peer responses, but is considering and utilizing the peer evaluations in its own evaluation.

§ Question 4

I assume temperature was set to 1 to ensure variation in the responses when n>1, but is there a reason this performs better than setting temp=0 and using n=1?

Analyze-Rate (https://arxiv.org/abs/2310.05657) showed in their paper previously that utlizing temperature of 1 gives better correlation than lower temperatures. And hence we used the temperature as 1 in all the setups. Temperature helps the model to be more creative in every response and High temperature would imply more variations. And with a larger value of n, you get multiple varied responses, which you can use to get a better output. Analyze-rate (appendix D) has already shown an ablation on how increasing temperature for n=20 increases correlation. We performed a small experiment using n=1 and temperature=0 to compare with n=1 when the temperature is 1. For the entire experiment we used the same peer evaluations to keep the input constant for the AC and used the same model (GPT-4o-mini pointed to the latest openai update). Table with spearman correlation coefficient:

We find that using temperature=1 gives better results for both n=1 and n=20 and also observe that when temperature=0, then varying n hardly improves the correlation score (gain of 0.003 for spearman correlation) while for the temperature=1 case we find that the total gain is 0.03 for spearman correlation.

§ Question 5

Could there be analysis on the rating distributions of the framework when $n>1$ ?

Assuming the reasoning behind the question being how much the ratings vary when n>1, we can get the standard deviation across n-samples for each data point and see the overall std-dev across the dataset. The below table shows that information:

Overall StdDev:

We can see that as n increases models tend to have higher std.dev/variance among the responses. This shows that as n increases, the AC’s responses have much variance, and would lead to better correlation.

§ Question 6

I am unsure how $C_{final}$ is produced when $n > 1$, because $S_i$ can be averaged but not sure how aggregations work for $C_i$.

The final correlation is calculated using the Scores and hence we averaged scores only for all the n responses. The Cfinal is only needed for the instruction tuning dataset, for which we randomly take the comment whose score is nearest to the Sfinal. We reworded line 806 in the algorithm in appendix B to explicitly mention this.

§ Question 7

If possible, it would be useful to see ablations on $n$, this could be done by choosing subsets from results of n=20 runs if the results are stored.

Yes it is possible to do an ablation like mentioned and here are the results for the same using the Topical Chat dataset on all 4 metrics averaged.

Topical Chat (ReFeR)

Since, G-eval and Analyze-rate have shown that the hyperparameter ‘n’ is integral to their system, we use n=20, and for fair comparison, we present our n=20 results too, as our ReFeR-Turbo. But, we can observe that for lesser ‘n’ values we have almost similar results. This was also mentioned by Analyze-Rate in their ablation of temperature(Appendix D) stating that theirs and G-Eval’s are robust for just n>5.

We hope that our responses have answered your questions and satisfied your concerns. We are happy to provide any additional details or clarifications that you need. We look forward to your response.

Thank you very much for replying back to us.

Weakness 1. Multi-Agent Framework Comparison Important: While I appreciate the clarification about ChatEval and ScaleEval, I maintain that ChatEval should be included as a baseline since: It evaluates on TopicalChat, making direct comparison possible Comparing against a non-hierarchical multi-agent framework would help demonstrate whether the hierarchical structure provides advantages. This is integral to show the importance of the contribution of this work.

ChatEval does a pair wise evaluation or comparison based evaluation (table-5 in page 10 of ChatEval shows an example of this paradigm). They have shown a benchmark on dialog response generation (section 3.2) on topicalchat dataset and they have not explicitly mentioned in their paper on how exactly they do this evaluation and report correlations. We assume that they must've evaluated the topicalchat dataset (which has 60 dialog contexts with 6 different systems generating a response) by choosing 2 responses at a time and comparing them (which is called as "comparison" evaluation and G-eval, analyze-rate and ReFeR follows a direct scoring evaluation) hence we think it is not possible to compare both the methods correctly.

Apart from this, ChatEval is not a real multi-agent framework as they only focused on using the same model with multiple role plays and also mention in their section 3.1. Nevertheless we would've compared our method with ChatEval's results but we don't find the codebase (in their official github repo) for experimenting on the TopicalChat hence it is not possible to show you any direct comparison. We are emailing the authors of ChatEval to ask for any code and if we hear back from them then we will present any comparisions before rebuttal.

Citation Issues: The introduction should cite these works(ChatEval, ScaleEval) when stating "limited research" to accurately represent the field. By italicizing the focus on "individual LLMs"(l36-37) for text evaluation research and then expressing surprise at limited multi-LLM research without citing any existing work, the text implies an absence of multi-agent approaches. This framing would be more accurate if it cited existing multi-agent frameworks like ChatEval.

We removed the italic for "individual LLMs" to remove that emphasis in the updated draft. ChatEval is not a true Multi-agent framework as it didn't explore heterogeous models and ScaleEval is for evaluating the chatbot (in the direction of ChatBotArena) hence we would like to stick to the statement that there has been limited research as multi-agent framework for scoring evaluation are very limited.

Weakness 2. Prompting Schema Contribution I understand that the prompt structure outperforms automated optimization methods. However, since the prompting schema is specific to your framework rather than a generalizable technique, it may be better positioned as part of the overall system rather than a standalone contribution and the system is listed as a contribution already.

We understand that prompting schema maybe specific to the framework for evaluation and is not a generalizable technique but we mentioned it as a contribution as it can be used to improve other frameworks like G-Eval or analyze-rate and give better results just by using the prompt structure.

Thank you for your suggestions on experiments to validate our statement, we will complete those experiments soon and post the results here.

Other Questions: What model with G-Eval is used?

We used same model as Area chair (GPT-4o-mini) for G-Eval , Analyze-rate and all the places where we compare the framework against other models.

also, as the new ablations suggest using n=5 not only performs similar to n=20, it also has less std dev so please consider adding to the manuscript about this.

Yes, we will mention about this in the draft, thank you for pointing it out.

Thank you for the rebuttal.

Weakness 1. Multi-Agent Framework Comparison Important: While I appreciate the clarification about ChatEval and ScaleEval, I maintain that ChatEval should be included as a baseline since: It evaluates on TopicalChat, making direct comparison possible

Comparing against a non-hierarchical multi-agent framework would help demonstrate whether the hierarchical structure provides advantages. This is integral to show the importance of the contribution of this work.

Citation Issues: The introduction should cite these works(ChatEval, ScaleEval) when stating "limited research" to accurately represent the field. By italicizing the focus on "individual LLMs"(l36-37) for text evaluation research and then expressing surprise at limited multi-LLM research without citing any existing work, the text implies an absence of multi-agent approaches. This framing would be more accurate if it cited existing multi-agent frameworks like ChatEval.

Weakness 2. Prompting Schema Contribution I understand that the prompt structure outperforms automated optimization methods. However, since the prompting schema is specific to your framework rather than a generalizable technique, it may be better positioned as part of the overall system rather than a standalone contribution and the system is listed as a contribution already.

Question 1 and 2: I appreciate the changes in the manuscript regarding the claim of robustness and explainability.

The work says on line 99-100 that the area chair should be a larger or more capable LM, but it is not clear why? Is using smaller peers justified because of efficiency? Given the ablation where a weaker area chair(section 5.2) also works, why use a larger one?

Question 3: Thank you for the clarifications. Some more questions:

Area Chair (AC) Performance Analysis The framework seems to improve Qwen's performance but I think the results relate to the best peer more than the framework, the claim that "larger/better AC models yield best results" needs more evidence:

1. ReFeR-Lite results are very close to best peer (Gemma-9B) performance (only 0.0157 difference in main expts and in sec 5.2, best peer outperforms)

2. Is AC performance bottlenecked by best peer or best model in the framework? this needs discussion and further evidence comparing:

   a. AC same as best peer vs current setup

   b. Refer-Lite vs Refer-Lite framework's best model alone

ReFeR-Turbo Performance Gap The substantial improvement in ReFeR-Turbo warrants investigation given the closeness of Refer-Lite to the best peer: Is improvement due to multiple AC samples or framework structure? Experiment to consider: Refer-Turbo framework's best model alone with repeated sampling vs ReFeR-Turbo

Other Questions: What model with G-Eval is used?

also, as the new ablations suggest using n=5 not only performs similar to n=20, it also has less std dev so please consider adding to the manuscript about this.

Thank you for the response and the experiments.

This clears a lot of things and shows the framework's advantages.

I am increasing my score by 2 points based on:

The thorough ablation studies that validate the framework's benefits analysis framework and experiments as it could potentially be impactful on future research in LLM evaluation systems Strong technical execution and clear presentation

However, some concerns remain:

Important: ReFeR system: ReFeR improvements range from 0.01-0.04 over baselines; ReFeR-Lite shows improvements <0.035; Using Gemma as AC instead of GPT-4o-mini reduces gains to <0.01. The relatively higher values for std deviation also does not help the case when the mean performance difference is small. Given the increased computational overhead and cost of multiple models, these incremental gains need stronger justification for practical applications. This point still stands even if prior work had similar improvements.

Other issues The architectural contribution(while valuable) requires stronger positioning. While ReFeR thoughtfully implements heterogeneous models in a hierarchical structure, this represents an engineering evolution of existing approaches(multi-agent perr review and multi-agent debate) rather than a fundamental change. Prior multi-agent frameworks' use of identical models was an implementation choice, not a technical constraint. Thus, while ReFeR's specific implementation produces measurable improvements, I am not sure of the novelty contribution.

The prompting contribution might be better presented as an integral part of the system rather than a standalone contribution as I have stated before because it is not a general schema. Therefore, I do not count it as a significant contribution.

Conclusion: The paper's strongest contributions lie in its empirical insights and analysis framework. The comprehensive experiments and error analysis provide valuable understanding of LLM evaluator behavior in both single and multi-agent settings. This analytical data may prove more significant than the absolute performance improvements, especially considering that future improvements in single-model sampling or prompting approaches might achieve similar gains(as they are small) with lower computational overhead.

We appreciate the reviewer for their time and for raising important points.

§ Weakness 1

The distinctions between this method and similar methods, such as multi-agent debate or multi-agent peer review, are not very clear. For example, debating or summarizing are merely different forms of prompts. In fact, for the multi-agent peer review method, if a reviewer can receive information from all other reviewers before refining their own review, they would essentially be playing a role very similar to that of an area chair.

Debating and summarizing or aggregating are different forms of communication between multiple agents, in debating format there can be agreement or disagreement and then each of the models has ability to revise their opinions over multiple rounds of debate based on others opinions. Whereas in the aggregating format we take another model which will aggregate all the opinions of models and then summarize them or use them to further continue the process. Here usually models may not communicate with each other or have multiple rounds of discussion. In general, our framework can allow peers to look at other evaluations and make changes before sending to AC but we have only explored the hierarchical way of communication as we are choosing smaller models as peers which can potentially be biased when seeing others evaluation. For these reasons we need an AC which can potentially use these assistant evaluations and give better overall evaluation. Our framework is quite different from other multi-agent frameworks mentioned, below are some differences.

• ReFeR vs Multi-Agent Debate:
  Multi-Agent debate uses the same model as multiple agents by invoking multiple instances of the model and primarily utilizes Debate format of communication with 3 rounds as default. Whereas ReFeR needs different models as peers and AC with a hierarchy of models where AC is required to be stronger or better model than peers, and we also utilize a specific format of prompt structure and it is a single round of evaluation where peers don’t communicate among themselves.

• ReFeR vs Multi-Agent Peer Review:
  Multi-Agent peer review also utilizes same model by invoking multiple instances and simulates multi-agent environment but instead of debate each of the models aggregate all the reviews (plus their confidence) of others and then revise their opinion and continue to do this for 3 rounds by default. Here each model acts like an AC at every evaluation. Whereas in ReFeR we have a single evaluation by peers and a single evaluation by AC in a hierarchy. We complete the evaluation in lesser steps/computations while matching or being better than other multi-agent models.

§ Weakness 2

The two-level hierarchical structure that summarizes through an area chair is not fundamentally different from ensemble methods like majority voting; it is merely implemented via prompts. Additionally, the difference between Turbo and Lite versions is simply one of 1 versus 20 rounds of integration.

Yes we agree that hierarchical structure is not fundamentally different from ensembling but it provides better ensembling ability by trying to understand the differences in the peers opinions and its opinion thereby having a possibility to reject any outliers during evaluation. For eg: if we refer to figure-3 b) we can see that when the majority of peers are wrong, the AC is still able to answer correctly atleast in 18.3%+14% (i.e “only 1 peer correct and AC is correct” (18.3%) and “0 peer correct but AC is correct” (14%)) of the cases. This cannot happen in a simple ensemble method like majority voting. This is evident in reasoning where the AC can find the differences in the peers' reasoning and can correct although the consensus is incorrect. And we agree that Turbo and Lite are simply the number of responses aggregated and they were designed by following the previous works like G-Eval and Analyze-Rate.

§ Weakness 3

The improvement in NLG evaluation is not particularly significant, and on reasoning tasks, the improvement is only more noticeable on AQuA, while it is not significant on other datasets.

The improvement in the correlation on evaluation tasks has always been comparatively low even in the previous works. G-Eval has shown only ~0.04 improvement in correlation over the best baseline in summeval dataset and ~0.02 improvement in topicalchat dataset, even though the baselines are just pre-trained models like BART/T5(BART Score, Uni-Eval respectively) and G-Eval uses a very capable LLM GPT-4. Although ReFeR’s improvement seems not so significant, we should also consider that Analyze-rate has shown even less improvement than its baseline G-Eval. This shows us the saturation problem in the inherent capability of the evaluator LLM. There is only so much we can improve in the zero-shot LLM Evaluators through prompting strategies/framework. This is also apparent in the Reasoning results, where many methods saturate at ~95-96% for GSM8k, ~92-94% for BBH-DU,etc, even though Reasoning has a much simpler metric like accuracy and the sample size is lesser. So, in the case of evaluation task, where the dataset size is much larger and the metric is correlation, we are bound to see very slight improvements compared to the changes.

Moreover, the differences in performance may also be due to different model selections, such as the models used as peers not being entirely consistent with those used in other methods, leading to unfair comparisons.

We understand that our comparisons with our methods may not be totally fair as we utilize different models and the rest of the baselines utilize the same model. But we can see that although they use the same model in multiple instances still the performance difference is not significant and moreover we utilized smaller models as peers (7B-12B) whereas the AC is GPT-4o-mini (single instance). We can also see that our method provides better performance to cost ratio.

§ Weakness 4

The two-level hierarchical structure has not been theoretically proven to be necessarily better than debate methods; it seems more that different methods have different strengths, weaknesses, and application scenarios

Yes we agree that 2 level hierarchical structure is not proven to be better than debate methods and Yes we also observe that both methods seem to have their own advantages and weaknesses.

We hope our responses resolve your concerns. We are happy to provide any additional clarification if there are any further questions or comments.

Dear Reviewer KWMK,

We hope this message finds you well. We kindly want to check if you had a chance to review our rebuttal as we have addressed all of your concerns, and if you have any further questions or comments we can address to help with your evaluation. Thanks again for your review and questions.

Sincerely,

Authors

Dear Reviewer KWMK,

We hope this message finds you well. We kindly want to check if you had a chance to review our rebuttal, and if you have any further questions or concerns left that we can address to help with your evaluation.

Sincerely,

The authors