Training Language Models to Critique with Multi-Agent Feedback

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1/Q2: The improvements demonstrated by the MultiCritique pipeline, while statistically significant, are relatively modest in scale ...; The multi-agent critique and meta-critique process likely incurs substantial computational expenses, especially given the high reliance on GPT-4 for critique refinement. A practical analysis of the computational cost relative to the quality gains would provide more context on the feasibility of scaling this approach for widespread or real-time applications.

A1/A2: We appreciate the reviewer's concern about the computation cost related to performance. We would like to address this as below.

While our approach does introduce additional computational overhead through multi-agent feedback and GPT-4 meta-critique, our proposed MultiCritique significantly outperforms previous works in data efficiency. As shown in Figure 2, our detailed analysis shows that MultiCritique achieves superior cost-effectiveness compared to existing approaches. With only 3K training samples (≈\$890 in API costs), MultiCritique significantly outperforms baselines using 100K-257K samples (\\$1,915-\$3,758 for UltraFeedback and Prometheus). This represents a 2.15-4.22× improvement in data efficiency. Since our current results already show state-of-the-art performance with modest computational resources, we didn't scale up the MultiCritique due to resource limitations.

Additionally, we want to emphasize that our MultiCritique pipeline is primarily designed for offline dataset construction and optimization rather than real-time applications. The improved data quality and efficiency justify the one-time computational investment, as the MultiCritique dataset enables the improvement of efficient critique LLM's critique ability.

We have supplemented these details in our revisions (Section 5.3).

Q3: Additionally, the paper would benefit from exploring where the MultiCritique pipeline's improvements become saturated. By examining performance gains across a broader range of model sizes or critique tasks, the authors could better identify the conditions under which MultiCritique delivers more substantial gains.

A3: We appreciate your valuable suggestions about the investigation of model sizes and critique tasks for MultiCritique. Let us clarify as below:

1. Model Scale Analysis

   While our experiments on 7B-8B models are comprehensive, our investigation of larger models is insufficient. We have supplemented a new limitation section (Appendix A.5) to clarify this question. Our preliminary study demonstrates that the improvements brought by MultiCritique on the advanced Qwen2.5-72B-Instruct model are not very significant. Our analysis suggests two primary reasons for this phenomenon:

   • These 70B models have been sufficiently fine-tuned, exhibiting very powerful critique and generalization ability. For example, Qwen2.5-72B-Instruct achieves 83.62% F1 on CriticBench, surpassing GPT-4's 78.75%. Besides, its objective critique performance in CriticEval is also superior to GPT-4 (78.72% > 76.09%). This high level of critique performance makes further improvements challenging.
   • Our experiments on 70B models utilize only MultiCritique data for optimization. We suggest that the data mix strategy that integrates our dataset with other high-quality instruction data, might play a crucial role in improving their performance further.

   However, due to constraints in our computational resources and scheduling, our experiments and studies on these advanced 70B models are insufficient. We have incorporated these details into the limitation sections, and plan to address this limitation in our future work.

2. Critique Task Diversity

   We want to highlight that our chosen CriticEval and CriticBench benchmarks have encompassed a comprehensive range of critique tasks, including alignment, NLP, and diverse reasoning tasks. It enables the observations of saturation conditions of our work. To the best of our knowledge, these two benchmarks are comprehensive in evaluating critique abilities. Meanwhile, we have also elaborated the critique protocol limitations of our work in Appendix A.1, which will be solved in our future work.

We appreciate the reviewer for bringing up these important points, which have helped us improve the quality of our paper.

We really appreciate your valuable feedback regarding the computational complexity and innovation aspects of our work. We would like to address these concerns with additional clarification and supporting evidence:

Regarding Performance Improvements and Computational Efficiency

As demonstrated in Section 5.3 and our previous evidence, MultiCritique represents a 2.15-4.22x improvement in data efficiency. This observation demonstrates that MultiCritique achieves comparable or even better performance with 2.15-4.22x times less computation cost compared to strong baselines. This also indicates that the quality of critique data is more crucial than quantity, justifying the additional computational cost during training for achieving superior critique capabilities.

Regarding Methodological Innovation

Unlike previous works, our work proposes a unified multi-agent feedback framework that improves the critique ability of LLMs from both SFT and RL fine-tuning phases:

1. Before Critique (Section 3.1) Our framework introduces three crucial information to reduce the complexity of the critique task.

2. In the SFT phase (Section 3.2) To address the bias of one single model, we propose to aggregate diverse and accurate critiques from multi-agents and mitigate the flawed critiques.

3. In the RL phase (Section 3.3) Beyond the behavior cloning of SFT, we propose to calibrate the preference critiques by utilizing the multi-agent feedback from the downstream revision task, facilitating the robust RL for improving LLM's critique ability.

Experiments in Section 6 (Table 4) prove the effectiveness of our designs.

Thank you again for your feedback, and we hope this detailed response could address your concerns. If you have other questions about our work, we would be more than happy to assist you.

Q7: ... authors applied SFT on InternLM2-7B-Chat-SFT ... How did they address potential issues of hallucination? Moreover, why did they not consider using Llama3-8B-Instruct as a base model?

A7: Thank you for these important questions. Let us address them separately.

Firstly, our preliminary experiments and human annotations have not observed any significant hallucination issues in critiques introduced by the additional SFT process. However, we observed error amplification problems when fine-tuning on GPT-4's flawed samples during the SFT phase.

Regarding the choice of the base model, our RL implementation is not designed for the Llama3 series due to its late publication date (April 2024, our project starts in March 2024). Following your suggestions, we have conducted MultiCritique-SFT fine-tuning on three additional advanced base models: InternLM2.5-7B-Chat, Llama-3-8B-instruct and Qwen2.5-7B-Instruct, detailed in our revised manuscripts (Appendix H). Experimental results show that even though these advanced LLMs have already demonstrated very strong zero-shot critique capabilities, our MultiCritique can further improve their critique ability. For example, the average improvement of these three additional models in CriticBench is 15.9%.

Q8: ...it appears to lack sufficient innovation compared to existing works on critique and debate. I recommend that the authors clarify this aspect.

A8: We want to emphasize that the implementations of our proposed multi-agent feedback have been described as the core contributions in Section 3.2 and Section 3.3.

We want to highlight that most previous multi-agent frameworks aim to improve the LLM's alignment and response quality, while our work aims to improve the critique ability of LLMs with multi-agent feedback through both the SFT and RL stages, rather than the alignment refinement. We will clarify more details in as below:

MultiCritique-SFT Stage

First, as described in Section 3.2, our proposed MutiCritique-SFT framework consists of three steps, differing from previous multi-agent debate approaches [7,8]:

1. Multi-agent Analytical Critique: multiple advanced LLMs independently generate Analytical Critique Units (ACUs). Each ACU is carefully designed to identify and address specific flaws in the evaluated response, consisting of five key values, such as error descriptions and suggestions.
2. Meta-Critique Classification: We introduce a meta-critique mechanism that evaluates the quality of each ACU through classification into human-annotated quality categories, given all the ACUs generated by multi-agent.
3. Critique Summarization: Rather than relying on debate outcomes, we employ critique summarization to consolidate diverse perspectives while filtering out flawed critiques. This approach preserves the unique insights of each agent while ensuring critique quality.

Regarding your question, Meta-Critique, and Critique Summarization can be viewed as a special communication mechanism among agents designed for the critique task. Besides, as described in Section 3.2 (second paragraph), we have already explained why we did not adopt the traditional multi-agent debate approach. Specifically, we observed that multi-agent debate reduces the diversity of critiques due to the complexity of critique tasks, which harms the diversity and accuracy of critiques.

MultiCritique-RL Stage

We calibrate to refine the preference critiques with the multi-agent feedback from downstream revisions, facilitating robust RL during fine-tuning.

We hope these distinctions demonstrate the innovation of our work compared to existing multi-agent debate and critique works. These details have been supplemented into the related work in our revised submission. Thanks for your questions.

[7] Debating with More Persuasive LLMs Leads to More Truthful Answers

[8] Improving Factuality and Reasoning in Language Models through Multiagent Debate

Q3: The terms “meta-cognitive” and “critique as a greater challenge than criticism” are ambiguous. Could the authors clarify these?

A3: We appreciate the reviewer's questions about these important concepts. Let us clarify them in detail:

The concept of "meta-cognitive capability" encompasses higher-order cognitive processes that involve understanding and regulating one's thinking processes [3,4], extending well beyond mere critique. This capability comprises three essential components:

• Self-understanding: the ability to comprehend its processes
• Reflection: The capacity to evaluate responses against criteria
• Self-regulation: The ability to adjust decisions (responses) based on reflection

Our work covers two aspects:

1. Self-understanding: understand the task by introducing the crucial information, which also reduces the complexity of the critique task (Section 3.1)
2. Reflection: collect diverse and accurate critiques by utilizing multi-agent feedback and a Meta-Critique process (Section 3.2).

We hope these clarifications help you better understand how our approach tackles the complex meta-cognitive aspects of critique generation, rather than merely producing surface-level criticism.

[3] Metacognition is all you need? using introspection in generative agents to improve goal-directed behavior

[4] Metacognitive prompting improves understanding in large language models

Q4/Q5: ... However, the reviewer does not know what these 123 tasks entail or where to find this information in the appendix in this paragraph ... Furthermore, Section 3.1 lacks a clear definition of these three levels.

A4/A5: We appreciate the reviewer's careful examination of our data preparation process. Let us address each point:

Tasks in MultiCritique:

We are sorry that we cited the wrong Section for the details about 123 tasks. The original text referenced Appendix C (last paragraph of Section 3.1), but the corresponding section should be Appendix D (Table 12). We have corrected this issue. Thank you very much for bringing this to our attention.

Response Quality Assessment with InternLM2-20B-Reward Scoring

Regarding InternLM2-20B for scoring, we want to clarify as following two points (motivation and performance):

• Motivation: Our utilization of the reward model is to solve the uneven distribution of response quality in our dataset. Therefore, we utilize the reward model to coarse-grained judge the quality of responses and classify them into three performance durations (low, medium, and high). Thus, the accuracy of the reward model has little impact on our final collection of critiques.
• Performance of InternLM2-20B-Reward: When we started our project, the InternLM2-20B-Reward was the top-tier reward model in the RewardBench leaderboard [5]. Besides, we explore the correlations between InternLM2-20B-Reward and human annotations at our work's earlier stage. Specifically, in discriminating between 200 pairs of high/low quality responses, InternLM2-20B-reward achieved a 95.3% consistency with human judgments, which is reliable to automatically evaluate the quality of responses. As described in Appendix C.1 (Table 7), these three quality levels are classified into low, medium, and high qualities, and the average quality scores are -1.41, 0.70, and 1.69, indicating a significant performance gap.

We have enhanced Section 3.1 and extended Appendix C.1 to better highlight these important details while maintaining full documentation in the Appendix.

[5] Rewardbench: Evaluating reward models for language modeling

Q6: In Section 4.1, the authors present Equation 1 ... How does this differ from the fine-tuning approach in the Aligner ...

A6: Thanks for your question. While there might be superficial similarities in the mathematical formulation, our SFT fine-tuning differs from Aligner [6] in both motivation and technique.

• Motivation: Aligner [6] focuses on optimizing response revision upon one upstream LLM, while our approach is specifically designed to enhance LLMs' critique abilities rather than revision capabilities.
• Technique: In Equation 1, we jointly optimize the crucial information and critiques in a unified framework, aiming to enhance the understanding of query-response, which is theoretically grounded in human metacognition [1,2]. This joint optimization approach differs significantly from Aligner's optimization.

While Aligner doesn't align with our topic of critique, we add it into the related work (line 161 in related work) since it can be viewed as a potential multi-agent solution that integrates upstream and downstream models. Thanks for your suggestion.

[6] Aligner: Efficient Alignment by Learning to Correct

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1: ... the authors emphasize that the inherent complexity of model-generated critiques ... However, MultiCritique originates from Multi-agent Feedback, which clearly adds to this complexity by involving multiple models.

A1: There appears to be a misunderstanding regarding the relationship between task complexity and technique complexity. While our approach involves multiple models, it reduces the overall complexity of the critique task rather than increasing it. Let us clarify it in the following two aspects:

1. Simplifying Critique Tasks with Crucial Information:

   As described in Section 3.1 (Crucial Information Collection), before generating critiques, our proposed MultiCritique specifically elicits three crucial information to reduce the complexity of critiques, including task description, customized evaluation criteria and reference response. Experimental results in Section 6 also prove the contributions.

2. Improve Critiques with Multi-agent Feedback:

   The inherent complexity of the critique task makes it challenging for a single model to perform effectively, often resulting in flawed critiques that negatively impact downstream fine-tuned LLMs. Therefore, we propose a multi-agent feedback framework by aggregating diverse and accurate critiques from multi-agents. Besides, we propose to calibrate the preference critiques by utilizing the multi-agent feedback from the critique's downstream revision task, facilitating the robust RL for improving LLM's critique ability.

   Extensive experimental results in Section 6 have proven the effectiveness of our designs.

Q2: In terms of research on critique, other researchers have already explored similar ideas with LLMs ... which results in some contributions being overstated.

A2: We sincerely appreciate the reviewer's valuable feedback on the literature review. While we acknowledge our oversight in not explicitly citing these fundamental works, we would like to respectfully clarify the distinct contributions of our work and demonstrate why we believe our claims are well-supported by evidence.

Novel Multi-Agent Feedback Framework for Critique Generation:

Although Saunders et al.'s work [1] on "Self-critiquing" pioneered the use of LLMs for critique, our approach fundamentally differs in several aspects:

• We propose a novel multi-agent feedback mechanism that automatically collects diverse perspectives from multiple LLMs, without the need for costly human annotations.
• We introduce crucial information (task description, evaluation criteria, and reference responses) to simplify the complex critique tasks, improving critique quality.
• Our framework proposes a multi-agent feedback framework to aggregate diverse and accurate critiques from multi-agents and calibrate the preference critiques utilizing multi-agent feedback from downstream revision tasks, facilitating robust RL.

Automatic High-Quality Dataset Construction for Both SFT and RL:

Unlike previous RLHF works, like InstructGPT [2], our key innovation lies in:

• Propose an automatic pipeline for generating high-quality critique datasets for both SFT and RL stages.
• Propose Meta-Critique and Multi-Agent-Revision-Scoring mechanism to refine quality of critiques and preference critiques, facilitating robust improvement for SFT and RL.

Our experimental results validate the positive contributions of our designs, showing that our 7B model achieves comparable performance to 70B models and approaches GPT-4 on CriticEval and CriticBench datasets. We have supplemented these two works in our revised manuscripts (Related Work) to better contextualize our work within the existing literature while maintaining our claims about its novel contributions.

[1] Self-critiquing models for assisting human evaluators

[2] Training language models to follow instructions with human feedback

We sincerely appreciate your response and thank you for taking the time to review our responses. We look forward to engaging in a constructive discussion about our work.

We really appreciate your valuable feedback and your time and effort in reviewing other reviewer's comments. We would like to address your concerns regarding computational complexity and innovation as follows:

Regarding Computational Cost

As demonstrated in Figure 2 and Section 5.3, MultiCritique represents a 2.15-4.22x improvement in data efficiency. This observation demonstrates that MultiCritique achieves comparable or even better performance with 2.15-4.22x times less computation cost compared to strong baselines, like Auto-J, UltraFeedback and Prometheus. This also indicates that the quality of critique data is more crucial than quantity, justifying the additional computational cost during training for achieving superior critique capabilities.

Regarding the Innovation in the training process and dataset collection

Our work primarily focuses on designing an effective multi-agent feedback framework to enhance the LLM's critique ability. Therefore, we don't modify the training loss function and training process, which is not the objective of our work.

The innovations of our work centers on a unified multi-agent feedback framework that enhances LLM's critique capabilities through both supervised fine-tuning (SFT) and reinforcement learning (RL) phases, effectively addressing two fundamental challenges in this domain. We have elaborated on these innovations in detail in our previous response (Q8/A8) and the "Global Response - Novelty" section.

We hope our clarification can address your concerns. If you have other questions about our previous comments, we would be more than happy to assist you.

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1: While the pipeline effectively aggregates critiques, the reliance on multiple LLMs (including GPT-4) for meta-critique classification may introduce considerable computational and financial overhead. Could you provide quantitative result and a discussion on efficiency, potential trade-offs, or future work focused on optimizing this aspect?

A1: We sincerely appreciate the reviewer's insightful questions regarding the efficiency and trade-offs of our proposed MultiCritique framework. We answer your questions as below:

1. Data Efficiency Analysis

   As shown in Figure 2 in Section 5.3, our detailed analysis shows that MultiCritique achieves superior cost-effectiveness compared to existing approaches. With only 3K training samples (≈\$890), MultiCritique significantly outperforms baselines using 100K-257K samples (\\$1,915-\$3,758 for UltraFeedback and Prometheus). This represents a 2.15-4.22× improvement in data efficiency.

2. Future Optimization Directions

   We acknowledge the computational overhead of MultiCritique, and future works could focus on following aspects to improve it:

   1. Investigating multiple samller LLMs for meta-critique, like PoLL [1]
   2. Distilling meta-critique ability from GPT-4 to efficient LLMs.

We have extended Section 5.3 to include these details. Thanks for your suggestions.

[1] Replacing judges with juries: Evaluating llm generations with a panel of diverse models

Q2: Although the multi-agent framework is well-justified, could you provide a more explicit differentiation from other multi-agent critique frameworks (e.g., Arena learning and Stable Alignment). Clearer explanations on how MultiCritique uniquely contributes to critique generation and feedback quality.

A2: We appreciate your questions regarding the differences between MultiCritique and previous works. The main differentiation lies in the targets and techniques. Let us clarify as below:

As described in our related work, current multi-agent frameworks are widely used in two applications: (1) multi-agent framework for LLM alignment; (2) multi-agent framework for LLM-based evaluation.

Multi-agent framework for LLM alignment

Numerous works prove the effectiveness of multi-agent framework for improving LLM's alignment and response quality. For example, Arena Learning leverages a Llama-3-70B model as a judge to evaluate battle results between multiple models for target LLM improvement. Similarly, Stable Alignment employs multi-agent critiques for alignment refinement. Unlike these works, our work propose a unified multi-agent framework to improve LLM's critique ability, rather than LLM's alignment.

Multi-agent framework for LLM-based evaluation

Multi-agent frameworks are also used to improve the reliability of LLM-based evaluations, such as ChatEval and PoLL (described in related work). While these works improves the evaluation quality during the inference stage, our work focuses primarily on improving the critique ability of LLMs through the SFT and RL fine-tuning, aiming to overcome two challenges in recent critique research: (1) limited critique dataset quality by single model bias; and (2) upper bound brought by mainstream critique SFT approaches.

More detailed clarifications are placed in "Global - Novelty". We also extend the "Multi-agent Framework" paragraph in our related work to better clarify our differences with previous works. Thanks for your suggestions.

We sincerely appreciate your feedback and are pleased that we have successfully addressed your concerns. If you have any additional questions or require further clarification, we would be more than happy to assist you.

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1/Q2: ... I think the paper ideas a straightforward and easy to follow ... ensembling for dataset collection ...; From a research perspective, I think novelty is lacking a bit, neverthless it is an important contribution

A1: We are delighted that you acknowledge our presentations and contributions. We would like to emphasize that our work is not an ensemble for dataset collection. To overcome two challenges of critique research field, i.e., flawed critiques of one single model and upper bound of SFT fine-tuning, we propose a multi-agent feedback framework that aggregates diverse and accurate critiques from multi-agents, exhibiting superior quality than each model's critiques (as evidenced in Section 6, Table 4). Furthermore, beyond the merely behavior cloning in SFT, we propose to calibrate the preference critiques by utilizing the multi-agent feedback from the critique's downstream revision task, facilitating the robust RL for improving LLM's critique ability.

You can find more clarification about out innovation in "Global Response - Novelty".

Q3: Performance is still limited by an upperbound of the the base LLM .. if sclae to 70b, would the performance be better

A3: We appreciate this insightful question. While our experiments on 7B-8B models are comprehensive, our investigation of larger models is insufficient. We have supplemented a new limitation section (Appendix A.5) to clarify this question. Our preliminary study demonstrates that the improvements brought by MultiCritique on the advanced Qwen2.5-72B-Instruct model are not very significant. Our analysis suggests two primary reasons for this phenomenon:

• These 70B models have been sufficiently fine-tuned, exhibiting very powerful critique and generalization ability. For example, Qwen2.5-72B-Instruct achieves 83.62% F1 on CriticBench, surpassing GPT-4's 78.75%. Besides, its objective critique performance in CriticEval is also superior to GPT-4 (78.72% > 76.09%). This high level of critique performance makes further improvements challenging.
• Our experiments on 70B models utilize only MultiCritique data for optimization. We suggest that the data mix strategy that integrates our dataset with other high-quality instruction data, might play a crucial role in improving their performance further.

However, due to constraints in our computational resources and scheduling, our experiments and studies on these advanced 70B models are insufficient. We have incorporated these details into the limitation sections, and plan to address this limitation in our future work.

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1/Q2: Mostly this paper feels like just a bit of a formalisation for distilling “critiquing” ability from GPT-4 ...; Novelty of this work is not very clear to me

A1/A2: We appreciate the reviewer's question. We would like to clarify that our work does not distill GPT-4's critique capabilities. Instead, we propose a multi-agent feedback framework that aggregates diverse and accurate critiques from multi-agents, which exhibit superior quality than each model's critiques (as evidenced in Section 6, Table 4). Furthermore, we calibrate the preference critiques by utilizing the multi-agent feedback from the critique's downstream revision task, facilitating the robust RL for improving LLM's critique ability (Section 3.3). This preference critique calibration process does not rely on GPT-4.

Overall, our work does not distill the critique ability of GPT-4. Please refer to "Global Response - Novelty", and "Global Response - GPT-4 dependency" for more clarifications.

Q3: What is the generalizabilty of this kind of fine-tuning ... can still help in generating critiques in unseen domains like math and code.

A3: We appreciate your important question about the generalization capability of our approach. Our extensive experiments demonstrate a strong generalization of MultiCritique to unseen domains, supported by the following evidence:

1. Strong Improvement on Unseen Domains in CriticBench

   Symbolic and Algorithm reasoning tasks in CriticBench are explicitly excluded from our training data (see Table 12, Appendix D), making them out-of-domain evaluation tasks. Table 1 demonstrates that MultiCritiqueDataset-SFT significantly improves the critique ability on symbolic (from 18.82% to 57.04%) and algorithm reasoning (from 14.29% to 51.85) tasks. Besides, RL further improves the critique ability on these two unseen tasks.

2. Ablation Study of Removing Math and Code Tasks

   Following your suggestion, we also supplement the ablation study to study the generalization of MultiCritique to math and code domains. Specifically, we exclude the math and code training samples in MultiCritiqueDataset-SFT, and evaluate the fine-tuned model's generalization on these two unseen tasks. Partial results in CriticBench are shown as follows, and complete results are shown in our revised submission (Generalization to Unseen Tasks in Section 6):

   Models	Math	Comm.	Symb.	Algo.	Code	Overall
   InternLM2.5-7B-Chat	59.46	48.26	42.63	37.21	63.97	53.98
   + MultiCritique w/o Math&Code	88.44	60.42	55.02	45.91	77.63	71.78

   Results demonstrate that model fine-tuned on dataset without math and code samples can still achieve significant improvements on these two tasks.

Q4: The performance improvement of the MultiCritiqueDataset-RL over MultiCritiqueDataset-SFT is not very clear for some tasks ... Also what about including both MultiCritiqueDataset -SFT and -RL?

A4: We appreciate the reviewer's insightful questions about the performance differences between MultiCritiqueDataset-RL and MultiCritiqueDataset-SFT. Let us address these points comprehensively:

1. Performance Variations Across Different Tasks

   Table 1 demonstrates that the improvement brought by MultiCritique-RL on mathematics, coding, and commonsense reasoning tasks in the CriticBench benchmark is modest. Since they are covered in MultiCritique, SFT's improvements are already significant. As a result, RL may not necessarily bring sufficient improvements. On the contrary, on unseen tasks, RL could bring substantial improvements (57.04% to 61.51% on Symbolic, and 51.85% to 57.76% on Algorithm).

2. Rationale for Separate SFT and RL Phases

   Following the established RLHF workflow, like InstructGPT [1], we adopt the typical two-phase learning approach (SFT followed by RL), where SFT first calibrates the basic output behavior, followed by RL for further alignment.

[1] Training language models to follow instructions with human feedback

Q5: Minor concern about terminology ... but I don’t see why this is termed “multi-agent”

A5: Thanks for your question. Following previous works [2], the term "multi-agent feedback" is chosen to accurately reflect the distinct characteristics of our MultiCritique, aligning with previous works in multi-agent frameworks (described in related work). Unlike traditional instruction tuning that relies on a single model's output, our approach fosters diverse thinking of multiple models on critiquing response quality, overcoming the challenge of flawed critiques of one single model.

[2] ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

We really sincerely appreciate your positive feedback on our unseen domain experiments. We would like to take this opportunity to further clarify the comprehensive nature of our work's main contribution.

Our primary contribution lies in developing MultiCritique, a unified multi-agent feedback framework that addresses two fundamental challenges in critique research from both SFT and RL fine-tuning phases: (1) The bias of single-model critiques; (2) The performance upper bound by conducting supervised fine-tuning on critiques.

The construction of the MultiCritiqueDataset (both SFT and RL parts) is an integral extension and implementation of the MultiCritique framework. The superior performance of models trained on these datasets validates the effectiveness of the MultiCritique framework.

We hope this clarification illustrates how our work's contribution extends beyond dataset construction to encompass a comprehensive solution for improving LLMs' critique abilities.

Thank you again for your valuable feedback.

We really appreciate your valuable suggestions and insightful questions. Your questions are addressed as follows:

Q1: Dependence on GPT-4: The project heavily relies on GPT-4 ... such as Claude 3.5 Sonnet, to see how the results might differ.

A1: We appreciate your valuable questions about our proposed method. We want to emphasize that our MultiCritique is a general and model-agnostic framework, and GPT-4 serves only as one possible meta-critique judge model. Following your suggestions, we utilize advanced Claude-3.5-sonnet and Qwen2.5-72B-Instruct to conduct the Meta-Critique and Critique Summarization (Section 3.2) and analyze their correlations with GPT-4. Our analysis reveals strong correlations between these models and GPT-4, confirming their viability as robust alternatives within our framework. You can find more details in "Global Response - GPT-4 Dependency" and Appendix G in our revised manuscript.

Q2: The model's focus on improving critique ability ... Evaluating ... like AlpacaFarm Arena-Hard ... degradation in general capabilities.

A2: We appreciate your valuable question about potential overfitting to critique tasks. Although solely fine-tuning on MultiCritique might lead to degraded performance in general ability, MultiCritique could be integrated into the existing SFT dataset pool to overcome this issue. Our experimental results prove that MultiCritique enhance both critique ability and general capabilities of LLMs.

Specifically, we integrate MultiCritique-SFT into open-source instruction-tuning datasets and optimize the InternLM2-7B-Chat-SFT model. Then, we evaluate both the critique and general capabilities of the LLM. The evaluation of general capabilities is split into two aspects:

1. Objective evaluation computes the average performance on 18 famous benchmarks, like MMLU and GSM8K.
2. Subjective evaluation uses famous CompassJudger toolkit [1] to evaluate model performance on 4 general alignment benchmarks.

Partial results are shown as below, and complete results are shown in Section 6 and Appendix C (Table 10) of our revised submission.

The results demonstrate that model optimized on our integrated dataset achieves consistent improvements in both critique and general capabilities.

[1] CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution

Q3: Table 4 is somewhat unclear to me ...

A3: We appreciate this important question. Let us clarify the motivation and experimental setup in detail:

Motivation

This ablation study aims to systematically evaluate the effectiveness of our MultiCritique-SFT pipeline compared to individual model feedback that is widely used in previous works. Specifically, we investigate whether the aggregated multi-agent feedback outperforms each one-single-model's individual critiques.

Implementation Details

1. Base Model: InternLM2-7B-Chat-SFT, the same as the main experiment.
2. Dataset Format: The dataset in this ablation study is slightly different from that in the main experiment (Table 1), consisting of two parts:
   1. Distinct parts: crucial information and analytical critiques (a list of ACUs), without the summarization and judgment score.
      1. MultiCritique-SFT Critiques: Critiques are generated through our MultiCritique-SFT pipeline, which aggregates accurate ACUs from multiple models via meta-critique classification.
      2. Four Individual LLMs Critiques: Critiques are extracted from our raw MultiCritique-SFT dataset, using feedback generated independently by four models.
   2. Shared parts: to enable the objective evaluation on CriticEval and CriticBench benchmarks, we supplement 5% of samples from MultiCritique-SFT to ensure the fair comparison, which consists of crucial information, analytical critiques, summarization and judgment score.

Experimental Results

Table 4 demonstrates that MultiCritique-SFT achieves substantial improvements over individual model feedback, validating that our multi-agent feedback framework effectively aggregates the strengths of multiple models while mitigating their individual limitations.

We have incorporated these details in our revision to better illustrate the experimental design and findings. Thank you for helping us improve the paper's clarity.