Tool-Augmented Reward Modeling

[1] https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2

[2] Dong, Hanze, et al. "Raft: Reward ranked finetuning for generative foundation model alignment." arXiv preprint arXiv:2304.06767 (2023).

Dear Reviewer zDVk,

Many thanks for your detailed and constructive comments.

Response to weaknesses:

For W1: Thank you for your valuable feedback. In response to your concern, we have conducted experiments on two standard reward datasets: WebGPT Comparison and HH-RLHF. The results are as follows: We partitioned the WebGPT Comparison dataset into 13.6K training samples and 2.4K test samples. Additionally, we randomly extracted 50K samples from HH-RLHF as training samples, combining them with our TARA dataset. The results reveal that our Themis outperforms other vanilla reward models, with the exception of RAFT (LLaMA-7B). It's noteworthy that RAFT (LLaMA-7B) performs SFT on the 112K positive training samples of the HH-RLHF dataset and then executes reward modeling training on 100K pairwise samples. In contrast, Themis does not perform SFT and is exclusively trained on 50K samples from HH-RLHF (vs. RAFT trained on 100k samples on RMs as well as SFT on 112K samples) in a multi-task learning setting (WebGPT Comparison + HH-RLHF + TARA), achieving comparable performance with RAFT (LLaMA-7B). We believe these additional experiments address your concerns and provide valuable insights into the effectiveness of our approach on standard reward modeling datasets. Thank you for bringing this to our attention, and we appreciate your thoughtful review.

For W2: There are two types of hyper-parameters in our work: training configurations ($\alpha, \beta, \omega \in \{0, 1\}$ in Sec 2.2) and experiment hyper-parameters (such as batch size and learning rate). The hyper-parameters $\alpha, \beta, \omega$ correspond to different training settings as shown in the following: We have provided a comprehensive list of experiment hyperparameters and training details in Appendix C.1, specifically in Table 8 and Table 9 of the submitted version. However, it's crucial to acknowledge that we did not conduct an extensive hyperparameter tuning (especially for batch sizes) in this study due to limitations in our training resources. To maximize resource utilization, we opted for the largest feasible batch size and set the learning rate to 1e-5. We appreciate your observation and acknowledge the potential value of a more extensive hyperparameter study, which could be explored in future work with additional resources.

For W3: Thank you for your insightful comment. We appreciate your attention to the dataset creation challenges, and we have incorporated a discussion of these limitations in the Limitation section of our paper. Despite the inherent complexities, we have implemented measures to streamline the process and facilitate scaling to larger preference datasets. To address scalability concerns, we have established an automatic pipeline for dataset construction involving multi-agent interactions using LLMs. Furthermore, we have integrated standard preference data (hh-rlhf) with our tool-augmented data during reward model training. The resulting model exhibits superior generalization performance compared to the vanilla reward model, as detailed in Section 4.4. While tool selection and heuristic design are integral to tool-augmented datasets, we have automated and accelerated these processes for scalability. By defining the tool invoke format, we can generate numerous instances through our automatic pipeline, contributing to efficient scaling. We hope this can answer your question. Furthermore, we will open-source our tool-augmented dataset and code to the research community. We believe this transparency will foster collaboration and further development in this area. Thank you for your valuable feedback, and we look forward to contributing to the broader research community.

For W4: We sincerely appreciate your valuable feedback. In response to your suggestion, we have enhanced the discussion on related works in Section 5 of the paper. The updated section now delves into the works you referenced, providing a more comprehensive context that helps better frame the contribution of our work in relation to existing literature. We hope this revision meets your expectations, and we thank you for bringing this to our attention.

We trust that the aforementioned clarification could address your concerns. We genuinely look forward to engaging in further discussion and appreciate your time and consideration. Thank you for valuable feedback!

Thank you for your feedback and for revisiting our experiments! The superior performance of Themis on the WebGPT and HH datasets can be attributed to two key factors:

1. Utilization of Search Tools:
   • Themis leverages the Wiki Search and Google Search tools, which prove highly beneficial for tasks associated with the WebGPT and HH-RLHF datasets. By tapping into these search tools, Themis gains access to additional pertinent information, enhancing its ability to comprehend and address the intricacies of the tasks.
2. Benefit of the Rationale Stage:
   • The incorporation of the Rationale stage in Themis prompts a Chain-of-Thought (CoT) approach to reward generation. During this stage, the tool-augmented reward model engages in information aggregation, synthesis, and self-reflection. This process empowers Themis to navigate through complex scenarios, providing a nuanced and contextually relevant basis for generating rewards. The Rationale stage plays a pivotal role in Themis's capacity to reason effectively and contribute to its superior performance on these datasets.

We value your continued engagement and hope that this response could adequately addresses your concerns. If you have any further questions or require additional clarification, please feel free to reach out.

For W5. Sorry for the confusion about the data collection steps. Due to the page limits, we have revised Section 3.1 and added detailed prompts for each agent in Appendix B in the submitted version for a more comprehensive understanding. Furthermore, we have also revised Section 3.1 in the draft version, hope this can address some of your concerns. We simplify the dataset construction steps as follows:

• Step 1: Initially, we collect a reward dataset featuring each instance comprising a question, a positive answer, and a negative answer. To construct this dataset, we employ two distinct approaches: resume open-source, high-quality datasets, and generation from scratch with some heuristic methods. However, the above methods usually only produce positive answers. To address this concern, we leverage a GPT-4 as a negative generation agent to generate antagonistic negative answers, which will be described in Step 3.
• Step 2: Subsequently, we define the name, and API calls of some tools including Calculator, Code Interpreter, Translator, Google, Weather, Calendar Search, and Wiki Search.
• Step 3: In this step, we leverage the capabilities of GPT-4 to generate the tool invocation processes and negative samples. To achieve this, we design a simulated environment where GPT-4 plays three different agents: the negative generation agent, the tool agent, and the rationale agent. Taking the Calculator tool as an example, we extract some instances from GSM-8K. However, the data only contain questions and positive answers. To address this limitation, we prompt the negative generation agent to generate the negative answer, constructing the pairwise preference samples. Subsequently, we query the tool agent to generate the tool invocation process, encompassing the Thought, Action, and Action Input stages. Then humans execute the tools based on the generated tool information, yielding the Observation. Finally, we amalgamate the above information (inputted question-answer, tool invocation process, observations) and prompt the rationale agent to produce the Rationale stage. For a more comprehensive understanding of this process, additional details are available in Appendix B.2.
• Step 4: We exclude tool-invoked processes acquired in Step 3 that exhibit invalid formats. Subsequently, we discard processes exceeding three interaction steps, lacking relevant function calls, or manifesting parsing errors in their execution results.

We hope the above can address some of your concerns. And we sincerely look forward to further discussion. Thank you for your time and consideration!

For W2. GPT-4 is a closed-source model that poses significant challenges in terms of cost and accessibility. Additionally, the practicality of conducting reward model training on GPT-4 is hindered by the necessity to add extra fully connected layers for reward prediction since OpenAI API did not support this feature. Given its high cost and limited availability for public use, employing GPT-4 directly as a reward model becomes impractical for wider applications. Therefore, we just utilize the strong in-context learning ability of GPT-4 and prompt GPT-4 to generate partial data, including tool-invoking processes (Thought and Action in Sec 2.2) and rationales (Rationale). Subsequently, we meticulously filter out invalid tool invocations and incorrect rationales to ensure the quality and reliability of the generated training data.

For W3. Your question is insightful, and we appreciate the opportunity to clarify. When alpha is set to zero, the model transforms into a vanilla reward model, specifically corresponding to RM (Vicuna-7B/Bert-Large) as indicated in Table 1 and Equation 1. In this configuration, the model generates a scalar reward solely through a fully connected layer, devoid of any tool-invoking or explanation-generating processes. With alpha set to zero, it's crucial to note that the reward model cannot spontaneously produce a tool use process without explicit loss terms on tool use. While we can resort to in-context learning (ICL) by prompting the model with a few tool-use examples, it is important to highlight that this approach may still lag behind in finetuning on different tools (with $\alpha=1$). Additionally, it is constrained by the maximum context length, as the contextual examples of ICL can consume significant length.

For W4. In Sec 4.2 (Scaling Trends in Themis), we explored the correlation between the performance and the scale of models. To enhance the training efficiency and reduce the resource cost, we utilized the parameter-efficient fine-tuning method LoRA which only trains specific modules, as shown in Table 1. The results elucidated a positive correlation between the scale of the model and its overall performance, and highlighted a substantial gap in the WikiSearch tool, as you rightly pointed out. Taking your valuable advice into consideration, we have conducted full-parameter fine-tuning with Themis (Vicuna-33B), and reported the results as follows. As illustrated in the table below, Themis with 33B demonstrates superior performance compared to its 7B counterparts, exhibiting an approximately 2% absolute improvement in preference prediction. Notably, the larger models showcase optimal capabilities in harnessing Wiki Search, Google Search, and Translator tools. This trend aligns with the findings from our LoRA experiments, which we have incorporated into Table 1. We value your thoughtful review and trust that these additional experimental results adequately address your concerns.

Dear reviewer wTHw,

Many thanks for your detailed and constructive comments.

Response to weaknesses:

For W1. Your concern about the potential bias introduced by the tools is indeed crucial. As you said, the tools may produce biased aspects or wrong aspects here, such as invalid tool invocations, incorrect tool-invoked results, biased content, and unexpected errors such as network failure. To avoid the reward model interference by these biases, we have implemented several measures:

1. Strict Quality Control: We rigorously control the quality of the training set, filtering out invalid tool invocations generated by GPT-4 to ensure the reliability of the training data.

2. Noise Injection: Recognizing the possibility of tools not always providing accurate outputs, we intentionally introduce "noise" instances in the training data. This strategy helps train the reward model to produce correct rewards even in scenarios where the tools may not work optimally. An example is shown in the following. Our Themis model encounters an error during the invocation of the Wiki Search tool and no results are returned. Importantly, our model demonstrates a capacity for reasoning based on its internal understanding, unaffected by the absence of tool output, which highlights the ability of our model to engage in self-reflection and form independent judgments when faced with unexpected tool behavior.

   Question: What do you call the chinese writing system?

   Answer: Standard Mandarin

   Thought: I need to invoke the Wiki Search tool to search Chinese writing system

   Action: Wiki Search

   Action Input: Chinese writing system

   Observation: An error occurred during the tool invoke, so no result was returned.

   Rationale: The answer provided is "Standard Mandarin," which is incorrect in the context of the question. The question is asking about the Chinese writing system, and "Standard Mandarin" refers to a spoken language, not a writing system. The tool's execution, however, resulted in an error, and no relevant information was retrieved. Based on this, the answer is already incorrect, and the tool's failure to provide information doesn't impact the correctness of the answer.

3. Rationale Generation: A key aspect of our Themis approach is the Rationale generation stage. Here, the tool-augmented reward model aggregates and synthesizes previously acquired information and undergoes a reasonable reasoning process, which involves RM's self-reflection instead of listening to the results of the tool. The emphasis on rationale generation encourages Themis to form reasoned decisions, relying on a process of synthesis and self-reflection rather than solely relying on the output of the tools. This approach enhances the model's ability to reason independently and avoid undue influence from tool biases. This is an example in which the model generates incorrect tool invocations, leading to biased observations. However, it is crucial to note that our model identifies the errors in tool results through the Rationale module. Importantly, even in the presence of biased information, our model maintains its capacity to provide reasonable answers, showcasing its robustness and ability to navigate through biases introduced by tool invocations.

   Question: Why is the flight/fight our defence mechanism when remaining calm and composed may serve better outcomes?

   Answer: The fight or flight response is an evolutionary response that happens in stressful situations. It can be helpful in situations where there is a perceived threat to survival. The fight or flight response is characterized by feeling bodily sensations of stress, such as an increased heart rate and faster breathing. This readies the body to either fight the danger or flee from it. The body’s sympathetic nervous system is responsible for this response, which triggers the release of adrenaline and noradrenaline...

   Thought: The information given in the answer is well-established knowledge within the field of psychology and does not pertain to a current event. Therefore, I don't think I need to use a tool to verify this.

   Action: Google Search

   Action Input: None

   Observation: 1 of 4. pronoun · ˈnən. singular or plural in construction. Synonyms of none. 1. : not any. 2. : not one : nobody. 3...

   Rationale: Although the executor's tool results did not provide any specific results or contradictions, the question asked why remaining calm and composed may serve better outcomes as a defense mechanism compared to the flight/fight response. However, the answer only explained what the fight or flight response is and how it works. The question about the potential advantage of remaining calm and composed in stressful situations wasn't addressed. Thus, it's reasonable to reason that the answer is incorrect.

We sincerely appreciate your thorough review and the time you took to consider our responses and revisions! We appreciate the reviewer's insightful feedback, which contributes to the robustness and clarity of our research.

Dear Reviewer bKyt,

Many thanks for your detailed and constructive comments.

Response to Weaknesses:

You accurately captured the essence of our approach. Our reward model initiates tool-invoking processes and generates explanations based on the inputted question and answer. Subsequently, we append an <eos> token, and the scalar reward is produced by a fully connected layer using the last hidden states corresponding to the <eos> token. To enhance clarity, we have revised the method description in Section 3, providing more details. We trust that these modifications improve the overall clarity of our method description.

Response to Questions:

For Q1: In Themis (Vicuna-7B + LoRA), we adopt a strategy where specific parameter-efficient modules (using LoRA) are trained, while the pretrained weights of Vicuna are frozen. This approach significantly reduces training resource costs but may lead to a performance decrease compared to the full-parameter fine-tuning setting.

For Q2: RM (Vicuna-7B) represents a vanilla reward model. In this configuration, the model generates a scalar reward solely through a fully connected layer, without engaging in any tool-invoking or explanation-generating processes. Therefore, in the context of our methodology, RM (Vicuna-7B) aligns with the setting where $α=0$.

For Q3: The observed lower performance in the zero-shot setting is primarily attributed to the fact that it was evaluated without training on HH-RLHF datasets. In contrast, the fine-tuning results demonstrate performance exceeding 50%. More specifically, the lower accuracy for RM (Vicuna-7B, Zero-shot) is a result of the specific design choices made in the RM model, particularly in the zero-shot setting. The evaluation on the HH-RLHF* dataset in the zero-shot scenario revealed unsatisfying performance, as indicated by an accuracy below 50%) As outlined in response to a previous question (Q2), RM (Vicuna-7B) employs a scalar reward generated solely through a fully connected layer. This approach, while chosen for its simplicity and efficiency, has inherent limitations, especially in the context of a shallow and weak signal. The model relies on the input question and answer, which may result in overfitting to training samples, particularly when evaluating with the complexities of the hh-rlhf* dataset in the zero-shot setting. In contrast, the proposed method outperforms vanilla RMs in both zero-shot and fine-tuning settings (as shown in Table 2). Moreover, it's essential to note that the task is not a traditional binary classification problem. RM (Vicuna-7B, Zero-shot) is designed to predict the preference score for both negative and positive pairs, then compute the correctness (acuracy) of the ranking order. We hope our response can address your questions.

We hope the above can address your concerns. We sincerely appreciate your time and thoughtful consideration. We look forward to any further discussion and appreciate your ongoing engagement with our work.

Re: GPT-4 as the RM Baseline. Thank you for your valuable suggestion. We appreciate the insightful idea of incorporating a ChatGPT or GPT-4 baseline in future experiments. Below, we outline the advantages and disadvantages of our proposed Themis model in contrast to GPT-4:

Advantages:

1. Ease of Application and Fine-tuning:
   • Themis: Can be easily applied and fine-tuned, making it compatible with traditional reward models.
   • GPT-4: A closed-source model currently not supported by fine-tuning in the reward modeling context. It lacks the capability to add an additional fully-connected layer and conduct contrastive reward modeling training.
2. Domain Specialization and Prompt-free Usage:
   • Themis: Facilitates easy domain specialization through fine-tuning, allowing adaptation to specific contexts. It also excels in prompt-free usage, automatically determining when and how to call pre-defined tools during inference.
   • GPT-4: Requires complex prompt design and execution in each stage for domain-specific tasks. In particular, it relies on in-context learning ability and intricate prompt design for generating various stages, such as tool invocation and rationale stages. Importantly, prompt designs may need adjustment with GPT-4 updates.
3. Better Training/Inference Stability:
   • Themis: Offers stability, avoiding recent downtimes associated with GPT-4/ChatGPT services due to increased calling burden. Themis is designed to be a reliable choice for model training.
   • GPT-4: Suffers from various factors during training, such as OpenAI API services, network failures, and other potential disruptions.
4. Consistent Reward Format:
   • Themis: Produces continuous scalar rewards through a fully connected layer, ensuring a consistent format.
   • GPT-4: Generates rewards in natural language format, necessitating the formulation of rules to extract reward values from prompted results.
5. Cost-effectiveness:
   • Themis: Serves as a reusable model that is significantly more cost-effective than GPT-4, providing an economical alternative for reward modeling tasks.
   • GPT-4: Requires extra cost for API calling, adding to the overall expenses.

Disadvantages: One notable disadvantage of Themis is the requirement for some data for fine-tuning. This contrasts with GPT-4, which can potentially leverage pre-existing knowledge without fine-tuning.

We highly appreciate your valuable input, and we are committed to integrating these insights into our upcoming experiments and the refined version of our work. Thank you for acknowledging the motivation behind our tool-enhanced reward models.

We hope these updates adequately address your concerns and kindly encourage you to reconsider our review score in light of these clarifications. Your contribution significantly enhances the quality of our research, and we thank you for your thoughtful feedback.

Re: Q3 Follow-Up. We appreciate the reviewer's continued engagement and would like to address the additional concern regarding the accuracy of RM using Vicuna-7B in a zero-shot setting.

The observed lower performance is primarily due to the challenges associated with domain generalization in vanilla RM. Specifically, the RM (Vicuna-7B) trained on tool-related data using the vanilla RM objective (without the ability of tool use) may encounter overfitting to the domain-specific data (including Calendar, Calculator, Weather, Code, Translator, Wikipedia, and Google Search). Consequently, it achieves less than 50% accuracy (worse than random) on the HH dataset concerning human alignment, highlighting the vulnerability of vanilla RM to overfitting within the in-domain data. On the other hand, Themis, which is also trained on tool-related data, attains a 55% accuracy on the HH dataset (human alignment domain) in the zero-shot setting. This underscores the exceptional out-of-domain generalization in preference prediction, credited to the effective integration of tool use.

Additionally, we would like to emphasize two key factors influencing zero-shot performance:

1. Dataset Distribution and Domain Differences: It is crucial to emphasize the substantial disparity in data distribution between our dataset, TARA, which is tool-related, and the HH-RLHF dataset, which focuses on human alignment with helpful and harmless content. The distinct nature of these datasets introduces challenges in achieving optimal zero-shot performance. The inherent differences in domain and data characteristics contribute to the observed lower accuracy in the zero-shot setting.
2. Impact of Model Parameter Size: The size of the model plays a significant role in zero-shot performance. Larger models, such as RM(Vicuna-7B), may exhibit a susceptibility to overfitting (easy to converge), as evidenced by the rapid convergence in just one training epoch (See Figure 3 Left). In contrast, a smaller model like RM (Bert-Large) requires more epochs to fit the reward data. The phenomenon of overfitting in a large model could explain the lower accuracy in the zero-shot setting, where the model may struggle to generalize effectively to unseen datasets. It's noteworthy that RM (Vicuna-7B) shows improved performance by +30% when finetuned on a smaller set of HH-RLHF instances (500), demonstrating the adaptability of the large model to relevant (in-domain) training data.

To sum up, the challenges observed in zero-shot accuracy can be attributed to the significant differences in dataset distribution and the impact of model parameter size. We are committed to providing a more comprehensive understanding of these challenges and implementing necessary refinements to address them in the final version of our work.

Dear Reviewers and AC,

We genuinely value the constructive comments and insightful suggestions you provided for our work. Recognizing the approaching end of the discussion period on November 22nd, we kindly request your valuable time and expertise for further discussion and re-review of our manuscript and response.

In light of the feedback received and to reiterate the key contributions of our work:

1. Introduction of the Tool-Enhanced Reward Model Framework: We advance the domain of tool-augmented preference modeling by introducing the Themis framework. This innovative framework firstly harnesses the power of external tools to augment preference modeling in large language models (LLMs). It effectively addresses inherent limitations observed in conventional reward models (RMs) and provides transparency and interpretability by unveiling the inner workings of the RM's decision-making process.
2. Creation of the Tool-Augmented Reward Modeling Dataset (TARA): We present the Tool-Augmented Reward Modeling Dataset (TARA), a novel dataset that includes comprehensive comparison data of human preferences and detailed tool invocation processes. Our commitment is to make this dataset publicly available, aiming to facilitate further research advancements in the field of tool-augmented preference modeling.
3. Comprehensive Experimental Validation: Our contributions are validated through extensive experimental evaluations conducted across eight diverse tasks, varying parameter scales, and experimental settings. Furthermore, we assessed the out-of-domain generalization of our approach using benchmarks like TruthfulQA and Retarded-bar (en) datasets. This evaluation was complemented by comprehensive RLHF investigations and in-depth analysis, highlighting the exceptional generalization capability and alignment ability of our method with humans and factual accuracy. The results of these experiments conclusively demonstrate the effectiveness of our approach in enhancing the performance of LLMs.

We believe our work can accelerate the use of tools in human alignment, providing more accurate and informative feedback to guide LLM fine-tuning. Additionally, we reaffirm our commitment to the open-source community by planning to release our code, data, and models, fostering collaboration and further research in this area.

We kindly urge you to participate in the ongoing discussion and provide any additional insights or clarifications you may have. Your expertise is invaluable to us, and we believe your input will significantly contribute to the improvement of our work.

Thank you very much for your time and consideration. We look forward to hearing from you soon.

Authors