StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs

W1: Novelty & technical contributions of this work.

A1: Thanks for your thoughtful feedback. We want to take this opportunity to clarify the novelty and contributions of our work.

Tool learning is an emerging scenario to extend the capabilities of large language models (LLMs) by enabling them to utilize external tools (APIs). Most existing approaches focus on data construction and supervised fine-tuning (SFT) methods to improve tool-use capabilities [1-4]. Our work is the first to explore tool learning from a reinforcement learning (RL) perspective, introducing a reward shaping mechanism to supervise intermediate steps and an optimization framework compatible with various policy-gradient-based RL algorithms. Unlike traditional RLHF methods [5], StepTool provides process-level supervision for intermediate steps and explicitly considers the interdependencies between multi-step actions in a trajectory during optimization.

Broader Implications of StepTool: This work provides a fresh perspective distinct from the dominant SFT-based approaches [1-4], demonstrating the potential of RL as a complementary method for optimizing tool-use tasks. As a foundational contribution, StepTool establishes a general framework that can inspire future innovations in RL techniques tailored to tool learning.

Thank you again for your insightful feedback. We view this work as a meaningful step toward broadening the application of RL methodologies in tool learning, and we look forward to contributing to continued advancements in this evolving field.

[1] Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121, 2024.

[2] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518, 2024.

[3] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.

[4] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.

[5] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

Q7: Does SFT on expert GPT-4 data followed by RL do better than RL alone?

A7: In our experiments, we adopted the standard pipeline [1], which involves using supervised fine-tuning (SFT) to initialize the model for instruction-following before applying RL methods. The SFT stage provides a strong foundation by ensuring well-formatted outputs for tool learning tasks and establishing robust instruction-following capabilities. Without SFT, RL would have to optimize from a less capable starting point, leading to less stable convergence and reduced effectiveness. This approach aligns with standard practices in related work that employ RL to fine-tune models [2-3].

Specifically, in our setup, we first applied SFT on expert GPT-4 data to initialize Qwen2 and Llama3.1. For ToolLlama, which has already undergone pre-training through SFT methods, we directly proceeded with RL training. Our experiments aim to demonstrate the effectiveness of step-grained supervision and optimization in enhancing model performance.

[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[2] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080, 2023.

[3] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935, 2023.

Q6: Pass@k metrics would be useful to understand whether the model discovers new knowledge throughout RL training (i.e. exploration), or is simply re-weighting its prio.

A6: Thank you for suggesting the use of Pass@k metrics to assess whether the model discovers new knowledge or re-weights its prior. As a widely-used metric in domains such as mathematical reasoning [1,2], Pass@k provides meaningful insights for tool learning tasks. We have conducted additional experiments under similar settings.

Specifically, we evaluated the ToolLlama model before and after optimization with our StepTool method under a temperature setting of 0.7, sampling 8 trajectories per task. To mitigate the time costs associated with trajectory sampling (due to real-world API interactions), we randomly selected 20 tasks from each subset as representative samples. We then computed Pass@2, Pass@4, and Pass@8 metrics, averaging results from three independent evaluations using the StableToolBench benchmark. The summarized results are as follows:

From the Pass@k results, we observe the following:

• Improvement with StepTool: ToolLlama optimized with StepTool outperforms ToolLlama across Pass@2, Pass@4, and Pass@8 metrics in most experimental settings.
• Evidence of Exploration: The improved Pass@k scores (across all values of k) suggest that the model is not merely re-weighting its prior knowledge but is also benefiting from the discovery of new knowledge during RL optimization.

Thank you again for suggesting the Pass@k evaluation. These results and insights will be included in the revised paper.

[1] Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially-correct solutions. arXiv preprint arXiv:2205.14318, 2022.

[2] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024.

Thank you for taking the time to read and review our paper. We respond to your comments and questions below.

Q1: How are rule-based and LLM-based reward annotation combined exactly? I couldn't find a statement of this in the text.

A1: Reward shaping in our framework includes three main components, SuccCalling, Contribution and IsSolved. Specifically, in our experiments:

• The values of SuccCalling were determined through a combination of rule-based and GPT-4-based annotations. Rules were employed to evaluate the format of the tool invocation (e.g., correctness of tool names and parameters). For invocations meeting the format requirements, GPT-4 was used to assess the actual success based on the tool response (e.g., a response like “Since the search parameter is empty, please provide the name of an author…” indicates a failed invocation due to incorrect arguments).
• The values of Contribution and IsSolved were obtained exclusively through automated annotations by GPT-4, which leverages its semantic analysis capabilities.

We sincerely appreciate your feedback and will incorporate these explanations into the "Step-Grained Reward Acquisition" section of the revised paper.

Q2: The “step-grained optimization” component of StepTool seems to just be the standard formulation of multi-turn RL (e.g. ArCHer). Are there any differences? If not, it should not be stated in an independent subsection.

A2: While both ArCHer and our framework aim to account for the interdependencies of multi-step actions in a trajectory, there are key distinctions in their implementation:

• ArCHer introduces an "action-level" critic that estimates a “multi-turn advantage,” capturing the influence of other steps within the trajectory (as detailed in Section 3.4 of the ArCHer paper). This value is passed to the actor for token-level optimization confined to a single step of action (see Eq. 3 in ArCHer).
• In contrast, our approach directly integrates multi-step rewards at the token level (see Figure 2 of our paper) for advantage calculation. Using the advantage calculation formula (Eq. 6), we explicitly incorporate the influences of other steps' rewards into the token-level optimization process, offering a more straightforward and interpretable solution.

The "step-grained optimization" component of StepTool is highlighted to emphasize how step-grained rewards are integrated into the optimization process, setting it apart from traditional RLHF implementations [1] by accounting for the interdependencies of multi-step actions.

We hope this explanation clarifies the differences and addresses your concerns. Thank you again for your valuable feedback.

Q3: Related to above: what is the difference between PPO and StepTool in the experiments?

A3: The core differences lie in how the training data is utilized. The "PPO" baseline, as a traditional RLHF implementation based on [1], is not designed to handle multi-step data and, as a result, applies rewards only at the final step of a trajectory. In contrast, our framework supports step-grained reward integration across the multi-step trajectory, considering the interdependencies of multi-step actions in a trajectory.

All other experimental configurations, including training data and reward annotations, were kept consistent between PPO and StepTool for fairness. Detailed experimental setups are available in the code repository to ensure transparency and reproducibility.

We sincerely appreciate your valuable feedback and will clarify this distinction further in the experimental setup section of the revised paper.

Q4: Appendix A/eq. 2 seems to be a standard result in RL and should be omitted (and just cited instead).

A4: The derivation in Appendix A was included to clarify the transition from $\nabla \log \pi(\tau)$ to $\nabla \log \pi(a_t|s_t)$, aimed at assisting readers less familiar with RL concepts, particularly those accustomed to SFT-based methods in tool learning.

Thank you for the valuable suggestion. Based on your feedback, we will remove this derivation in the revised version and provide a citation instead.

Q5: What discount factor (γ) is used in the experiments?

A5: The discount factor ($\gamma$) used in our experiments was set to 0.99. Detailed experimental configurations are included in the code repository to ensure transparency and reproducibility.

[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

W1: What is the difference between StepTool and PPO besides reward shaping in the experiments?

A1: The core differences lies in the utilization of training data: the "PPO" baseline, as a traditional RLHF implementation following [1], is not designed to handle multi-step data and therefore applies rewards only to the final step of a trajectory. In contrast, our framework supports step-grained reward integration across the multi-step trajectory, considering the interdependencies of multi-step actions in a trajectory.

All other experimental configurations, including training data and reward annotations, were kept consistent between PPO and StepTool for fairness. Detailed experimental setups are available in the code repository to ensure transparency and reproducibility.

We sincerely appreciate your valuable feedback and will clarify this distinction further in the experimental setup section of the revised paper.

[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

W3: If StepTool has no core difference from the above works at the algorithm level, reward shaping alone does not seem to be enough to support an ICLR accepted article.

A3: Thank you for your thoughtful feedback. We would like to take this opportunity to clarify the novelty and contributions of our work.

Tool learning is an emerging scenario to extend the capabilities of large language models (LLMs) by enabling them to utilize external tools (APIs). Most existing approaches focus on data construction and supervised fine-tuning (SFT) methods to improve tool-use capabilities [1-4].

Our work is the first to explore tool learning from a reinforcement learning (RL) perspective, introducing a reward shaping mechanism to supervise intermediate steps and an optimization framework compatible with various policy-gradient-based RL algorithms. Unlike traditional RLHF methods [5], StepTool provides process-level supervision for intermediate steps and explicitly considers the interdependencies between multi-step actions in a trajectory during optimization.

Broader Implications of StepTool: This work provides a fresh perspective distinct from the dominant SFT-based approaches [1-4], demonstrating the potential of RL as a complementary method for optimizing tool-use tasks. As a foundational contribution, StepTool establishes a general framework that can inspire future innovations in RL techniques tailored to tool learning.

Thank you again for your insightful feedback. We view this work as a meaningful step toward broadening the application of RL methodologies in tool learning, and we look forward to contributing to continued advancements in this evolving field.

[1] Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121, 2024.

[2] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518, 2024.

[3] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.

[4] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.

[5] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

W2: There exists many works [1,2,3,4] leveraging RL methods for language-argumented sequential decision-making problem. And the tool learning is a subset of language-argumented sequential decision-making problem. Thus authors should compare more carefully with these works instead of saying "Due to the lack of publicly available implementation details for some concurrent works, we choose two classic optimization methods: supervised fine-tuning (SFT) and PPO" in lines 346-347.

A2: Thank you for your insightful comments regarding the connection between tool learning and language-augmented sequential decision-making problems. We deeply value the advancements in Language RL Agents [1-4], which highlight the potential of RL to align large language models with complex goals in interactive and dynamic environments. However, we would like to elaborate on two major challenges that arise when directly applying Language RL Agent approaches to tool learning:

• Training paradigms: Language RL Agent tasks typically adopt an online training paradigm, where agents interact with the environment, receive explicit reward signals, and iteratively improve based on those signals [1-4]. In contrast, tool learning operates in a "semi-offline" context, more akin to RLHF setups [5]. In our framework, training involves improving the model’s performance on pre-generated (sampled) multi-step trajectories in a round-based process, where training data is iteratively collected. These offline trajectories include tool invocation responses but lack explicit or real-time reward signals. This difference in paradigm makes a direct transfer of Language RL Agent methods to tool learning nontrivial.
• Task Characteristics and Action Space: Language RL Agents often operate within relatively constrained environments, such as Overcooked, VirtualHome [2,4], or BabyAI-Text [1], where the actions usually can be grounded within a relatively limited action space. The overlap between training and evaluation actions in such tasks is substantial, which is typical for RL-based agents. Tool learning, however, features a significantly larger action space, encompassing thousands of APIs and parameter configurations. This scale introduces challenges in grounding and action enumeration. Moreover, in tool learning, it is common for evaluation tasks to involve tools that were not seen during training, as described in the ToolBench task design [6]. This introduces challenges related to generalization, grounding, and handling unseen tools.

Given these challenges, directly applying Language RL Agent methods to tool learning could result in inefficiencies or performance degradation. While exploring this transferability is an exciting direction for future work, it would require significant adaptation to align with the unique characteristics of tool learning environments.

We appreciate your thoughtful feedback and will revise the phrasing in the manuscript to better reflect the scope and context of our work. Furthermore, we believe that investigating the applicability of Language RL Agent methods to tool learning could open valuable avenues for future research and significantly enrich the field.

[1] Carta, Thomas, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. "Grounding large language models in interactive environments with online reinforcement learning." In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.

[2] Tan, Weihao, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." In The Twelfth International Conference on Learning Representations.

[3] Zhou, Yifei, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. "Archer: Training language model agents via hierarchical multi-turn rl." arXiv preprint arXiv:2402.19446 (2024).

[4] Wen, Muning, Ziyu Wan, Weinan Zhang, Jun Wang, and Ying Wen. "Reinforcing Language Agents via Policy Optimization with Action Decomposition." arXiv preprint arXiv:2405.15821 (2024).

[5] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[6] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.

Q3: What is the average number of tools utilized per task, and how does this correlate with the difficulty of the tasks? If the step length is excessive, could this indicate deficiencies in planning or task decomposition?

A3: Thank you for highlighting this area for deeper analysis. First, let us provide a detailed description of the task setup in the StableToolBench benchmark [2]. In StableToolBench, each task typically requires 2-3 API calls for successful completion, accounting for 88.5% of the tasks (68.6% requiring 2 APIs and 19.9% requiring 3 APIs). Task difficulty is primarily influenced by factors such as tool diversity (e.g., intra-tool, intra-category, or intra-collection variations) and whether the required tools are included in the training set. More details on task difficulty design can be found in the ToolBench paper [1].

Regarding the model's performance on the average number of tools utilized, we calculated the average tool invocations per task across subsets and model configurations. The results are summarized in the following table:

From the results, we observe the following key points:

• The average number of tool invocations per task shows minimal variation (<0.1 tool calls per task) across models under the same experimental settings. This observation is highly related to the task design in the benchmark, as mentioned above.
• The number of tool invocations does not directly correlate with model performance. This is because fewer tool invocations may indicate omitted critical API calls, while more tools invocations could reflect redundant steps.

In summary, we believe that step length could serve as a valuable metric for evaluating a model's planning and task decomposition capabilities. However, this aspect cannot be fully assessed within the constraints of this benchmark.

We appreciate your insightful feedback and will include these results and discussions in the Appendix of the revised paper. Additionally, we are grateful for your correction regarding our statements on Line 176 and will revise it in future versions of the paper.

[1] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.

[2] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. arXiv preprint arXiv:2403.07714, 2024.

Q1: Could the authors elaborate on how the auxiliary rewards might conflict with the ground truth reward and the implications of such conflicts?

A1: Thank you for raising this question. We are not sure whether the "conflicts with the ground truth reward" refers to discrepancies between manually designed rewards and environment-provided rewards, or potential estimation errors during reward annotation.

If the concern involves conflicts between manually designed rewards and environment-provided rewards, it is worth noting that tool learning environments often lack explicit or ground-truth reward signals. In such cases, our work is the first to provide a practical solution by defining auxiliary rewards to approximate the intended outcomes of tool usage. However, since ground truth rewards are unavailable, such conflicts cannot currently be detected.

If the concern relates to the potential reward annotation errors, these could arise during automated or manual labeling of auxiliary rewards or result from reward hacking. To improve the robustness of reward annotations and mitigate such errors, several strategies could be employed:

1. Reward Model Ensembles: Using multiple reward models and averaging their outputs can reduce the impact of individual model biases.
2. Multiple Annotations: Aggregating annotations from diverse sources can provide a more balanced reward signal.
3. Iterative Refinement: Refining reward annotations iteratively based on experimental results can improve alignment with the task's ultimate objectives.

We appreciate your feedback on this critical issue and hope this explanation addresses your question. If further clarification or discussion is required, we would be happy to elaborate and include these points in the revised paper.

Q2: How exactly are the contribution metric/reward trained?

A2: The values of our rewards are obtained from both rule-based judgments and GPT-4-based annotations, without training. Reward shaping in our framework includes three main components/functions, SuccCalling, Contribution and IsSolved. Specifically, in our experiments:

• The values of SuccCalling were determined through a combination of rule-based and GPT-4-based annotations. Rules were employed to evaluate the format of the tool invocation (e.g., correctness of tool names and parameters). For invocations meeting the format requirements, GPT-4 was used to assess the actual success based on the tool response (e.g., a response like “Since the search parameter is empty, please provide the name of an author…” indicates a failed invocation due to incorrect arguments).
• The values of Contribution and IsSolved were obtained exclusively through automated annotations by GPT-4, which leverages its semantic analysis capabilities.

We sincerely appreciate your feedback and will incorporate these explanations into the "Step-Grained Reward Acquisition" section of the revised paper.

W1: The author should consider more benchmark algorithm for comparison to illustrate the proposed method.

A1: As an emerging scenario, most existing work in tool learning relies on Supervised Fine-Tuning (SFT) to enhance the tool-learning capabilities of LLMs [1-3], corresponding to the SFT baseline in our paper. While these works differ in dataset construction, they all share the same optimization method: SFT.

Since our work is a pioneering effort in introducing RL-based optimization for tool learning, relevant baselines are scarce. Therefore, we implemented a classic PPO as a naive baseline to directly adapt RLHF [4] for tool learning. Our experiments demonstrate that a direct transfer of RLHF is not fully compatible, whereas our framework provides a more suitable and effective solution.

We appreciate your feedback and will further clarify these points in the revised paper.

W2: The authors should provide more implementations for the benchmark algorithm PPO for fair comparison.

A2: As mentioned in Q1&A1, the key distinction is that the PPO baseline, as a traditional RLHF implementation following [4], is not designed to handle multi-step data and applies rewards only to the final step. In contrast, StepTool supports step-grained reward integration across the entire trajectory. Both approaches used the same training data, reward annotations, and adaptive KL penalties [5] with an initial KL coefficient of 0.3. Additional experimental details are provided in the configuration file in the code repository for transparency and reproducibility.

We sincerely appreciate your feedback and will clarify this in the experimental setup section of the revised paper.

[1] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.

[2] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.

[3] Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121, 2024.

[4] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[5] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.

Thank you for taking the time to read and review our paper. We respond to your comments and questions below.

Q1: What are the differences between PPO and StepTool in the experiment? Do they use different reward models or apply different KL penalties?

A1: The core differences lie in how the training data is utilized. The "PPO" baseline, as a traditional RLHF implementation based on [1], is not designed to handle multi-step data and, as a result, applies rewards only at the final step of a trajectory. In contrast, our framework supports step-grained reward integration across the multi-step trajectory, considering the interdependencies of multi-step actions in a trajectory.

For a fair comparison, both PPO and StepTool utilized the same training data and reward annotations. Additionally, both applied identical KL penalties using the adaptive KL controller [2], with an initial KL coefficient of 0.3. Detailed experimental configurations are provided in the code repository to ensure transparency and reproducibility.

We sincerely appreciate your valuable feedback. We will clarify this distinction further in the experimental setup section of the revised paper.

Q2: Could the authors provide more details on the reward shaping used in tool learning? Specifically: Do we know the values of r_t^{SC} and the IsSolved() function? Are these values predefined, or does the reward function need to learn them? What are the main challenges of reward shaping in the context of tool learning?

A2: Reward shaping in our framework includes three main components/functions, SuccCalling ($r_t^{SC}$), Contribution, and IsSolved. Our work, as a general framework, supports various methods to assess these values, such as rule-based judgment, human annotations, and automated annotations using GPT-4. Specifically, in our experiments:

• The values of SuccCalling ($r_t^{SC}$) were determined through a combination of rule-based and GPT-4-based annotations. Rules were employed to evaluate the format of the tool invocation (e.g., correctness of tool names and parameters). For invocations meeting the format requirements, GPT-4 was used to assess the actual success based on the tool response (e.g., a response like “Since the search parameter is empty, please provide the name of an author…” indicates a failed invocation due to incorrect arguments.).
• The values of Contribution and IsSolved were obtained directly through automated annotations by GPT-4, which relies on its semantic analysis capabilities.

From our perspective, the main challenge in reward shaping for tool learning is the lack of "ground-truth" reward feedback from the environment. Tool learning operates in an environment that only offers tool invocation responses, without explicit reward signals. To address this, we utilize prior knowledge to manually design process rewards according to the characteristics of tool learning scenarios.

We sincerely appreciate your feedback and will incorporate these explanations into the "Step-Grained Reward Acquisition" section of the revised paper.

References:

[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[2] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.

We appreciate the time and effort you have devoted to reviewing our submission. We kindly hope that our previous responses have addressed your concerns regarding the identified limitations and the contributions of our work. If there are any remaining questions or areas where further clarification is needed, we would be more than happy to provide additional explanations.

Your feedback has been invaluable to improving our work, and we look forward to any further input you may have. Thank you again for your thoughtful review and consideration.

I appreciate the authors' detailed responses and clarifications. However, my concerns still remain since the proposed method requires external evaluations in reward shaping like rule-based judgement and GPT-4 evaluations, which can be a major limitation of the proposed method. In contrast, standard PPO pipeline in RLHF / LLM training constructs an estimated reward model, which can be a more general paradigm in LLM training. Hence, I will maintain my evaluations.