MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

We thank the reviewer for their detailed responses, and we are glad that the reviewer find our paper important and easy to follow.

… no comparison to simple baselines, such as self-critic ...

We evaluate self-critic in section 3.4 when LLMs provide feedback for themself. We find that even GPT-4 suffers from performance degradation, suggesting the limited performance gain from self-critic.

… can the authors provide any benefit (such as efficiency) comparing using a large LLM as a feedback provider, compared to using a LLM to generate the results directly…?

We believe such a comparison is irrelevant to our research questions. MINT never intends to use a large model to improve the performance of the smaller model, but to evaluate the latter's capabilities.

One of MINT’s goals is to quantify a model’s ability to leverage language feedback, which, in the real-world setting, is provided by users in their multi-turn interaction with LLM. The decision to employ GPT-4 (a larger/stronger LLM) as a feedback provider was driven by the need to establish a reproducible way to simulate users' language feedback for the evaluation of LLMs.

It seems that the only tool used is the python execution tool…

Please refer to the general response.

… dataset … biased towards math problems that specifically require code execution and can benefit from explicit human feedback … not the more natural and systematic tasks … only 43 examples for HotpotQA … may not be statistically large enough to represent the performance…

We argue that MINT is not biased toward math (~25% of instances are math). It covers a wide range of tasks, including math, code generation, decision-making, and multihop QA, that reflect the representative and commonly occurring tasks that can benefit from tool & feedback (our goal of evaluation).

Regarding the size of our dataset, we had to trade-off between the number of examples and the cost of evaluation. Our original dataset contains 29k entries; while we could have included all these data points for evaluation with sufficient funding, it would have been prohibitively expensive (for GPT-4 to provide feedback) for the academic community to use and hindered the broader research and dissemination.

In preliminary experiments, we found that the models' performance trend across the selected data points is consistent and representative of that on a larger selection. Our decision on downsampling largely reduces the cost of MINT without negatively impacting the conclusions one can draw with it. We believe that a sample of 500 entries that requires interaction is an appropriate compromise, as seen in other datasets in the field (e.g., TheoremQA, AlfWorld).

We will also release the script we used to down-sample these datasets so users may choose to up-sample the HotpotQA subset when deemed necessary.

results … are limited to the setup that those worse performance would largely be due to "less viable of incorporating feedback" rather than necessarily indicating that they are indeed worse on multi-turn interactions with agents in general...

We would like to clarify that these findings hold regardless of whether the GPT-4 simulated feedback is included, as discussed at the end of the introduction. Particularly, in Section 3.2, we find supporting evidence that even when no GPT-4 feedback is provided at all, the claims the reviewer mentioned still hold.

If the reviewer is referring to “feedback” from the Python interpreter (i.e., observation), we believe that the capabilities of improving from observations in multi-turn interaction and incorporating natural language feedback are highly correlated. We are happy to hear the reviewers' thoughts on how to disentangle them and include experiments to verify them accordingly.

Where did you compare to the "Lazy User-LLM Interaction"? Or is this only for k=1?

As explained in Section 2.1, “Lazy User-LLM Interaction” defines the behavior of users in interaction (i.e., provides at most one response when the solution is wrong). This alone corresponds to the setting of k=2 (2 turns of LLM response): LLM proposes answer (at most twice) for each response with no tools used, which deviates from our goal of evaluating tool and feedback incorporation capabilities. Therefore, “Lazy User-LLM Interaction” is just a description of the user behavior when no GPT-4 simulated feedback is involved. We will make this clearer in the next revision.

The proposed method is very similar to the Tree-of-thought paper (Yao et al., 2023) where the authors used LMs as a feedback/reward provider

We emphasize that our focus differs significantly from that of Tree-of-thought (ToT). Despite some similarity in methodology, MINT focuses on evaluating any LLM under the setting of multi-turn interactions with tools and feedback, which diverge substantially from the objectives of the ToT that shows prompting LLM in ToT improves problem-solving performance.

<execute>
from sympy import symbols, Eq, solve
x, b = symbols(‘x b’)
equation = Eq(x + (x+7) + (1/2) * b, 43)
solve(equation, x)
</execute>

We thank the reviewer for their detailed responses, and we are glad that the reviewer found our paper novel, important, and easy to follow.

... Figure 1 … assume a "collaborative work" interaction between the LLM and the user, as opposed to the more common case where a user seeks assistance from the LLM ... not practical to assume such detailed feedback from the user because the user does not know how to solve the problem... While MINT's feedback interaction is still valuable …its kind of feedback may not reflect the realistic case, and this should be clarified in the paper… The human evaluation of GPT-4 generated feedback does not look at the discrepancy …

We thank the reviewer for suggesting relevant work, which we will include in our next revision. For ease of evaluation, in MINT, we disentangle a user’s “assistance-seeking” and “collaborative” feedback-providing behavior.

The “assistance-seeking” (not depicted in figure 1) corresponds to (1) the initial user-provided instruction to LLM and (2) templated user feedback LLM received (e.g., Your answer is wrong) when proposing a solution that users disagree with (Lazy User-LLM Interaction section on Page 3, Table 2). As a side note, we think this is similar to real-world user interaction with LLM: even when the user does not know how to solve the problem, it is still possible for the user to provide feedback. For example, I might not know how to write code to solve a particular problem from scratch, so I ask LLM to help, but I might be able to understand and judge the LLM-produced code (and its intermediate code execution result) to see if it fits my expectations and can response with minimal feedback (e.g., "this is not what i want" - similar to the templated feedback). MINT focuses on evaluating LLM on these types of “hard to solve directly, but easy to verify” problems (Table 1).

For the latter “feedback-providing” behavior (a more collaborative setting), we use GPT-4 to simulate feedback to mimic the “collaborative” feedback aspect of the user, which is the dotted box shown in Figure 1 with results in Table 3. As described in Section 2.1, we separately evaluated these two as “LLM-Tool interaction with Lazy User-LLM Interaction” and “Informative User-LLM interaction with Language Feedback.” We will use the suggested wording to explicitly distinguish and clarifies these type of interactions in the revision.

MINT leverages GPT-4 (a stronger / larger model) to simulate human language feedback, which is reproducible and costs 14x less (footnote 4) compared to hiring real humans. We do agree that GPT-4 simulated feedback can be different compared to real-world human feedback, which is evidenced in our human evaluation in Table 5, where we found most human feedback is comparatively less helpful than GPT-4 feedback. Still, such GPT-4 simulated feedback can serve as an initial step to help us gauge LLM’s ability to incorporate feedback (i.e., can they incorporate feedback that’s already very detailed and helpful?), which can provide future research that studies the difference between real-world human feedback and GPT-4 simulated feedback a good starting point. We will include clarifications in our next revision to make this more clear.

As indicated in footnote 4, evaluating one LLM using MINT costs around 100 USD, due to the use of GPT-4... can be an issue for iteratively improving (and then evaluating) an LLM... thoughts on reducing the benchmark cost?

MINT, similar to the widely used MT-Bench [1], chose to use GPT-4 due to its effectiveness and reproducibility. We are aware that such a cost might become a roadblock for iterative improvement, and therefore, we made significant efforts to keep MINT’s evaluation dataset compact and representative to reduce the GPT-4 feedback cost down to ~100 USD (98% less compared to the original size of the datasets). Furthermore, Human evaluation costs on a similar quantity of examples are likely to cost 14x more (footnote 4), which already makes GPT-4 feedback simulation more cost-effective.

In our design, we deliberately made the feedback provider replaceable (Section 3.4), which allows potential future cost-saving by replacing GPT-4 with a cheaper model that specializes in providing feedback (e.g., [2]).

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[2] Shepherd: A Critic for Language Model Generation

The current tool use has been limited to Python function calls. It offers better reproducibility but also loses the benchmark scope.

Please refer to the general response.

Can MINT be integrated with AgentBench (Liu et al., 2023) or InterCode (Yang et al., 2023), so that a larger task scope could be facilitated with both tool and user interaction?

Yes! We do plan to continuously integrate additional tasks (e.g., AgentBench, InterCode) into MINT to further expand its scope in the future (e.g., a MINT 2.0 version) as MINT’s evaluation framework is general and task-agnostic.

We thank the reviewer for their detailed and thoughtful responses, and we are glad that the reviewer found our paper important, sound, and easy to follow.

…it wasn’t clear to me which ones are new findings vs. confirmation of existing findings……Situating the findings in the existing literature will give this work more validity and help readers understand the contributions of the paper… As some of the prior work covers a range of diverse tasks/measures multi-turn interaction capabilities of LLMs/simulates human feedback with LLMs, the main difference of this work is (presumably) accounting for the use of tools and user feedback on top of all of these elements… why it’s important to look at these elements altogether (as there are prior work looking at the tool use and NL feedback) and what we expect to see?… I wouldn’t say that the current setup is strictly “more aligned with real-world applications” than prior work as there is a clear trade-off due to simulating human-written feedback with LLMs and potential drawbacks with the simulation (e.g., Koo et al. (2023), Rajani et al. (2023)).

We acknowledge the contributions and findings from existing literature on tool-augmented language models (e.g., ToolLLM) and human-AI interactions (e.g., HALIE). Different from existing work that independently contributed to the understanding of a particular element (e.g., tools, multi-turn, human feedback), to the best of our knowledge, MINT is the first to put everything together.

MINT, by unifying the evaluation dataset for both the tool and language feedback aspect, allows us to examine the interplay between these two elements, i.e., the trade-off between tool performance and the model’s ability to leverage language feedback, which is common in real-world applications (e.g., chatGPT with plugins). We actually can observe such a trade-off in MINT’s evaluation of the Lemur model [1]: we find that their incorporation of reasoning (e.g., OpenOrca) task-specific single-turn data in SIFT improved the model’s reasoning performance (tool-only) without language feedback. However, training on such single-turn data sacrifices their ability to leverage language feedback (e.g., only +0.9 improvement when language feedback is given in Table 3). We will include more discussion on relevant literature and better situate our findings to make it clearer.

[1] Lemur: Harmonizing Natural Language and Code for Language Agents

… Although the surface-level similarity and helpfulness are similar, my guess is that GPT-4 generated feedback lacks the variability that we’d normally observe in human-written feedback, which can easily affect these models’ performance (e.g., Lee et al., 2023)...

We acknowledge that although it has been more and more widely accepted to use GPT-4 to provide feedback, this approach has the limitations that the reviewer points out (i.e., they might not capture the variability that might present in real human feedback). Possible solutions include explicitly prompting GPT-4 to play various personas, which would complicate the evaluation and out of the scope of MINT but is nonetheless worth exploring. We conjecture that even though our setting (with less variety) does not perfectly reflect the models' performance with real human feedback, it highly correlates with it and provides a good starting point for future research that studies the difference between real-world human feedback and GPT-4 simulated feedback. We will include more discussion and clarifications about this in our next revision.

In finding #3, could the size of a model be the main confounder? Is the finding only applicable to multi-turn settings or also to single-turn settings?

This is a great question! As discussed in section 3.2, size does play a role in the multi-turn performance: multi-turn tool-augment interaction performance of LLM with the same training data improves with scale. However, since commercial models did not release their actual model size, it is hard to say whether their good performance in multi-turn interaction comes from model scaling versus better data. We believe that finding #3 mainly applies to multi-turn settings since we did not comprehensively test LLM’s single-turn performance with well-established benchmarks (e.g., MMLU, GSM8K).

We thank the reviewer for their detailed and thoughtful responses, and we are glad that the reviewer found our paper novel, important, and easy to follow.

The authors claim that the SIFT can benefit models' capabilities of tool-augmented task-solving in multi-turn interaction (section 3.2), while the authors also claim that the SIFT can hurt models' ability to leverage feedback (Section 3.3). Why do these two claims seem to contradict each other?

We thank the reviewer for pointing out this confusion! In summary, we find that SIFT trained on multi-turn conversations (e.g., Vicuna, Lemur was trained on ShareGPT) have improved tool-use (Section 3.2) and leveraging feedback capabilities (Section 3.3); While for other models (e.g., LLaMA, CodeLLaMA), SIFT can hurt their capabilities. We will make this part clearer in the revision.

I am also very curious about the trade-off between tool-use capabilities and abilities to leverage human feedback. SIFT on general domain multi-turn interaction data (like ShareGPT, in section 3.2) can bring performance gain in problem-solving. I think this might be because of the multi-turn feature that caters to the final application. I also think if the model can SIFT on some task-specific data (if exists), the performance can also improve. I am very curious about which is more important for SIFT.

We agree that it is an interesting and important trade-off regarding the use of general-domain multi-turn conversation (e.g., ShareGPT) versus task-specific data in improving models with SIFT. In MINT’s evaluation of the Lemur model [1], we find that SIFT on task-specific data (e.g., OpenOrca for reasoning) significantly improved the model’s ability to perform reasoning without language feedback; However, the task-specific dataset OpenOrca only contains single-turn data, which we hypothesize could make the trained model more “stubborn,” i.e., sacrificing their ability to leverage language feedback (e.g., only +0.9 improvement in Table 3 when given language feedback).

If our ultimate goal is to improve the model’s single-turn performance (single question-answer pair), we think training on task-specific data might be sufficient. However, If the ultimate goal is to build a helpful interactive assistant that is not only good at providing good answers in the first turn, but can also collaborate with users in multi-turn settings to correct its previous answers (e.g., an almost perfect piece code that only needs a small tweak to work), balancing the model’s ability of direct problem-solving AND leveraging language feedback in a collaborative setting would be essential. We believe a more rigorous study could be done as a follow-up to further analyze each SIFT component’s effect on the model’s different capabilities.

[1] Lemur: Harmonizing Natural Language and Code for Language Agents