LLF-Bench: A Benchmark for Interactive Learning from Language Feedback

Thank you for your thoughtful review. We address your concerns point by point below:

The paper provides good motivation for why we should expect language feedback to be more efficient than traditional rewards, but fails to provide evidence for this hypothesis.

Large Language Models have been fine-tuned to follow instructions. Feedback, especially future positive (FP) or future negative (FN) can be seen as a form of instruction (e.g., “recommend an action movie” or “do not recommend a horror movie”). In Table 2 and 3, we already showed that by providing reward and different kinds of feedback types, there is a large variation in performance differences between LLMs. We highlight the comparison in the Movie-Rec env below as an example.

We can see this type of difference for all LLM models across all of our environment in Table 2 and 3, showing a sensitivity to the type of feedback it receives. We believe this motivates the need to investigate different feedback types.

Table 2 and Table 3 gives us an ablation of how removing Future Feedback affects performance, but the separate presentation of the tables makes it very difficult to compare, and the text does not discuss the findings as far as I can see.

Thank you for bringing up this great point. We will add a new plot that contrasts the performance differences caused by different feedback types (Reward+Hindsight+Future, vs Reward+Hindsight, vs Reward) in the revision. (See the table below for the comparison).

Thank you for your feedback! Our answers to your questions are as follows:

...aligned with current missions of agent-human interaction...

These missions can be understood in at least two ways: (a) a person teaching an AI agent do a task for them and (b) a person and an AI agent working together to accomplish a task. Both interpretations are being actively pursued by the community, and ours fall under (a): LLF-Bench is meant to assess agents’ ability to learn a task from human feedback.

1. Task Selection for Rich Human Feedback:

Per the above clarification, our benchmark is meant to model situations where a human is trying to automate a task by teaching its AI assistant in natural language to perform this task. So, the scenarios our benchmark models are indeed cooperative in the sense that the agent’s incentive is fully aligned with the human one.

2. Varied Levels of Language Feedback

This is indeed one of the motivation behind this benchmark's design. The levels of feedback in LLF-Bench already work as you propose. In particular, your Level-1 feedback is what we call “hindsight negative (HN)” feedback, and Level-2 is “future positive (FP)”. In experiments (Table 1 an Table 2), we compare how agents learn with and without Future Feedback.

3. Diversity in Language Feedback

Good suggestion, we will provide a wrapper for generating feedback on the fly with an LLM by seeding the response with existing templates. At the same time, we will keep the current method of choosing from hand-picked templates an option, due to its higher reproducibility.

4. Effect of Vision Input on Performance

The vision input can indeed help an agent that is “smart” enough to use it, but our agent is very basic and has a limited 3D understanding. Please note that our aim in constructing this agent was to provide a conceptually simple baseline that can serve as a natural starting point for other researchers to improve upon.

Response: We thank you for your review and we address your responses in the order in which they arise.

1. LLF vs Past work on language feedback:

You’re right that past works have studied using language feedback to instruct or teach an agent. We do not wish to claim we’re the first to study learning from language feedback. Instead our aim is to provide a formal abstraction of these problems, just as the Markov decision process was proposed to abstract sequential decision making. (We will better clarify this in the revision). Having a formal problem setup of learning from language feedback is a timely need, as this area gets more attention. One important novelty of the LLF setup here is the perspective that language feedback can be viewed as a generalization of reward in MDP, separate from the observation that the agent receives. (e.g., in the LLF setup, observations and actions can be non-text while feedback is natural language). This perspective highlights that both reward and language feedback are means (e.g. of a human) to teach an agent, which are not necessarily needed for a learned agent to execute tasks later on. Following this idea, we propose a taxonomy of feedback to an agent (e.g. reward, hindsight, future feedback). This insight connects the Education (for humans) literature to (AI agent) learning literature, and allows us to systematically construct comprehensive benchmark environments in LLF-Bench to study how agents learn from different language feedback, In comparison, previous works tie language feedback to a specific agent embodiment or applications and often implicitly assumes one kind of feedback (e.g. future feedback in [2]) without discussing effects of other feedback types.

3. Reason for randomizing prompts

It is well-known that LLM agents are sensitive to prompt changes; see e.g., this well-cited study of Sclar et al., 2024. For this reason, we randomize the agent’s textual description and feedback to prevent prompt hacking where an agent is particularly good with one particular prompt but not robust to meaning-preserving changes in the prompt.

Sclar et al., 2024: Quantifying Language Model’s Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, at ICLR 2024

4. Designing rewards vs designing feedback

The main challenge with designing rewards over designing language feedback is that humans think in language so giving language feedback is more intuitive to us than giving scalar numbers. Humans are particularly bad at thinking about the effects of maximizing “accumulated” rewards, which is the objective of e.g. MDP used in RL. Previous studies have shown that when defining a task using accumulated rewards, humans at times inadvertently create positive reward loops which lead to the agent running in circles and getting infinite rewards (e.g., throwing trash, putting it back in the bin, and throwing it again, etc.). We argue that this intuition of human language will lead to more aligned language feedback. There is another big advantage of language feedback over rewards in that it provides a rich learning signal as language can communicate many more bits of information than a scalar reward, which can enable designing faster learning agents.

5. Non-text-based problems

Learning from Language feedback is not limited to tasks where observations and/or actions are texts. In fact, language feedback is particularly useful in many multimodal real-world applications such as robotics (like in [2]), driverless cars, and an OS agent. E.g., imagine telling the driverless car, “you made a wrong turn, you should have stopped at the cafe on the previous street”. Our image-based experiments are an example of these applications. Our experimental results show that basic agents lack the ability to effectively use images and more work is needed to solve these tasks.

6. Availability of optimal action

Future positive assumes suggestions of things to do, which may not necessarily mean knowledge of the optimal action. In the llf-metaworld envs, the future positive feedback contains actions that a suboptimal expert policy would take (which cannot solve the task with 100% success). You can see Appendix F.1 for another example. There the future positive feedback part corresponds to “Make recommendations that are Western, like True Grit and The Good, the Bad and the Ugly. Identify movies that were released during 2000s or 80s, like Pearl Harbor and Black Hawk Down.”. which are helpful advice but not necessarily actions to follow exactly. We also highlight that LLF-Bench allows users to determine which feedback to provide the agent. E.g., users can study learning without future positive feedback (as we did in Table 2). Such a feature of controlling what information feedback can provide is missing in existing benchmarks. This is a key strength of LLF-Bench.

7. Task as part of the environment vs agent:

A common problem with the design of LLM agents is prompt hacking where people find prompts that works really well for a given task but fail in general, and release task-specific implementation of algorithms which are non-trivial to extend to new applications. As a result, these benchmark performance does not reflect real-world performance. By having prompt be part of the environment and requiring users to design agents that can solve multiple environments, we prevent the agent’s limit to hack to it. We also randomly select prompts from a prompt set which further reduces prompt hacking. This means that to do well on various tasks in LLF-bench, an LLM agent must be general enough to work with prompts given the environment across a range of tasks.

8. RL vs learning from instruction:

By RL here, we imply that classical definition that agents learning from observation, action, and reward -- without any language feedback or instruction. We will clarify this in the revision.

the definition of LLF is neither mathematically formal

Thanks for the response. However, we strongly disagree with this statement. The LLF setup is formally defined in the paper, in Sec 2.1 (starting at line 110). It's sequential decision problem of [instruction, observation_1, action_1, feedback_1, observation_2, action_2, feedback_2...], which is in contrast to RL of [observation_1, action_1, reward_2,, observation_2, action_2, reward_2...]. We also describe taxonomy of feedback and how this interaction structure can enforced in the gym API in Appendix C.

MDP is a Markovian version of sequential decision problems. Similarly, one can say an LLF problem is Markovian when the observation represents the state and the feedback is generated purely based on the current state.

sample efficiency comparison between LLF and RL agent

We want to highlight that in LLF we're targeting at ultra efficient learning which the traditional RL setup cannot approach. Here we show agents learning to solve the problem within a single episode of 15-20 steps of learning. This is a regime where no RL algorithms can touch. For example, for Meta-World [1] (see their Figure 12, 13, 14), an RL agent needs an order of 1 x 10^8 steps across multiple episodes to learn and generalize across tasks. In particular, the PPO algorithm you mentioned only performs update after at least one episode. So if we were to run it in the current experimental setup of one-episode, no learning would happen and the evaluation score would be just of the initial random policy.

Thank you for your thoughtful feedback. We address the weaknesses and questions you raised below:

Inclusion of dialogue tasks

We agree that dialogue tasks such as negotiation or persuasion are important and challenging areas for RL fine-tuning. However our focus in this work is on learning from language feedback in a "cooperative" setting, where the feedback generator assists the agent in learning. This is a specific subset of learning from text interaction, which includes other dialogue tasks. We believe that understanding this cooperative learning process is crucial before tackling adversarial or competitive dialogue tasks like negotiation. Thank you for the suggestion, we will clarify the setting and the relationship between LLF and dialogue tasks in the paper.

Computational Cost of Textual Feedback:

In all our environments, feedback is procedurally generated using code that takes the environment state as input. This process does not introduce any noticeable computational overhead. In all of the experiments, the computation time is dominated by the agent's inference, with feedback generation time being negligible. We will clarify the computation cost of the textual feedback, thank you.

Comparison with Other Methods

The primary goal of Section 5 is to establish a baseline of task difficulty rather than to compare different LLM agent architectures. That said, we conducted experiments with another agent architecture (detailed in Appendix E) and found the same empirical trends. Regarding LATS, its codebase has specifically engineered prompts for different applications, so its code cannot be directly applied here. One of the objectives of LLF-Bench is to encourage the development of learning agents that are not hand-engineered for specific applications like LATS code, and promote a more general approach to learning from language feedback.

Thank you for your rebuttal. I will maintain my score. Regarding introducing additional baselines such as LATS, the authors are correct in that they require hand-engineered prompts. But to my knowledge, so does Reflexion, which the authors do evaluate on. Is there a reason why Reflexion can be evaluated that the authors could clarify?