LETI: Learning to Generate from Textual Interactions

We thank the reviewer for their responses, and we are glad that the reviewer found our approach highly relevant and sample efficient.

…Reinforcement Learning based methods are probably state-of-the-art for integrating feedback (whether human-generated or automatic). …it lacks a comparative analysis with RL-based state-of-the-art approaches like RLHF… It would be beneficial to conduct a comparison between the LETI FCFT approach and the RL-based fine-tuning techniques…

The baseline we consider in this paper (LETI w/o textual feedback) is a strong online reinforcement learning (RL) algorithm Quark [1], which was shown to perform better than traditional MDP-based RL (e.g., PPO) that requires value function in their paper. LETI directly compares with this stronger baseline Quark and demonstrates the advantage of leveraging textual feedback on both evaluation metrics and sample efficiency (Section 3.3).

[1] Quark: Controllable Text Generation with Reinforced Unlearning

Limited application to broader NLP tasks: While LETI shows promise in code generation tasks … its effectiveness in a broader range of NLP tasks remains uncertain. The approach's reliance on feedback akin to that found in programming environments may not translate well to tasks where feedback is less structured or where the notion of "correctness" is more subjective.

As other reviewers recognized that “such bootstrapping methods are important for further improving LLMs given their data-hungry nature,” and “its successful application to both programming language and natural language tasks suggests that this paradigm can be extended to other domains, making it an impactful contribution to the field of language model fine-tuning,” We argue that deriving an algorithm that can improve code generation and NLP tasks that can be formulated into code generation is already valuable.

As discussed in the introduction, we chose code generation as the main testbed because it can reliably provide automated feedback for any given solution. However, this does not constrain LETI from being applicable to broader NLP tasks with less structured feedback. In fact, we did not do any post-processing with the textual feedback (raw error message and stack traces) that comes from the Python Interpreter, and the LM is able to improve upon this textual feedback. Furthermore, in Section 3.4, we show that LETI can work on NLP tasks like EAE by receiving free-form natural language feedback (Figure 4, stack trace is NOT provided) as opposed to structured stack traces in code generation tasks. The EAE experiment in Section 3.4 shows that LETI is not constrained by the format of stack traces, and can improve based on textual feedback like free-form natural language.

Could you illustrate what the prompts look like for each inference round in the Figure 1? Specifically, does the feedback from iteration n−1 get incorporated into the prompt for the subsequent iteration n?

The prompt for each inference round is illustrated as is in Figure 1. That is, the feedback from iteration n-1 will not get incorporated into the prompt for interaction n. Instead, as described in Algorithm 1, the feedback, prompt, and solution from iteration n-1 will be used to fine-tune the LM, and then the fine-tuned LM will generate solutions for iteration n.

Algorithm 1, states that the 'bad' completions are excluded. However, would it be more accurate to state that these 'bad' solutions are only disregarded until they meet the criteria of 'good' on a future iteration? … The observation that exclusively fine-tuning on 'bad' solutions yields subpar outcomes is not unexpected, considering there's no mechanism within the loss function to penalize such solutions.

Thanks for pointing out this potential confusion! The “bad” completions that did not pass test cases are never thrown away. Instead, all LM’s completions are judged by a solution evaluator (i.e., a test suite) to determine whether they are “good” or “bad” (i.e., binary reward token), then this binary reward token, along with the textual feedback (e.g., error messages and stack traces if the completion is bad) are used to iteratively fine-tune the model (Section 2.2).

We believe the confusion comes from “conditioned on <|good|> for fine-tuned $p_{\theta}$” in the 5th line of Algorithm 1. The reason why we conditioned on a “good” reward token for a LETI-tuned model is that, in previous LETI iterations, we trained the model on both good and bad solutions conditioned on corresponding reward tokens, and our goal is to improve the model to produce more good solutions in the next iteration; therefore, we always sample solutions by conditioning on a good reward token. In the first LETI iteration, the pre-trained model is not trained with LETI; in that case, we just sample the solution from the LM without conditioning on anything. We will make this clearer in the revision.

We thank the reviewer for their detailed and thoughtful responses, and we are glad that the reviewer found our paper interesting, well-written, sound, and easy to follow.

Missing references. There have been many more works about "using feedback to improve code generation"..

We thank the reviewers for pointing out relevant work! We will include them in the next revision.

As mentioned in Section 2.1, LETI assumes a code pre-trained LM which can give a decent, initial performance on code generation. I wonder if this is also a reason for the smaller LM benefiting less from LETI (they may be too weak initially)? It can be helpful if the authors could provide the initial good vs. bad instances distribution in each LM's training set.

We think this hypothesis is reasonable; the capability of the base model performance could have an impact on how well each model benefits from LETI. We provide a breakdown analysis of the top-5 most frequent types of LM-generated code (on the training set). We find that the larger model produces more functionally correct instances, which support the hypothesis.

Experiment in Table 5: Why PAL for GSM8K but CoT for BBH? Could you also show the \delta_{PAL-direct} amount on GSM8K, or is there a reason for not including this result?

The CodeGen model we use has a relatively smaller scale (up to 2B) with limited text-based reasoning capability. Therefore, we follow PaL to reformulate GSM8K as code generation for the model to solve. We did evaluate GSM8K directly without code generation (direct), and there is little distinction before and after LETI fine-tuning as the performance is generally too low. Therefore, we did not include it due to space constraints. We will clarify this in our next revision.

The main experiments say that relatively larger LMs can benefit more from LETI, but in Table 5 LETI shows to hurt the GSM8K performance for 2B (40.03 -> 38.97) while aid for 350M (13.04 -> 16.68). Is this saying that LETI impairs an LM's CoT reasoning performance?

The degradation is not related to CoT, since we use PaL [1] instead of CoT to evaluate GSM8K (i.e., generate code to solve GSM8K question) due to the abovementioned reasons. The degradation of the 2B model (w/o post-processing) compared to the base model is minor (40->39) compared to LETI w/o regularization (40->32) on GSM8K.

We believe that the slight performance degradation we observe on GSM8K is potentially related to the slight degradation we observed on the HumanEval of 2B model in Table 2. We hypothesize that training a larger capacity (i.e., number of parameters) model with LETI allows the models to capture more generic patterns of common success and failure solutions reflected in the consistently improved pass@10 and pass@100 performance of HumanEval (generic programming capabilities, similar to recall), while trade-off a bit of pass@1 (e.g., precision) that might cause slight degradation in GSM8K.

[1] PAL: Program-aided Language Models

The event argument extraction experiment: Can the authors provide results for the supervised, fine-tuned baseline? As LETI in this application assumes ground-truth annotations, the supervised baseline is necessary for justifying its advantage.

We are using ground-truth annotation to simulate automated feedback for EAE. The information the model obtained is less than directly finetuning on those ground-truth annotations since it might take multiple rounds of feedback to guess out the ground-truth annotations.

Therefore, we think LETI is probably a less effective method to improve EAE compared to directly fine-tuning a ground-truth solution (since the construction of the EAE solution evaluator requires ground-truth annotations). Instead of trying to justify the advantages of LETI on EAE, the purpose of Section 3.5 is more to demonstrate the feasibility of LETI to learn from the feedback of different sources (e.g., heuristically constructed natural language feedback) rather than demonstrating the superiority of LETI in the domain of EAE.

We thank the reviewer for their detailed responses, and we are glad that the reviewer found our paper useful and easy to follow.

Evaluation on larger models is needed to provide insights into its scalability and effectiveness… Evaluating LETI's effectiveness in other domains and tasks would further validate its generalizability… How would LETI perform on larger LM (e.g., 6B or 16B)? Can you provide any insights or expectations on the scalability and effectiveness of LETI when applied to these larger models?

We agree that testing LETI on the model of a larger scale (e.g., 6B, 16B) would yield a more comprehensive understanding of the method, and we’d like to do it. However, due to resource limitations, we are not able to run such large-scale experiments. Yet, given the existing possessive results on both in-domain improvements as well as a generalization to HumanEval, we think scaling LETI is likely to bring more benefits than harm. When resource permits, we will include more experiments on other domains.

Comparing LETI with other RL-based approaches that leverage rewards or value functions would help establish the advantages of using textual feedback…

The baseline we consider in this paper (LETI w/o textual feedback) is considered an online reinforcement learning algorithm Quark [1] that performs conditional training on a quantized reward token, which was shown to perform better than traditional MDP-based RL (e.g., PPO) that requires value function in their paper. LETI with textual feedback directly compares with this stronger RL baseline Quark and outperforms it on both evaluation metrics and sample efficiency (Section 3.3).

[1] Quark: Controllable Text Generation with Reinforced Unlearning

Investigating the impact of different solution evaluator designs on LETI's performance would be informative, as biases may be introduced when optimizing towards certain metrics… Since the performance of LETI relies on the solution evaluator's implementation, could you elaborate on the potential biases that may arise from different evaluator designs? How might these biases impact the performance of LETI in optimizing towards certain metrics? … Would it be possible to provide more examples of how LETI can be applied to other domains and tasks?

We agree that different implementations of a solution evaluator could define different optimization goals. We find the problem of overfitting to a solution evaluator is similar to the problem of reward hacking in reinforcement learning [2]: the policy might overfit to the provided reward function (i.e., a solution evaluator), which might cause the policy to produce a worse outcome on the true reward function we care. In the case of code generation, the test case pass rate we use for LETI training is usually considered the true reward function, and the “reward hacking” phenomenon of LETI is relatively less concerning since it is optimizing for a desired goal (i.e., produce functionally correct code).

On domains where the ground-truth reward function (i.e., solution verifier) is challenging to obtain (e.g., need to trade-off between precision and recall for EAE in Section 3.5), different solution verifier designs might impact the optimization goal differently. For example, if we want to improve recall for EAE (as opposed to precision) in Section 3.5, we can instead not penalize outputs that hallucinate arguments that do not exist in Figure 4 and count generations that contain all ground-truth arguments as success. Note that LETI is probably a less effective method to improve EAE compared to directly fine-tuning on ground-truth solution (since the construction of the EAE solution evaluator requires ground-truth solution); the purpose of Section 3.5 is to demonstrate the feasibility of LETI to learn from the feedback of different sources (e.g., heuristically constructed) rather than demonstrating the superiority of LETI in the domain of EAE.

A heuristic solution evaluator is less feasible to design for a lot of real-world open-ended NLP tasks (e.g., chit-chat, summarization, information-seeking dialog). Yet, LETI’s framework is still applicable under such settings – LETI can be used to perform Reinforcement Learning from Human Feedback (RLHF, similar to [3]) with the additional benefit of leveraging language feedback: we can employ humans as solution evaluators that provides preference (thumb up or down, i.e., binary reward), and their language feedback as the textual feedback used in LETI. This is similar to our code generation experiment: LETI’s generalizable framework is applied to optimize the true objective/reward function (human preference).

[2] Defining and Characterizing Reward Hacking

[3] Pretraining Language Models with Human Preferences

We thank the reviewer for their detailed responses, and we are glad that the reviewer found our approach important and easy to follow.

W1.1: Using TheStack as part of the "pretraining dataset". As the authors noted in footnote 4, CodeGen-mono is trained on BigPython… adding such ablation (i.e., simply further finetune on TheStack w/o any of the proposed methods) in the main results is important.‘

We thank the reviewer for the suggestion. In fact, we include the ablation of continued pre-training in Appendix Figure A.10. We find that simply performing continued pre-training for the same amount of training interactions results in no performance improvements on MBPP. We will make this clearer in the next revision.

W1.2: Comparing results w/o post-processing. Note that the post-processing is mostly fixing formatting errors …I think comparing the results w/o post-processing will mix the formatting and semantic errors, making it harder to measure the actual improvements from the proposed method. I would suggest showing all the major results w/ post-processing (e.g., Figure 2).

Thanks for the suggestion! We will include updated figures and results of w/ post-processing in the next revision. We originally did not include it due to space constraints (e.g., the post-processing curve does not fit nicely with the curve w/o post-processing in Figure 2).

On the other hand, despite w/o post-processing potentially mixing the formatting and semantic errors, the LETI still demonstrates consistent advantages over the baseline without textual feedback on both in-domain (i.e., MBPP) and most other metrics (e.g., HumanEval).

W2: One other concern about this work is the cost. If I understand it correctly, for each iteration, LETI needs to perform sampling and finetuning. And from Figure 2, it seems that the majority of the improvements happen after the first 2 iterations. Q2. Despite the fact that LETI doesn't need any gold program, can you comment on the computation cost between LETI and simple finetuning (w/ gold program)?

This is of similar cost compared to online RL algorithms (e.g., PPO, Policy gradient) that need both sampling & finetuning phases, which are widely applied in RL for improving code generation [1]. In fact, our baseline (LETI w/o textual feedback) is considered a strong online reinforcement learning (RL) algorithm [2] that is trained by conditioned on a reward token.

Similar to how online RL is computationally expensive compared to behavior cloning (e.g., fine-tuning with the gold program), LETI requires at least N times (i.e., the number of generated programs per problem) more computation on fine-tuning per iteration compared to directly fine-tuning on the gold problem, with the additional overhead of sampling N programs.

We expect the cost of LETI could be reduced by designing better sampling algorithms that produce more diverse examples with the same sampling budget (e.g., better exploration); therefore, the model could maintain similar performance improvements with a smaller sampling budget, therefore reducing cost. LETI could potentially be modified into an offline RL setting similar to [3], where we generate a comprehensive set of trajectories that contains various error types and only fine-tune it once to reduce cost.

[1] CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

[2] Quark: Controllable Text Generation with Reinforced Unlearning

[3] Pretraining Language Models with Human Preferences

Q1. The models might have already seen such textual feedback (i.e., from Python interpreter) during pretraining… have you tried to apply the iterative method during prompting? Maybe it can be another baseline to be compared with?

Thanks for the suggestion! Iterative prompting approaches are orthogonal to our approaches since they are applied at inference time to iteratively improve a model’s answer to a particular problem. However, the fundamental programming capability of the model is not improved (e.g., the next time you ask the model the same question without the context of the error message, the same model would still fail). Instead, LETI aims to improve the model’s fundamental programming capabilities; That is, the model is less likely to make mistakes it has seen before, as demonstrated in Table 1. Furthermore, our base model CodeGen is a pre-trained model without going through any alignment (e.g., SFT, RLHF) with limited scale (350M, 2B) and context window (2048), so its ability to engage in a dialog and perform advanced prompting can be limited. Therefore, we did not compare with these methods as it is unclear whether we can perform an apple-to-apple comparison on CodeGen. We are happy to hear the reviewers' thoughts on how to perform fair comparisons and include experiments to compare them accordingly.

We thank the reviewer for their detailed responses, and we are glad that the reviewer found our paper well-motivated and easy to follow.

…a. In Table 2, The improvement over HumanEval is mixed for pass@1... could be attributed to error ... averaging over different runs/seeds would be preferable….

As discussed in Appendix C.1, we follow Codex [1] to sample 256 samples to estimate pass@k using an unbiased estimator to reduce the mentioned randomness. Based on this strategy and observations, we believe the performance trend is consistent across multiple runs, and we will highlight this in the revision.

[1] Evaluating Large Language Models Trained on Code

b. In Table 5, the pretrained 2B model performs better than LETI 2B without postprocessing, and regression is observed for BigBench-Hard ...

The degradation of the 2B model (w/o post-processing) compared to the base model is minor (40->39) compared to LETI w/o regularization (40->32) on GSM8K and even negligible compared to BBH (less than 0.3%). The general trend established in Table 5 shows that the generalization performance stays the same.

LETI hasn’t been tested with larger available CodeGen models (6B, 16B) … difficult to determine whether performance results hold consistently…. Table 2: Pretrained 2B seems to be better than LETI on pass@1

We agree that testing LETI on the model of a larger scale (e.g., 6B, 16B) would yield a more comprehensive understanding of the method, and we’d love to do it. However, due to resource limitations, we are not able to run such large-scale experiments. Based on the evidence below, we conjecture that LETI will equally benefit larger models.

As shown in Table 2, regardless of whether post-processing is used during training, we can observe a consistent performance gain for both the 350M model and the 2B model to generalize on Pass@10 and Pass@100 of HumanEval, which we consider to be more reflective of a model’s inherent coding capabilities (e.g., similar to Recall). Such performance gain is even larger when the model is trained with a post-processing heuristic. All these happen without training on any ground-truth solution.

While HumanEval (e.g., Precision) pass@1 did take a minor hit on the 2B model, we hypothesize that training a larger capacity (i.e., number of parameters) model with LETI allows the models to capture more generic patterns of common success and failure programs which sacrificing pass@1 (e.g., precision) for improvement on generic programming capabilities (e.g., recall). Note that such pass@1 (e.g., precision) can be easily improved with established instruction-tuning methods (i.e., alignment) given a base model with good inherent capability, which LETI improves by training the model with both incorrect and correct code solutions with textual feedback.

Results haven’t been compared against strong baselines. … Self-Refine (cited in the paper but not compared), LEVER, and Self-Debugging are relevant methods which use self-refinement and execution-based training respectively, and it is difficult to gauge how FCFT compares…

These approaches are orthogonal to our approaches since all of these three approaches are applied at inference time to improve a model’s answer (e.g., self-refine and self-debug apply iterative prompting, LEVER train verifier to re-rank output programs). However, the fundamental programming capability of the model is not improved (e.g., the next time you ask the model the same question without the context of the error message, the same model would still fail). Instead, LETI aims to improve the model’s fundamental capabilities; That is, the model is less likely to make mistakes it has seen before, as demonstrated in Table 1. Therefore, we did not compare with these methods as it is unclear whether we can perform an apple-to-apple comparison. We are happy to hear the reviewers' thoughts on how to perform fair comparisons and include experiments to compare them accordingly.

...could be tried on Spider/other code generation tasks

We had to prioritize the evaluation tasks due to the page limit, and we believe our current tasks properly evaluate the contributions of LETI. According to CodeGen’s original paper [1], its multi-lingual version does not include SQL capability that is required for tasks like Spider. That said, we can include more evaluations based on the CodeGen-Python we used in the next revision when the resource permits.

[1] CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

In table 2 and table 5, are the pretrained model performances on HumanEval measured with or without postprocessing?...

HumanEval pre-trained performance is measured with postprocessing. We will make this clear in the revision.

…why “AssertionErrors” and “Other Errors” go slightly up with postprocessing …

The generated program is less likely to have SyntaxError after applying post-processing, therefore could cause other error types to increase.

We thank the reviewer for their detailed responses, and we are glad that the reviewer found our paper useful and easy to follow.

Execution-oriented solutions further are heavily dependent on run time environment and specific to it… error messages or stack traces might sometimes be valid in generated code for improper prompts so just their presence is not sufficient to call the generated code as a fail.This is especially true for snippets of code being generated as part of the interactive development scenario which has been most successful for code gen so far.

We thank the reviewer for pointing out the complication of execution-oriented code feedback, especially in the interactive development scenario.

LETI’s goal is to show that it is feasible to allow LM to directly benefit from textual feedback in addition to binary reward (e.g., pass of test cases). The current experiment setting (e.g., basic Python programming that is less likely to produce runtime-specific stack traces) we chose is partially limited by the data we have access to: Collecting reliable natural language (i.e., textual) feedback for other domain can be challenging, and controlled Python programming environment allows us to easily obtain automated textual feedback (i.e., stack traces). We are happy to add experiments if the reviewer has any suggestions on how to extend this (e.g., to interactive development scenario)!

What's the difference between FCFT and RLAF (Reinforcement learning with automated feedback)

We’d appreciate it if the reviewer could provide a specific paper on RLAF. To the best of our knowledge, as discussed in Section 4, the most closely related line of work that leverage reinforcement learning to improve code generation using code execution feedback (e.g. [1]), which mostly uses binary/scalar reward that requires human-designed heuristics to convert different execution error type (e.g., SyntaxError, RuntimeError) into one scalar reward value used to optimize the policy model through reinforcement learning.

Instead of directly relying on a manually defined reward function that converts traceback errors into scalar reward values, LETI only requires a binary reward (pass test or not) and a textual string of the error message and traceback to perform optimization. This allows us to utilize fine-grained feedback information from stack trace without a manually designed reward function, which can potentially scale to different domains that manually designing a reward function for each is not feasible, while informative textual feedback is relatively easy to obtain.

[1] CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

I would love to see more programming languages handled in this work; it feels very narrowly defined using Python problems and runs the risk of the specific interpreter's error generation capabilities overfitting the solution… Did you try the solution on other languages than Python? (CodeGen model supports multiple)? Did you try on different versions of python interpreter to see if differing error outputs have implications?

We agree that adding more programming languages would certainly be helpful in providing a more comprehensive evaluation of LETI. Another recent line of work [2] shows that most of today’s LLM can benefit from execution textual feedback (i.e., stack trace and error messages) across different programming languages (Python, C, C++, Go, Java), suggesting that with a good base LLM (i.e., a multi-lingual CodeGen base model), LETI can scale to different languages. We plan to add evaluation across different languages and interpreter versions when resources permit.

[2] Understanding the Effectiveness of Large Language Models in Code Translation