Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

We thank the reviewer for useful feedback. We address their questions below:

1. Prompt-Based Adaptation

One can easily combine prompt-engineering approaches with fine-tuning approaches. E.g., one can fine-tune an LLM and still prompt it with previously seen relevant in-context learning examples (a popular prompt adaptation approach). While prompt-based adaptation is orthogonal to our main focus on fine-tuning, we will add a short discussion on results with a prompt-adaptation approach. We will try to share preliminary results during the author-reviewer discussion period itself.

$$

2. Users' edit data privacy topics

We included a discussion on how to best protect the user’s privacy when handling their user-edit data in the NeurIPS checklist. Please see our response to the broader impact on lines 756-764. If accepted, we will use the extra page to move this discussion to the main paper.

$$

3. Scalability and Human Studies

In this paper, we focus on laying the algorithmic foundations of learning from user-edits. Expanding our study to a larger scale and doing human studies are natural follow-up work. These studies can refer to our paper to decide what algorithms to run. In terms of scalability, both individual algorithms and their ensembling approach, including Algorithm 1, are easy to run with any standard LLM fine-tuning library. Algorithms such as SFT and DPO are routinely used to train LLMs with big datasets.

One advantage of using simulated LLMs in our study is that they provide reproducible and quick studies. This is especially useful for algorithm development, which is our main focus. The use of simulated users has increasingly become common due to improvements in LLMs [1, 2, 3, 4]. Further, doing a real human study by collecting enough training data and having to do repeated human evaluations can be prohibitively expensive.

However, any industry lab that deploys a writing or coding assistant can easily test our algorithms with a real human study. They can do this by collecting user edits and using them to fine-tune LLMs. They can then do A/B testing to compute total user edits on generations of the trained model (assuming they have all the required permissions to store and train on user edits). We feel our research is particularly suited to make a practical impact in these settings.

References:

1. Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems, Sekulić et al. 2024
2. Quantifying the Persona Effect in LLM Simulations, Hu and Collier (ACL 2024)
3. Simulating User Agents for Embodied Conversational-AI, Philipov et al. 2024
4. USimAgent: Large Language Models for Simulating Search Users, Zhang et al., SIGIR 2024

$$

4. Theory and Real World Validity

In section 4.2, we provide a variety of insights on how our theoretical results relate to practice (lines 248 to 255) where we discuss when SFT, DPO or RL are preferable depending on different user edit ``parameters" such as $\gamma_{\min}$ or the preference coverage $C_{\mathrm{PREF}}$ parameter. When deployed, these parameters have interpretable descriptions that characterize how thorough user edits are when generating a response ($\gamma_{\min}$), or the richness of the edits training data ($C_{\mathrm{PREF}}$). If accepted, we will use the extra page to make some of these connections clearer.

$$

5. Contributions in Intro

We will revise the introduction to explicitly list our contributions (theoretical bounds, unified framework, and ensemble strategies).

We thank the reviewer for their helpful feedback. We answer their questions below:

$$

1. Protocol 1 Line Number Mismatch

We want to clarify that Protocol 1 is correctly written and complete. However, there are two typos in the text in the line numbers. On page line 109, the protocol line should be line 4, and on page line 111, the protocol line should be line 7. We appreciate the reviewer for catching this typo, and we will fix these in the revision.

$$

2. Using different LLMs for teacher and user models

We want to clarify that we experimented with using a different combination of LLMs for user and teacher at test time. In Table 2 in the main paper and in Table 6 in the Appendix, we tested with a setting where the user was either a Qwen or Gemma model while the agent LLM remained Llama 8b. Please also see the discussion on lines 334-342. These results add a new dimension of generalization to our studies by showing the robustness to user LLM. In both cases, the results were generally the same as in Table 1. Notably, our late-ensemble approach performed the best overall.

If, however, the reviewer meant having different LLM families (e.g., Qwen vs Llama) for the user and teacher during training time and then using the same families at test time, then we are also happy to add those results in our revision. This will be an easier setting since, unlike the transfer learning results, the model will be training and testing on the same combination of LLMs.

$$

3. Why DPO Underperforms the Base Method (Table 2)

We have noticed some issues with DPO, including the well-known observation that DPO-trained models can be very verbose. This verbosity can be especially hurtful in the summarization task, where the goal is to generate a concise summary. We also noticed that DPO can enter repetition loops, where it repeats the same text, something that we fix with an explicit filter (see discussion around lines 306-308). We found that in this particular case of summarization with a strong user and a different LLM, the DPO was generating either really long summaries or really short ones, both of which are sub-optimal. We believe this is due to the instability in DPO, and testing with a different LLM model results in a higher edit distance penalty for these deficiencies.

We thank the reviewer for their detailed feedback. We address their questions below:

1. Clarification on assumptions and their validity

We first provide some important clarifications regarding our assumptions.

“How is the learning outcome affected if real users sometimes do not fully correct every error (violating the contraction property)”

We do not assume that users fully correct every error. In fact, this would be a very strong assumption. The contraction lemma states that the user will improve a given policy's response with the rate of improvement determined by \gamma_\min. This does not mean the user will fix all issues in one round of edits. Instead, what this implies is that if the user keeps on editing a given response to make it better and better, then they will eventually converge to their optimal policy. This is a much weaker assumption and is similar to our lived experience, where constantly editing one’s work (e.g., for a paper deadline) leads to an eventually acceptable version.

“How is the learning outcome affected if $\gamma_{\min}$ varies across contexts”

This is a great question. Suppose we have a context $x$ dependent value $\gamma_{\min}(x)$ where Equation 9 holds. The simplest way will be to define \gamma_\min = \min_{x \in X} \gamma_{\min}(x) as the minimum over all contexts. However, we can do even better as follows:

Firstly, in Lemma 1, for any context $x$, we will get:

||q \circ \pi_{ref}(. | x) - \pi^\beta_\star(. | x) || \le (1 - \gamma_\min(x)) ||\pi_{ref}(.|x) - \pi^\beta_\star(. | x)||,

Then, in the proof of Theorem 4, we can use Cauchy-Schwarz’s inequality to get:

E[||q \circ \pi_{ref}(. | x) - \pi^\beta_\star(. \mid x) ||] \le E[(1 - \gamma_\min(x)) ||\pi_{ref}(.|x) - \pi^\beta_\star(.|x) ||]

\qquad \qquad\qquad \qquad \qquad \qquad \le \sqrt{E[(1 - \gamma_\min(x))^2]} \sqrt{E[||\pi_{ref}(.|x) - \pi^\beta_\star(.|x) ||^2]}

where we now only require \sqrt{E_{x}[(1 - \gamma_\min(x))^2]} to be small. This is a weaker assumption as \gamma_\min(x) no longer needs to be close to 1 for all contexts. We will update our proof to use this softer notion of \gamma_\min.

More generally, we believe our results are robust to assumptions being violated. The biggest evidence is our experiments, where the assumptions may not necessarily hold as we use complex transformer-based LLMs and perform optimizations that may not necessarily converge to an ERM solution; however, findings from our theory still apply. Finally, assumptions similar to ours are routinely made in RLHF and RL literature.

2. Challenges of Implementing Equation 6

Unfortunately, computing an ``exact’’ pessimistic $\bar{f}$ estimator for $f$ requires solving a challenging optimization problem (see line 157 for a definition of $\bar{f}$). This is because enforcing the constraint set $\gamma(\mathcal{F}, \delta)$ is impractical to achieve in real settings. There are some heuristics to implement pessimism using an ensemble, but these tend to be brittle in practice and/or have their own challenges. Our theoretical analysis nevertheless provides insights into the potential of this approach. We will add this clarification in the revision.