We also really appreciated all your questions --- we've been mulling over similar ones for follow-up work. We couldn't carefully empirically evaluate all of them, but we are considering a FAQ in the final paper answering these questions. Regardless, we hope these responses help!

1-shot learning. From our demonstration scaling experiments (Figure 2 and Sec 5.2), getting the model to learn demonstrated feedback might need at least three samples. Qualitatively, we’ve played around with DITTO-ed models on a single demonstration—there’s definitely a behavioral shift, but I think three is where you see generalization beyond the specific demonstrated behavior.

Finetuning with LoRA. See general response (point 3)

Does this look indistinguishable to an LLM from some other real distribution? Personally, we think the generations from DITTO look surprisingly “real.” Beyond our lexical analysis (Table 3), there are a bunch of odd linguistic quirks we see in the outputs (e.g. typos, coordinating conjunctions, sentences starting with “and”, etc.). We didn’t think of the red-teaming application, and we’ll definitely add it to the future work.

Does it help few-shot to explicitly instruct needed to be very close to few-shot examples in style? Or was that just tried for fitting zero-shot? We tried explicitly instructing the few-shot variant too! Apologies if that was unclear—we’ve revised that line in our paper (models)

True few-shot learning. Our setting is indeed a true few-shot setting. We have a withheld validation set that is approximately the same size as our actual test set! We agree that a much larger validation set, or engineering directly on the test set, would result in the effect documented by Perez et al. Additionally, our hyperparameter sweeps were done on a randomly selected author—still, DITTO generally outperforms all baselines across most authors where no tuning was done.

Model Scaling. Unfortunately, there isn’t a Mistral model larger than 7B parameters; and moving all our experiments to a larger model was cost-prohibitive. We couldn’t test this hypothesis while keeping everything else fixed—we’ve mentioned this in the limitations.

Data mixes, splits, and higher data regimes. We don’t know (yet). Unfortunately, there aren’t many datasets where many demonstrations come from a single user, and where we can ablate these properties. DITTO’s design choices are definitely focused on low-resource settings; applying our method to higher-resource settings will require additional algorithmic / engineering work!

Thank you for your comprehensive and careful review! We’re glad that you found DITTO’s performance impressive, and our method clever. We’re also glad you appreciate the user study and our lexical analysis. Finally, we appreciate the insightful application ideas (e.g. improved red-teaming) — we’ve included these suggestions in our future work!

Generalization beyond the demonstration and tasks with more expertise

We agree that this is an important application area! Please see the general response (see point 1). The TL;DR is that our current setup indeed covers generalization beyond niche tasks, but we’re not sure if DITTO can teach new complex reasoning skills—we’ve addressed this in our revised limitations section.

Confounding effects of RLHF

We agree that RLHF can have confounding effects, and we’ve introduced a new experiment to test this. See point 2 in the general response. The TL;DR is that general instruction following capabilities may be required as a “starting point” — jointly learning instruction-following and demonstrated feedback is too difficult a task to learn from a handful of demonstrations.

Comparisons to Constitutional AI

Even though Constitutional AI uses a small set of principles, it's still bottlenecked by the same issues of pairwise preferences. Consider the following algorithm.

1. Take demonstrations from the user and convert the demonstrations into principles.
2. Follow Constitutional AI:
   1. Sample generations from the LLM.
   2. Use the principles to:
      • (a) Label pairwise preferences (with an LLM).
      • (b) Train the model.

If none of those generations at step 2.1 are close enough to the user's desired output, then the method will not succeed.

We effectively tried an “upper bound” of this in Section 5.3, where a human annotated many pairwise preferences with an ideal set of demonstrations in mind. In other words, we used a human in step 2.1 instead of an LLM. This is similar in nature to Constitutional AI, where an LLM would instead annotate pairwise preferences. The big problem we ran into occurred when we sampled pairwise preferences from π_ref. We observed that generated pairs were out-of-distribution relative to the demonstrations—pairwise preferences do not reach a user’s demonstrated behavior. In other words, the samples generated by an LLM (2.1) are so far from the user’s ideal behavior that the samples from the LLM never get close, and the labeled LLM preferences (2.2b) are irrelevant. We think something similar will probably happen for Constitutional AI too. We added a few lines discussing this intuition in the updated version of section 5.3.

Examples + Explain DITTO earlier.

We're glad you liked the examples! We will definitely move some of the Appendix examples to the main text given the extra space — we’re working on a new figure for that. We’ve additionally revised the abstract, adding a few sentences to explain the high-level algorithm in more detail.

DITTO operates by having an LLM generate examples that are presumed to be inferior to expert demonstrations. The method iteratively constructs pairwise preference relationships between these LLM-generated samples and expert demonstrations, potentially including comparisons between different training checkpoints. These constructed preference pairs are then used to train the model using a preference optimization algorithm (e.g. DPO).

We thank 41aT for their thorough and thoughtful review! We're glad that 41aT thinks DITTO "tackles a significant challenge and presents an interesting solution that is well-supported in theory and through empirical evidence"; and that our work has "strong impact on making LLMs more customizable and accessible." Below, we address your questions:

Wider Range of Domains and Modalities

See the general response (point 1). The TL;DR is that while our current setup indeed highlights generalization beyond niche tasks, we would not expect that DITTO to teach new reasoning skills that haven’t been seen by the the pretrained LLM. We also do indeed observe some forgetting on general alignment post-DITTO-ing on specific demonstrations, but we propose a prompt-based routing mitigation that addresses this (see general response, 1.2).

Sensitivity to Demonstrations

This is a great point!

We ran an experiment to see if performance improvement compared to the few-shot baseline was correlated with “demonstration cohesiveness.” We prompted an LLM to score the cohesiveness of demonstrations, modifying prompts from [1] on LLM-based document clusterer. Then, we computed Pearson’s R correlation coefficient between cohesiveness scores (1 - 5 likert scale) and performance increases. We find a moderate positive correlation (R = 0.42) between the cohesiveness of author demonstrations and the downstream performance.

One could automatically cluster a large set of documents into specific sub-styles; and then train DITTO models individually on each cluster. Since LLM-judged cohesiveness correlates with downstream performance, automatically assembling a set of DITTO adapters from a training corpus is a potential avenue for future work. We’ve referenced this analysis in our limitations and have outlined a new section in the Appendix.

[1] Lam et al. 2024. Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

Confounding Effects of RLHF

We agree that RLHF can have confounding effects, and we’ve introduced a new experiment to test this (see general response, point 2). Please see the general response. The TL;DR is that general instruction following capabilities may be required as a “starting point” — jointly learning instruction-following and demonstrated feedback is too difficult a task to learn from a handful of demonstrations.

Computational Efficiency

See the general response, point 4. TL;DR is that our primary bottleneck is in sampling, but work on faster inference (e.g. VLLM) can easily mitigate this. We’ll add this to the limitations and future work.

We thank yVim for their thoughtful and thorough review! We appreciate that yVim appreciates our user study, our focus on few-shot alignment, and our strong performance improvements over available baselines.

Metrics and Static Benchmarks are Not Convincing

Metrics

Beyond just GPT-eval, we did try both sentence embeddings and perplexity measures. We abandoned both for performance reasons. We found that both perplexity and sentence embeddings did not discount degenerate outputs. Repetitions of phrases that appear in a generation result in inflated scores from both PPL and sentence embeddings [1]. Our observation—on degenerate text yielding low PPL—is already a well-documented finding (see [1]). While we reference these reasons in 4.1 (automatic evaluation) we will explicitly mention alternative metrics, namely embeddings and perplexity, and the associated challenges. We will also cite related work on text degeneration and its relationship to perplexity. We additionally validated GPT-eval in Appendix F.2, and found that it was quite good at judging authorship (98% accuracy).

[1] Holtzmann et al. 2020. The Curious Case of Neural Text Degeneration

Beyond static benchmarks

We agree that GPT-eval is not perfect, so we spent a significant amount of time working on a user study to complement our static benchmarks. Many of the tasks are quite diverse in nature—from writing recipes to asking for advice. In addition, we sourced preferences from each user: our setup ensured that the user writing the demonstration also evaluated their own DITTO-ed model. We document many of the provided demonstrations in the Appendix. In the final paper, we will move a handful to the main text.

Generalization Ability

To clarify: we have ten authors for the test set, not three. Our splits are not done at the author level, but at the demonstration level. Each author has 7 train demonstrations, ~3 validation demonstrations, and ~3 test demonstrations. For each author a_{1…10} we train a DITTO model on 7 demonstrations, and then validate / test on ~3. All of the results in Table 1 are done on the test split. We understand that Table 5 in the appendix is confusing, and we’ve revised the caption to be more explicit.

In terms of generalization beyond writing, please see the general response (1.1 and 1.3).

What’s the purpose of Section 3.3?

In Section 3.2 we derive and present the DITTO method intuitively, demonstrating how by treating the demonstrations as "gold data" that we can use to generate more preferences. In comparison to prior work, we make several design choices (e.g., a constant reference policy) that lead to improved performance in the low-data regime, per our results in Table 1.

Section 3.3 is designed to complement the intuitive explanation in 3.2 with a more formal grounding in an imitation learning / RL perspective. Specifically, we demonstrate that the intuitive choice of treating demonstrations as preferred to model samples has a mathematical grounding in the area of online imitation learning. Specifically, DITTO's objective can be viewed as optimizing the min-max Max-Ent IRL game popularized by [2]. The result is a more theoretical verification of our design choices to complement our empirical results.

If you have any further questions about this, we are happy to answer!

[2] Ziebart et al. 2008. Maximum entropy inverse reinforcement learning.

How did you determine the percentage distribution of the paired data, specifically the 70% online data, 20% replay data, and 10% intermodel pair data?

This is a hyperparameter we optimized in the hyperparameter search setup. Apologies for the oversight—we included this in the revised version of Appendix D (Hyperparameters). To summarize, we tried a handful of setups with varying amounts of paired data. In general, we qualitatively observed that online and replay data comparisons were most stable, and intermodal comparisons less-so. Increasing intermodal percentages beyond 30% resulted in degenerate output.

What’s going on with Author 9?

We looked into A9’s demonstration data to identify some reasons. First, A9’s demonstrations stylistically vary a lot from task to task; and second, A9’s opinions are bit… polarizing (re: A9 has fairly conservative opinions on gay marriage and religious freedoms). We suspect that few-shot GPT doesn’t improve because A9’s opinions are likely in conflict with the values encoded in GPT-4.

As for the tied performance between SFT and DITO: A9’s demonstrations are qualitatively quite different from one another. We suspect that DITTO is especially useful when demonstrations are more cohesive. To mitigate this, one could cluster demonstrations beforehand and train DITTO on individual clusters. We ran some preliminary experiments on demonstration sensitivity and our clustering approach (Appendix H), and have mentioned sensitivity in the revised limitations.

Thank you for the reply! Taking a crack at your observations:

Further discussion on why GPT-4-based evaluation appears to be more robust against these degenerated phrases.

There've been a few papers that study LLM-based evaluation. These papers compare against other metrics / human evaluation, and find that LLM-based evaluators often align with human evaluation more than other metrics. One reason for doing so is handling degenerate outputs---LLM-based metrics evaluate outputs more as a whole, according to prior work. We've cited a handful of papers that support GPT eval in general below. We'll include this discussion in the revised paper (under automatic evaluation, Section 4.1).

Zheng et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS

Chiang et al. 2023. Can Large Language Models Be an Alternative to Human Evaluations? ACL

Dubois et al. 2023. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. NeurIPS

Kim et al. 2023 Prometheus: Inducing fine-grained evaluation capability in language models. ICLR

Exploring whether the occurrence of repeated phrases could be attributed to specific generation hyperparameters, such as repetition_penalty and temperature settings.

We did explore repetition_penalty ablations, but at this point our models were already too overfit to a specific demonstration, memorizing and repeating sentences from the train demos. Model outputs would not generalize to new tasks. In our setting, we think GPT-eval captured both of these failure modes and was likely the best option. Still, the gold standard is human-eval; the user study in our paper additionally aligns with our results.

While the current examples in general responses effectively illustrate certain capabilities, a more comprehensive evaluation across diverse scenarios would be beneficial. Regarding generalization to domains like code and mathematics, incorporating established reasoning benchmarks could help validate the model's practical applicability in real-world human conversations.

We agree! However, established reasoning benchmarks do not have gold standard or high-quality demonstration-based data---mostly just questions and answers. While we would have loved to include more writing domains and authors, unfortunately there aren't many sources of gold-demonstrated feedback---just model-generated (potentially unfaithful!) CoTs. In fact, for our paper, we had to repurpose author attribution datasets for our evaluation. Creating a new benchmark for demonstrated feedback would be a wonderful avenue for future work! For example: collecting a reasoning dataset across which gold v.s. subpar trajectories are carefully labeled, or a writing dataset with orders of magnitude more demonstrations per author. We're happy to outline this in the revised paper.

Thank you again for engaging! We hope our rebuttal addressed most of your concerns :) Let us know if you need anything else!

We thank 7eqT for their thoughtful review! We appreciate that 7eqT thought our work was particularly advantageous for writing and user-specific finetuning (we agree!); and that our approach is an effective blend of RLHF and RLAIF. Below, we address your questions:

Handling Performance Degradations in General Alignment

See the general response (point 1.3) — we introduce a new experiment to evaluate this! TL;DR: while we do observe degradations on tasks unrelated to what we train DITTO on, we can proactively mitigate this by selectively routing/dropping the LoRA adapter.

Computational Efficiency

See the general response (point 4). TL;DR is that our primary bottleneck is in sampling, but work on faster inference (e.g. VLLM) can easily mitigate this. We’ve added this to the limitations and future work.

Limitations of DPO

DITTO’s overarching setup is agnostic to the specific “*PO” optimization method. One could swap out DPO with KTO, ORPO, SimPO, etc. etc. We did some early experimentation with alternatives. In our setting, we observed no statistically significant difference—in practice—across the specific PO method. We’ve revised section 3.2 to note this! There may be some way to make training more efficient with reference-free approaches (e.g. SimPO), but we wanted to make sure our approach worked with vanilla DPO first. We leave this exploration to future work.

Do you think DITTO would be useful for coding or math?

Potentially! Please see the general response (point 1.1).

Have you tried full finetuning?

Yes! Please see the general response (point 3). TL;DR is that we observe no significant difference between LoRA, and we stuck with LoRA to save money / compute.

How are you generating online responses?

Right now, we’re using vanilla Huggingface code (see the TRL repository) to generate online responses—an anonymous repo of how we do this is in the paper (https://anonymous.4open.science/r/demonstrated-feedback-3531/). There are some tricks TRL employs with LoRA significantly that reduce memory usage: because we’re only finetuning the adapter as $\pi_{t}$, we do not have to save a separate reference model in memory—we can just disable the adapter and run a forward pass.

In general, we think our codebase could be adapted to train much larger models. Our codebase’s bottleneck is primarily at inference. While we use FlashAttention, applying recent work on speeding up inference further improve performance (see vLLM). We’ve revised the future work/limitations sections to address this.

3. Have you tried full finetuning or do you use just LoRA? (7eqT HDJC)

We did try full finetuning for a few authors and noticed little / no change in performance (t-test). Since we didn’t observe much of a difference (and because full finetuning was significantly more resource-intensive across 20 models we needed to train) we stuck with LoRA for the entire paper. We include a reference to this in Section 4.1 (models and baselines) of the revised paper.

4. Have you considered computational efficiency? (7eqT and 41aT)

Yes! In general, DITTO does take longer than SFT. DITTO is slower than training-free approaches (prompting) and SFT (15 minutes with DITTO vs. 2 minutes with SFT on 7 demonstrations). The largest bottleneck lies in sampling—our current implementation relies on vanilla HF code. However, we suspect a mix of prior (e.g., vLLM [25]) and future work in LLM inference optimization can improve DITTO’s speed. Once a DITTO model is fully trained, however, it yields a single LoRA adapter $\pi_n$ that has no inference overhead compared to any other adapter—we do not have to save any of the intermediate policies ($pi_0 … pi_{n-1}$).

In addition, we are quite excited about inference time extensions of DITTO! We think that applying DITTO in-context—sampling negatives and using demonstrations as in-context feedback—is a promising approach. We’re leaving this as an avenue for future work; and have added an excerpt related to efficiency in the revised paper’s limitations section.

1.2 What about forgetting general capabilities/alignment?

Prompts from both our user study and our static benchmark (from the same author) are quite diverse, varying from writing recipes to asking for advice. Within this range, we observe no degradation. Still, all of these domains are related to writing.

We suspect that the reviewer is interested in domains like coding, so we additionally evaluated DITTO on HumanEval [2], using a randomly sampled author (a_10) from CMCC. We can easily mitigate degradations by selectively dropping DITTO’s LoRA adapter, and routing instructions between the general instruction-following model (Mistral 7B) and the specialized LoRA adapter (ala MoE). We experimented with the following zero-shot prompt, prompting the general model.

I have a specialized model trained on data of the form:

{demonstrations}

Should I use the specialized model or a more general-purpose model for the following task?

{human_eval_task}

Respond with just SPECIALIZED or GENERAL.

Answer:

This approach completely mitigates degradation.

If one tries to use a specialized writing model for mathematical reasoning tasks, we would expect degradation: performance on HumanEval drops significantly for a DITTO-ed model (Instruct 0.31 -> DITTO 0.13).

Finally, we updated the limitations section in the revision to be more explicit about the effects of forgetting, and we’ve added a section in the Appendix that outlines our mitigation approach. We think routing requests to specialized, demonstration-aligned models is a very interesting avenue for future work! While our prompted approach works, there are likely faster, more accurate, and more general methods.

[2] Chen et. al 2021. Evaluating Large Language Models Trained on Code.

1.3 Models must non-trivially generalize to perform well on our benchmark.

Within-author demonstrations span a diverse range of tasks, from both our static benchmarks and user study. DITTO-ed models must perform non-trivial generalization to perform well on our provided tasks. The submitted paper did not highlight this sufficiently. Here, we want to highlight the diversity of tasks in our author attribution benchmarks. Here are a handful of train-test prompts that highlight differences—we’ve included them in the Appendix.

train: Discuss a recent movie or TV show you watched
test: Share a new recipe you tried and loved.

train: The city of Denver has decided to legalize small amounts of marijuana for persons over 21. How do you feel about this?
test: Do you feel the Catholic Church needs to change its ways to adapt to life in the 21st Century?

train: Write an email to your professor seeking advice on research topics for an upcoming project.
test: Outline an agenda for a project meeting with a new collaborator.

train: Share personal writing rituals and habits for inspiration.
test: Highlight a fellow writer's work and encourage support within the community.

To summarize, these tasks span opinion pieces, blog posts, recipe writing, requests to meet, etc. Performing well on these benchmarks requires non-trivial generalization. Across these, DITTO-ed models generalize substantially across different train/test prompts and topics, extrapolating from a very limited number of demonstrations and domains. We’ve revised portions of our dataset (Section 4.1 and Appendix C) and user study (section 4.2) to make this remark more explicit!

2. RLHF Priors (HDJC and 41aT)

One observation shared by reviewers HDJC and 41aT is that our evaluated models are already instruction-finetuned and have strong RLHF priors—our baselines might be stronger on just base LLMs.

To test this, we evaluated few-shot prompting and SFT on the base mistral model and compared to DITTO on CMCC. We found that even when using the few-shot prompted/finetuned base model, DITTO still significantly outperforms baselines.

We suspect that general instruction-following capabilities are required as a “starting point” — jointly learning instruction-following and demonstrated feedback is too difficult a task to learn from a handful of demonstrations. We’ve included this analysis in Section 5.1.