Instruction Following without Instruction Tuning

We thank reviewer Pxjk for their review and suggestions.

We’ve added IFEval and MTBench as extra evaluations, and added 9 samples for each model for each finetuning task, to help provide more context to our AlpacaEval results. We discuss this more in the general response.

We agree that we don't provide new state-of-the-art methods, but we hope that the work provides insight into future instruction-following work by providing a range of ablations (of surprising quality) that wouldn't otherwise have been considered by the community.

Hope the best to your paper.

We thank reviewer eN6V for their enthusiastic review.

We appreciate the feedback on the abstract, and have streamlined the abstract in our new draft.

We've also added Gemma-2-9B experiments (discussed in our general response.)

Regarding controlling variables at pretraining time, we agree this is a fascinating direction. The best we could do here was to check with the Olmo authors to guarantee that no intentional instruction-tuning data made its way into the pretraining mix. Of course, much naturalistic data is instruction-like (which one must imagine is part of why this is all possible.) One thing we’ll note here is that recent work on pretraining data filtering (DCLM) found that filtering to documents that are most like instruction-tuning data leads to the best pretraining mixture, suggesting that quite a bit of why LLMs perform well on our current evals is due to instruction-like data at pretraining time.

We thank reviewer aAKp for their evaluation and proposed improvements to our work.

aAKp was concerned with our use of two language models that are not as recent as the Llama-3 or Gemma-2 lines of recent models (we used Llama-2 and Olmo-Feb-2024.) As aAKp notes, we did so to avoid studying models that are likely “mid-trained,” that is, explicitly instruction-tuned throughout the pretraining process. (e.g., the Olmo authors told us that this Olmo version was not mid-trained.) While we stand by this argument, we’ve decided to extend our experiments to the strong Gemma-2-9B models to provide extra context. We’ve updated the paper with Gemma experiments. We find (1) roughly the same result for response tuning (40% win rate of response tuning over instruction tuning, compared to 43% for the other models). (2) we directly ported our rule-based adapter to Gemma-2-9B, and found that it also yielded instruction following there, despite being written for the Llama-2 models. We find this really exciting! Finally (3) we found weaker single-task finetuning results, though still some improvements over the base Gemma-2-9B model (for Poetry and GSM.)

aAKp was concerned that we only used length-controlled AlpacaEval2 metrics. We believe that the combination of these metrics and the qualitative examples provide a clear distinction between the behavior of the base models and the behaviors of the other models. More evaluations can of course help readers interpret the results however, so we added IFEval and MTBench evaluations for all of our finetuned models (including the new Gemma models.) We’ve also included many more example outputs, as requested.

aAKp asks whether response-tuning works in a single-task setting. In preliminary experiments, we found that it does not. One seemingly needs at least one of either a broad distribution of responses (response-tuning) or instruction-response pairs (single-task instruction-response).

aAKp was concerned that we only used LIMA for our instruction tuning dataset. We respectfully disagree that LIMA is a particularly non-standard instruction-tuning dataset; we think that most strong instruction-tuning datasets include long, detailed responses. However, to provide even more analysis, we took the Tulu-v2 dataset and subsampled 1030 instruction-response pairs from it (for comparison to LIMA) and re-ran our response-tuning vs instruction tuning evaluations using Gemma-2-9B. We found that the base model wins against the Tulu instruction-tuned models at roughly 6%, while tulu response-tuned win roughly 25% of the time, a similar result to the 11% base and 40% response-tuned from LIMA. We did find that the exceptionally short responses in some Tulu examples leads to lower response-tuning quality (as shown in our newly provided samples in Table 35.)

Finally, aAKp is concerned that the simplicity of our rule-based adapter shows that we must be gaming AlpacaEval instead of really causing the base models to follow instructions. For example, aAKp suggests that AlpaceEval “will give higher scores as long as the output is long and diverse.” There may be a misunderstanding here; our EOS rule makes outputs shorter, not longer. Certainly, the rule-based adapter doesn’t make the strongest instruction-following model, but its simplicity points to just how easy it is to elicit some level of instruction following from strong language models.

Here’s an example (markup added by us):

User: Why can't you see the stars in the city, but you can in the countryside?

Rules+Gemma-2-9B: Because there is less light pollution.

And another example,

User: I'm trying to stop my neighbour's cat from coming in to my garden and fighting with my cat. Any suggestions on how to do that?

Rules+Gemma-2-9B: Sure! Here are a few tips:
1. Keep your cat indoors as much as possible, especially at night when cats are most active.
2. Install a fence or barrier around your yard to keep the neighbor's cat out.
3. Use motion-activated sprinklers or ultrasonic devices to deter the neighbor's cat from entering your yard.
4. Talk to your neighbor about the issue and ask them to keep their cat inside or use a leash when outside

Without the diversity rule, the output is fine, just less useful:

User: I'm trying to stop my neighbour's cat from coming in to my garden and fighting with my cat. Any suggestions on how to do that?

EOS+Uniform rule+Gemma-2-9B: Sure! Here are a few tips:
1. Keep your cat indoors: This is the most effective way to prevent your cat from fighting with your neighbour's cat.
2. Use a cat deterrent: There are many products available that can help deter cats from entering your garden.
3. Install a fence: A fence can help keep your cat in your garden and prevent your neighbour's cat from entering.
4. Talk to your neighbour: If your neighbour is aware of the problem, they may be willing to help

We thank reviewer uWBN for their evaluation of our work. We’ve incorporated the recommended paper, Wu et al[1], into our related work. Reviewer uWBN recommends that we provide theoretical analysis or parameter-level insight like [1]. We believe that the success of, e.g., response tuning provides interesting context to work like [1]; for example, while [1] finds that instruction tuning “empowers LLMs to recognize the instruction parts of user prompts”, the success of response tuning demonstrates that if this is the case, it isn’t done explicitly, since one can yield instruction following without any instruction. For analysis (though not theoretical), our rule-based model provides, we think, the clearest explanation why simply telling the model not to say a few words (i.e., simple changes) can cause strong instruction following. We believe our behavioral analyses–experimental ablations to traditional instruction tuning methods–provide complementary insights to parameter-level or mechanistic analysis. To this end, our goal with this work was not to provide an immediate improvement to instruction tuning, but to dive deeply into the phenomenon of instruction following and give insight into how simple it is to yield it in language models.

We agree that instruction-following behavior in language models likely stems from instruction-response-like data naturally occurring in pretraining data; we see no promising other option when (as we show,) so little extra is needed to elicit instruction following from pretrained models. While we do not in this work identify the specific aspects of pretraining that make this possible, we believe we provide considerable clarity relative to the field's prior understanding of just how little is necessary after the pretraining process.

Hi Pinzhen,

Thanks for your question. So, the key difference is in what LIMA responses are like--structured as responses to some relatively concise question--vs. what pretraining documents are like. Yes, any text (including pretraining text) is a suitable response for some instruction; e.g., for any text $T$, I could say "Generate the string $T$". One could say that it's the style/format of responses, and we think that's a lot of it, but it's not quite so obvious to us it's all that's going on---there's also a notion of the responses being good responses for some instruction in some reasonable distribution of instructions people tend to ask. I'd guess that most pretraining texts are not good responses for the kinds of questions that are evaluated in modern LLM evaluations. Responses in instruction-tuning datasets are generally self-contained given the dialogue so far, whereas a pretraining document might not make a ton of sense as a response due to a lack of context, e.g., people might not tend not to ask "Give me the twenty-third through twenty-sixth page of the unix manual for grep," but that might be a pretraining document.

As another simple effect, response tuning also teaches the model not to generate the instruction-response delimiter tokens, like <|assistant|> or <|user|>; I mention this because it seems important in our rule-based system to explicitly disallow tokens that would begin the generation of these strings. Base models see the formatting tags and really tend to like to repeat them; response tuning (and single-task finetuning for that matter) teach the model not to repeat these tokens.

So, response tuning is certainly maximizing the likelihood of a bunch of text, as is pretraining -- response-tuning differs in that (1) the distribution of responses in instruction-tuning datasets is different from the distribution of pretraining data, and (2) (though you could consider this part of (1)) you condition on these formatting tags and learn not to generate them when you see them.