Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks

Thank you for the feedback. We've now included additional sections that study model families -- effects of scale (within a family), and effects of distractors within a model family at different scales.

Performance within Model Families: We report the performance on Full Benchmark for models from the Llama family and Qwen family of models in Figures in the updated appendix. We observe a consistent pattern of improvements in instruction following-ability with increase in model capacity for the Llama family. However, this is not the case for Qwen family of models. Specifically, for some instructions like *print_correct_answer, print_correct_answer_label, *sort_only_incorrect_answers the Qwen 1.5B model outperforms 3B model. Qwen 3B model is better than Qwen 7B and 14B variants for the print_correct_answer_append_string instruction. We consistently see 32B and 72B variants outperforming other models by a significant margin.

Distractors: We report details of how different model families (Llama, Qwen and Phi) are affected by distractors, at different scales. We find that the Llama family and Phi of models are extremely distracted by non-applicable instructions that require reversing and casing of text (eg: such instructions are inapplicable on numerical data), and report a drop of nearly 75-78% while Qwen family of models (at all scales) is relatively robust to such distractors. On the other hand, distractor instructions that are based on numeric operations lead to a minor drop in performance in Llama and Qwen models but still affect Phi family of models significantly.

Our goal was to release a benchmark that could be used for studying instruction following with verifiable outputs. Constraints such as formatting style, length are harder to verify along with task performance (not just instruction performance) and there has been additional work in that space as you mentioned.

However, work on assessing instruction-following with verifiable answers is limited. Our list of instructions demonstrate: 1. Simple changes, such as answering with text instead of labels results in a drop in performance at all model scales (and families). 2. Simple tasks of counting, concatenation, conditional exclusion/inclusion/application as well as, using distracting instructions all result in significant drop in performance.

To the best of our knowledge, there is no prior work that demonstrates this with verifiable results and benchmarking of current models. We hope the reviewer agrees that this is valuable and non-verfiable instructions can lead to inflated assessments of instruction-following performance. We further, agree that the list of instructions could be increased (our goal is continuously maintain and improve the collection through open-source contributions).

Thank you for the question -- as we discuss in our related work, existing benchmarks either do not assess task performance when studying instruction following, or do so in ways that are not subjective assessments by LLMs (which have their own biases and limitations). In the case of IFEval, for example, if the task requires a model to answer the question in three words -- simply answer "the the the" would get a perfect score. We believe this is a major limitation of disentangling task performance from instruction-following as real-world performance is always a reflection of both. We use knowledge tasks as existing LLMs do perform well on these tasks (and give us a baseline performance for each model) -- adding simple instructions (instructions which are otherwise considered easy) leads to very observable drop in performance. Our goal was not to show that LLMs cannot follow, instructions but to provide a benchmark against which improvements can be objectively studied. For each model, we include the print correct answer text (PCA) and print correct answer label scores -- contrasting instruction following scores against these disentangles knowledge and instruction-following performance. We would like to further state that we also plan to continuously extend this benchmark.

We apologize for the this oversight and inconvenience caused. We are in the process of regenerating the figures for better rendering and we will further update the rebuttal version shortly.

We apologize of the typographical errors and we have incorporated changes and also proof-read the paper again. We shall be updating the rebuttal version of the PDF shortly.

Yes that is correct -- this instruction requires returning a list. It is conditional on the correct answer in the sense, that the list includes all options (incremented by one) except for the correct answer.

We apologize for the oversight -- the instruction category for use_incorrect_options_to_create_string is indeed conditional and the main category label was misaligned for this row in LaTeX.

The instruction is as follows: Create a string by concatenating the last character of every option value (not option label). If the last character is a special character (such as period, comma, quotation, etc) use the previous character.

Options values are shown to LLM in a fixed sequence and the results of this operation are thus deterministic and verifiable. We generate ground-truth data by implementing these actions in code.

Comment on Instruction Categories Our goal was to release a benchmark that could be used for studying instruction following with verifiable outputs. Constraints such as formatting style, length are harder to verify along with task performance (not just instruction performance) and there has been additional work in that space. However, assessing instruction-following with verifiable answers is limited. Our list of instructions demonstrate: 1. Simple changes of return text instead of labels results in a drop. 2. Simple tasks of counting, concatenation, conditional exclusion/inclusion/application as well as distracting instructions all result in significant drop in performance.

To the best of our knowledge, there is no prior work that demonstrates this with verifiable results and benchmarking of current models. We hope the reviewer agrees that this is valuable and non-verifiable instructions can lead to inflated assessments of instruction-following performance. We further, agree that the list of instructions could be increased (our goal is continuously maintain and improve the collection through open-source contributions).

Limitations of IFEval We would like to further add, the instructions that we include, while simple, also contribute to "practical instructions" - for instance, IFEval includes instructions to check for capitalization -- however, it does not assess whether the target string being capitalized is correct or not. Using our benchmark, we demonstrate, that simple capitalization results in a 20% absolute drop in performance (as compared no capitalization) on models such as Llama 70B Instruct but the Qwen family of models aren't as affected by such an instruction. Such insights cannot be drawn using existing benchmarks. Another example is that, all models fail to correctly append strings to correct answers of the original knowledge task. IFEval instructions that require the inclusion of a prefix strings for Titles etc are similar in spirit but fail to account for task performance/correctness.

Thank you very much for the detailed reading of our paper and your insightful comments. We hope to get back by addressing all your questions/comments shortly. In the meantime, we address some of your concerns in our responses below.

Thank you for pointing out the discrepancies with Figure 1 and the performance of Qwen. We have updated the rebuttal version with corrections and added details regarding the error analysis, and have also uploaded more figures showing errors across different model families. Here is the summary for your reference.

Qwen performance

1. We took a closer look at the raw output from Qwen-2.5-3B and noted a consistent pattern of *Response: $Label* for the *print_correct_answer_label* (i.e.) extra$ symbol. While this could arguably be considered an instruction following error, we decided to modify our evaluation to not penalize this minor issue. With the new PCA label score, we see that Qwen-2.5-1.5B and Qwen-2.5-3B have comparable performance for PCA and PCA label.

2. We also realized that we had accidentally reported the strict version of the errors instead of the loose (that we had mentioned in the paper). This means in the updated version, our error detection sets continue to be high-precision, however the categorization of errors into the different buckets is high recall. Not doing so, would automatically result in the unclassified categorization becoming high recall, which is not helpful for error analysis.

3. We have modified Figure 1 (and Figure 5) to additionally show the unclassified score, as well as the updated error analysis scores. We find that the Knowledge and IF-errors of Qwen-2.5-1.5B and Qwen-2.5-3B are comparable (within 4-5% of each other) , leading us to believe that the Qwen-2.5-3B model does not offer superior instruction following capabilities compared to its 1.5B counterpart.

Error Analysis

Additionally, from the updated Figure 1, we observe that the errors decrease as the inherent model capability and size increases. Note that incorrect answers could correspond to both knowledge and instruction following errors. The figure also shows that we capture most errors. We observe larger models making fewer errors ($20\%-80\%$ reduction) for the Llama and Qwen models. The Phi model family however does not show this trend, calling for a closer look at their instruction training methodology. We also perform a deep-dive on the error distribution for each instruction category across model scales. We observe that models make the most errors on string manipulation tasks, and model scale does little to mitigate this. For the other categories, errors reduce as the model size increases. All these figures are available in the appendix of the revised pdf.

Naming: Thank you for the feedback - we refer to knowledge tasks in the sense of verifiable grounding; perhaps grounding could be an alternative phrasing but that too has existing connotations in literature.

Lite Benchmark: We would like to clarify that the Lite Benchmark has 150 samples from each dataset on which each instruction is applied i.e, we have 150 samples per instruction, for each dataset and for MMLUPro we have 25 samples per instruction. The total size of this benchmark still exceeds 12000 instances.

We would like to clarify that the task of printing the correct answer label is an unperturbed assessment of the model performance. We demonstrate simply asking the model to print the text instead of the answer label results in a performance drop (See Figures 1) and are other instructions demonstrate that additional (simple) operations result in a further drop in performance.

We would like to thank the reviewer for their detailed feedback and comments.

Thank you for sharing the reference on the verbalizer paper - we had not come across this. Indeed our approach to study instruction following is based on verbalizers as a method. The reference does not include a benchmark and is based on label manipulation as a task. We have updated the related to work to incorporate this reference.

Performance analysis with model family and distractors

We had previously included all models in different sizes and the intra-model family trends can be studied from the tables 4 in the paper but agree with the reviewer that trends may be hard to study with that view. Thank you for the feedback. We have now also included additional model family-specific figures in the appendix as per your suggestion.

On Applicable Instructions: We report the performance on Full Benchmark for models from the Llama family and Qwen family of models in Figures in the appendix. We observe a consistent pattern of improvements in instruction following-ability with increase in model capacity for the Llama family. However, this is not the case for Qwen family of models. Specifically, for some instructions like *print_correct_answer, print_correct_answer_label, *sort_only_incorrect_answers the Qwen 1.5B model outperforms 3B model. Qwen 3B model is better than Qwen 7B and 14B variants for the print_correct_answer_append_string instruction. We consistently see 32B and 72B variants outperforming other models by a significant margin.

On inapplicable Instructions (distractors): We report details of how different model families (Llama, Qwen and Phi) are affected by distractors, at different scales. We find that the Llama family and Phi of models are extremely distracted by non-applicable instructions that require reversing and casing of text (eg: such instructions are inapplicable on numerical data), and report a drop of nearly 75-78% while Qwen family of models (at all scales) is relatively robust to such distractors. On the other hand, distractor instructions that are based on numeric operations lead to a minor drop in performance in Llama and Qwen models but still affect Phi family of models significantly.