TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation

We would like to thank the reviewer for their precise and informative review. We are glad that the reviewer sees our paper as “significantly improving the scalability of automated instruction-following benchmarks” and finds it “interesting that the checklist can help LLMs refine their initial responses”.

We directly address the weaknesses raised by the reviewer in the updated manuscript, as described below.

Lexical-matching metrics should be replaced with more semantic ones [such as] BERTScore.

We thank the reviewer for suggesting this and have included BERTScore as a column in Table 2 of the updated manuscript. These results further demonstrate the similarity between LLM-generated and gold standard human-written checklists, thus strengthening this part of the paper. We acknowledge that reporting the percentage of recalled gold standard checklist items would be another informative metric, however doing so would require further costly human annotations, as judging whether items from two different checklists are precisely the same is an ambiguous task. However, we hope that the addition of BERTScore in combination with reporting the count MAE provides sufficient evidence that the checklists are similar in terms of count and content. Finally, we would also like to point the reviewer to Table 3 (a), which shows that the downstream evaluation scores from using gold standard or LLM-generated checklists are highly correlated, demonstrating that the checklists also lead to similar evaluations on aggregate.

The paper fails to discourse the details of human study.

We would again like to thank the reviewer for identifying this shortcoming. We have updated the manuscript to include further details on the training of annotators and report the inter-annotator agreement. This information is now at the top of Appendix H.

We are confident that these changes strengthen our paper and hope that the reviewer finds that their suggestions have been directly addressed.

We would like to thank the reviewer for their deeply engaging with our paper during this discussion period. We are pleased to take the opportunity to address the reviewer's remaining concerns.

I still look forward to seeing the precision/recall metrics for such a rule-decomposition task.

Given that there is subjectivity in assessing whether or not two items from two different checklists are the same, we have made an effort to deliver on this request by using an LLM (GPT-4o) to label matching items between a model generated and human-written checklist. From this we can compute precision, recall and F1 with respect to the gold standard human-written checklists.

This analysis gave us the following results:

Command-R+ - Precision: 0.766, Recall: 0.702, F1: 0.712

GPT-4o - Precision: 0.811, Recall: 0.781, F1: 0.773

Given that we are unable to use human annotation for this analysis within the discussion period, we hope that this approach of using a strong LLM to label equivalent checklist items addresses the reviewer's request.

It provides some general information about human study without clarifying any basic numbers, like the total number of annotators. In addition, I am very confused about one statement in their attached content: "... if a prompt or response should be flagged as unsafe." Since the whole paper has not discussion about the safety alignment problem, where is the "unsafe" coming from? Combining these two factors, I suspect that the attached human study details in Appendix H are written by an LLM, which cannot discourse the real procedure of the human study. I will turn down my confidence because of this concern.

We assure the reviewer that the additions to Appendix H were not written by an LLM. We have included a number of expansions on the points that were added to that appendix in a new revision of the manuscript in the hopes that this clears up specifically what was meant by each point. We also include the revised content here:

The annotator pool consists of 143 annotators. All annotations were completed by native-level English speakers. The annotators are predominantly from the western hemisphere, with most living in the USA or Canada. Annotators were paid hourly, above the minimum wage of their country of employment.

The training undertaken by these annotators consists of being given documentation detailing the purposes of AI chatbots and detailed descriptions of common desirable and undesirable behaviours, accompanied by many examples and explanations. Some specific examples of undesirable behaviours are "leaps in logic'', "mechanical errors (e.g., incorrect reasoning, grammar, or formatting)", "factual errors", "being uninformative"}. Some specific examples of desirable behaviours are "expressing useful and accurate information", "writing in a suitable tone for the context"}. Annotators are also provided with safety guidelines that detail how to assess if a prompt or response should be flagged as unsafe. This is simply intended to identify any NSFW or unethical behaviour by the model and is no more than a sanity check, as we do not deal with the alignment problem here.

Finally, annotators are provided a small set of annotation tasks for which ground truth annotations are known. Specifically, they are required to complete 25 of these training annotations for any new annotation instructions (i.e., 25 for checklist question answering, 25 for direct scoring, etc.). Where there is disagreement between an annotator and the ground truth annotation at this stage, new annotators are able to discuss any sources of confusion or uncertainty with us and annotators who have successfully completed the training.

The inter-annotator agreement for pairwise preference annotations on the Internal dataset, computed as Krippendorff's alpha, is 0.684.

We sincerely hope that this addresses the reviewer's remaining concerns, especially regarding the accusation of using LLM generated content, which we take very seriously.

My first concern has been addressed. However, I still look forward to seeing the precision/recall metrics for such a rule-decomposition task.

My second concern has not been addressed. In particular, it provides some general information about human study without clarifying any basic numbers, like the total number of annotators. In addition, I am very confused about one statement in their attached content: "... if a prompt or response should be flagged as unsafe." Since the whole paper has not discussion about the safety alignment problem, where is the "unsafe" coming from? Combining these two factors, I suspect that the attached human study details in Appendix H are written by an LLM, which cannot discourse the real procedure of the human study. I will turn down my confidence because of this concern.

I am happy to see these discussions and my concerns have been addressed. Please include them in your manuscript. I will turn back to my initial evaluation scores.

We are pleased that the reviewer finds that the paper “conducts extensive automated and manual consistency experiments to quantify and demonstrate the advantages of the TICK evaluation method”. However, we find it a shame that the reviewer has not acknowledged Section 4 of the paper, beyond claiming that using checklists for refinement is not new, despite the fact that no prior work on using checklists for in-context self-improvement exists to the best of our knowledge. Our paper demonstrates that Self-TICK (STICK) significantly improves in-context self-refinement, an increasingly common practice for improving response quality by expending more compute at inference time [1]. Table 1 demonstrates that end-to-end checklist self-evaluations enable purely in-context self-correction in challenging code, reasoning and mathematics problems, despite the reviewer’s claims that we do not consider these task types. We are therefore led to believe that the reviewer has not considered these results in their assessment and would like to highlight their significance, especially in light of several recent works suggesting that purely in-context self-correction is yet to be demonstrated in these settings [2, 3].

Employing decomposed checklists for instruction evaluation, validation and refinement is not new, as seen in work like FollowBench, InFoBench, and Self-Contrast.

FollowBench and InFoBench use expensive-to-gather, human-written checklists, which limits the use of checklist-based evaluations to their predefined prompt datasets, whereas TICK is substantially cheaper and generally applicable (as acknowledged by Reviewer sSrg), which is what enables Section 4 of the paper, where an LLM can perform checklist-based self-evaluations on-the-fly. Self-Contrast is not closely related to our paper, being a similarity-based method for alignment involving two fine-tuning phases. To the best of our knowledge, no prior work uses decomposed evaluation structure to enable improved iterative self-refinement and even self-correction, which recent work has suggested requires RL fine-tuning to achieve [2].

There is a lack of in-depth discussion regarding the efficiency of the proposed method.

We thank the reviewer for identifying this. Scaling inference compute by sampling more or longer generations is an increasingly common practice for improving LLM capabilities on problems that are otherwise challenging [1, 4, 5]. We therefore see the improvements of TICK and STICK as being due to an effective way of improving evaluation and refinement quality in exchange for additional inference costs. To further address this concern, we additionally compare to the most common approach to inference scaling of majority vote among K generations sampled in parallel (i.e., Maj@K) [5] in Table 4 of the updated manuscript. We do this for preference-based LLM-as-judge evaluation and direct scoring with K=32 and still using Chain-of-Thought for the evaluator in each case. The results demonstrate that this improves both LLM-as-judge preferences and direct scores, but that both still perform worse than TICK, highlighting that TICK makes more efficient use of additional tokens than majority vote.

Answering questions

1. We acknowledge that checklists do not capture sequential dependencies by default and see the automatic construction of evaluation rubrics for agentic tasks as an exciting direction for future work. Whilst ComplexBench explicitly constructs a dataset of instructions with constraint dependencies and has human annotators write checklists that reflect this, simply prompting the LLM to “opt for ’NO’ if the generated text provides no information that could be utilised to answer the question” implicitly captures the fact that a checklist question should be answered ‘NO’ if a question higher up a dependency chain is answered ‘NO’, as is done in this work and InFoBench.

2. We explicitly prompt the LLM to include “implicit criteria that are generally important for an instruction’s problem domain” in the checklist (line 174 in the manuscript), the positive effect of which can be observed in the examples of generated checklists in the appendix and in the positive STICK results on precisely the fields mentioned by the reviewer (Table 1 of the manuscript).

3. We have included results for Llama-3.1-8B-Instruct in Table 2 and Table 3 (b) to address this. We see that it performs only marginally worse than larger models at both generating and answering checklist questions.

[1] Snell et al, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024

[2] Kumar et al, Training Language Models to Self-Correct via Reinforcement Learning, 2024

[3] J. Huang et al, Large Language Models Cannot Self-Correct Reasoning Yet, 2023

[4] Madaan et al, Self-Refine: Iterative Refinement with Self-Feedback, 2023

[5] X. Wang et al, Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022

We would like to thank the reviewer for their clear and focused review. We are glad that the reviewer sees the presented method as “novel and significant”, including “comprehensive experiments”. In light of this, we are surprised that the reviewer’s concerns should warrant the current score and thoroughly address each one below.

Previous work has also used decomposition techniques...

Whilst it is true that prior work decomposes the evaluation task, our work is the first to take an approach to decomposition that has proven powerful in fixed datasets and fully automate the decomposition and evaluation itself using an LLM. Our work is also the first to show that such a decomposition technique enables in-context self-improvement/ self-correction in settings where unstructured self-critiques fail.

The construction details and statistics of the Internal dataset are not sufficiently explained.

We thank the reviewer for drawing attention to this and have included further details on both the annotator pool and Internal dataset construction in Appendix H of the updated manuscript. The Internal dataset and its full construction details are scheduled for public release within the next month (footnote 1 of the manuscript).

The authors use metrics like ROUGE and BLEU.

Due to the expense of human annotation, we were unable to additionally acquire human annotation results for this comparison. As suggested by reviewers KpU7 and sSrg, we have included semantic similarity (BERTScore) between checklists in Table 2 of the updated manuscript, where we see that LLM-generated checklists maintain strong similarity to gold standard human-written checklists.

The preference labelling approach of annotators does not fully align with the checklist-based method.

We would like to raise two key points that we are confident address this claim. Firstly, as can be seen in the prompt for checklist generation and as is stated in line 174 of the manuscript, the LLM is prompted to include “implicit criteria that are generally important for an instruction’s problem domain” in the checklist. Secondly, the superior agreement with gathered human preferences achieved by TICK relative to asking an LLM-as-judge for a preference or score empirically demonstrates that it is better aligned with the preference labelling approach of annotators.

The low inter-annotator agreement for direct scoring raises concerns.

TICK’s effectiveness was demonstrated by comparing to preference annotations on Internal, for which we provide the inter-annotator agreement in Appendix H of the updated manuscript (0.684 Krippendorff’s alpha). This difference reflects the fact that WildBench involves particularly long and sometimes low quality instructions, direct scoring yields lower agreement than preference labelling, and annotators are familiar with the Internal instruction set.

Evaluations with fine-tuned models or well-established frameworks could provide a fairer assessment.

Given that TICK requires no additional data or fine-tuning, we firmly disagree that comparing to fine-tuned evaluator models would be a fairer assessment. As an alternative inference scaled baseline, we have additionally provided results for a majority vote (Maj@K) [1] version of preference evaluation and direct scoring among K=32 parallel samples in Table 4 of the updated manuscript. Notably, both remain inferior to TICK.

The baseline comparison is limited to vanilla self-refinement, which is insufficient.

Self-refine [2] is itself a relatively new method, with no well-established, fine-tuning free alternatives. There are numerous papers indicating that purely in-context self-refinement in fact generally fails, with a prominent recent paper [3] claiming that RL fine-tuning is absolutely necessary to achieve this behaviour in self-correction settings. Yet, in Table 1 we show that STICK is able to reliably self-correct across almost all task categories in the challenging benchmark LiveBench. We believe that this is a very significant result.

Answering questions

1. We thank the reviewer for identifying a potentially out-of-sequence figure caption, but are unable to identify which they mean. Could the reviewer please clarify whether they mean Table 3 (a) or Figure 3 (which has no subfigure labelled (a))?

2. As shown in [3, 4] and in Table 1, in-context self-refinement is typically prone to response quality degradation, as the LLM can misidentify issues with its own response. The small performance dip in the fourth iteration on WildBench simply shows that the number of iterations STICK can sustain improvements is still limited.

[1] Wang et al, Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022

[2] Madaan et al, Self-Refine: Iterative Refinement with Self-Feedback, 2023

[3] Kumar et al, Training Language Models to Self-Correct via Reinforcement Learning, 2024

[4] Huang et al, Large Language Models Cannot Self-Correct Reasoning Yet, 2023

We thank the reviewer for their detailed review. We are pleased that the reviewer sees our paper as “offering valuable insights” and that the “overall experimental analysis is thorough”. We strongly believe that the weaknesses raised by the reviewer are addressed by the following clarifications.

[The approach] requires the LLM to first follow a complex set of instructions to generate the checklist.

Our results demonstrate that current LLMs are already capable of generating checklists that are similar to gold standard human-written checklists (Table 2 and Table 3 (a)), including smaller, open-source models, such as Llama-3.1-8B for which results have been added in the updated manuscript. Additionally, checklist self-feedback (i.e., STICK) proves effective at enabling self-correction where unstructured self-feedback fails (Table 1), demonstrating that checklist generation and answering in fact eases the problem of answering the original instruction and thus cannot be more difficult than answering the original instruction.

It only examines three benchmarks: Internal, InFoBench, and WildBench.

As shown in Table 1, we also evaluate on LiveBench, which spans a range of task categories covering reasoning, mathematics, code, language and more. Additionally, both Internal and WildBench cover a very broad spectrum of instructions, with WildBench instructions being taken from a wide range of real-world interactions with chatbots. We believe that the four benchmarks considered cumulatively provide strong evidence for the benefits of using automatically generated checklists to structure automatic evaluation.

The existing design is computationally expensive during inference time.

Scaling inference compute as an alternative to scaling training compute has emerged as an exciting paradigm for further improving LLM capabilities [1, 2], with self-refinement [3] and self-correction [4, 5] becoming popular research directions. We convincingly demonstrate that checklist-based self-evaluations are an effective way obtaining greater benefits from increased inference compute, whether by iterative refinement (Table 1 & Figure 3), or Best-of-N selection (Table 5). As a further investigation of how TICK compares to alternative approaches to assigning more inference compute to the task of evaluation, we have added a comparison to majority vote among 32 parallel sampled evaluations (i.e., Maj@32) for preference and direct scoring in Table 4 of the updated manuscript. We see that doing so improves agreement between the subsequent preferences or scores and human evaluations, but that they remain worse than TICK.

Answering questions

1. We thank the reviewer for this suggestion. We have included results using the semantic similarity metric BERTScore in Table 2 of the updated manuscript.

[1] Snell et al, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024

[2] OpenAI, o1, 2024

[3] Madaan et al, Self-Refine: Iterative Refinement with Self-Feedback, 2023

[4] Kumar et al, Training Language Models to Self-Correct via Reinforcement Learning, 2024

[5] Gou et al, CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, 2024