Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

Thank you for your valuable review and suggestions. We appreciate your detailed and informative concerns. Below, we address the comments on Weaknesses (W).

Note: We've marked "D9R8" next to relevant changes in the general comments to make it easier for you to navigate.

W1. Clarification re: few-shot baseline

The contrastive task measures specificity - how well a qualitative evaluation captures model-specific characteristics. When humans compare models, they naturally examine a few examples to identify consistent behavioral traits that could predict performance on new tasks. Our few-shot baseline implements this intuitive process: using k training examples per model to simulate how humans derive and apply their understanding of model behavior. Since Report Cards are also generated using the training set, this baseline provides a fair comparison to evaluate whether our structured summaries can capture model-specific traits as effectively as the natural example-based approach.

W2. De-stylization results suggests few-shot baseline captures stylistic information

We appreciate the reviewer's insight about the few-shot baseline's limitations. We agree that the de-stylization results reveal that few-shot examples primarily capture superficial stylistic patterns rather than substantive behavioral traits. However, this finding actually strengthens our evaluation: Report Cards maintain strong performance even after de-stylization, particularly on Advanced AI Risk topics where the few-shot baseline's high performance was driven by exploiting stylistic cues. Together, the few-shot baseline and de-stylization experiments demonstrate that Report Cards successfully capture meaningful behavioral characteristics rather than superficial patterns. In addition, the one-pass generation method also servers as a baseline (Figure 8), supporting the performance of PRESS.

W3. Rename to Oracle ELO

We agree, and we will rename it to Oracle Elo. Thanks for pointing that out!

All results (contrastive/faithfulness metrics) are reported based on held-out test sets that have no overlap with the training sets used to create the Report Cards and/or select samples for the few shot baselines. In one out of all of our datasets (MMLU High school mathematics), we did use the test set as a validation set in order to tune prompt. Otherwise, the test sets have been held out. See Page 15 for details (the train set is 40 examples and test set is 60 examples). You mention other methods to compare against, is there something specific you have in mind?

We hope this answers your question and are happy to clarify further.

Thank you for your valuable review, concerns, and questions. We appreciate that you recognize our work as potentially addressing the need for “more than model cards”. Below, we address the comments on Weaknesses (W) and Questions (Q).

Note: We've marked "f1vs" next to relevant changes in the general comments to make it easier for you to navigate.

W1. Use LLMs as Report Card writers at a scale add some cost. We used Claude 3.5 Sonnet as the card writer model. In the current PRESS algorithm, it would cost ~$0.1 to generate a Report Card, which is not too expensive. Notice that only one card is needed for each model under a topic, so in actual use cases when users want to qualitatively check the performance of models on a specific task (topic), the total number of cards would equal to the number of models. The costly tasks are the metrics - assessments of Report Cards themselves - which are not required for typical use cases.

In addition, we have conducted an ablation study on different card writer models with varying capabilities and costs. Using the same experiment setup as the ablation studies in our paper (Appendix Section C), we report the specificity and faithfulness on the MMLU dataset. The results are aggregated over topics as in Appendix Table 6.

As the table shows, smaller and cheaper models like Claude 3.5 Haiku, Llama 3.1 70B, and GPT-4o-mini can generate reasonably good Report Cards at a lower cost (it costs only $0.005 to generate a Report Card with GPT-4o-mini). We also see that Claude 3.5 Haiku produces Report Cards of the same quality as Claude 3.5 Sonnet while being more cost-effective.

Q1: Assuming a specific case, a question that cannot be reliably solved by the judge model as well, but a fine-tuning of the base model does solve it, can a judge model reliably generate a summary for the same?

As we've shown in response to W2, we can generate Report Cards with high faithfulness by using reasonably performing models (like in the W2 case, gpt-4o-mini). We can also include the ground truth answers (if available) when generating Report Cards to mitigate the issue. In cases where there are no ground truth answers, Report Cards can be easily augmented with few-shot samples to show both the interpretable and condensed qualitative evaluations (Report Cards), and the actual examples of model behaviors. We've attached a HTML file report_cards.html to the supplementary materials for you to access the augmented Report Cards across many topics and models.

Q2: Is the method generally constrained by the capability of the larger, better model?

As noted in the response to weakness 1, smaller and less-capable card writer model like gpt-4o-mini and claude-3.5-haiku can generate Report Cards with similar level of specificity and faithfulness. However, with stronger card writers, the Report Cards can indeed capture more nuances and faithful behaviors, and we expect that Report Cards will improve as our models improve.

Q3: Is the readability of the report card considered at the macro level? What is done to balance specificity vs general comments?

Indeed. Readability at a global level is impacted by our focus on nuanced behaviors. However, each bullet point in the Report Card focuses on independent subtopics or criteria. During human evaluations, a LLM is to extract relevant parts from the whole Report Card. We anticipate that for general usage, a practitioner can get the relevant excerpt (such as via search or LLM extraction) instead of the longer complete card.

In addition, we also proposed the Hierarchical Bullet Point Format for Report Cards (Appendix Section A), which captures the behaviors with both high level descriptions (overview and thinking_pattern) and fine-grained details (strength and weakness). HBP performs slightly worse than the bullet point format as shown by Figure 12 (d), so we chose the Bullet Point format as our main format.

Q4: Could you showcase a full report card for any model of your choice?

We have attached some example complete Report Card with many augmented Report Cards examples. Please check the general comments for details. Thanks!

W2: The reliance on LLM as a judge model perhaps also showcases a shortcoming, would a model be good at verifying or recognizing a model better than it or rely on its own (perhaps wrong) judgement.

To address the concern about whether a card writer might rely on its own potentially incorrect judgments when evaluating a more capable student model, we conducted an ablation study. By comparing Report Cards generated with and without access to ground truth answers, we can determine if the card writer model's assessments are overly dependent on its own understanding rather than objective evaluation criteria. To test this, we specifically selected claude-3-haiku, a model that consistently performs poorly on the MATH dataset, as our card writer model. The ablation study is conducted with the same setup as the card writer ablation, but on the MATH dataset. The results of this investigation were:

This indeed shows that claude-3-haiku is relying on its own errored judgements when generating Report Cards. However, we also show that a more intelligent model with higher dataset accuracy, but far from good, would achieve consistently high faithfulness.

In conclusion, the quality of Report Cards relies moderately on card writer capability, but (1) we can mitigate the issue by including ground truths and (2) Report Cards generated by a reasonably performed card writer have high faithfulness.

W3: It is not wholly clear if the report card works for advanced topics, MMLU while it covers a variety of topics, is not a benchmark covering all the depth and breadth of any one subject.

To show that Report Cards works for advanced topics, we sample from the most difficult problems in the MATH dataset (level 5).

Note that due to computational constraints during the brief rebuttal window, the results below use gpt-4o-mini as the guesser to assess the specificity and the faithfulness instead of Llama-3.1-405B (as in our paper), while gpt-4o-mini is a good indicator, the accuracies would be higher with a stronger guesser. To maintain methodological consistency in our paper, we decided not to include these results in the current version, as they were obtained using a different guesser than our main experiments. We plan to replicate these experiments using Llama-3.1-405B when we have access to adequate computational resources.

Thank you for your valuable review, and criticisms. We appreciate your detailed and informative concerns. Below, we address the comments on Weaknesses (W).

Note: We've marked "BghS" next to relevant changes in the general comments to make it easier for you to navigate.

W0. The utility of these report cards is very unclear, the few concrete examples given in the paper give little confidence that any actionable insights can be derived.

While we agree that our paper does not provide many examples of actionable insights, this was not our stated intent, and we respectfully disagree with the reviewer’s characterization of the utility as “unclear”. As noted in our work, quantitative metrics such as MMLU accuracy provide limited insight into underlying capabilities, and some form of accompanying, interpretable explanation is necessary for humans to understand the causes of the quantitative performance. We propose Report Cards, as they possess several advantages over sample-based explanations (e.g., the few shot baseline in our work). Our experiments are not intended to prove actionable insights (the existence of which is independent of the quality of the qualitative evaluation), but rather to show specificity, faithfulness, and interpretability, which we argue is more important for purposes of evaluating qualitative evaluations (in particular, we disagree that immediately actionable insights are necessary for utility; see e.g., the GPT-4 System Card or the Sparks of AGI Paper — understanding behavior is, in and of itself, a utility).

We elaborate on your more specific criticisms below.

W1. A much more concrete way of [trying to identify useful subtopics for a task, and describe a model's performance on such sub topic] …

If we understand correctly, you are proposing that given a dataset, we identify a subset $T$ of questions that pertain to some semantically meaningful topic (e.g., combinatorial reasoning) and then see whether a quantitative metrics computed on the training subset $T_{\text{train}}$ generalizes to the test subset $T_{\text{test}}$. However, the $T_{\text{train}}$ performance will always generalize to $T_{\text{test}}$ long as the method of defining subset $T$ is the same for both the train set and test set, so we believe this suggestion would not prove anything about the quality of Report Cards.

While your method would allow us to obtain model-specific accuracy figures for certain subsets, which naturally meets our “specificity” requirement, this still leaves open the question of whether the subset identified is faithful to its natural language description.

If we have misinterpreted the suggestion, we would appreciate your clarification.

W2 “Contrastive Accuracy” has confounders; confusion of $k$ in the few-shot baseline

Besides stylistic features, which we address in our experiment, could you please identify the confounders you are concerned with? Regarding the confusion in the size of quizzes $k$:

• In the contrastive task, the guesser LLM receives two Report Cards (one for each student LLM) and two sets of responses (one from each student in random order) to a quiz consisting of $k=3$ questions. For each topic and student pair (72 pairs in our paper), we perform this task 120 times. As we mentioned in the paper, the accuracy for each topic is averaged over 8,640 samples for each topic, and the variance is negligible.
• We’ve revised the few-shot baseline description. The few-shot baseline involves $l$ examples of model responses. These $l$-shots are then used in place of Report Cards, as a baseline, in both the contrastive task and the Elo task. We apologize for the confusion.

W3. As a measure of report card utility, [contrastive accuracy] seems quite farfetched

As noted above, the contrastive metric is not a measure of the utility of Report Cards, but rather a measure of the specificity of qualitative evaluations. The specificity is defined as the nuances specific to the model, and by contrasting and matching Report Cards with correct responses, we can show how good our Report Cards are capturing such nuances.

W4. Card Elo is not a substitute for the raw scores.

Similarly, the Card Elo task is a measure of the faithfulness - how good do qualitative evaluations capture the true model capability - instead of a substitute of the raw score. By comparing with ChatBot Arena Elo, we are showing that Report Cards are better in capturing true capabilities of models on a fine-grained level. We are not trying to rank models based on Report Cards on a dataset/benchmark where quantitative scores are available.

Thank you for your valuable review and criticism. Please find our comments below.

Note: We've marked "EuP9" next to relevant changes in the general comments to make it easier for you to navigate.

W1. Other works focusing on qualitative evaluation

Thank you for bringing [1] to our attention. The Self-Challenge framework emphasizes exposing model failure patterns and generating an evaluation dataset that poses challenges for frontier models. Rather than focus on creating a challenging dataset that can be used to identify failure modes, our focus is on fine-grained qualitative evaluations that can be used to compare and contrast the strengths of different models—-we are not aware of other works that propose to do this in an automatic way. Indeed, the evaluation set produced via Self-Challenge (which is evaluated quantitatively) might be used to create complementary Report Cards that summarize relative differences between models. We leave this for future work, but will include the citation to [1] in our paper. We also note that a large part of our contribution is our approach to evaluating the qualitative evaluation (i.e., the 3 metrics that assess specificity, faithfulness, and interpretability of qualitative evaluations).

W2. Limited task selection; cannot be easily extended to generation tasks

Generation tasks where LLMs produce extended and open-ended are hard to evaluate. However, we’ve made progress toward such open-ended tasks with no ground truth answer. The Chinese grammar correction dataset (Appendix Section B.5) contains a suggested correction for each erroneous sentence, but there usually are multiple ways to do the corrections. In our experiments, we chose to exclude the suggested corrections from both the PRESS algorithm and Report Card assessment tasks like the contrastive tasks. Therefore, this task is open-ended. Our results in Figure 7 validate show that Report Cards maintain high faithfulness even for such open-ended generation tasks.

Since Report Cards are designed to evaluate specific model capabilities against some criteria (or sub-topics), evaluating completely open-ended generation tasks is a significant challenge without an established rubric. This is why we limited our scope to tasks that maintain some degree of structured evaluation criteria, such as the Chinese Grammar correction task, rather than including purely open-ended generation scenarios.

W3. unreliable human evaluation

As shown in the Appendix Section E, volunteers need to read the question, model’s answer, and an excerpt of a Report Card for each question, which is time-consuming and difficult to administer at scale. For this reason the collection process is rather slow, and we had to choose between collecting labels for more questions or collecting duplicate labels in order to compute human agreement. We agree that human agreement would be an asset here, and are in the process of collecting more human annotations. As of this response we have collected 100 more labels (330 in total now), and we are confident that we will be able to collect sufficient annotations to compute statistically meaningful agreement statistics.