We thank the reviewer for the helpful and insightful comments! Please see our general response above and our responses to the reviewer’s main concern below:

Skill picking: Section 3.1 (and the appendix) describe how the authors pick the skills by hand based on textbooks on logical reasoning, rhetoric, and theory of mind. But what is the justification for using these types of skills? For example, i’m sure there are many skills not related to any of those topics which still require some sort of intelligent capability to combine meaningfully. The authors should explain this better.

We apologize for any confusion. The evaluation tests skills that are important in intelligence but is not meant to be a complete test of intelligence. Please see the response to all reviewers Q3.

In Appendix C.2, we discuss details of our skill selection process and how we remove skills from the original large set due to various reasons. Conceptually, the choice of topic should not affect the use of language skills. The capability to combine seemingly unrelated skills and topics is indeed what Skill-Mix aims to test.

We thank the reviewer for the helpful and insightful comments! Please see our general response above and our responses to the reviewer’s main concerns below:

I'd love to get some suggestion on how this framework can be extended beyond language to Vision - Language Models.

We sketch an example of how to extend Skill-Mix to evaluate vision-language models below:

• The set of skills can include both vision skills (e.g., object recognition, object generation, 3D reconstruction, spatial relation between objects) and language/reasoning skills (e.g., description of the scene, object interaction, reasoning of past and future events).
• The set of topics can be the environment or background of the image.
• Task can either be answering questions based on given images, or generating images based on instructions (the instruction may include images as well).
• The images fed to models (as part of the prompt) can be generated by graphics programs.

I'm curious to see how the experiments with fine-tuning while releasing 10% of skills pan out.

We include some preliminary results on finetuning LLaMA2-7B on 50% skills and topics, and observe improvements across the various Skill-Mix metrics. We are excited to release a more detailed exploration of this in future work! Please also see our response to all reviewers, Q2.

Can you please provide reference for "While both models are creditable graders, they, like all current LLMs, were unreliable at simple arithmetic, which is important for calculating a total score”

We provide some references below. We also include our experience in Appendix C.3.3.

[1] Exposing Attention Glitches with Flip-Flop Language Modeling Bingbin Li, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang

[2] Table 10, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, Denny Zhou

Below, we address the other concerns brought up.

I'd love to see some correlation of the presented results with human spot-checking and evaluation and if such correlation does not exist, discussion on what might be the reason.

We thank the reviewer for bringing up this point. In Appendix B, we mentioned that the difference between GPT-4 grading and the average human grading is ~0.25, meaning that the human and GPT-4 agree on most of the points. We will include details in the final version.

Do you think that instead of having one strong evaluator, you can have a mixture of experts each capable to evaluate a single task exceptionally well?

This is a good point! If one decides to fine-tune the grader, it is unclear to us whether it is better to fine-tune one model or use a mixture of fine-tuned models. One concern of using a mixture of experts is that some experts may be harsher than others. However, in terms of human grading, we believe it would be more efficient to split skills and assign them to specific graders. We leave this a future work to explore.

We thank the reviewer for the helpful and insightful comments! Please see our general response above and our responses to the reviewer’s main concerns below:

lack of IRB or ethics approval

We thank the reviewer for bringing up this point. The human graders were carrying out research too and we understand IRB is not required in such cases. The final version will fully explain this status.

the paper doesn't make it clear … how these 7 desiderata are related to the 3 pressing issues with current evaluation methods.

Please see our response to all reviews, Q1.

no reason to believe that, once released, skill-mix won't also suffer the problem of 'cramming for the leaderboard'

We release 10 skills and 10 topics (randomly sampled) to avoid “cramming” for Skill-Mix. For more detailed explanations, please see our response to all reviewers, Q2.

The theoretical justification for skill-mix is also unclear to me. For instance - does it make sense to combine any k of the N skills together, or are there some incompatible skill-tuples? The connections with pedagogy theory and the empirical observations from the Arora & Goyal 2023 paper are a good starting point, but it's not clear in what way combining skills relates to generalized intelligence.

We apologize for any confusion and would like to emphasize that as of now, the goal of our evaluation is to test general-purpose text generation capability, rather than generalized intelligence. Please also see our response to all reviewers, Q3.

With respect to incompatible skill tuples: We eliminated skills that in our judgement inherently compose poorly with other skills (see Appendix C.2). It is possible that this intuitive weeding still left some pair of skills that don’t compose well. But this would equally affect the final performance number of all models, and not be expected to affect the ranking.

I like the core idea of N-choose-k as an evaluation approach that scales quickly with k. The paper and results gave me the impression that you only consider linguistic skills, which seems like a big limitation.

For our first version of Skill-Mix, we acknowledge that we only use language-related skills. However, we note that Skill-Mix can be expanded to other types of skills (e.g., domain-specific skills related to coding, science, economics, math, law, etc).

Below, we address the other concerns brought up.

Some 'late breaking' paper results are missing from the submitted version (e.g. footnote 3, Sec. 6) - the paper will be strengthened with the inclusion of these results in a future version.

We thank the reviewers for pointing this out, and have updated these results in our revised version of the paper. In particular,

• For k = 5 and k = 6, we show that GPT-4 is generating correct answers a good fraction of time, and a reasonable portion (say more than 1/3) of these correct answers contain combinations of skills and topics that have never jointly appeared in the training.
  • We verify this claim by providing upperbounds on the average frequency of skills, the average frequency of topics, and the number of sentences in the training corpus (see Sec. 6 and Appendix D).
• We include preliminary finetuning results for LLaMA2 7b in Appendix E.

(End of Sec. 5) "Difference between model ranking on popular LLM leaderboards vs. SKILL-MIX provide evidence of "cramming for the leaderboard", further validating that SKILL-MIX is a good evaluation benchmark" - the conclusion here doesn't follow from the premise.

We apologize for any confusion. We meant to convey that we believe Skill-Mix is a good evaluation benchmark because model rankings on Skill-Mix coincide with anecdotal human experience with these models (and that this is not the case with model rankings on e.g., Open LLM). Many derivatives of Llama2 models scored higher than the Llama2 model on the Open LLM leaderboard but performed much worse on Skill-Mix, suggesting their general-purpose language skills were harmed along the way.