Benchmarks as Microscopes: A Call for Model Metrology

This paper has a relatively high quality, as the authors construct a compelling case against current large language model (LLM) benchmarking practices, meticulously exposing their flaws and providing clear explanations for their shortcomings. The inclusion of real-world examples, such as the Air Canada chatbot incident, effectively illustrates the consequences of relying on flawed evaluations. By introducing "model metrology" as a novel discipline dedicated to LLM evaluation, the authors offer a persuasive solution to a pressing problem.

The paper's clarity is good enough. The logical and coherent structure of the argumentation enables readers to effortlessly follow the authors' reasoning, while precise definitions of key terms ensure a unified understanding. As a result, readers can gain a deep understanding of the challenges inherent in LLM evaluation and fully appreciate the proposed solutions.

The concept of "model metrology" is genuinely novel and sets this work apart from others although i am not an expert in this field. By challenging the status quo, the authors advocate for more ecologically valid and dynamic evaluation methods, such as using adversarial LMs to generate diverse test cases. This innovative approach has the potential to revolutionize LLM assessment and positions the paper as a contribution.

This work addresses a critical issue in AI: the lack of reliable and informative LLM evaluation methods. By proposing the model metrology framework, the authors provide a path towards a more nuanced understanding of LLMs, ultimately leading to responsible development and deployment. The emphasis on collaboration between researchers, developers, and users is crucial for building effective benchmarks and advancing the field. Overall, this position paper is promising for LLM evaluation, offering a fresh perspective and innovative solutions to a pressing challenge.

This position paper proposes a new discipline: "model metrology", aimed at generating new benchmarks for Language Models (LM) evaluation that overcome the limitations of current benchmarks (saturation, memorisation, non-constrained scope, etc.). First, the paper analyses current limitations of current benchmarks, then proposes a number of desired features for such a new generation of benchmarks: constrained, dynamic, and plug-and-play. Finally, it analyses the characteristics of the proposed "model metrology" discipline and proposes a roadmap to achieve such a vision.

The topic is a relevant and interesting one. Limitations of current benchmarks for LLM evaluation are well known, and there are active research efforts in the direction of mitigating them, to which this work could contribute. The paper contains interesting ideas (the idea of model metrology as a new discipline is appealing) and can be the basis for stimulating a debate in that direction.

However, the paper has some flaws in my view. In general, there are a number of good ideas, but some of them not enough elaborated, some of them are bit vague (e.g., section 4.3) and some others not so novel, such as the "constrained" feature (task-based and extrinsic evaluations have been around for decades). In contrast, the "dynamic" aspect is a more contemporary need in NLP. However, I miss the "human in the loop" aspect as well.

In the paper, there are too subjective claims such as "[OpenAI, Antrophic, Google] still cling to the idea that general intelligence is quantifiable by these benchmarks", which are not sufficiently supported. (Do these vendors really think so? If so, citations would be welcome).

As an example of the limitations of current benchmarks, the Air Canada Chatbot incident is presented. In fact, as stated by the authors, is "implausible that this failure case could have been predicted through generalized benchmark results". However, this does not contradict the value of such benchmarks to allow testing (and improving) the model along several specific dimensions, before integrating them in a final system. The tests of the system as a whole (model + policies specification + UI, etc.) in the Air Canada case were obviously incomplete, but this is possibly because good test engineering practices were not followed, not invalidating the current LLM benchmarks per se.

From a stylistic point of view, the writing style is a bit wordy at times and difficult to follow, lacking the conciseness of academic writing. Some grammar constructions are a bit weird, for instance in section 3: "One reason [for what?] is that static benchmarks of that form [which form?] are easily memorized, […] “, or in "outlook and roadblocks to this discussed in §4.2", where the auxiliary verb is missing. There are also many contractions along the whole text (aren't, it's, can't), which is unusual in academic written English.

Despite the extensive references section, there are some omissions that could have complemented the author's analysis, such as: Tedeschi et al. "What’s the Meaning of Superhuman Performance in Today’s NLU?", ACL 2023 (a critical analysis of superhuman claims derived from using traditional benchmarks, with ideas to incentivize benchmark creators to design more solid and transparent benchmarks).

This paper offer a critique the current paradigm of benchmarking LMs. They identify several issues with the current paradigm, including benchmark saturation, lack of ecological validity, an over-focus on benchmarking generalized capabilities. The authors make a case for a new field of study - model metrology - that focuses on studying and designing LM measurements.

Overall, this paper is well written, well researcher, and well motivated. Parts of the papers argument have been made elsewhere, but the authors bring together a broad set of literature in a well organized may to make a compelling case for concentrated focus on LM measurement.

This position paper argues that there should be a division of labor between researchers developing language models, and “model metrologist” that should devote their efforts to measure the quality of LLMs wrt their actual usage.

Thanks for your time, reviewers. Here are some parting thoughts to close out the discussion.

We appreciate your recognition of the timeliness, relevance, and importance of addressing the evaluation problem now, praising our work for “addressing a critical current issue” (nguQ) that is “relevant and interesting” (GTUZ) with “a compelling and timely argument relevant for COLM” (3uTB).

The reviewers broadly agreed with and recognized the novelty in our proposed solution to this problem—developing a new subdiscipline of model metrology. GTUZ found the idea appealing on first review, and was won over to raise their score to an accept following our rebuttal clarifying why a focused community is important. 3utb pointed out that centering “expertise outside the field of ML” as essential, and organizing an evaluation-centered cross-disciplinary community will enable this. nguQ pointed to the metrology idea as a core reason to accept for its “innovative approach” with “the potential to revolutionize LLM assessment.”

We are pleased to see such a broad range of perspectives shared by all reviewers. A great position paper cannot please everyone, as their purpose is to put forth debates that advance the field. Considering this, we are glad to see spirited debate in our reviewers’ positions. GTUZ directly recognized this strength, saying that the “interesting ideas [in our work] can be the basis for stimulating a debate” toward better practices for benchmarking.

This disparity in reviewer perspectives underlines why organizing a unified field is instrumental. For example, WdB7 noted that we did not put much focus on how the information retrieval community in particular has handled evaluation. Our authors’ are experts in other domains. The fact they overlooked IR illustrates why a dedicated metrology discipline is so important—if we only worked in a text generation silo, we may never have benefited from the “30 years of devoted work” in IR. WdB7’s excitement to share this with us will make a great case study to complement the others we planned to add following nguQ’s request in our appendix.

We appreciate all reviewer suggestions that will improve our camera ready version. While a position paper is not meant to provide a comprehensive literature review, and appreciate GUTZ’s compliment on the extensiveness of our 100 reference bibliography, we are excited to add discussion of the reviewer-suggested related works. Finally, we are excited to make the clarifications we arrived at in our initial responses to all reviewers.

Thank you all for your detailed and constructive feedback.

The topic is a relevant and interesting one, and this position paper contains interesting ideas that can be the basis for stimulating a debate in that direction. It also identifies several issues with the current approaches to evaluation/benchmarking, including benchmark saturation, lack of ecological validity, an over-focus on benchmarking generalized capabilities. The author's proposal for a new field of study - model metrology - is novel and promising. As noted by reviewers, it would be good to: -engage with other forms of evaluation and/or discussing elements of a user's experience/model capabilities that cannot be fully captured through the benchmarking paradigm. -provide more concrete examples or case studies of how these benchmarks could be designed and used in practice. -explore in more depth the practical challenges of creating and implementing dynamic benchmarks. -add missing references. It would be good to add these elements in the camera-ready version of the paper.

[At least one review was discounted during the decision process due to quality]