Metritocracy: Representative Metrics for Lite Benchmarks

We thank the reviewer for the detailed list of questions. Some of these questions seem to stem from misunderstandings or are already addressed in the paper, as we explain below.

1. The paper highlights the lack of a formal definition for "representativeness" in metric subset selection, while existing lite benchmarks [...] rely on manual curation. What specific flaws in representativeness do these existing subsets exhibit? For example, are there systematic under-representations of certain alternatives? Does the paper quantify these flaws using empirical data?

Given the lack of existing formalism for representation of metrics, our paper serves three purposes: with our new formalism, (1) there now exists a quantitative method to check to what degree existing manually curated metrics are representative, and (2) if not representative enough, an algorithmic intervention to add new metrics to improve representation. Additionally, in domains where existing manually curated “lite” metric subsets don’t exist, (3) our algorithm can propose a subset of metrics from scratch that can serve as a base to seed additional manual curation from domain experts, thus saving human hours.

Regarding existing manually curated metrics: we indeed demonstrate empirically that existing manually curated metrics may not be representative by the definitions we provided, depending on the tolerance for error. The dashed lines in Figures 1 and 2 show the minimum group size $g$ and tolerance $\epsilon$ for which the manually curated subsets satisfy the representation definition. For any $g$ and $\epsilon$ lower than that, the manually curated sets would not be representative.

We also show that it’s possible to significantly improve representation by adding relatively few additional metrics using our proposed algorithms: for example, Figure 2 shows that on the HELM dataset, $\epsilon$ can be halved by adding four metrics to the existing manually curated lite set.

2. The paper draws parallels between LLM evaluation and hospital quality assessment [...]. However, LLM metrics and hospital metrics differ significantly in semantics, distribution, and evaluation goals. How does the paper justify generalizing the proposed framework across such diverse domains? Are adjustments to parameters (e.g., or ) needed for domain-specific contexts?

The core evaluation goals are actually quite similar between public LLM evaluation platforms and the hospital honor roll: the evaluator wants to reward and encourage good performance, and the evaluator wants the evaluations to be useful in some sense to stakeholders, like users or patients. Of course, how “strict” a notion of representation an evaluator needs absolutely depends on the specific application, and the parameters $g$ or $\epsilon$ must be chosen by the evaluator to reflect this. It’s possible that stricter representation is more important in the healthcare domain: if $\epsilon$ is not low enough, a high-performing hospital might not make it onto the honor roll, and a hospital might get on the honor roll that actually doesn’t perform well. Whether one would use PR or PP also depends on the domain: with LLMs leaderboards, PR might be enough to ensure that LLMs that are good across a variety of tasks are ranked highly in the lite benchmark set. On the other hand, PP might be important in healthcare to ensure that one can identify hospitals that perform poorly using the lite metric set. To summarize, our framework and results provide sufficient flexibility to apply to diverse domains.

3. Positional Representation requires that if an alternative is ranked in the top $r^2$ positions by $\ell \cdot g$ metrics in N, it must be ranked in the top positions by at least metrics $\ell$ in $K$. Why is chosen as the positional mapping? Is this choice theoretically validated or empirically tested? How would switching to a linear relationship affect the theoretical bounds (upper and lower)?

Positional Representation as defined in Definition 1 says that if an alternative is ranked in the top $r$ positions by $\ell \cdot g$ metrics in $N$, then it must be ranked in the top $r$ positions by at least $\ell$ metrics in $K$. This is already a linear relationship (and we don't believe we use $r^2$ anywhere in the paper). A linear relationship is conceptually natural because it guarantees proportional representation for every position. Finally, we prove (Theorems 2 and 3) that the linear relationship is (almost) tight, so in this sense the choice is indeed "theoretically validated."

4. The paper claims that Positional Proportionality approximates any social choice scoring rule with an error bounded by $\epsilon$. However, scoring rules weight ranks differently, Borda assigns higher weights to top positions. How does Positional Proportionality adapt to such weight variations? Are there scoring rules that cannot be effectively approximated?

Theorem 7 states that positional proportionality guarantees an additive approximation of the scores of every normalized scoring rule. Informally, this means that for any alternative, that alternative's score when evaluating the scoring rule on $N$ is within $\epsilon$ of that alternative's score when evaluating the scoring rule on just $K$. Therefore, every normalized scoring rule (including Borda) can be approximated, no matter how the scoring rule weights the ranks.

5. The greedy algorithm for Positional Representation Alg 1 uses a "coloring" step[...]. What is the rationale for this coloring strategy? Could alternative strategies, e.g., grouping by rank positions, yield better subsets?

The greedy algorithm is guaranteed to return a subset $K$ that satisfies positional representation as shown in Theorem 3, and by Theorem 2 this is within a logarithmic factor of being optimal. Therefore, no other algorithm can in the worst-case yield better subsets than Algorithm 1. The coloring approach is motivated by the greedy algorithm for set cover in theoretical computer science, which solves a related problem and gives strong approximation guarantees.

6. Positional Proportionality uses an additive error $\epsilon$,[...]. Why is additive error preferred over multiplicative error? In domains like hospital quality assessment, relative errors in metric proportions may be more critical—does the framework need extension to support multiplicative error?

If we were to require a multiplicative factor of approximation, then there exist instances of the problem where no subsets of $N$ except for $N$ itself actually satisfy the property for any $\epsilon \le 1$. Formally, suppose we want that $|\frac{C(N,r,a)}{|N|} - \frac{C(K,r,a)}{|N|}| \le \epsilon \frac{C(N,r,a)}{|N|}$. Then suppose we have that $n = m$ and that every alternative is ranked first by exactly one benchmark. In order to satisfy the multiplicative property described above, we must have that each alternative is ranked first by at least one benchmark in $K$, which requires that $K = N$. This example shows that a multiplicative form of PP is too strong of a requirement.

7. The paper notes that Positional Representation and Positional Proportionality are "mutually non-implied,"[...]. However, in scenarios with highly conflicting rankings, which property better ensures representativeness? Can experimental data compare their applicability?

For this problem, there is no notion of "better" representativeness. Positional representation and positional proportionality are two different types of representativeness, each with different interpretations and theoretical guarantees described in the paper. In Figures 1 and 2, we have experimental results on the same data sets for both notions of fairness, but in the end it is up to a user to decide which is the form of representativeness they desire.

8. Figure 1 shows "minimum for existing subsets", but the calculation method is unclear. How is the smallest for which an existing subset satisfies Positional Representation determined? Does this account for biases in manual curation?

For a given existing subset of metrics $K$ (e.g. $K =$ the existing BIG-bench Lite metrics), the “minimum $g$ for the existing subset” plotted in Figure 1 is the minimum group size $g$ such that Definition 1 is satisfied. That is, we plot $\min\{g : \forall r \in [1:m], \forall a \in A, C(K,r,a) \ge \left \lfloor \frac{C(N,r,a)}{g} \right \rfloor\}$. Code for calculating this is included with the code submission, and is simply done by binary search over $g$. Any biases in manual curation will be reflected in the resulting minimum value of $g$: a more “biased” manual curation (relative to the existing PR definition) will end up with a higher minimum value of $g$.

9. What criteria guided the selection of [the additional metrics in the hospital quality case study]? Was domain knowledge incorporated? Would results remain robust if selection were fully algorithmic?

The additional metrics were selected by solving the following optimization problem: What is the minimal set of additional metrics that can be added to the existing 12 metrics such that PP or PR is satisfied for each given value of $g$ or $\epsilon$? This is solved using the same algorithms proposed (Algorithm 1 and integer programming).

The algorithmic selection of these additional metrics can be seen as a hybrid approach that uses our algorithms to augment domain knowledge rather than replace existing human metric selection processes. We emphasize that we do not argue that human curation should be replaced, but rather that our algorithm and formalism can be an effective tool to support domain experts. A comparison to fully algorithmic selection is also included in Figures 1 and 2, which we show is able to achieve stricter representation using fewer metrics. This fully algorithmic selection can be used to create a starting point for a Lite metric set from which human domain experts can build.

In the paper, different notions of positional representation typically look at item-wise guarantees. Is it possible to extend the techniques in this work when we require, for instance, pairs of items to be treated similarly by the sub-small subset of metrics selected and the large input subset of metrics?

This is an interesting question, and the answer depends on what you mean by "treated similarly" for a pair of items. For example, suppose we have alternatives $a$ and $b$ that are 'clones' of each other so they are ranked adjacent to each other in every benchmark. Then positional proporionality immediately guarantees that the fraction of time $a$ and $b$ are ranked at position $r$ in the subset $K$ will be within $2\epsilon$ for every $r$. The same will approximately hold even if $a$ and $b$ aren't perfect clones of each other and are only approximate clones of each other. Now if we want to guarantee that $a$ and $b$ have a similar representation in $K$ when $a$ and $b$ have very different rankings in $N$, then that becomes much more difficult. In fact, $a$ and $b$ having similar representation in $K$ while having different representation in $N$ would directly contradict our notions of position proportionality and position representation. Therefore, such a guarantee would not be compatible with our notions of fair representation.

Since the notions of representation studied in this work assume that the underlying metrics output a ranking, I think they cannot directly be applied to metrics which output scores for individuals, such as Borda. Can the authors comment on extensions to such score-based voting rules?

Any metric that outputs scores for individuals can be directly converted to a ranking (by score), in which case the notions of representation studied in this work can be directly applied. That being said, doing this effectively "throws away" any information in the scores for a metric. Another option is, if a metric involves a Borda count generated from a bunch of submetrics, then we can instead guarantee positional representation/proportionality with respect to the entire set of submetrics.

We do also have results in our paper for score-based aggregation rules of benchmarks; see Theorem 7, which informally says that if $K$ satisfies position proportionality, then any scoring rule (e.g. Borda) evaluated on $K$ will be a good approximation for the same score-based rule evaluated on $N$.

How would the framework extend to settings where metrics have differing levels of importance, quality, or computational cost?

This is a great question, and our results can in some cases incorporate these additional aspects. Suppose that instead of caring about the size of the subset of metrics, we instead have a computational cost for each metric, and our goal is to keep the total computational cost below some threshold. In this case, a user could easily modify the integer programs in Appendix A to include weights in the objective function. For a more computationally efficient solution, you could also run the greedy algorithm and select the set that gives the lowest cost per new color (this mimics the greedy algorithm for the weighted set cover problem). This algorithm would run in polynomial time, and give a $\log(mn/g)$ approximation to the optimal. However, depending on the costs of the benchmarks, there no longer exists a nice guarantee for the total cost of $K$ as in Theorem 3. We are happy to include further discussion of this in the final version of the paper.

If there is an importance weighting of benchmarks then we functionally have two objectives: minimizing the size of $K$ and maximizing the sum of importance weights. Our results do not directly generalize to this multi-objective setting — this is an interesting question for future work.

Can fairness, diversity, or cost-sensitive constraints be integrated into the subset selection procedure (e.g., ensuring representation across language groups or domains)?

Yes! Using the generalized properties defined in Section 4, a user can use our subset selection procedure for any general form of representation. As in your example, we could choose the collection of groups $\mathcal{G}$ to include groups of different languages, in which case any set $K$ satisfying generalized representation/proportionality will ensure repesentation for every language. Similarly, if we choose $\mathcal{G}$ to include sets of benchmarks from different domains, then these properties guarantee representation for these groups. See Lines 293-298 for some explicit examples of how we can use these general properties. All of the results from Sections 2 and 3 generalize to this broader setting, which we show rigorously in Appendix B.