Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks

We would like to warmly thank reviewer R5ne for their review we are glad that they are excited by the paper and find our contribution strong and novel.

Below we answer the questions of the reviewer:

1. On the choice of Borda aggregation/lack of ground-truth/theoretical advantages of Borda. To comment on the validity of using the ranking as ground truth. It's important to recognize the absence of a definitive ground truth ranking. Indeed, the complexity of social choice theory, exemplified by Arrow's impossibility theorem and the Gibbard–Satterthwaite theorem, underscores the inherent challenges in establishing a universally satisfying and consistent ranking system within the specified criteria.

In this paper, we choose to sum the ranking because it is exactly the Borda Count. The Borda Count is a 2 approximation of the Kemmeny consensus [1] and is an excellent choice in practical applications (see [2,3]).

2. We did not investigate the impact of misleading evaluation results in this paper, but we do agree this is an interesting question. We did add it to future work (see Appendix).

3. On the robustness analysis in the experiments. $\eta$ varies from 0 to 0.3 and the corrupted tasks are randomly chosen. The change in the final ranking is measured by the Kendall $\tau$. We did add a clarification in the main paper (see updated manuscript).

4. In Figure 5. We agree that the label of the x-axis was misleading. We have updated it by setting it to 1- \eta to make it coherent with the discussion. We attribute the decline in performance on Flickr to the data distribution. However, our investigation into the score distribution did not reveal any discernible differences.

5. In Tables 1 and 2, we quantified the variation in rankings by calculating the average Kendall's tau (τ) between the generated rankings as we varied the proportion of missing rankings. We have included a clarification on this point in the revised version of the manuscript.

6. We added the comparison with ANOVA in the future work section. We agree with the reviewer this is outside the scope of the paper.

References

[1] John G Kemeny. Mathematics without numbers. Daedalus, 88(4):577–591, 1959.

[2] Alnur Ali and Marina Meila. Experiments with kemeny ranking: What works when? Mathematical Social Sciences, 64(1):28–40, 2012.

[3] John J Bartholdi, Craig A Tovey, and Michael A Trick. The computational difficulty of manipulating an election. Social Choice and Welfare, 6(3):227–241, 1989.

We would like to warmly thank reviewer qUQT for carefully reading our manuscript and for their enthusiasm about our work. We indeed hope that our work will be widely adopted by the community as we firmly believe it provides a more robust way to evaluate NLP systems.

Below is a response to the reviewer's question:

1. About the baselines. To the best of our knowledge, this is the first work that addresses this issue of benchmarking in the presence of missing data for NLP systems. This is indeed a strong contribution to the paper.
2. About the correlation between tasks. Numerous studies in the literature explore the correlation between metrics (see [1] for example). We did consider this aspect in the early stages of the real data analysis, however, delving into it extensively proved challenging. Additionally, given the amount of experiments in the paper, we believe this would require work on its own. We do agree this would be an interesting follow-up work and added it to the paper's next research directions (see updated version of the manuscript). A promising avenue for studying this problem could lie in investigating confidence intervals. If all metrics exhibit strong correlations, it could lead to a reduction in the size of the confidence interval—a potential starting point for a more in-depth examination.

References:

[1] Colombo, P., Peyrard, M., Noiry, N., West, R., & Piantanida, P. (2022). The glass ceiling of automatic evaluation in natural language generation. Findings AACL 2023.

We hope our answers address all concerns of reviewer qUQT and we hope they would be keen to consider raising their score.

We would like to warmly thank reviewer qUQT for their review. The main concerns of the reviewer are focused on the novelty and the choice of the baseline.

On the novelty.

The first novelty of the paper is to identify and formalize the problem of benchmarking in the presence of missing data. Previous work in NLP often simply ignores the missing data when choosing the best system (see Pfeiffer et al., 2022; Lin et al., 2022; Martin et al., 2020; Guibon et al., 2021; Peng et al., 2019). This is mainly due to the difficulty of collecting data/private datasets or more recently the cost of running expensive models such as GPT.

To the best of our knowledge, this is the first paper to tackle the problem of benchmarking with missing data in the NLP community.

On the technical contribution.

Our work builds on Colombo et al 2023. However, the addressed problem is different as such their method cannot be naively applied. Second, it seems that the reviewer missed the technical aspect of our paper. Specifically: we introduce a new ranking algorithm for missing data. Our newly designed method can be adapted to both instance- and task-level aggregations. This method is not trivial and relies on counting the number of system orderings compatible with a partial rank. We respectfully refer the reviewer to Section 3. Our algorithm’s complexity is polynomial $O(n^3)$ while the naive version would be factorial. Additionally (see response to cmy7) our estimator is unbiased. We introduce confidence intervals for the ranking see Section 3.3 to better understand the uncertainty linked to the ranking.

Additionally, we conducted an extensive data collection and gathered a new dataset with a public release. Our effort is crucial for future work. Concretely, our dataset gathered over 131M score which is an order of magnitude larger than all existing datasets. We believe that this will spur more research in benchmarking, a critical area in NLP, especially considering the surge in Generative Models.

On the choice of the baseline: To the best of our knowledge, this is the first work that addresses the issue of benchmarking in the presence of missing data for NLP systems. This is indeed a strong contribution to the paper.

On the readability issue. We have diligently incorporated the reviewer's feedback into our manuscript. This includes rectifying typos, addressing citation issues, and increasing the size of figures as suggested.

We answer the questions of the reviewer below:

1. On the toy experiments. The selection of these two factors, namely robustness to scaling and confidence, stems from the specific demands of NLP evaluation. In the realm of generative models, NLP practitioners frequently encounter metrics on diverse scales, some of which may even be unbounded, as exemplified by BartScore. Furthermore, practitioners often neglect confidence intervals, making this paper's focus on them a notable contribution. Given the already dense nature of the paper with numerous experimental results, our decision to explore additional factors, such as robustness to noise in real data experiments, serves to further substantiate the efficacy of our approach on authentic rankings.

2. On the difference between $\sigma^l$ and $\sigma^{2l}$. It's important to recognize the absence of a definitive ground truth ranking. Indeed, the complexity of social choice theory, exemplified by Arrow's impossibility theorem and the Gibbard–Satterthwaite theorem, underscores the inherent challenges in establishing a universally satisfying and consistent ranking system within the specified criteria. However, it is worth noting that the proposed method is more robust than the widely adopted method (namely the mean aggregation in NLP) on all the considered datasets.

We hope our answers address all concerns of reviewer yPnM and we hope they would be keen to consider raising their score.

We thank reviewer cmy7 for their careful reading of the manuscript. We are glad they acknowledge that they found the problem interesting and impactful. We are particularly thrilled by their positive assessment, like our technical contribution, and find that “the method is clever”.

Below is a response to the reviewer's concerns:

1. On the method. The reviewer expresses concern regarding the appropriateness of assuming uniformity for unobserved rankings. In response, we demonstrate that in the realm of ranking data, unlike real-valued data, the estimation process remains consistent despite considerable noise. The conventional proofs affirming the consistency of the estimator for the Borda algorithm under various assumptions rely on demonstrating that for any pair $i,j$, if the data is drawn from a model satisfying the strong-unimodality property (such as Plackett-Luce or Mallows model just to name a few popular ones) with a median $\sigma_0$ for which $\sigma_0(i)<\sigma_0(j)$, then $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, and thus, the Borda algorithm accurately ranks such pairs [1,2,3] Here, we demonstrate that if Borda accurately ranks the pair $i,j$ without noise, i.e., if $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, then it will also correctly rank $i$ and $j$ in expectation, given an equal noise rate $\eta$ for both $i$ and $j$. To see this, note that the expected imputed ranking value will be the average of all possible rankings, $0.5n$. Therefore, the expected ranking of $i$ will be equal to $\mathbb{E}[0.5n\eta + (1-\eta)\sigma(i)] = 0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)]$, where the equality holds by linearity of the expectation. By noting that $\mathbb{E}[\sigma(i)]\mathbb<{E}[\sigma(j)]$ implies $0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)] <0.5n\eta + (1-\eta)\mathbb{E}[\sigma(j)] = \mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$ we conclude the proof.

References:

[1] Irurozki, E., Perez, A., Lobo, J., & Ser, J. del. (2021). Online Ranking with Concept Drifts in Streaming Data. Joint European Conference on Machine Learning and Knowledge Discovery in Databases.

[2] Fligner, M. A., & Verducci, J. S. (1988). Multistage Ranking Models. Journal of the American Statistical Association, 83(403), 892–901. https://doi.org/10.2307/2289322

[3] Caragiannis, I., Procaccia, A. D., & Shah, N. (2013). When Do Noisy Votes Reveal the Truth? Proceedings of the Fourteenth ACM Conference on Electronic Commerce, 143–160. https://doi.org/10.1145/2482540.2482570

2. On the readability issue. We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

3. On the two argsorts. The argsorts are used to convert the combined scores of the systems into rankings. For instance, scores [1.0, 3.5, 2.0, 2.2] would be transformed into [0, 3, 1, 2], indicating that system 1 is ranked first, system 2 is ranked fourth, system 3 is ranked second, and system 4 is ranked third. The initial argsort provides an ordering of systems based on their scores, while the second one generates rankings of the systems relative to this ordering.

We hope our answers address all concerns of reviewer cmy7’s and we hope they would be keen to consider raising their score.