A Statistical Framework for Ranking LLM-based Chatbots

We appreciate your thoughtful feedback, which has helped us refine and strengthen the manuscript. Below, we address each of your comments in detail:

Variability in Rankings and Relationship with LLM Characteristics

The observed stability of rankings for high-scoring LLMs and variability among lower-ranked ones appears predominantly data-driven rather than tied to the intrinsic characteristics of the LLMs themselves. In the Chatbot Arena framework, high-ranking models benefit from substantially more pairwise comparisons, with the top 50 LLMs receiving one to two orders of magnitude more matches than lower-ranked competitors. This abundance of data simplifies the ranking task for statistical models, fostering agreement among different methods.

High-ranked LLMs consistently dominate in match outcomes, and the availability of sufficient data ensures that statistical models with differing assumptions converge to similar rankings. In contrast, lower-ranked LLMs, which are paired less frequently, face limited data availability, leading to greater variability in their inferred rankings. This sparsity makes the rankings of these models more sensitive to the assumptions of specific statistical methods, allowing for greater model-specific interpretations. Models with more parameters (e.g., those incorporating tie modeling and Thurstonian covariance) are better equipped to capture nuanced relationships and uncertainties in pairwise data, thanks to their improved fit to the data.

We also found the reviewer’s suggestion to explore the role of LLM characteristics in relation to ranking insightful. To investigate further, we've added a detailed analysis in Appendix F. This section examines the relationship between scores from our framework and three key LLM attributes: number of parameters, computational budget (FLOPs), and dataset size. Using data from Epoch.ai matched with our models, we summarize regression results in a table and provide scatter plots illustrating the log-linear relationships between these characteristics and the scores. Our findings reveal consistent positive associations across all three characteristics, with computational budget and dataset size exhibiting stronger trends, while the number of parameters shows a comparatively weaker association. These observations align with prior studies on scaling laws (e.g., Kaplan et al., 2020; Hoffmann et al., 2022), which emphasize the importance of scaling compute and data alongside model size for optimal performance.

Overall, while variability in rankings is primarily data-driven, this additional analysis highlights intriguing patterns in how LLM characteristics relate to their performance scores.

Effectiveness of Handling Ties

Incorporating ties into our statistical models significantly enhances predictive performance, as demonstrated both visually and numerically in our manuscript.

Figure 1 (formerly Figure 2 in the appendix) compares pair-specific win and tie probabilities among 25 competitors. The first column shows empirical data, while subsequent columns depict predictions from various models: our generalized Rao-Kupper model with factored ties (second column), the original Rao-Kupper model with ties (third column), the Bradley-Terry model with ties as half win/loss (fourth column, as used in Chatbot Arena), and the original Bradley-Terry model without ties (fifth column). Notably, our generalized model (second column) aligns closely with the empirical data, capturing both tie and win/loss predictions with fine-grained accuracy.

This strong visual alignment is supported by numerical results in Tables D.1, D.2, and D.3, where models incorporating ties consistently achieve improved metrics, including lower negative log-likelihood, cross-entropy, RMSE error, and KL and JS divergences. Additionally, Figure D.1 highlights a significant reduction in prediction error for marginal probabilities (both ties and win/loss outcomes), decreasing by up to two orders of magnitude with tie factor parameters.

Our analysis demonstrates that incorporating ties significantly improves model predictions, enabling better alignment with observed data.

Model Favorability to Certain LLMs

We did not conduct an analysis to determine whether specific models are more favorable to certain LLMs. Addressing this would require detailed information about LLM characteristics, such as architecture or training data, which falls outside the current scope of our work, focused on developing a purely statistical framework. However, we see this as a valuable direction for future research.

We’ve revised the manuscript to incorporate these changes and address your comments. Please let us know if there are any remaining concerns or areas where further clarification would be helpful. We appreciate your feedback and look forward to your thoughts.

If I do not receive a response to my questions, I may have to consider lowering the score for the paper. I hope to hear from you soon.

We appreciate your thoughtful suggestions—they’ve helped us improve the manuscript significantly. Below, we’ve addressed your comments by expanding our analyses and providing additional insights.

Examination of Ranking Consistencies and Model Characteristics

To address your suggestion regarding ranking consistencies and their relation to model characteristics, we’ve added Appendix Section F: Relationship Between LLM Characteristics and Scores. This section explores the correlation between performance scores and key LLM attributes, such as the number of parameters, computational budget (FLOPs), and dataset size. Our analysis reveals strong correlations between scores and both computational budget and dataset size, offering insights into the factors driving model performance variability. These results align with established trends in scaling laws and provide actionable insights into the observed consistency of high-ranking models.

Analysis of Ranking Variability Across Models

In response to your feedback on examining ranking differences across models more deeply, we expanded the comparative analysis in Appendix Section E: Comparative Analysis of Ranking Variability, which now includes:

• Quantifying Ranking Similarity Across Models (Section E.3): We systematically quantified ranking similarities across all 30 models using Kendall’s ranking correlation. Notably, Model 1 in our analysis corresponds to the Elo-based ranking algorithm employed by Chatbot Arena. This provides a direct comparison between Leaderbot rankings and Arena’s Elo rankings, offering a more rigorous analysis beyond the visual bump chart presented in the original submission.

• Identifying Ranking Similarities via Hierarchical Clustering (Section E.4): By clustering the Kendall correlation matrix, we uncovered distinct clusters of ranking behaviors. This analysis reveals a fascinating finding: the inclusion or exclusion of the Thurstonian covariance model is the primary driver of these clusters. These results show how covariance modeling shapes ranking consistency, complementing earlier findings on the impact of generalized tie modeling on inference accuracy.

Incorporation of Complementary Methods for Correlation Analysis

In response to your comment about exploring methods beyond PCA, we’ve expanded our correlation analysis in Appendix Section C. Specifically:

• Multidimensional Scaling (Section C.2): We introduced MDS as a complementary method to kernel PCA. MDS provides an intuitive spatial representation of chatbot relationships and, interestingly, uncovers ranking orders solely from score differences and covariance-derived uncertainties, without access to absolute scores.

• Hierarchical Clustering (Section C.3): We extended the correlation analysis with hierarchical clustering to identify performance tiers among competitors. This approach, driven by the optimal leaf ordering of the dissimilarity matrix, offers an intuitive grouping of competitors into tiers. Notably, the method recovered the rankings almost perfectly without direct access to the scores, relying on the dissimilarity matrix derived from our generalized models incorporating Thurstonian covariance. This integration of performance scores and covariance-based dissimilarities underscores the interpretability of the clustering approach.

Clarifying the Use of Semantic Clustering

Regarding the use of semantic clustering, we believe that statistical clustering techniques, such as hierarchical clustering, are well-suited for our pairwise comparison framework. Our focus was on extracting performance-based groupings and relationships that reflect the pairwise comparisons and score-based dissimilarities of the models. Exploring semantic relationships is an interesting direction, but it would require additional datasets or features, such as linguistic embeddings, which fall outside the scope of this work. To address this point, we’ve added a clarification at the end of Appendix Section C.3 to highlight this reasoning and emphasize the alignment of our approach with the statistical focus of the manuscript.

We’ve revised the manuscript to reflect these changes and address your feedback. Please let us know if there are any remaining concerns or areas where further clarification would be helpful. We’re grateful for your suggestions and look forward to your thoughts.

Thank you for your follow-up question regarding Yi-Large in Figure F.1. Following your observation, we reviewed the source data from epoch.ai and identified that the values for Yi-Large, along with a few other models, were derived from external estimates rather than directly published details. In response, we refined our methodology in Appendix F to include only models flagged as "confident" in epoch.ai's dataset, ensuring the accuracy of our analysis.

This refinement adjusted the number of LLMs in the regression analysis from 37, 34, and 28 to 31, 27, and 24 for the number of parameters, computational budget (FLOPs), and dataset size, respectively. Notably, this adjustment led to improved regression results, with stronger $R^2$, $r$, and $p$-values, indicating more reliable associations between these characteristics and the scores. Below is the updated regression Table F.1:

The updated figure F.1 is now available in our package repository (anonymized link provided on page 31 of the manuscript) under /notebooks/epoch_ai.pdf. This plot includes the confidence intervals for the regression and reflects the updated analysis. If permitted, the direct anonymized link to the file is: https://anonymous.4open.science/r/leaderbot-CA90/notebooks/epoch_ai.pdf. The manuscript has also been updated to reflect the new data in the table and this plot.

We appreciate your attention to this detail, which has ultimately improved our findings.

We appreciate your feedback and for highlighting strengths as well as areas for improvement/clarification. Below, we address your concerns and clarify points raised.

On the Absence of Ground Truth in Ranking

We agree that rankings lack a definitive ground truth, as they are inferences drawn from models and data. However, in statistical modeling, the goodness of fit and predictive performance of a model can and should be rigorously evaluated against empirical data, which serves as the ground truth for observed outcomes.

To this end, we systematically compared the predictive performance and the goodness of fit of our models against empirical data:

• Figure 1 (formerly Figure 3 in the appendix) demonstrates the predictive performance of our model in capturing tie and win/loss probabilities. Notably, our generalized model achieves fine-grained accuracy, closely aligning with empirical data, and outperforms simpler approaches, including the Elo-based method.
• Appendix D evaluates model selection and goodness of fit metrics (Tables D.1, D.2), including negative log-likelihood, cross-entropy, RMSE error, and divergences, demonstrating superior fit of generalized models.

While rankings themselves are an inference without a definitive ground truth, the framework underlying these rankings can be rigorously validated based on its ability to model and predict empirical data accurately.

Dependence on the Chatbot Arena Dataset and Broader Impact

We acknowledge the reviewer’s observation that our experiments focus on the Chatbot Arena dataset. This pioneering dataset, widely used by OpenAI, Google, and Hugging Face, is among the largest and most credible for pairwise comparisons in LLM evaluation.

While our experiments leverage this dataset, our contributions extend far beyond it. The framework we propose—incorporating ties and covariance—addresses fundamental challenges in paired comparison frameworks, applicable to diverse domains regardless of dataset. To clarify the general applicability of our work, Appendix A highlights paired comparison applications across sports analytics, marketing, psychometrics, election studies, and clinical trials.

Additionally, paired comparison methods are increasingly applied in modern machine learning, such as reinforcement learning with human feedback (RLHF) and direct preference optimization (DPO), key to fine-tuning large language models. Recent studies (e.g., Rafailov et al., 2024; Karthik et al., 2024; Liu et al., 2024) showcase the growing relevance of paired comparison models in these settings. Our contributions enhance the flexibility of these models, underscoring their potential to address emerging challenges.

Balancing Sophistication with User Simplicity

We understand the concern about balancing complexity with user accessibility. The statistical framework operates behind the scenes and does not affect the simplicity of the user interface. Rankings remain straightforward for users, supported by an intuitive Python package with an easy-to-use interface.

In practice, grounding rankings in a robust statistical framework can enhance user trust in the platform, even if the underlying methodology is sophisticated.

Visualization of Ranking Differences

Thank you for suggesting a figure illustrating ranking differences between Chatbot Arena’s method and ours. Figure E.2 (bump chart), previously included as Figure 5 in the earlier submission, tracks ranking differences across models, including Model 1, representing Chatbot Arena's Elo-based system. To clarify, we updated the figure caption to explicitly label Model 1 as the Chatbot Arena method. We are also open to including additional visualizations if you believe they would add further clarity.

Beyond the bump chart:

• Appendix E.3 quantifies ranking similarities using Kendall’s rank correlation across all 30 models, providing a systematic metric for ranking comparison.
• Appendix E.4 applies hierarchical clustering to Kendall’s correlation matrix, revealing patterns in ranking behavior and how model features drive similarities.

Interpretation of PCA in Figure 2

We acknowledge the challenge of interpreting the 3D PCA scatter plot in Figure 2 (formerly Figure 1), as overlapping points and labels limit clarity. To address this, Appendix C.2 introduces a 2D Multidimensional Scaling (MDS) plot, offering similar insights into the covariance structure with improved interpretability. As kernel PCA and MDS are dual methods visualizing the same dissimilarity matrix, both provide consistent insights. This additional visualization aims to enhance clarity while preserving the depth of analysis.

We've revised the manuscript to address your comments. If there’s anything else you’d like us to clarify or discuss further, please let us know—we’d be happy to engage. Thank you again for your feedback and for helping us improve our work.