Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

W1 OOD Results: We understand that using the top static model might appear an intuitive baseline. However, practically, this static model is unknown ahead of time, and P2L’s value is precisely in selecting the best-performing model dynamically. This means P2L, which never sees ground truth labels or model responses, performs as good as running all models on LiveBench, scoring them using the benchmark’s ground truth labels, and selecting the best model after the fact. We will communicate this more clearly in future revisions.

W2 Baselines: During development, we attempted non-deep learning approaches such as KNN and embedding based methods. We found these did not perform better than, and sometimes worse than, the marginal baseline– with a log-loss delta of less than 0.01. Ultimately, we chose the marginal BT as baseline, since this was the most effective and stable baseline– we will ensure to communicate this in the revision. In the end, the goal of this work is to create a scalable (along data and parameter count) method to provide granular model evaluations, which our deep learning approach provides.

W3 Clustering Novelty: We are not claiming any novelty in the clustering algorithm. We are claiming that ours is the first approach that allows us to create a leaderboard for small, possibly singular, clusters, moreover doing so label-free. We will clarify in the final revision that our clustering mechanism is for demonstrative purposes and will relegate extensive discussions to supplementary materials.

W4 Fig 7: We recognize our current presentation was insufficiently clear. We will enhance the caption of Fig. 7 to explicitly state what the function distance represents, how it relates to prompt-specific leaderboards, and how this concretely supports our claim. Specifically, the large function distance between P2L’s leaderboard and the marginal leaderboards for a smaller number of prompts (<= 10) suggest P2L's predictions differ from the marginal.

W5 Uninteresting Results: We appreciate the reviewer’s feedback on Figures 6, 8, and 9, and agree that some of these results align with existing expectations—such as o1 performing well on mathematical tasks and ChatGPT-4o excelling in creative tasks. However, we argue that these confirmatory results are precisely what demonstrate the reliability and effectiveness of P2L. Moreover, P2L systematically captures these known performance trends without requiring expensive per-prompt annotations, thus validating our approach as robust and practically useful for model selection and routing. Additionally, our finding that GPT-4o-mini matches or surpasses substantially more expensive (up to 100x) models (e.g., o1 or GPT-4o) in certain prompt clusters highlights an important, practically valuable insight into cost-effective model utilization that is only clearly revealed by our method. We will revise the manuscript to more explicitly emphasize these practical insights, clarify how they support the validity of our approach, and ensure the results' practical implications are clearly communicated.

W6 Public Release: We will release all P2L models, routing code, training code, evaluation code publicly. We will host a P2L endpoint to provide a way for readers to calculate raw P2L outputs for any prompt of interest.

W7 Related Work: We will elaborate our connection to benchmark compression in related work. P2L uses deep learning to compress evaluation signals parametrically, uniquely allowing it to be label-free at test time.

W8 Figure Clarity: We will comprehensively revise all figure captions in the revision, ensuring they succinctly summarize methods, key findings, and implications. Where necessary, we will include hyperlinks to relevant text.

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

W1 Arena Score Computation: We deploy the P2L router onto Chatbot Arena, routing between the models detailed in Appendix D1. We collect blind human preference votes between responses from the P2L router against active public models on Chatbot Arena. We use these preference votes to calculate the Bradley-Terry regression over all Chatbot Arena battles to produce an Arena Score. This is the standard method to add a model to the Chatbot Arena leaderboard. We will ensure Arena Score calculations are well documented in future revisions. This Google Colab Notebook details exactly how Arena Scores are computed on the Chatbot Arena leaderboard. Our results, including comparison vote data, will be publicly available after double blind is lifted.

W2 Categorization: We acknowledge that the automated clustering strategy has risk for noise. Note P2L can create an aggregate leaderboard over any subset of input prompts; this is detailed in section 2.1.1. Thus, any clustering strategy can be employed with P2L– our paper only details one such strategy as an example. We believe that co-developing clustering methods with P2L based rankings is an interesting future research direction.

W3 Theoretical Assumptions: The reviewer is correct, and we try to be clear about this in the paper. The proof says that under the Bradley-Terry model, these routers are equivalent. We explicitly say, after the theorem statement, that “It is important to note that deviations from the BT model—for example, any non-transitivity—will break this relationship" (line 162). Thus, we believe this limitation has already been sufficiently communicated.

W4 Cost Estimation: This is correct. We will mention this limitation explicitly in the paper.

W5 Paper Density: We thank the reviewer for feedback. We will aim to increase accessibility in future revisions with intuitive visualization illustrating P2L’s functionality.

Q1:

What explains the outlier performance of P2L-1.5B in Figure 3?

We attribute this to noise. The bootstrapped 95% confidence intervals overlap, indicating this variation is well-within sampling variability. We view the trend in the plots as more informative than the specific values.

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

W1 Scalability: We agree with the reviewer that reducing the cost of adding new models is of interest, and are excited to continue exploring methods, such as online learning, to optimize this in future work. However, we also note our P2L models are fairly inexpensive to train: P2L-7B on 1.5 million data points costs less than \$250 to train end-to-end using a relatively unoptimized Deepspeed + Huggingface Trainer infrastructure (\\$23.92 per hour for 8xH100 on Runpod). The well-performing 3B and 1.5B variants train with negligible cost. We will include these exact training hardware, time, and cost numbers in the appendix for future revisions, which we believe will clarify cost concerns.

W2 Cost: The cost of the P2L model is negligible, both in terms of compute and time. P2L is, at its largest, a 7B model inferencing a single forward pass on the prompt only, which is very fast: P2L 7B add around 5% overhead on first token latency on an A40, $0.40 per hour on Rundpod, and less than 1% of the average routed LLM’s cost. The performance gain, on the other hand, is substantial. Therefore we do not see this point as a concern in practice. We will update the paper to communicate this fact. W3 Validation Metrics: We will include clear full definitions detailing the loss calculation for additional clarity in future revisions. Moreover, we will include straight forward accuracy metrics in the next revision. For context, we have computed accuracies for the grounded RK models which classify {A, B, Tie, Tie Both Bad}:

W4 Deployment Clarity: We will update our revised paper to include greater detail on deployment to Chatbot Arena. Specifically, we currently detail the P2L router’s model list in Appendix D1. We will additionally specify that we collect blind pairwise comparisons against all active public models hosted on Chatbot Arena in a process identical to how standard models are added to the Chatbot Arena Leaderboard.

W5 Training Params: When training P2L we do full parameter training. This means we update the weights of both the pretrained transformer and the newly initialized linear head. We will ensure this section is fully specified in section 3.1 in future revisions.

W6 Missing from Leaderboard: It was deployed in battle mode on Chatbot Arena for some time, and with that data, we are able to calculate its leaderboard position even though it is not displayed. The model does not appear on the Chatbot Arena leaderboard because routers are not allowed on the leaderboard– however, results, including comparison vote data, will be publicly available after double blind is lifted.

W7 Clustering Method: We leverage a topic modeling approach using BERTopic. We first encode each prompt using OpenAI’s embedding model, text-embedding-3-small reduce dimensions with UMAP, and apply a hierarchical-based clustering algorithm (HDBSCAN) with min size cluster 8. This process generates distinct topic clusters. Each topic is then summarized and named using an LLM (GPT-4o-mini).

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

W1 Intro: We will revise our introduction to explicitly define 'leaderboard of LLMs' as a prompt-specific ranking of multiple LLMs to improve clarity.

W2 LLMSelector: We believe LLMSelector is an excellent work which shares some philosophies with P2L. Importantly, it is concurrent work, and we should not be responsible for establishing novelty over LLMSelector. Additionally, the goals of the two works are different. LLMSelector aims to select models in compound AI systems using an LLM judge. Instead, P2L’s methodology instead gives full rankings of chat models for a single input.

W3 Regarding MOE: Good point, this does bear some resemblance to MOE. In fact, it would be interesting future work to see if our strategy can be incorporated in model training as a principled approach to MOE. However, at the moment the two are different. MOE typically trains model parameters end-to-end for inference optimization, whereas P2L's approach predicts leaderboard coefficients over fixed sets of external, possibly black-box APIs. The P2L framework provides granular evaluation of external models without retraining these models or modifying their parameters.

W4 Method Clarity: We thank the reviewer for pointing out clarity issues regarding terminology. In our revision, we will simplify and clearly define key terminology (e.g., BT regression, leaderboard vectors, routing policy notation) to improve readability, especially in Section 2.

W5 Comparison w/ Previous work: We will make sure to provide more detail on our training method in future revisions. Our architecture is similar to reward modeling, with a pretrained transformer initialized with a new output linear layer. However, we output a dimension per model in the leaderboard, and use a loss that supervises 2 dimensions of the output per datapoint– this, to our knowledge, is novel.

W6 Baselines: We understand the reviewers' concern around solid method baselines. We previously tested retrieval-based methods nearly exactly as described by the reviewer (e.g., retrieving k nearest prompts, weighting similar prompts). Empirically, we found the log-loss did not improve over marginal BT regression, with average loss differences smaller than 0.01 across various similarity metrics. Primarily, k must be quite large to obtain stable BT regression, which nullifies any per-prompt estimation advantage. Moreover, existing embeddings may encode “similar” in “semantic” subspace, and not “model performance” subspace. For example, “Explain BT regression” vs “Implement BT regression” are semantically similar, but fundamentally different tasks (explaining vs coding). Ultimately, we chose the marginal BT as baseline, since this was the most effective and stable baseline.

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We thank the reviewer for their comments. We thank the reviewer for considering the novelty and soundness of our work. We agree with the reviewer on generalization to new LLMs; this is of interest, and we are excited to continue exploring methods, such as online learning, to optimize this in future work. Roughly, we have seen it takes roughly 6k votes to get stable rankings for a given model— similar to what is needed for the marginal case. Fast adaptation to new models is a promising future direction. We also note our P2L models are fairly inexpensive to train: P2L-7B on 1.5 million data points costs less than \$250 to train end-to-end using a relatively unoptimized Deepspeed + Huggingface Trainer infrastructure (\\$23.92 per hour for 8xH100 on Runpod). The well-performing 3B and 1.5B variants train with negligible cost. We will include these exact training hardware, time, and cost numbers in the appendix for future revisions.