We sincerely thank the reviewer for the positive assessment and for pointing out the presentation gap between Eq. 3 and Eq 5. Below we state the missing bridge explicitly and justify the approximation.

The step from Eq. (3) to Eq. (5).

Eq. (3) defines a per-prompt token NLL that requires access to an expert’s next-token distribution $p_h(\cdot \mid \cdot)$. For API-only (black-box) experts, logits/probabilities are not exposed, so Eq. (3) is not computable. Our remedy is to

1. let the API expert $h$ produce a deterministic continuation $\hat{y}_h(x)$ for prompt $x$ (greedy decoding), and
2. evaluate that sequence under a single shared, public scorer $p_S$.

This yields Eq. (5), a pseudo-perplexity:

which we then normalize (similar to Eq. 4) and use as the fingerprint coordinate(s) for black-box experts. If $\hat y_h$ is a typical (high-probability) output of $h$ (i.e., $\hat y_h \sim p_h$) then averaging the pseudo-perplexity $\tilde{\ell} _h(x) = -\tfrac{1}{|\hat{y}_h(x)|} \sum _{t} \log p_S ( \hat{y} _{h,t} | \hat{y} _{h,<t})$ over many prompts/tokens is equivalent to taking an expectation over $y \sim p_h$:

Here $H(p_h,p_S)$ is the cross-entropy of $p_h$ with respect to $p_S$, which decomposes into the entropy of $h$’s own distribution $H(p_h)$ and its divergence from the scorer $\mathrm{KL}(p_h\|p_S)$.

Because $p_S$ is fixed for all experts, $H(p_h,p_S)$ is a stable, model-specific quantity that makes descriptors comparable across experts (same yardstick). It is not the true NLL under $p_h$, but it preserves differences between experts via $H(p_h)$ and their mismatch to $p_S$ via $\mathrm{KL}(p_h\|p_S)$. In practice we use deterministic (greedy) decoding to reduce variance; averaging over many prompts/tokens makes the empirical $\tilde{\ell}_h(x)$ closely track the expectation above. Figure 2 in the paper empirically validates this argument.

We will make sure we add an explanation so the transition from Eq. 3 to Eq. 5 is smooth.

We thank you for your feedback again. This improves the clarity of our work. We have also conducted additional ablation experiments which we invite you to read in our response to reviewer PkxW.

We sincerely thank the reviewer for the positive assessment, the thoughtful feedback and actionable suggestions.

1. On statistical significance

Our router consistently outperforms all baselines: AUDC is higher on every dataset, and the deferral curves align with or extend the empirical accuracy–cost Pareto frontier, rather than simply trading wins and losses across different regimes. Furthermore, AUDC is an average metric across the full deferral curve, not just a single accuracy point on a split dataset. If the reviewer is referencing a specific result, could they please clarify which values?

We had considered running robustness experiments with different probe sets to report mean and standard deviation, but this would require repeating each experiment at least three times, which is computationally prohibitive. That is what we referred to in the checklist.

As requested by the reviewer, we re-ran the full evaluation on EmbedLLM using paired, prompt-level significance tests to concretely assess statistical significance. Specifically, we computed a paired bootstrap [1] to obtain a $95\\%$ confidence interval for $\Delta_{AUDC}$ (Difference between AUDC of CSCR and AUDC of UMR as the best baseline overall). We also report a one-sided p-value for the hypothesis $\Delta > 0$. Additionally, we ran McNemar’s test [2] at a matched budget (using the median of the combined cost grids) to compare per-prompt wins and losses at equivalent operating cost. These tests quantify uncertainty over the test prompts, which we believe matches the reviewer’s intended notion of significance.

$95\\%$ Confidence Interval and the one-sided $p$ come from a paired bootstrap over prompts ($N=3000$, $B=5000$; $H_1 : \Delta > 0$). $c^\star$ is the matched budget used for McNemar. $n_{10}/n_{01}$ are discordant counts (CSCR correct / UMR correct), and “$p$ (McNemar)” is the one-sided exact binomial $p$ for CSCR $>$ baseline at $c^\star$.

$\Delta_{AUDC}$ is positive with CIs that exclude zero, and McNemar shows win rates above 0.5 with very small p-values at the matched budget. In other words: CSCR’s deferral curve encloses more area (higher accuracy at the same or lower cost on average), and at a fixed budget it wins on more prompts than it loses. This complements the Pareto-frontier plots: the gains are not an artifact of a single operating point or random variation, but hold paired, prompt by prompt. The establishes statistical significance.

2. Writing clarity

We will remove the bold/italics in the introduction and other sections, clarify writing, and do another full proofread. We appreciate the pointers and have fixed the two typos the reviewer noted.

3. More Ablations

Most of our choices for the contrastive loss were based on our past experience training contrastive models. We conducted additional ablation studies during the rebuttal period as per the reviewer's request.

3.1 $K$ in Eq. (9)

We selected a default of $K=4$ and it worked reasonably well so we did not do further hyperparameter tuning. However, we agree with the reviewer that an ablation would be useful, so we performed one on a subset of EmbedLLM prompts.

Increasing $K$ generally provides modest improvements before reaching a plateau. Beyond a small $K$, the gains in accuracy or AUDC become minimal, while both latency and the likelihood of choosing unnecessarily expensive experts increase. The baseline UMR also included an ablation on $K$ in a K-NN router. The trends are similar.

3.2 Banded Costs

We measured AUCD and peak accuracy with and without bands to validate having banded cost.

3.3 $\gamma$ in Eq. 8

$\gamma$ is a soft deterrent against assigning probability mass to costly, wrong experts during training (the denominator of the band-softmax in Eq. 8). If too small, the router learns to over-consider expensive hub experts. If too large, it over-penalizes cost and undertrains on valid high-cost positives, hurting hard prompts. We picked 0.2 as it’s a light regularizer: enough to push apart costly negatives first, but not so strong that it drowns the similarity signal for truly necessary expensive experts.

3.4 $\alpha$

In the band-specific temperature schedule $\tau_k=\tau_{\min}+\alpha\,\bar c_k$, the slope $\alpha$ sets how much flatter the softmax is in higher-cost bands. Increasing $\alpha$ raises $\tau_k$ for expensive bands, which flattens their per-band softmax over experts. This reduces gradient variance in those bands (where each query has fewer suitable positive) mitigating collapse onto a single rare positive and preventing cost-creep during training.

As $\alpha$ increases from $0$ to a moderate value ($0.1$–$0.25$), AUDC improves and the cost@max_acc drops, indicating we reach peak quality at lower cost, while max_acc remains comparable. For larger $(\alpha \ge 0.4$), the high-cost bands become oversmoothed, weakening discrimination among expensive experts; max_acc and AUDC decline despite further cost reductions. This validates our choice of adopting a moderate setting (default $\alpha=0.25$, with $0.1$ performing slightly better).

3.5 $\tau_{\text{min}}$

$\tau_{\min}$ is the softmax temperature for the cheapest cost band; all other bands inherit $\tau_k=\tau_{\min}+\alpha\,\bar c_k$. A very small $\tau_{\min}$ makes the cheap-band softmax sharp (highly discriminative but prone to noisy, peaky gradients) while a larger $\tau_{\min}$ smooths the distribution, lowering variance but also blurring differences among cheap experts.

Raising $\tau_{\min}$ from $0$ to $0.02$ increases AUDC and peak accuracy, showing that a touch of smoothing stabilizes learning without hurting discrimination. At $\tau_{\min}=0.05$ we keep nearly the same AUDC while cutting the cost needed to achieve peak accuracy by $\approx14\\%$ (from 51.7 to 44.6). Pushing to $\tau_{\min}=0.08$ oversmoothes the cheap band: accuracy at low cost rises slightly, but max_ac and AUDC both fall. Thus a moderate setting ($\tau_{\min} \approx 0.02$–$0.05$) offers the best efficiency–stability trade-off.

4. Q2: What if we ablate or enlarge the encoder?

We’re glad you asked as this is exactly the direction we’re exploring next. In this work we deliberately kept the router small: a frozen sentence-transformer plus a 2-layer MLP. We have added the following ablation where we increased the dimension of the middle layer of the MLP. As the router size increases, AUDC improves, but latency also increases.

We’re now testing larger routers by replacing the frozen query encoder with a larger LLM to see whether richer semantics improve difficulty judgment. More broadly, we’re excited to answer: How big must a model be to recognize a problem’s difficulty even if it can’t solve the problem itself? We want to study whether size and fine-tuning helps; we’re exploring whether RL on LLMs makes them able to purely as routers and sweep model size to pinpoint when difficulty awareness kicks in.

5. Qualitative Insights and Interpretability

We agree that interpretability is extremely valuable. We performed an analysis on embedllm, which features a large pool of models with diverse costs. Note that these observed trends are specific to the dataset and may not generalize to other datasets or prompt pools with more challenging examples.

5.1 Selection Profiles

$`Qwen/Qwen1.5-0.5B-Chat`$ (selected 91 times), $`google/gemma-2b-it`$ (233), and $`microsoft/phi-2`$ (292)—are smaller experts that are selected frequently. Some expensive experts (e.g., $`Qwen/Qwen-72B`$, $`ibivibiv/alpaca-dragon-72b-v1`$) are rarely chosen, since a less costly correct expert typically exists in the dataset.

We also report per-expert selection rates by cost band.

5.2 Taxonomy of Errors

We present a confusion-style breakdown of routing errors:

1. Too cheap—the router selects a cheap but incorrect expert
2. Too expensive—the router selects an unnecessarily costly expert, though correct
3. Optimal—the router selects a correct and minimally costly expert
4. No correct—no available expert produces a correct answer

We will add more comprehensive analysis and charts to the Appendix.

6. References

[1] An Introduction to the Bootstrap, Efron and Tibshirani, 1994 [2] Note on the sampling error of the difference between correlated proportions or percentages, McNemar, 1947

We truly appreciate the reviewer’s suggestions; the above additions directly address significance, interpretability, and clarity or our work.

We sincerely thank the reviewer for the careful read, the positive assessment, and the great questions. We address the questions below and will incorporate the suggested clarifications into the paper.

1 why take the $K$ most frequent tokens? Will they be trivial? Should we use TF-IDF instead?

In short, we use the most frequent tokens so the basis is shared and stable: they appear in all models, give low-noise estimates with few probes, and make calibration comparable across experts. They give less noisy estimates because they get non-negligible probability across many contexts, so their averaged log-probs vary less than rare or OOV tokens. In all experiments, we set $K=256$ and $T=10$ (Appendix C1), which is large enough that the basis isn’t dominated by a few function words. We will add a short paragraph to the paper stating this rationale explicitly.

1.1 Are frequent tokens "trivial"?

These tokens aren’t used for their meaning. They’re probes of each model’s output behavior. Even common words get scored differently across models (temperature, punctuation/number handling, style). By averaging over many prompts and steps, the descriptor captures overall model behavior, not any single word’s semantics. Also, a recent work shows that LLMs exhibit stable, word-level idiosyncrasies (as the authors call them) that enable near-perfect model attribution using only the first few generated tokens (even after paraphrasing or translation), implying that common tokens still provide discriminative signals about a model’s predictive calibration [1].

1.2 Shared token set vs. per-model selection.

A shared basis gives all descriptors a common coordinate system. If each model used a different token set, cosine distances would mix basis changes with true behavior, hurting comparability. It would also require computing many more probes to align per-model bases that are different across models.

1.3. Beyond raw frequency.

We deliberately kept the descriptors simple to isolate and quantify the contrastive router’s contribution. We agree that frequency is a pragmatic, not necessarily optimal, choice. Two variants that we considered and could be explored are: (i) TF–IDF-weighted selection over the probe corpus, just like the reviewer suggested. (ii) picking tokens with the largest across-model log-prob variance. These can be dropped into Eq. (2) without changing downstream training or inference.

If the reviewer believes that ablating this choice would strengthen the paper, we are happy to include it.

2. How could dense human annotations help?

Great question. We considered using human annotations but intentionally avoided them: model pools change quickly, so adding/replacing experts would require fresh labels that are costly and often unavailable. Instead, we train with sparse correctness and cost signals, which remain portable across experts. If dense feedback is available, it could help in several ways:

• Positive sets $P(i)$ with preference structure. Replace binary “correct expert” labels with pairwise preferences (cheap‐and‐good $\\succ$ expensive‐and‐similar $\\succ$ clearly wrong), yielding band-aware positives and margin constraints. This can be implemented by expanding $P(i)$ and adding a lightweight pairwise ranking (DPO-style) regularizer within each cost band.
• Difficulty-aware reweighting. Use human “difficulty” scores to upweight rare/hard prompts when computing the contrastive loss, especially in higher cost bands. this could balance the effective sample sizes across easy vs. hard (and cheap vs. expensive band) cases so the gradient isn’t dominated by the abundant, easy examples.
• Band calibration. We can ask users how much quality they’re willing to trade for a lower cost, then use that to set the cost bands and the penalty for picking more expensive models, so the router’s choices match what users actually prefer.

We will add a brief paragraph in Appendix describing these straightforward extensions.

3 Expanding L167: why do band-specific temperatures yield smoother gradients? What if gradients are not smooth on hard examples?

Thank you for catching this. We originally included a full explanation in the paper (lines 183-186), but it was accidentally removed during our edits for space and wasn’t moved to the appendix.

Most prompts in everyday interactions (and in our datasets) can be handled by cheaper models; plus there are usually fewer very expensive experts overall. These expensive experts are only needed for a small fraction of hard prompts, so within those high-cost bands there are fewer suitable positives per query. With few positives, the similarity distribution becomes very peaky.

Effect on gradients: Increasing $\tau_k$ makes the softmax less concentrated, so (i) gradient magnitudes shrink roughly like $1/\tau_k$ and (ii) directions average over more positives, reducing variance and preventing updates from collapsing onto a single rare positive, especially in the high-cost bands.

If gradients are not smooth for hard examples. Without band-specific temperatures, we observed the router can have unstable training (possibly due to oscillatory updates on hard prompts) and slower convergence. It also performs worse (please see the additional ablations in our response to Reviewer PkxW).

4. References

[1] Idiosyncrasies in Large Language Models, Sun et al., ICML 2025

This concludes our response to the reviewer’s question, but we invite them to read section 3 of our response to Reviewer PkxW for additional ablation experiments including ablations on cost spectrum hyperparameters. We hope these clarifications address the reviewer’s questions. We truly appreciate the insightful questions as they help increase the clarity of our work.