A Unified Approach to Routing and Cascading for LLMs

We thank the reviewer for their detailed review and questions. We are pleased to read that they find our paper well-written and the research topic valuable. Below, we address their questions and concerns.

Could you include statistical error bounds on your reported numbers?
Yes, we have now included a thorough analysis of statistical error bounds in Appendix F. Specifically, we underline values in Tables 1 and 2 that fall within the $2\sigma$ error bounds of the highest number. This analysis shows that, in Table 1, cascade routing consistently and significantly outperforms our baselines. In Table 2, cascade routing outperforms the baselines in three out of six cases and performs comparably in the other three cases.

Q2. Is the 1-4% accuracy improvement minor?
Please see Q2 of our main reply.

What criteria should guide the decision on which models are assigned to the same cascade level?
In our approach, all models are automatically assigned to every cascade level. Unlike traditional cascading methods, which require user-defined selection criteria, cascade routing dynamically determines the optimal selection strategy across models. This eliminates the need for restrictive pre-defined criteria, enabling a more flexible and efficient solution.

Does combining cascading and routing hold the key to solving this problem, or would factors like quality estimation approaches be more promising topics for research?
Both approaches are orthogonal and can significantly increase the performance of selection strategies. With our cascade routing approach we prove that, under our modeling assumptions, our given strategy is optimal. We therefore believe that research should now focus on more accurate quality estimators. Please refer to Q1 of our main reply for a full discussion on the impact of our optimal strategy and how it can help the community to shift focus to better quality estimators rather than algorithmic improvements.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and questions. We are pleased to read that they find our paper provides a solid theoretical basis for both routing and cascading and appreciate our significant performance gains. Below, we address the remaining questions and concerns.

Does cascade routing require extensive data to tune it effectively?
The hyperparameter optimization process for our proposed cascade routing method requires a minimal amount of training data. For routing algorithms, a single hyperparameter determines the trade-off between cost and quality, and this can often be set manually without extensive tuning. Cascading and cascade routing involve optimizing a slightly more complex set of parameters related to model selection in the cascade. However, we found that in cascade routing, assuming equal hyperparameters across stages already gives strong performance, significantly reducing the need for tuning.

In contrast, thresholding-based cascade algorithms typically require explicit hyperparameter tuning to perform optimally. Despite this, the overall data requirement for all methods is limited: even a few hundred training samples provide sufficient signal to achieve robust parameter estimation for routing, cascading, and cascade routing. This ensures that our proposed method remains competitive and scalable in scenarios where training data availability is constrained. Furthermore, the data on which the hyperparameters are tuned do not have to be labeled, since it mainly depends on the quality and cost estimates rather than the actual qualities and costs.

Is the hyperparameter optimization scheme time-consuming and computationally expensive?
No, the optimization scheme is highly efficient. In our experiments, we parallelized the search for hyperparameters over 20 CPU cores, evaluating 1,000 configurations. This process completed in under 6 hours on a standard laptop. By assuming equal hyperparameters across models, manual tuning can replace automated optimization in some cases, further reducing the effort required. Compared to the computational cost of LLM inference, the resources needed for hyperparameter optimization are negligible.

How can we address very noisy quality and cost estimates especially for real-time or adaptive applications where these are challenging to maintain with high accuracy?
Cascade routing relies on accurate estimates to make optimal decisions, and noisy signals—particularly in quality estimates—can degrade its performance. However, we emphasize that cost estimates in most tasks, such as classification, are exact or very good. The main challenge, therefore, lies in obtaining robust and accurate quality estimates.

Our current approach uses certainty-based metrics as proxies for quality, but these are inherently limited in high-noise scenarios. To address this, we propose two potential directions for improvement: (1) leveraging reward models specifically designed to predict output quality, and (2) integrating additional signals such as syntactic correctness or error metrics (e.g., runtime errors in code) to refine quality predictions. These strategies could enhance the system’s adaptability to noisy environments. However, initial experiments with the first technique we performed did not improve quality estimation. Therefore, further work on this aspect is required.

Despite the noise in quality estimates, our experimental results demonstrate that cascade routing outperforms traditional routing and cascading frameworks under a wide range of conditions. For further notes on the significance of our 1-4% improvement, please refer to Q2 in our main reply.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and questions. We are pleased to read that they find our paper well-written and appreciate our theoretical analysis. Below, we address the remaining questions and concerns and aim to clarify any outstanding points.

Why did you not include other routing and cascading methods?
The work mentioned by the reviewer is an instantiation of the baseline cascade with a different quality estimate. Existing cascading algorithms, as outlined in prior work, rely on thresholding mechanisms for quality estimates, which RouterBench does not directly support due to the pre-availability of answers. In our second experiment, we chose specific quality estimates from [Gupta et al., 2024], which are state-of-the-art for the given problem. While other estimates (e.g., from [1]) are viable, cascade routing can naturally integrate them, making it adaptable.

Could you include an analysis on runtimes, showing that the additional runtime required is sufficiently minimal?
Good question! We now present a new, detailed analysis of the expected latency of selection strategies in Appendix G and find that cascade routing slightly increases latency by 0.6 seconds compared to routing. This minor increase can be attributed to two factors: (1) smaller models incur low computational overhead, and (2) early stopping when a smaller model meets the required confidence level reduces latency. Moreover, we note that latency can be directly incorporated into our cost estimates to optimize a weighted combination of latency and monetary costs if the application values latency as well.

Can the authors provide the amount of training data required for hyperparameter optimization and compare it to the other methods?
The hyperparameter optimization process for our proposed cascade routing method requires a minimal amount of training data. For routing algorithms, a single hyperparameter determines the trade-off between cost and quality, and this can often be set manually without extensive tuning. Cascading and cascade routing involve optimizing a slightly more complex set of parameters related to model selection in the cascade. However, we found that in cascade routing, assuming equal hyperparameters across stages already gives strong performance, significantly reducing the need for tuning.

In contrast, thresholding-based cascade algorithms typically require explicit hyperparameter tuning to perform optimally. Despite this, the overall data requirement for all methods is limited: even a few hundred training samples provide sufficient signal to achieve robust parameter estimation for routing, cascading, and cascade routing. This ensures that our proposed method remains competitive and scalable in scenarios where training data availability is constrained. Furthermore, the data on which the hyperparameters are tuned do not have to be labeled, since it mainly depends on the quality and cost estimates rather than the actual qualities and costs.

What are the contributions associated with the proposed method, given that it only expands the search space?
See Q1 of our main reply.

What are the practical applications of cascade routing given that current quality estimates are noisy?
Although quality estimates are inherently noisy, our algorithm demonstrates resilience to such inaccuracies. The consistent improvements observed, even under high-noise conditions, underline its practical utility. On a large scale, such incremental advancements translate to significant benefits in real-world applications, where even modest cost or accuracy improvements can have substantial impacts (see also Q2 of our main reply).

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and questions. We are pleased to read that they appreciate our strong theoretical foundation, assess our performance improvements to be consistent, and acknowledge that our paper is well-written. Below, we address their questions and concerns and aim to clarify all outstanding points.

How does the computational cost of cascade routing, including the pruning strategy, scale with the number of available models?
Yes, we have added new experiments in Appendix F, showing that "Slow" becomes significantly slower as more models are included. We also introduce a fifth variant, "Max-Depth," which limits the maximum depth of a cascade to three models. Notably, this approach improves speed without compromising the performance of cascade routing when the optimal depth is three models or fewer. Specifically, we evaluate five cascade routing variants for up to 80 models on random data. Our findings demonstrate that the "Slow" variant becomes infeasible beyond 20 models due to its high computational cost. Standard cascade routing can handle up to 40 models within a 1-second runtime, while "Max-Depth" further scales to 80 models within the same runtime. Furthermore, we now also observe a clear difference between the “No-Expect” and “Greedy” variants in terms of runtime. While Greedy remains computationally very cheap even for $80$ models, No-Expect has a significantly higher runtime, even obtaining higher runtimes than Max-Depth for $80$ models.

Could you elaborate on potential future directions for improving these estimations? Are there specific features or model architectures that you find promising?
Yes, as the reviewer correctly identifies, improving quality estimation is crucial to enhancing cascade routing's performance. In fact, due to our theoretical analysis, we can now be sure that any future improvements to model selection strategies are to be found in better quality estimation rather than in algorithmic improvements. While we experimented with reward models trained on user preferences, we found that these metrics correlated less with answer correctness than the mean log probabilities of generated tokens. Future work could explore hybrid approaches that combine token probabilities with contextual embeddings or fine-grained task-specific features. Additionally, using dynamic feedback loops to refine estimations based on live user interactions may further enhance robustness.

Could you provide a more detailed analysis of the results across different benchmarks?
We now include benchmark-specific results for the experiment in Section 5.2 in Appendix H. These results reveal that cascade routing’s relative improvement varies depending on the used model set. We could not find any specific structure in these improvements.

Does the paper provide any contribution beyond the introduction of cascade routing, which is a combination of two known methods?
See Q1 of our main reply.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.

We thank the reviewer for their detailed review and questions. Below, we address their remaining questions and concerns and aim to clarify any outstanding points.

How does latency compare to simpler approaches like routing-only or ensemble methods?
Good question! We now present a new, detailed analysis of the expected latency of selection strategies in Appendix G and find that cascade routing slightly increases latency by 0.6 seconds compared to routing. This minor increase can be attributed to two factors: (1) smaller models incur low computational overhead, and (2) early stopping when a smaller model meets the required confidence level reduces latency. Moreover, we note that latency can be directly incorporated into our cost estimates to optimize a weighted combination of latency and monetary costs if the application values latency as well.

For quality estimation of model responses, have you considered more sophisticated evaluation methods beyond perplexity and uncertainty features?
Yes, we have. However, a key requirement for cascading is the efficiency of response quality evaluation. Sophisticated methods such as LLM-as-a-Judge, while effective, are computationally expensive and would significantly diminish the utility of the cascade framework. Current estimates based on perplexity and uncertainty are informed by state-of-the-art methods in cascade quality estimation (Gupta et al. 2024).

That said, we acknowledge the importance of exploring alternative evaluation strategies that balance cost and sophistication. Techniques such as lightweight heuristic-based scoring or smaller secondary models to approximate quality could serve as viable alternatives. For this purpose, we experimented with several classification reward models trained on user preferences to predict quality. However, we found that they do not provide a better signal for the quality estimator: their inclusion does not increase performance of the quality estimator. We therefore omitted them from the experiments.

Could you elaborate on specific use cases or scenarios where the trade-off between the additional complexity and the 1-4% accuracy improvements would be clearly beneficial?
We appreciate the reviewer’s concern about the modest improvement. While 1-4% might seem minor in isolation, the value of this gain is context-dependent and substantial in certain real-world scenarios:

• High-stakes decision-making: Domains such as healthcare diagnostics, financial fraud detection, or legal document analysis often rely on models for large-scale classification or analysis. In these settings, small gains in accuracy directly translate to measurable improvements in outcomes. For instance, a 1% reduction in diagnostic error rates or false positives can have significant real-world impact, both ethically and economically. Further, these domains are often not latency-critical as they are run in batches.
• Large-scale operations: In applications such as e-commerce recommendation systems, social media moderation, or news summarization, even fractional percentage gains in accuracy scale massively with large datasets. A 1% improvement in a system processing millions of requests daily could enhance user experience and engagement, while also reducing downstream errors. Once again, latency is not a key concern here.

Is the 1-4% accuracy improvement modest given the additional complexity and latency?
Please see Q2 of our main reply.

Should Chain-of-thought methods be another baseline to compare with the cascading methods?
We agree that chain-of-thought reasoning is a valuable technique. However, it is orthogonal to cascading algorithms, as both can be used in conjunction. For instance, our open-form experiment in Table 2 explicitly incorporates chain-of-thought reasoning to enhance response quality before cascading decisions are made.

We hope that this addresses all the reviewers' questions and concerns. We are happy to discuss any further questions the reviewer might have.