Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems

We thank the reviewer for the feedback. We answer the questions as follows.

The results are not surprising: We appreciate your comment but we would like to respectfully argue that our results are interesting. First, many practitioners and researchers believe that they can consistently improve performance with more LLM calls (e.g. https://arxiv.org/abs/2402.05120). In this context, our finding that more LLM calls can hurt performance in specific settings is unexpected and a useful contribution to the literature. Furthermore, we provide an explanation of this counterintuitive phenomenon based on query difficulty and provide both mathematical and empirical justifications. The prediction based on our theory matches the empirical performance accurately. These results have not been not studied in existing work and opened the door to understanding and optimizing the design of compound LLM systems.

model prediction is based on strong assumption: Thank you for this question. We would like to clarify that the algebraic formula (K*) shown in Theorem 4 is used for a special case. On real-world datasets, we use the scaling law function empirically fitted on a small dataset for prediction purposes, which does not require strong assumptions. The scaling law function’s predictions are empirically accurate (see Figure 4(c)). We will highlight this in the revised paper.

Comparison with hyper-parameter search algorithms: Thank you very much for pointing this out! We conducted additional experiments to compare our proposed scaling laws with two popular hyper-parameter search algorithms, bayesian optimization and tree structured parzen estimator, with the same number of performance evaluation budgets. Specifically, we evaluate these hyper-parameter search algorithms on GPQA, AVERITEC, and MMLU PHYSICS, with 5 calls to the performance oracle (the same input to our scaling law method). As shown in the following table, our scaling law reaches the most accurate prediction and overall accuracy. We will add this discussion in the revised version. Notably, both the Parzan and Bayesian optimizers would have recommended the users to use a large number of LLM calls (>100), which would have incurred much more cost while performing worse than our estimator.

We thank the reviewer for the helpful feedback and support of our paper! Please see below for our response.

present the overall results of all datasets in the experiment section if the page limit permits: Thanks for your suggestion! We will move more empirical results to the experiment section in the revised paper.

answer extraction: We prompted the LLM to generate its final output in a specific format (i.e., “the answer is (X)”) and then used regular expression to extract the output.

Did you (Or do you think whether it is worthwhile to) conduct additional robustness tests (e.g. by shuffling the option orders, or adding input perturbations) to validate the LLM outputs?: Thank you very much for pointing this out! Robustness studies are orthogonal but complementary to the focus of this paper. For example, it would be interesting to study how input perturbation affects query difficulty distribution. We will add a discussion in the revision.

Recent evaluation papers: Thanks for the reference. We will add a discussion to them in the revised paper.

Dear authors,

thanks for the response. I think this is overall very interesting work and my initial score should properly reflect the quality of this work. And I would like to see it included to the proceedings.

One final remark: thanks to the author for pointing the use of regex to extract the response and I would suggest the author include the details of the extraction process in the appendix, as well as the rate of the missuccess (if any). Especially the template "the answer is (X)" reminds me of the title of a very recent paper from ACL this year that might be more or less relevant. Perhaps also worth reading: https://aclanthology.org/2024.findings-acl.441/

We thank the reviewer for the helpful feedback and support of our paper! Please see below for our response.

Simple System: We indeed focus on simple systems. There are two reasons. First, they both represent real-world systems. For example, the Cot@32 approach by Google Gemini is indeed a simple vote method. Second, the observed phenomenon is pretty surprising, and we want to understand it empirically and analytically in the simplest place it appears. We believe it is always important to find a minimal working example. Of course, it will be valuable to investigate in challening settings in future work.

Is the optimal number of LM calls model-specific or not?: Thanks for this great question. We actually predict the optimal number of calls in two ways (K* for a special case and the scaling law function G in general). They are both model-specific. This is an important point and we will emphasize it in the revision.

the least number of calls for a query to get correct answer: Thank you so much for pointing this out. We think this is a great intuitive way to explain how to characterize query difficulty. It is very closely connected, in fact, to the characterization we currently use. It is a kind of soft version of our difficulty level. Indeed, let’s take Vote as an example.

Here, the inverse of $Pr[G(x,\theta)=y]$ is the expected number of calls to obtain at least one correct answer, which is one formalization of the reviewer’s soft difficulty. We focus on the binary difficulty because it offers a natural explanation of the non-monotonic behavior. We appreciate the suggestion for providing this nice intuition and will put a sentence about it into the revision.

We thank the reviewer for the helpful feedback and support of our paper! We have addressed the questions as follows.

Are there methods beyond Vote and Vote-Filter that can be applied here? Can this approach generalize if difficulties are probabilistic to begin with?: Thanks for bringing up this point. We indeed believe that our methods can apply beyond just Vote and Vote-Filter. We chose to focus on those two systems because … [paste] … We have in mind to do some followup work on more complex systems. One example is Ranking-Vote: Each LLM call provides a ranking of possible answers, and then we perform ranking aggregation (e.g., Borda count) to produce a final answer. Here it’s indeed important to consider probabilistic query difficulty. The effect will be significant because while in Vote and Vote-Filter we have a law of large numbers that governs the resulting prediction (polarizing the difficulty to a binary easy or had), in more complex models, we expect that the full distribution of query difficulty over a dataset will impact model performance.

Can this approach generalize if difficulties are probabilistic to begin with?: Yes. For example, one might take a Bayesian view: difficulties can be initially quantified as a prior distribution, and invoking more LLM calls offers a more accurate estimation of the posterior.