Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?

We sincerely thank Reviewer Zmym for their thoughtful assessment of our work, particularly noting that our paper is "pretty clear" and addresses an "important" question with "good experimental settings."

Quantifying Score-Compute Decorrelation Per Transformation

Since we know that the trasnformations bring the detoriation to the correlation, I would like to how how much each step contribute to this.

This is an incisive question. While the answer depends on the model family and the benchmark, Figure 4 suggests the answer, but we can be much more quantitative. We will create a new figure with 4 subfigures, one for each of the statistics of the score-compute correlation distributions (mean, median, AUC of CDF, and negative Wasserstein distance). The x axis will be the sequence of metrics, the y axis will be the statistic’s value, and the hue will be the model family. This will show exactly how much “predictability” is lost under each transformation.

If the paper is accepted, we will include this detailed quantitative breakdown in the final version, including a new table summarizing these contributions across benchmarks.

On Visualizing Distributions

using a pdf is way more straightforward than using a cdf, people can well distinguish where there is of high density.

We appreciate the suggestion about using PDFs instead of CDFs. We actually explored both approaches during our analysis. While PDFs can be more intuitive in some contexts, we found that CDFs (specifically complementary CDFs) offered three advantages for our particular analysis: (1) they avoid bandwidth parameter tuning artifacts that emerged with KDE-based PDF estimation; (2) they directly visualize the tail behavior critical for our analysis; and (3) they more clearly quantify "what fraction of samples have correlations above X" - the key relationship we wanted to highlight. We'd be happy to include a comparison of both visualization approaches in an appendix if you think this would benefit readers.

Terminology

Thank you for suggesting clearer wording. We used 'fraction of accuracy=1' to precisely distinguish between two concepts: (1) the proportion of individual samples where the model answered correctly (binary per-sample outcome), versus (2) the overall accuracy score across the dataset (continuous value between 0-1). While mathematically equivalent, maintaining this precision helps readers follow our sample-specific correlation analysis. We'll revise this wording to improve clarity while preserving this distinction.

Request for Additional Constructive Criticism

We value your assessment and would appreciate any additional specific suggestions that might strengthen the paper. In particular, are there specific analyses or clarifications about the transformation process that would make our findings more compelling or accessible to the broader ML community?

We appreciate your 'weak accept' assessment and are committed to strengthening the paper to earn a more enthusiastic endorsement. We believe the quantitative breakdown of correlation degradation across transformations will substantially enhance the paper's contributions and clarity.

Thank you again for your constructive engagement with our work.

We sincerely thank you for your thorough and thoughtful review. We greatly appreciate that you found our analysis "interesting," "quite comprehensive," and that it "gets at the main cause" of unpredictability in downstream capabilities. We're particularly pleased that you found our story "presented in a clean way" and that the "paper is easy to read." Thank you also for highlighting Figure 3 as effectively demonstrating the degradation of predictive power on the ARC benchmark.

On Specificity to Multiple Choice Question Answering (MCQA) Benchmarks

We agree that our focus on MCQA benchmarks represents a scope limitation. We are upfront about this but made this decision for several reasons:

1. MCQA benchmarks remain a widely used evaluation format for frontier models due to their objective scoring and standardization. They're employed by major benchmarking efforts (e.g., MMLU, HELM) and by leading labs to track progress.
2. The MCQA format is particularly valuable for investigating scaling predictability because it allows us to isolate specific transformation steps in a controlled manner. This methodological clarity helped us identify the core mechanism behind unpredictability.
3. We view our manuscript as contributing to the science of scaling-predictable evaluations. While any given task or type of task will have its quirks, our paper shows how one can dive into the mechanisms of apparent unpredictability For example, new work on studying predictable scaling behavior of inference-time compute (https://arxiv.org/abs/2502.17578) also applied a per-sample analysis, albeit in a different context. We will add a citation to this work and clarify how analyses like ours can be applied to generative evaluations.

Fundamental Limitation or Temporary Obstacle

It mentions that the problem mainly stems from unpredictable movement of probability mass on the incorrect options, but does not discuss any ways in which this can be fixed, or is this a fundamental limitation?

This is certainly not a fundamental limitation and we have subsequent work showing how to overcome this limitation by predicting how probability mass changes on incorrect choices. However, demonstrating and dissecting the problem thoroughly is itself already quite long (as you can tell by our manuscript). We will move our Future Directions section into the main text if accepted.

Clarification of How Mass Changes on Incorrect Choices

Intuitively, one might imagine that the probability mass on all incorrect samples should reduce monotonically with scale...

This was our belief initially, but probability mass on incorrect choices almost always increases with compute, at least in the models we were able to study. For example, suppose we have a question about what pet a child has. As models train, (1) they place more mass on syntactically valid words and phrases, so if an incorrect option is "airplane", its mass will still increase relative to a randomly initialized model, even though airplane isn't a valid pet ; (2) they place more mass on semantically plausible alternatives e.g., "cat" can be a likely pet, even if it isn't the correct option for this question; (3) they learn the nature of MCQA tasks and place more mass on the available options regardless of content.

At some point, probability mass on incorrect choices must decrease, but it seems the models we studied are not close to this tipping point.

Responses to Specific Questions

In Figure 6, do the trends change depending on the number of incorrect options present for each question?

For True/False questions, is the scaling more predictable?

This is an excellent question. Most widely-used MCQA NLP benchmarks have 4 options. MMLU-Pro (https://github.com/TIGER-AI-Lab/MMLU-Pro) but offers 4-10 choices per question, but was publicly released after our data were collected. We expect that if there are more incorrect options, scaling is less predictable and if there are fewer options (e.g., True/False), scaling would be more predictable. We will add this to our Discussion.

Visual Presentation Improvements

We appreciate your feedback about plot label sizes. We will increase font sizes in all figures.

Conclusion

We believe addressing these concerns will substantially strengthen the paper and hope they address your "weak reject" assessment. Thank you again for your constructive feedback, which has genuinely helped us improve our work.