Thank you for your constructive and encouraging review -- we appreciate your comments and the time you took to evaluate our work!

Benefits of benchmark refinement

Thank you for raising this matter.

First, while a single evaluation run may appear inexpensive relative to pretraining, evaluation becomes significantly more costly when it is repeated frequently, as is typical in modern LM workflows. For example, evaluating a single model across the full HELM benchmark suite can cost over 4,000 GPU hours or $10,000 in API usage (Liang et al., 2023). Costs escalate further in in-loop setups, where frequent evaluations are needed during pretraining. In fact, when pretraining a 1B model and evaluating every 100 steps using the OLMES evaluation framework (Gu et al., 2025), the evaluation time is longer than the pretraining time.

Second, benchmark refinement is not only about efficiency, but also about improving other aspects, particularly validity, variance, and saturation -- critical properties for meaningful model evaluation. Gains in these dimensions can be orthogonal to efficiency: for example, even when evaluating on all items of a benchmark, Fluid Benchmarking puts less weight on mislabeled items, leading to a higher validity empirically (see the next point below).

We appreciate the opportunity to clarify and will include a more explicit discussion of these points in the final version of the paper.

Full-benchmark evaluation

This is an excellent question -- thank you for raising it.

When evaluation is performed on all items of a benchmark, the dynamic item selection of Fluid Benchmarking is neutralized. In this scenario, our method effectively reduces to the Random IRT baseline, which applies IRT-based ability estimation across all items. Crucially, even in such a full evaluation regime, IRT-based ability estimation offers significant advantages over raw accuracy. Specifically, while accuracy treats all items equally, IRT takes into account item difficulty and assigns less weight to mislabeled items.

Empirically, this distinction is reflected by consistent gains across validity, variance, and saturation when comparing IRT-based ability estimation vs. standard accuracy on all items:

Comparing these numbers with Fluid Benchmarking (Table 2 in the paper), we observe that Fluid Benchmarking achieves similar (validity, saturation) or even better results (variance) than the full-benchmark IRT ability estimate, with much fewer items. The advantage becomes even more striking when comparing Fluid Benchmarking to full-benchmark accuracy: with just 50 items, Fluid Benchmarking outperforms full-benchmark accuracy across all three evaluation dimensions: validity (8.8 vs. 9.1), variance (6.5 vs. 10.0), and saturation (0.86 vs. 0.85). In other words, even if efficiency is not a concern, Fluid Benchmarking can improve evaluation quality.

We agree that these results provide a valuable contextualization, and we will include them in the final version of the paper.

Generalization beyond train models

Please note that IRT ability estimates are not upper bounded by the abilities of the train models (i.e., the models used to estimate item parameters). For instance, if the most capable train model correctly answered $k$ items from a benchmark, a new model that correctly answers those $k$ plus at least one additional item will receive a higher ability estimate.

That being said, you are absolutely right that fixed IRT models can be limited in how they adapt to better models. Specifically, consider the subset of items of a benchmark that were not answered correctly by any train model. These items are effectively assigned the same maximum difficulty. If we conduct Fluid Benchmarking with a model that is better than any train model, evaluation will quickly move to those most difficult items. However, the fixed IRT model cannot distinguish finer levels of difficulty among them. Therefore, in this extreme region, ability estimation reduces to something close to accuracy over those hardest items, rather than leveraging graded item properties to make finer distinctions.

This does not invalidate the IRT estimates and hence is not a terrible mode of failure, we believe, but it should be clearly understood by users of the method. Future work could explore extensions such as dynamic recalibration of item parameters. We will include a discussion in the final version of the paper.

I have no further questions and will keep my recommendation to accept.

Thank you for your kind and thoughtful review of our paper -- we really appreciate the positive feedback and helpful suggestions!

Fluid Benchmarking for post-training

We fully agree that Fluid Benchmarking is not inherently limited to pretraining and has potential value in post-training settings as well. We focused on pretraining here because evaluation efficiency is especially critical in this phase, due to both the high cost of runs and the central role of repeated in-loop evaluation. That being said, we see extending the framework to post-training as an exciting next step and will add a discussion in the final version of the paper.

Standardized IRT reference models

Thank you for raising this important point. We agree that the utility of Fluid Benchmarking depends on stable and up-to-date IRT models. To support wider adoption, we will release all IRT models trained for this paper, which we hope can serve as a starting point for a standardized reference. We will place more emphasis on our released artifacts in the final version of the paper, and we will also emphasize the importance of regularly building new IRT models on fresh evidence.

Clarification for Figure 2

Thanks for pointing this out. We will add a clarification to the caption of Figure 2. Additionally, we will include a supplementary figure in the appendix that illustrates how item discrimination influences Fisher information for a given ability level.

Dynamic stopping criterion

This is an excellent suggestion. We did in fact experiment with dynamically stopping the evaluation once the standard error of the provisional ability estimate dropped below a predefined threshold value. We found that the number of items required to meet this criterion varied significantly over the course of pretraining: earlier checkpoints required fewer items, while later ones needed more. For example, on HellaSwag, using a maximum standard error of 0.1, only 16 items were needed at the first checkpoint, compared to 32 at the final checkpoint. We will include these results in the final version of the paper.

Multidimensional IRT models

Thank you for raising this. We indeed experimented with fitting separate multidimensional IRT (MIRT) models with two to five latent traits to each benchmark to investigate whether they could capture multiple underlying capabilities. However, the MIRT models did not yield improvements in model fit over unidimensional IRT models, which led us to not pursue this direction further. We will report these results in the appendix. In general, we fully agree with you that a systematic exploration of dimensionality in LM-based IRT models is a promising avenue for future work, and we will add a discussion in the final version of the paper.

Thank you for your thorough review of our paper -- we very much appreciate the positive and constructive comments!

Non-English and multimodal benchmarks

Thank you for suggesting to evaluate our framework on other languages and modalities. Extending Fluid Benchmarking to these settings is indeed feasible, provided that evaluation results to fit an IRT model are available. For example, Fluid Benchmarking of vision-language models could leverage leaderboards like VHELM. While a comprehensive evaluation is beyond the scope of this paper, we will include a discussion in the final version to clarify how our method could be applied more broadly, including potential challenges.

Overfitting to known capabilities and capability trajectories

Thank you for raising this. Fluid Benchmarking, and IRT-based evaluations more generally, rely on historical evaluation data, which implicitly assumes that future models will exhibit broadly similar capability patterns -- for example, not being dramatically better at the hardest items while worse at the easiest ones. While we have not observed such shifts empirically, they are theoretically possible. In such cases, retraining the IRT models would be necessary to maintain evaluation accuracy. We will add a discussion and recommend ways to mitigate this risk.

(Please note that the pretraining trajectories are not used -- neither during IRT parameter estimation nor in dynamic item selection -- so they cannot directly bias the evaluation.)

Validity of unidimensional IRT model

Thank you for raising this important point. We considered the question of dimensionality at several stages of the project.

We first experimented with fitting a single unidimensional IRT model across all benchmarks, following prior work suggesting that one latent trait can capture overall model behavior (Kipnis et al., 2025). However, we found that this substantially reduced construct validity. For example, the performance of Amber-6.7B on TruthfulQA decreases during pretraining (Liu et al., 2023). By contrast, when we evaluated Amber-6.7B using a unidimensional IRT model trained across all benchmarks, the estimated ability increased -- the IRT model effectively emphasized TruthfulQA items aligned with general trends, obscuring the fact that Amber-6.7B actually becomes less truthful during pretraining.

Next, we fit separate multidimensional IRT models (with two to five latent traits) to each benchmark. These models, however, did not yield consistent improvements in model fit compared to unidimensional IRT models.

Ultimately, we adopted a compromise: fitting separate unidimensional IRT models to each benchmark. This struck the best balance between interpretability and model fit.

We realize now that we should make this line of analysis more explicit, and we will add the additional experiments in the appendix. In general, we fully agree with you that a systematic exploration of dimensionality in LM-based IRT models is a promising avenue for future work, especially in settings where heterogeneous capabilities are expected to play a central role. We will incorporate a discussion of this much-needed future work in the final version of the paper.

Novel architectures and tasks with no prior data

These are excellent points.

Regarding novel architectures, we believe the answer is "yes" -- our method does not make assumptions about model internals, so it remains applicable regardless of architecture.

Regarding tasks with no prior data, IRT-based methods require some initial evaluation data. For a new task, this means benchmarking several models first to estimate item parameters.

We agree that these are important considerations in practice, and we will make sure to mention them in the final version of the paper.