The Value of Prediction in Identifying the Worst-Off

Thank you for taking the time to carefully read our manuscript and provide comments. It is very much appreciated.

Other Domains. Yes, performing similar evaluations in other empirical domains would be valuable, but beyond the scope of this paper due to the substantial effort required to access administrative, government datasets. While each context should be analyzed individually, the close alignment between our theoretical results and empirical findings might suggest that the PAR could behave similarly across different settings. Our paper exemplifies how to do this for a specific real-world, high-stakes setting and we hope it encourages future work in other domains.

Fairness Evaluation. Thank you for raising this point. We fully agree that fairness concerns, such as differential performance across protected subgroups (e.g., gender), must be considered in the design of allocation systems. We view our analysis of policy levers as a necessary—but not sufficient—step toward understanding when investments in predictive systems are justified. We believe these subgroup concerns are an important direction for follow up work.

We appreciate your suggestion of relevant references and will more explicitly connect our work to the literature on fairness in algorithmic decision-making (including Hardt et al., 2016 and Chouldechova et al., 2018) in the revised related work section. We will also include additional figures in the revision that show model performance across demographic subgroups.

Additional Models. We selected CatBoost as it is widely regarded as a strong, out-of-the-box model for tabular data. Government agencies often rely on standard, readily available ML models, and thus our primary aim was not to optimize predictive accuracy but rather to illustrate trade-offs that arise with commonly used predictive methods. Our findings remain robust when using alternative models, such as random forests.

Costs. Thanks for bringing this up. The costs of improving prediction accuracy versus expanding screening capacity will typically differ and be domain-specific. A complete cost-benefit analysis would multiply the PAR by the ratio of marginal costs of both policy levers. Policymakers often have clearer estimates of these direct costs, while the PAR helps them determine how much more they should be willing to pay for one lever relative to the other. We will expand these cost considerations in the introduction of our revised manuscript.

Presentation. We will streamline the introduction and link the informally stated main results to their formal counterparts to ensure technical clarity.

Thank you for your detailed, thoughtful questions and suggestions for improving the paper! We appreciate your encouraging feedback on our conceptual framework and empirical analysis.

Cost Ratio and PAR = 1. Thank you for raising this. We fully agree that $\Delta_{\alpha}$ and $\Delta_{R^2}$ represent changes with different costs. For clarity and generality, we deliberately chose not to explicitly model these costs, as they naturally vary across application domains. We believe that examining the PAR independently remains valuable to disentangle benefits from costs. Policymakers typically find it easier to estimate relative costs of practical policy levers (e.g., hiring additional staff or conducting more frequent surveys to improve predictive accuracy) compared to evaluating how these changes will impact the policy value. We therefore focused on shedding insight on the welfare ratio with the understanding that domain experts have knowledge of the cost ratio and can easily factor these into the cost-benefit analysis (as in your expression).

While our results hold for general $\Delta_\alpha$ and $\Delta_{R^2}$, we chose PAR =1 with these fixed ($\Delta_\alpha = \Delta_{R^2}$) purely for the sake of discussion. We are happy to point out how this choice is made for simplicity and will add further discussion on how the true threshold is at (cost of prediction) / (cost of access) as you point out. In the final version, we will move up and expand the discussion on marginal costs to the introduction, and we will aim to provide further context on the PAR = 1 threshold to avoid any confusion. Your perspective and comments on this issue are very helpful. Thank you.

Why differ from Perdomo 2024. This previous work focuses on the impact of different policy levers on average social welfare. By contrast, we focus on identifying the impacts of various policy levers on the welfare of the worst-off individuals. This captures a (mathematically and conceptually) distinct as well as a practically important setting often seen in the public sector. For instance, employment agencies commonly aim to identify and support those in greatest need, rather than maximizing average employment outcomes across all job seekers. We will clarify this distinction from Perdomo (2024) in the related work section of the revised manuscript.

Choice of $R^2$. We will clarify the choice of $R^2$ and discuss other measures that could also be applicable in the final version of the manuscript. The short answer is that $R^2$ tends to exactly characterize predictability in our theoretical models.

Presentation. Thank you for your suggestions on improving the presentation—your comments are very helpful in this regard. We will streamline the introduction to reduce redundancy and clarify the presentation to ensure that key concepts are formally defined before they are used. Given the additional space that comes with the revision, we will happily include a discussion at the end of the paper and will include subcaptions for the figures in the revised version of the manuscript.

Thank you very much for your positive feedback. We're especially glad you found the paper clear and relevant, and appreciate the time you took to engage with our work.

Thank you for engaging with our work and for your constructive feedback. We very much appreciate it!

Beta Values. 0.15 reflects the official government threshold used in Germany. However, we included additional values to account for differing long-term unemployment thresholds used in other countries. To improve clarity, we will reorganize the figures and expand the captions in the revision.

Comparison to Perdomo 2024. Thank you for the question. Our work conceptually builds on Perdomo (2024), who introduced the prediction-access ratio (PAR) to analyze trade-offs between prediction quality and resource availability. The main difference between our work and his is that we develop a new theoretical analysis of the PAR focused specifically on the welfare of the worst-off population rather than average welfare, as in Perdomo 2024. These two settings are conceptually and mathematically distinct. Furthermore, relative to Perdomo 2024, we introduce a set of empirical tools to analyze these tradeoffs in practice (the previous work was purely theoretical). We will clarify and expand the comparison to Perdomo (2024) in the revised related work section.

Section 4. We are happy to improve the transition into Section 4 and connect it more explicitly to the empirical case study in the revised manuscript.

Figure 4(c). Figure 4(c) shows the rate of improvement in policy value with respect to small changes in prediction quality in a high $R^2$ regime (predictions close to perfect), as a function of available resources $\alpha$. Incremental prediction improvements have the greatest impact when resources precisely match the targeted fraction of the population ($\alpha = \beta$). For both insufficient ($\alpha < \beta$) and surplus ($\beta > \alpha$) resources, further prediction improvements are less impactful. We will expand and improve the caption for Figure 4 in the final version. Thanks for raising this.