A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding

Thank you for your feedback and interesting questions about our axiomatic comparison. Below we address some of your questions:

• Figure 2 in Appendix A gives the main intuition for why MERIT violates monotonicity. We will add clearer axis labels. The Figure has four intervals where interval 1 lies entirely above intervals 3 and 4, while intervals 2,3, and 4 overlap. When k=1, the top 1 proposal (given the overlaps in the intervals) could either be proposal 1 or proposal 2 since these two intervals overlap, while 3 and 4 are strictly below 1. Hence, MERIT chooses uniformly at random between proposals 1 and 2. Meanwhile, when k=2, proposal 1 is always in the top 2 so it gets accepted with probability 1. This leaves 1 probability mass to allocate between proposals 2,3,4. Since any of these 3 proposals could be the other proposal in the top-2, they each receive probability 1/3 of acceptance. Therefore, proposal 2 is worse-off when selecting 2 proposals rather than 1 proposal. Can you give an intuition as to why the acceptance rate between the two algortihms is more varied for the ICLR dataset? (Table 1).

• In our formal definition of “maximal instability” we specifically define the axiom for budget (k) ranging from 2 to n-2, because k=1 is not a particularly meaningful case to consider for the stability of selection algorithms. Just by definition when k=1, changing one proposal (highest one) can fully change the outcomes. This holds even choosing deterministically or for any other reasonable method. Hence k>1 is a more meaningful parameter range in which to consider the stability of an algorithm. We will clarify this in the main text.

Thank you for the review and for your attention to the paper's comparison to existing methods. We are glad that you found the problem formulation and axiomatic analysis interesting. Below we address some of your concerns:

1. Novelty of the formulation: While Stackelberg / min-max formulations are indeed well-established and widely used in many different applications, our paper makes two novel contributions:

   • First, as we discuss in Section 4 of the paper, it is computationally challenging to solve for an equilibrium in the specific Stackelberg game we describe. Many related optimization problems—minimum weight k-ideal problem [FK94], maximum cut on interval graphs [ABMR23], interval graph completion [RAK91] and minimum weighted completion time of jobs subject to interval ordering precedence constraints [Woe03]—are known to be NP hard. Our work develops a novel, polynomial-time algorithm to solve for the Stackelberg equilibrium in our problem setting and this algorithm is also practically efficient.
   • Second, to our knowledge, modeling uncertainty in selection decisions as a Stackelberg game has not been done previously and represents a new take on the problem.

2. Manski bounds: The statement in the paper about the use of Manski bounds on the Swiss NSF data is indeed accurate. The Swiss NSF setting only has ~300 proposals and panels meet and discuss each proposal and then vote on all proposals. Hence, the only reason for missing evaluations is conflict-of-interest. Therefore, the Manski intervals indeed cover most review pairs.

   For conference settings, like NeurIPS and ICLR you are correct that Manski bounds would result in meaningless intervals that cover the entire range of scores since there are few reviews per paper, so we do not use Manski bounds there. We test a number of methods for generating intervals in conference settings, detailed in Appendix F.

3. Axiomatic criteria: Our interpretation of the maximum instability axiom is that all natural algorithms should obey even this very weak notion of stability. It was quite surprising to us that existing methods did not respect this axiom and could be "maximally unstable."

4. Post-processing for monotonicity: We agree that other algorithms could also be post-processed in order to enforce monotonicity. MERIT is intrinsically better with respect to the robust objective over interval uncertainty and also is guaranteed to meet the ex-post validity principle, which prior methods do not guarantee. Additionally, our algorithm to enforce monotonicity for MERIT empirically seems like it does not lead to much worse performance with respect to worst-case ex ante optimality. In the modified algorithm, we obtain a locally optimal solution at each step (i.e., at budget k we solve for the ex ante optimal vector of probabilities that is monotonic with respect to the solution obtained at budget k-1). It will be an interesting area for future work to try to prove that this algorithm (or a suitable modification) is near the global optimum when one wishes to maximize the ex ante objective with the additional constraint of monotonicity.

References

[FK94] Ulrich Faigle and Walter Kern. Computational complexity of some maximum average weight problems with precedence constraints. Operations Research, 42(4):688–693, 1994.

[ABMR23] Ranendu Adhikary, Kaustav Bose, Satwik Mukherjee, and Bodhayan Roy. Complexity of maximum cut on interval graphs. Discrete & Computational Geometry, 70(2):307–322, 2023.

[RAK91] Ramamurthy Ravi, Ajit Agrawal, and Philip Klein. Ordering problems approximated: Single-processor scheduling and interval graph completion. In Automata, Languages and Programming: 18th International Colloquium Madrid, Spain, July 8–12, 1991 Proceedings 18, pages 751–762. Springer, 1991.

[Woe03] Gerhard J Woeginger. On the approximability of average completion time scheduling under precedence constraints. Discrete Applied Mathematics, 131(1):237–252, 2003.

Thank you for the thoughtful and detailed review, and for recognizing the theoretical and practical contributions of the work.

1. Constructing intervals: Our approach is designed to not be too closely tied to any specific model, so there are a number of ways that a decision maker could generate intervals. We had limited space to describe approaches for interval generation in the main text but give more detail in Appendix F on methods for generating intervals in our experiments. For example, the funder could simply take intervals as the range from minimum and maximum reviewer score given to each proposal or they could compute bootstrapped estimates. Alternatively, they could accommodate subjectivity in the reviewer recommendations by allowing for different mappings from sub-criteria scores ("quality", "clarity", "significance", and "originality" in NeurIPS reviews) to an overall score.

2. Interval characteristics: The width of the constructed intervals varies between methods and settings. Below, we provide the mean width of intervals for the methods we tested (after normalizing the units of all intervals to lie in [0,1]):

   Dataset	Average Width
   Swiss NSF	0.0625
   Swiss NSF (Manski Bounds)	0.0198
   NeurIPS LOO	0.1259
   NeurIPS Gaussian	0.1671
   NeurIPS Subjectivity	0.0700
   ICLR LOO	0.0917
   ICLR Gaussian	0.1196
   ICLR Subjectivity	0.0684

   In terms of the skew of the intervals, because we do not know the ground truth for the "true quality" of each candidate in real-world datasets like ICLR and NeurIPS, it is not possible to measure where in the interval the true quality lies. This is the motivation for optimizing over intervals where the true quality could lie anywhere within each interval.

3. Complexity of communicating MERIT: This concern is valid, as we discuss in the Limitations section, but we have reasons to be optimistic about adoption of MERIT. First, funders are using the current Swiss NSF method—this method has a simpler decision making part than MERIT, but a much more complex method of generating intervals (hence the overall method is complex.) In spite of the complexity of the Swiss NSF's interval generation method it has been put into use for real-world funding decisions. Second, we are currently speaking with organizers of such selection processes, and so far, complexity has not come up as an issue.

Thank you for your thoughtful comments on our theoretical results and detailed attention to the writeup. Below we clarify your questions:

1. Relation to existing work: While our setting shares the high-level motivation of making robust decisions under uncertainty, classical robust and distributionally robust optimization techniques typically operate over convex uncertainty sets in continuous spaces (e.g., real-valued parameters that can take any value within a range). In contrast, our problem optimizes over a discrete uncertainty set of rankings: we optimize worst-case utility over a combinatorially defined set of feasible permutations consistent with interval constraints. This ranking-based uncertainty set leads to a non-convex inner problem that cannot be directly addressed by standard RO/DRO formulations.

2. Monotonicity vs. ex-ante optimality: This is an interesting question — we propose a simple algorithm to enforce monotonicity for MERIT in Appendix D, which allows for violation of ex-ante optimality. Empirically, it seems like it does not lead to much worse performance with respect to worst-case ex ante optimality. In the modified algorithm, we obtain a locally optimal solution at each step (i.e., at budget $k$ we solve for the ex ante optimal vector of probabilities that is monotonic with respect to the solution obtained at budget $k-1$). It will be an interesting area for future work to try to prove that this algorithm (or a suitable modification) is near the global optimum when one wishes to maximize the ex ante objective with the additional constraint of monotonicity.

3. Generalizing Theorem 4.3: Thanks for the great observation—yes the statement holds for any solution. Precisely, we could generalize the theorem statement as follows:

   Given a feasible solution $p$ to linear program (2), applying Algorithm 3 yields a post-processed solution $p′$ such that any sampling of $k$ proposals with marginal probabilities $p′$ satisfies ex post validity. Moreover, the objective value achieved by $p′$ is greater than or equal to that of $p$.

   We intend the current statement of the Theorem to capture the usefulness of the post-processing algorithm to the MERIT algorithm, which takes as input an ex ante optimal solution to the problem. We will add the above statement as a remark to capture that the post-processing algorithm could be more generally useful.

4. Discussion of fairness: Our algorithm satisfies two notions of "fairness." First, one can think of the "ex-post validity" criteria as a type of fairness in randomized decisions that states that if one candidate is clearly better than another candidate, the stronger candidate should not be randomly rejected when the weaker candidate is accepted. Second, the motivation for our work starts from the premise that it is unfair that two candidates can have similar point estimates of quality, but large differences in outcomes under deterministic selection rules. Randomization is more "fair" in tie-breaking between these similar quality proposals than accepting one and rejecting another. In fact, existing literature [FC16,HTB22,GLM+25] argues that deterministic grant peer review processes introduce unfairness by tie-breaking between proposals on arbitrary, spurious signals like applicant seniority or preference for a certain subject area. We will add discussion of such fairness considerations to our Discussion section.

We will make sure to fix the typos you point out in the final version of the paper.

References

[FC16] Ferric C Fang and Arturo Casadevall. Research funding: The case for a modified lottery, 2016.

[HTB22] Serge PJM Horbach, Joeri K Tijdink, and Lex M Bouter. Partial lottery can make grant allocation more fair, more efficient, and more diverse. Science and Public Policy, 49(4):580–582, 2022.

[GLM+25] Joanie Sims Gould, Anne M Lasinsky, Adrian Mota, Karim M Khan, and Clare L Ardern. Threats to grant peer review: A qualitative study. BMJ Open, 15(2):e091666, 2025.

Thank you for your careful reading of the paper and thoughtful comments on the problem formulation and axiomatic analysis. We address concerns below:

1. Motivating ex-ante optimality: this is a great point, as the average case and worst case models are the two most common modeling assumptions in the ML/CS/Statistics literature for selection and estimation problems. An “average case model” would assume that the funder has a probabilistic model of the data (e.g., a probability distribution over candidates’ true qualities and reviewers’ errors in assessing candidate quality). Our ex-ante optimality formulation starts from the assumption that a funder lacks such a probabilistic model of their data and therefore operates under Knightian uncertainty. We believe this assumption that a funder lacks a probabilistic model is appropriate for settings where quality is determined by peer review for a number of reasons motivated by prior work. First, there are many proposed probabilistic models of reviewer errors, but these models have generally performed poorly in real deployments [Sha22, Section ‘Miscalibration’]. One possible reason for this poor performance is that in practice, human miscalibration is more complex than simple models [BGK05]. Further, decision makers generally lack ground truth data with which to evaluate whether a given probabilistic model of review data is appropriate in a given setting, making it difficult to rely on a probabilistic model. Additionally, even if a decision maker had a reasonable model of review scores, these probabilities may not represent meaningful probabilities to the funder, since review scales are arbitrary and likely do not map linearly to the utility of selecting a candidate. Finally, the future success of candidates in the settings we consider may be inherently hard to predict. In the context of scientific peer review, many works find that it is difficult to predict future citations and peer review scores are poor predictors [SWL+22, CL21, WDH+24]. In the related area of college admissions, recent work demonstrated that predictive models used to rank candidates are highly unstable with respect to their training data, resulting in arbitrariness in the ranking from any single model [LHZ+24]. For these reasons, we assume that the funder does not have a reliable probabilistic model of their data.

   Thus, the funder operates in a setting of “Knightian uncertainty” where they estimate intervals that convey information about the relative quality of proposals but are not trusted as probabilistic estimates. We answer the question—how should they make decisions given this interval ordering? Given that any ordering consistent with the intervals is consistent with the decision maker’s information, taking the worst case over this set of orderings is a natural objective to optimize over their uncertainty.

   We will expand on this motivation in the final version of the paper.

   Additionally, one interesting direction along the lines you propose would be a smoothed analysis, which combines the average and worst case model. We conjecture that our MERIT algorithm would continue to be near optimal for that as well, but leave a formal investigation to future work.

2. Monotonicity vs. ex-ante optimality: We provide a simple algorithm to enforce monotonicity with MERIT in Appendix D, which allows for violation of ex-ante optimality, thereby prioritizing the monotonicity property. We agree that monotonicity is a natural property. Interestingly, our analysis shows that existing randomized mechanisms of the Swiss NSF and randomize-above-threshold (which do not satisfy ex-ante optimality) also violate monotonicity.

3. Results in Table 1: Since there is no ground truth, we cannot say which algorithm is “better” at selecting high quality candidates based on this real world data. Therefore, our results in table 1 are meant to provide an interesting qualitative comparison of applying the Swiss NSF’s method and MERIT in real settings. The biggest takeaway in qualitative differences is that MERIT does in practice assign variable probabilities to different applicants (allowing for more nuanced lotteries), while Swiss NSF enforces a uniform lottery.

References

[Sha22] Nihar B Shah. An overview of challenges, experiments, and computational solutions in peer review, June 2022.

[BGK05] Lyle Brenner, Dale Griffin, and Derek J Koehler. Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment. Organizational Behavior and Human Decision Processes, 97(1):64–81, 2005.

[CL21] Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment. arXiv Preprint arXiv:2109.09774, 2021.

[WDH+24] Aidan S Weitzner, Matthew Davis, Andrew H Han, Olivia O Liu, Anuj B Patel, Brian D Sites, and Steven P Cohen. How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal. Regional Anesthesia & Pain Medicine, 2024.

[LHZ+24] Jinsook Lee, Emma Harvey, Joyce Zhou, Nikhil Garg, Thorsten Joachims, and Rene F Kizilcec. Algorithms for college admissions decision support: Impacts of policy change and inherent variability. arXiv preprint arXiv:2407.11199, 2024.