Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies

We sincerely appreciate the reviewers' acknowledgement of this paper’s writing quality and real-world impact. Below, we provide clarifications addressing the weaknesses and questions:

Weakness 1 & Question 1: Group Fairness Definition

Our definitions of group fairness are inspired by utilitarian social welfare, which measures the average desk-rejection damage across all authors (see lines 282–283 on page 6). This differs from the conventional notion of group fairness, which assesses fairness between distinct groups. In light of this, we plan to rename our “group fairness” metric to “average fairness” (or “entire fairness”) in the next version to avoid confusion.

Weakness 2: Simulation Setups

Although direct testing on real backend desk-rejection data is difficult, we have added evaluation on our approach using public ICLR (2021–2023) data from the OpenReview API. We compared the current desk-rejection method (Algorithm 2, p.8), with our proposed fairness-aware method (Algorithm 1, p.7). Using the Python PuLP library with submission limits $x=[4,6,8,10,12,14]$, we obtained the results shown below:

Table R.1: ICLR ’21 Results

Table R.2: ICLR ’22 Results

Table R.3: ICLR ’23 Results

Here, we take the group fairness metric (Definition 5.5), which represents the average cost function (Definition 5.2) across all authors. For instance, a value of 0.02 indicates that on average an author experiences a 2% paper desk-rejection rate. For more details, please refer to W2 of our response to Reviewer UGt2.

Weakness 3: Approximate Solution for Individual Fairness

In practice, one could relax the discrete constraints to make $r$ continuous and use softmax to substitute the infinity norm, leading to an alternative objective: $\min_{r \in [0,1]^m} \mathsf{Softmax}(1_n - D^{-1}Wr) \quad \text{s.t.} \quad \frac{Wr}{x} \leq 1_n.$ This formulation allows for smooth optimization with standard convex solvers. We note that the minimizer of this relaxed objective is not guaranteed to be optimal for the original problem, and we leave a detailed empirical evaluation of this approximation for future work. Additionally, existing mixed programming solvers can handle some medium-scale and non-worst-case problems. For additional discussion, please refer to W3 of our response to Reviewer UGt2.

Weakness 4: Proposition 5.6

We thank the reviewer for pointing this out. We agree that the proof in Appendix B is correct, and the version in the main text contains a typo where $\leq$ should be $\geq$. We will correct this in our next revision.

Weakness 5: Baselines

We appreciate the reviewer’s concern regarding baselines. The number of baselines is limited because we are addressing a novel problem that, to the best of our knowledge, has not been studied before (its novelty has also been acknowledged by Reviewer UGt2 and Reviewer q5TE). Our chosen desk-rejection method serves as the only existing baseline and is widely adopted in top conferences, not just CVPR but also other conferences like KDD.

Question 2: Difference Compared with Fair Selection Under Quotas

Thanks for introducing these relevant works, and we will cite them in future revisions. We believe these works are quite different from ours. First, considering [EGGL’20], they introduce a two-group setting where candidates are selected from two different groups, and the selection rates from these groups are forced to be close. In contrast, in our paper, we use the term group fairness in Definition 5.5 and individual fairness in Definition 5.4 to denote the average or maximum cost function across all authors without any pre-defined group division. Besides, although we also have a quota (i.e., paper submission limit) similar to the ranking fairness paper [YXA’23], we do not incorporate mechanisms such as ranking, consumer probability, or examination probability. This means our setting is fundamentally different.

References

[EGGL’20] On Fair Selection in the Presence of Implicit Variance. EC 2020.

[YXA’23] Vertical Allocation-based Fair Exposure Amortizing in Ranking. SIGIR 2023.

We sincerely thank the reviewer for recognizing the novelty, mathematical rigor, and potential social impact of our work. We appreciate the constructive feedback and address the concerns as follows:

Weakness 1: The Severity of the Problem

We acknowledge that direct evaluation on real conference desk‐rejection data is challenging due to data access restrictions. Although a direct evaluation is infeasible at this stage, several indirect observations support the issue’s significance.

1. First, submission numbers in conferences have surged. For instance, NeurIPS increased from 6,743 submissions in 2019 to 15,671 in 2024, and ICML from 3,424 to 9,653 in the same period.
2. Moreover, Table 1 of our paper shows that many major conferences (e.g., CVPR, ICCV, AAAI, IJCAI, KDD, WSDM) have adopted submission-limit-based desk-rejection policies.
3. Besides, we conducted a new analysis on the ICLR 2023 data, collecting 12,451 authors and 3,793 papers using the OpenReview API. Under an 8-paper limit (as applied by IJCAI 2021–2025), we found that 506 papers (13.3%) had authors exceeding the limit, affecting 2,114 authors (17.0%). This new evidence further strengthens the urgency of addressing fairness issues in current policies.

Weakness 2 & Question 1: Effectiveness of the Method and Real-World Validation

Although direct testing on real backend desk-rejection data is difficult, we have empirically evaluated our approach using public ICLR (2021–2023) data from the OpenReview API. We compared the current desk-rejection method (Algorithm 2, page 8), which rejects all papers with non-compliant authors based solely on submission order, with our proposed group fairness optimization method (Algorithm 1, page 7), which prioritizes rejecting papers from senior researchers with many submissions to better protect junior researchers. Using the Python PuLP library and setting submission limits $x=[4,6,8,10,12,14]$, we obtained the following results:

Table R.1: ICLR ’21 Results

Table R.2: ICLR ’22 Results

Table R.3: ICLR ’23 Results

The results in these tables use the group fairness metric (Definition 5.5), representing the average cost function (Definition 5.2) across all authors. For example, a value of 0.02 indicates that, on average, an author has 2% of their papers rejected. Since a lower fairness metric indicates a stronger fairness guarantee, we can conclude that our method consistently achieves a significant cost reduction compared to the conventional approach.

Weakness 3: NP-Hardness of Individual Fairness

We agree that it may not be practical to directly solve the mixed programming problem in Definition 5.7 and Definition 5.9. First, we would like to clarify that proving the hardness for the individual fairness could be considered a strong technical contribution of our work. By establishing that under submission-limit-based desk-rejection the individual fairness objective may be impossible to optimally optimize, we highlight critical fairness concerns inherent in the desk-rejection mechanism.

In real-world scenarios, however, we can relax the discrete constraints on $r$ and employ a softmax transformation to make the infinity norm continuous. Concretely, we consider the alternative objective: $\min_{r \in [0,1]^m} \mathsf{Softmax}(1_n - D^{-1}Wr) \quad \text{s.t.} \quad (Wr)/x \leq 1_n.$ This formulation allows for smooth optimization with standard convex solvers. We acknowledge that the minimizer of this relaxed objective is not guaranteed to be the minimizer of the original problem (Definitions 5.7 and 5.9), and we leave an empirical evaluation of this approximation to future work.

Another feasible approach is to use existing mixed-integer programming solvers. Considering some instances that are not worst-case and in moderate-sized conferences, this alternative may also yield acceptable solutions in practice.

Other Comment 1: Name of Fairness Metrics

We thank the reviewer for the suggestion on the names of fairness metrics. We will adopt your advice and improve the naming in the next version of our paper.

We would like to thank the reviewer for acknowledging the novelty of our work and the strong support for our claims. We are pleased to further clarify the motivation behind our paper and to discuss the equilibrium perspective.

Weakness 1 & Question 1: Equilibrium Effects

Thank you for this interesting and inspiring comment. We believe that the issue raised is highly worthwhile for future study. However, it relates more to the motivation behind the problem than to the flaw in our technical contributions.

1. Our work demonstrates that even before reaching equilibrium, a submission-limit desk-rejection policy can lead to unfair outcomes. The gaming toward equilibrium mentioned in your comment does not mitigate this unfairness but may, in fact, worsen it. Therefore, we propose alternative approaches that may address the fairness issue.

2. We agree that in the long term, gaming toward equilibrium might occur after applying a fairness-aware desk-rejection strategy. This is an important question for analyzing the effects of different policies at equilibrium, and we leave a detailed study of this topic for future work. We hope that our current work, as the first analysis of the fairness of desk-rejection methods, serves as a strong starting point.

3. To incorporate equilibrium effects into this work, we can formally define the problem as follows. Consider that there are $n$ authors. Each author must decide which co-authors to collaborate with and how many projects to pursue. The cost represents the time or resources required for each project. The naive reward is that a paper is not desk rejected, while the final reward is the paper being accepted. The key question is what the optimal strategy is for each author in selecting collaborators and projects. We acknowledge that this formulation omits several realistic considerations, such as relationships among authors, project idea ownership, resource allocation, and more. Solving for the Nash equilibrium of this game would require a more detailed and concrete formulation, which is beyond the scope of the current paper and rebuttal. Nonetheless, we recognize this as an interesting direction for future work.

Weakness 2: Why Authors Submit More Papers Than the Limit

This phenomenon is common among well-known professors who supervise many students and often do not have the time to verify whether the submission limit is exceeded. They may not be aware of the exact number of papers they have submitted. For instance, in conferences like CVPR and ICCV, a 25-paper limit is sometimes necessary to manage submissions, which highlights how busy these prominent professors can be.

Weakness 3: Excluding Senior Authors from the Author List

Different research communities have varying traditions regarding authorship. In some cases, it is difficult to exclude senior researchers because they may have provided essential funding or other support. Excluding them could lead to lab politics and harm the relationships between junior and senior researchers, with potentially severe consequences, such as affecting recommendation letters or graduation permissions.

Weakness 4: Senior Researchers Working with Fewer Junior Researchers

Most projects require significant hands-on contributions, which senior authors often cannot provide due to time constraints. To maximize their outcomes, senior researchers typically prefer to collaborate with junior researchers on each project. Additionally, many senior researchers have a responsibility to work with junior colleagues, such as PhD advisors with their students or tech leads with their engineers.

In this paper, we examine the desk-rejection problem from a conference organizer’s perspective, where over-submission has already occurred. Our goal is to minimize the cost imposed on junior researchers, thereby protecting them from the negative impacts of submission-limit violations.