Pairwise Maximum Likelihood For Multi-Class Logistic Regression Model With Multiple Rare Classes

We thank the reviewer for the constructive suggestions. The concerns have been well addressed as follows.

1. Motivation and Practical Impact.
   • Theoretical Motivation. The focus here is the theoretical investigation of logistic regression with rare classes. Specifically, we find that the asymptotic covariance of the resulting MLE is block-diagonal, which further inspires the novel PMLE method. All the discussions in the introduction about customer purchase behavior are used to demonstrate the practical importance of our problem. However, with your kind reminder, we have realized that we might have overemphasized this point. We now have rewritten the section for better elaboration.
   • Comparison with Two-Stage Approach. We consider a very ideal situation with all car plates having been correctly detected. All rare classes have been perfectly separated from the major one. We then only need to focus on rare classes. The detailed classification results on the TikTok Screenshots (TTS) dataset are given in Table A.1. Even under this ideal situation, the resulting classification accuracy (RARE) is much worse than our one-stage strategy (GMLE, PMLE and SPMLE). This is not surprising since the useful information contained in the major class was not effectively used in the second stage. In contrast, all the information from both major and rare classes is comprehensively used by the one-stage strategies.

Table A.1: Prediction results for the TTS dataset.

2. Theoretical Properties.

   • Assumption. The theoretical assumption $\alpha_N \to -\infty$ and $\alpha_N+\log N \to \infty$ are used for allowing rigorous asymptotic analysis. This leads to the block diagonal structure of the asymptotic covariance for MLE, and further leads to the novel PMLE method. Without this specific condition, we can never obtain those theoretical findings and then be inspired to develop PMLE method. Moreover, those conditions seem to be very standard in the literature. See, for example, Equation (2) in Wang (2020, ICML) and Section 2 in Wang et al. (2021, NeurIPS).
   • Convergence Rate. The convergence rate reflects that sufficient amount of samples must be provided for both major and every rare class. Otherwise, no consistent parameter estimates can be obtained. The much slower convergence rate of $1/\sqrt{Ne^{\alpha_N}}$ is as expected. In fact, this is a phenomenon having been well documented in the literature. See, for example, Wang (2020, ICML), Song and Zou (2024, TIT), and Wang et al. (2021, NeurIPS). We now have made this point clear in the revision.

3. Empirical Comparison with Baseline Methods. We have included the following methods for comparison on the TTS dataset: the focal loss (FL) of Lin et al. (2017), the class-balanced loss (CBL) of Cui et al. (2019), the cost sensitive loss (CSL) and random downsampling (RDS) of Fernández et al. (2018). All methods are optimized according to the suggestions of the original papers. The results are summarized below in Table A.2. We are happy to report that our method outperforms all competitors.

Table A.2: Prediction results for the TTS dataset.

4. Minor Issues. For the other issues, we have also addressed them carefully.
   • “NR algorithm” has been referred to as the Newton-Raphson algorithm in the revision.
   • The code has been made publicly available at https://anonymous.4open.science/r/Anony-63CC.

Additional References

1. Cui, Y., Jia, M., Lin, T. Y., Song, Y., & Belongie, S., 2019, Class-balanced loss based on effective number of samples. CVPR.
2. Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P., 2017, Focal loss for dense object detection. ICCV.
3. Song, Y., & Zou, H., 2024, Minimax optimal rates with heavily imbalanced binary data. IEEE TIT.
4. Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., & Herrera, F, 2018, Learning from imbalanced data sets. Springer.

We thank the reviewer for all the constructive suggestions. All the concerns have been well received and carefully addressed as follows.

1. Relationship between Two Models. Note that the model (2) is a special case of the model (1), which is a more general model. As you have correctly noted, we can indeed start with the model (2) directly. However, we then lose the opportunity to explain why the model (2) is specified in its current form. Specifically, by starting from the general model (1), it seems clear that we have to set $\alpha_{N k}\rightarrow -\infty$. Otherwise, the rare class phenomenon cannot be theoretically reflected. Moreover, we cannot have balanced rare classes unless $\alpha_{N k_1}-\alpha_{N k_2}=O(1)$ for any $k_1\neq k_2$. Both the conditions (i.e., $\alpha_{N k}\rightarrow -\infty$ and $\alpha_{N k_1}-\alpha_{N k_2}=O(1)$) enable us to re-parameterize $\alpha_{Nk} = \alpha_{N}+\beta_{0k}$. That leads to the model (2). We now have explained this issue in the revision more clearly. Thank you so much for this kind reminder.

2. Imbalanced Class Assumption. Requiring the rare classes to be sufficiently balanced is a relatively loose sufficient condition and not very necessary. In fact, if rare classes are unbalanced, PMLE can still be applied without any difficulty. However, there is one critical condition. That is the sample size of the target rare class must be sufficiently large. Otherwise, no consistent classifier can be learned. This seems to be a very basic and reasonable condition, which has been widely used in the past literature (Wang, 2020, ICML; Li et al., 2024, JMLR; Wang et al., 2021, NeurIPS). However, this very basic and reasonable condition can be easily violated, if rare classes are highly unbalanced. In such cases, some classes may be even rarer than others. We refer to those even rarer classes as tiny classes, whose sample sizes are often too tiny to support any meaningful statistical learning. This is the only reason we focus on the balanced case in our theoretical analysis. We now have made it clear in the revision.

3. Multiple Major Classes. Our method can be readily extended to a more general setting with multiple major classes. To fix the idea, consider, for example, the case with two major classes. Denote the two major classes by $k = 0$ and $k = 1$, respectively. Then, the logistic regression model becomes:

We thank the reviewer for all the constructive suggestions. All the concerns have been well received and carefully addressed as follows.

1. Diverging Number of Classes. With a diverging $K$, the expected percentage of rare classes should be even smaller. To ensure a diverging sample size for each rare class, the technical assumption $\alpha_N \to -\infty$ should be replaced by $\alpha_N + \log K \to -\infty$. The resulting theoretical behavior of our proposed methods remains fairly the same. More specifically, the convergence rate of the PMLE remains $\sqrt{Ne^{\alpha_N}}$-consistent. The PMLEs associated with different rare classes remain mutually independent asymptotically. More importantly, the resulting PMLE should be asymptotically as efficient as GMLE. To numerically demonstrate this point, we replicate the simulation example but with $K = [N^{0.25}]$. See below Table A for the detailed simulation results. We find that (1) both GMLE and PMLE remain statistically consistent, and (2) their asymptotic efficiency seems to be the same with extremely similar RMSE values. We now have made this point clear in the revision.

Table A: Simulation results with diverging $K$.

2. Minor Issues. For the other issues that you have mentioned, we have also corrected them carefully.

(1) In the revision, we have reorganized Lines 629-639 in the Appendix.

(2) In Line 93, it should be $1 \le i \le N$. We now have made it clear in the revision.