Towards Realistic Long-tailed Semi-supervised Learning in an Open World

1. The novelty of the proposed method seems limited. As stated in the Strengths, the proposed method is simple and easy to implement because it is pretty much built upon existing techniques. However, this also makes the contribution of the method minor.

2. The rationale behind the method is not well-explained. It is unclear why the logit adjustment strategies for labeled and unlabeled data are different. Also, the design of Eq. (3) is not straightforward and needs in-depth understanding and justification. In Eq. (4), the impact of hyperparameters $\alpha$ and $\beta$ are not studied.

3. The writing needs improvement. For instance, the paper should include the definition of $\mathcal{L}\_{pair}$ and $\mathcal{L}\_{reg}$ to be self-contained.

4. It would be better if the paper could include real-world semi-supervised datasets which follow long-tailed label distribution and also include novel classes in unlabeled data.

5. It seems that the logit adjustment strategy in open-world semi-supervised learning has been considered in [1].

[1] Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning. IJCAI 2024.

1. Since readers may not be quite familiar with the development of open-world long-tail SSL, it is suggested that more background about the existing OLSSL algorithms be included in Section 3.2, especially for the definition of Eq(1).
2. The dual-stage post-hoc logit adjustment section is not easy to read. Specifically,

• why the aremax is used in Eq(2)? According to the Fig 1, it is expected to be a distribution adjustment.
• What is the definition of F_{y_i^l} in Eq(3)? If the labeled samples are limited, will this sample frequency be biased? How C_base and S_base are defined?
• What is the relationship of the equation in L244 and the Eq(2)?
• for unlabeled samples, how the sample ratio \pi_c^r in Eq(4) is obtained when the class of the unlabeled samples is unknown? How alpha and beta are defined? What is the meaning of (alpha-beta) in Eq(4)?

3. Does the \tau in Eq(7) denote the ones in Eq(2)? If so, why the operation become plus but not minus as in Eq(2)?
4. The proposed method is argued to be a two-stage method, does it mean training labeled and unlabeled samples in different stages? However, according to Eq(8), the model seems optimized on labeled and unlabeled samples together.
5. In Figure 1, the right side of the diagram shows a question mark ("?") symbol, does it represent the unlabeled samples? If so, why does the labeled branch also include a depiction of unlabeled samples? Is there a particular significance or reason for this choice in the illustration?
6. The paper seems primarily focused on addressing the long-tailed problem while offering little innovation for the open-world problem.
7. In the defined Realistic Open-World Long-Tailed Semi-Supervised Learning (ROLSSL) framework, it is assumed that the number of novel classes is known (L200). The proposed method leverages this prior knowledge for model calibration (Eq(3)). However, this assumption seems somewhat unrealistic, as it is uncommon to have prior knowledge of the exact number of novel classes present in the unlabeled samples.

1. A key concern is the complexity of the proposed pipeline, which involves multiple stages and numerous hyper-parameters. With additional stages and hyper-parameters to tune, it becomes more challenging to generalize this pipeline to new problem settings and datasets. For instance, as shown in Tables 3 and 4, using non-optimal scaling factors can significantly impact performance on CIFAR-100 and ImageNet-100, despite both datasets having the same number of classes. Have you explored methods to automatically tune the hyperparameters or analyzed which parameters are most sensitive?

2. Technical contributions. A primary contribution of this paper is the proposed logit adjustment method for pseudo-labels, which often exhibit class imbalance. However, the issue of naturally imbalanced pseudo-labels has been previously addressed by DebiasPL [1], which also applies a logit adjustment based technique to mitigate class imbalance in pseudo-labels, resulting in improved classification performance on long-tailed datasets. A discussion on the technical distinctions between this approach and DebiasPL would be beneficial.

3. While this paper presents experiments on several benchmarks to demonstrate performance gains, most datasets used are small to medium-scale, such as ImageNet-100 or CIFAR. I believe it is essential to include experiments on larger datasets, such as iNaturalist or ImageNet-1k or ImageNet-22k, to better assess the generalizability of the proposed method. It will be nice if you could provide any potential challenges or modifications needed to apply your method to larger datasets.

4. In addition to presenting results on novel and known subsets, I believe it is essential to include performance metrics for many-shot, medium-shot, and few-shot subsets. This would help clarify whether the current logit adjustment methods compromise performance on many-shot classes to improve results on few-shot classes—a common issue with most logit adjustment techniques. It would be valuable for the authors to discuss whether their proposed method faces this challenge and, if not, to explain why it avoids this problem. If it does, providing insights into potential strategies to address this limitation in future work would be beneficial.

5. I may suggest authors to include a specific breakdown of performance metrics for these subsets in their results tables. Add a discussion section that explicitly addresses the trade-offs between performance on different shot categories and potential strategies to balance these trade-offs. In my opinion, while improving performance on few-shot classes is important, it is equally crucial to avoid jeopardizing performance on many-shot classes, as they are more prevalent in real-world applications.

[1] Wang, Xudong, Zhirong Wu, Long Lian, and Stella X. Yu. "Debiased learning from naturally imbalanced pseudo-labels." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657. 2022.

1. The comparative methods are outdated, and the most recent comparative method of this paper is ECCV 2022. In the literature, OSSL is also referred to as generalized category discovery (GCD). Some recent GCD methods, including GCD [R1] and SimGCD [R2], should be included in Table 2 and 3. There are also some works [R3, R4] in GCD considering long-tailed scenarios, which are highly related to this work and should be compared.
2. The novelty of this paper is limited. The basic loss functions $L_{pair}$ and $L_{reg}$ are well-known techniques in novel class discovery and GCD, while the logit adjustments are also prevalent methods in long-tailed classification. Simply applying model predictions in logit adjustments is naïve and might not provide great insights to the community.
3. The method still requires strong priors of the ground truth new class numbers, which is unrealistic.
4. Experiments with the ViT backbone should be conducted.
5. The notations should be concise and clear.

References: [R1]. Generalized Category Discovery. CVPR 2022. [R2]. Parametric Classification for Generalized Category Discovery: A Baseline Study. ICCV 2023. [R3]. Novel Class Discovery for Long-tailed Recognition. TMLR 2023. [R4]. Generalized Categories Discovery for Long-tailed Recognition. ICCV 2023, workshop.

1. The writing of this paper is not very clear. For example, 'S/N Consistency' in Tab. 1, a key feature of the proposed setting, is never explained.
2. Open-world Long-tailed Semi-supervised Learning seems quite similar to [*1], which is not discussed in related works. Therefore, the novelty of this paper may be weak.
3. Experiments may be insufficient. Only some tiny datasets are considered. Large-scale benchmarks (e.g., ImageNet-LT or iNaturalist) should be included.

[*1] SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning. ICML'24