A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning

References:

We appreciate the reviewer's suggestion and confirm that the highlighted literature is relevant to our study, and we will include citations in the revised manuscript.

Realistic cases (Weaknesses 1):

In the medical field, when collecting information from various patients, we may obtain a long-tailed dataset from non-specialized hospitals, i.e., a large number of common disease cases (head classes) accompanied by very few rare disease cases (tail classes). However, if we consider specialized hospitals focused on specific rare diseases, this scenario would yield an inverse long-tailed dataset characterized by abundant rare disease cases and scarce common disease cases. We will add this example in the final version.

In practical applications, where the distribution of unlabeled data is unknown, we follow recent works to investigate three representative extreme distribution scenarios. If a model can perform well on all those three extreme cases like ours, it is expected to fit different unlabeled data distributions.

Model design (Weaknesses 2):

To effectively address three different unlabeled data distributions, we constructed three MLP-based experts with logit adjustment, specifically designed for long-tailed, uniform, and inverse long-tailed distributions, respectively. Furthermore, we observed that different depth features and differnt experts exhibit its own properties (as evidenced in Tables 1-3 and Fig. 1). To fully leverage the characteristics of multi-expert and multi-layer features, we employed DEA and MFF for respective expert and feature integration. Table 6 validates the effectiveness of DEA and MFF. Moreover, we investigated the performance differences between addition-based and concatenation-based feature operation strategies for feature fusion in Table 9, ultimately adopting the superior addition-based feature operation strategy.

Training Stability (Weakness 3):

Our empirical validation demonstrates consistent optimization behavior across multiple runs, as evidenced by the lower standard deviations reported in Tables 4-5 and 7-8 (which are comparable to or better than those of baselines). We further provide the accuracy curve in https://anonymous.4open.science/r/Acc-curve/Acc-curve.png, which shows that our method performs stable in the later stages of training.

Computational Complexity (Weankess 4):

Please refer to the response to reviewer wN5j's Computational overhead (Weakness 2).

More experiments (Experimental Design 1):

Following recent works like CPE and BaCon, we conducted experiments across three datasets (CIFAR-10-LT, STL-10-LT, and SVHN-LT), each evaluated under two different imbalance ratios (150 and 200). As suggested, we have extended our evaluation to include CIFAR-100-LT, along with conducting evaluations on CIFAR-10-LT and STL-10-LT under lower imbalance ratios. The results presented in Tables R2 and R3 [refer to the responce to reviewer sReE's Questions 1 (Aligned experimental setting)] demonstrate that our method achieves comparable or superior performance even on these standardized benchmarks, with more significant performance improvements observed under higher imbalance ratios.

Results of CPE (Experimental Design 2 & Weakness 3 & Question 2):

We rigorously evaluated baselines (like CPE) using their official codes under the specified conditions, re-implementing experiments with matched hyperparameters and three independent runs.

Effectivness of DEA and MFF (Experimental Design 3):

The ablation results in Table 6 demonstrate that: the DEA module provides an average 1.68% accuracy improvement across all imbalance ratios (γ_u = 200/1/1/200), showing relatively greater performance gains; while the MFF module delivers a 0.76% average gain, which though comparatively smaller, remains statistically significant. The combined DEA+MFF configuration achieves 2.42% improvement, confirming their complementary effectiveness and synergistic interaction. These improvements conclusively validate the effectiveness of our proposed modules in enhancing model robustness across diverse imbalance ratios.

Multi-scale feature (Weaknesses 1 and Questions 5):

We acknowledge existing discussions about multi-scale feature characteristics in deep learning. However, to the best of our knowledge, no prior work has investigated how long-tailed distributions differentially affect shallow and deep features. Our analysis reveals that shallow features are relatively balanced although less discriminative, and deep features improve the discriminative ability but are less balanced. We strategically leverage this complementary characteristic to achieve performance gains through fusion of different depth features. Specifically, the ablation results in Table 6 demonstrate that the MFF module delivers a 0.76% average gain, confirming its significant effectiveness.

Expert attention (Weaknesses 2 and Questions 4):

The expert attention scope in our DEA module follows the partitioning in CPE while maintaining comparable performance to alternative scoping strategies. As shown in Table R5, we conducted additional experiments with alternative splits on CIFAR-10-LT. The average performance fluctuation of our method during partitioning changes is only 0.37%, which is smaller than CPE's 0.81%, verifying the DEA module's robustness regardless of specific partitions.

Table R5: CIFAR-10-LT with (N_1, M_1, γ_l) = (1500, 3000, 200)

DEA training (Questions 1):

To address parameter optimization for the DEA module under poor initial pseudo-labels, we introduced a confidence threshold (t = 0.95) to filter unreliable pseudo-labels following CPE and ACR. To quantify its impact, we evaluated the pseudo-label accuracy with and without the threshold in the following Table R6, which suggests incorporating threshold improves accuracy by 3.85% on average.

Table R6: CIFAR-10-LT with (N_1, M_, γ_l) = (1500, 3000, 200)

Fusion strategy (Questions 3):

CPE does not use fusion strategy and employs three experts simultaneously to produce pseudo-labels for all samples, along with a uniform expert to make predictions. In contrast, our DEA module learns class membership automatically and utilizes it to select the most appropriate single expert for each sample. This strategy avoids error accumulation caused by conflicting predictions in CPE, thereby significantly improving pseudo-label quality. Our ablation study in Table 6 quantitatively demonstrates that DEA achieves an average 1.68% accuracy improvement across all imbalance ratios (γ_u = 200, 1, 1/200) compared to CPE.

Aggregator (Questions 6):

The aggregator model processes both strong and weak augmentation views simultaneously, generating $y_{m, j}^{u}$ and $\hat{y_j}$ respectively. The aggregator’s parameters are updated through a unified optimization process. Specifically, the model is trained to minimize the consistency loss between $y_{m, j}^{u}$ and $\hat{y_j}$, ensuring that predictions align across augmentation views.

Questions 1 (Aligned experimental setting):

Our work focuses on advancing long-tailed semi-supervised learning, thus primarily investigating settings with higher imbalance ratios (the more challenging scenarios) compared to recent works. As suggested, we have conducted supplementary experiments on CIFAR-10-LT, CIFAR-100-LT, and STL-10-LT under lower imbalance ratios aligned with recent works. As empirically demonstrated in Tables R2 and R3, our method achieves comparable or superior performance even in these standardized benchmarks: attaining better results in 5/6 cases and comparable performance in 1/6 cases on CIFAR-10-LT (with maximum gains of +1.19%), showing +1.06% improvement on STL-10-LT and +0.58% average improvement on CIFAR-100-LT. Moreover, our advantages become even more pronounced under higher imbalance ratios: on CIFAR-10-LT with (N_1, M_1) = (1500, 3000), our method achieves performance gains of +0.23% (γ_l = 100), +1.26% (γ_l = 150), and +1.64% (γ_l = 200) over previous SOTA methods.

Table R2: CIFAR-10-LT with (N1, M1, γl) = (500, 4000, 100) (left three columns) and (N1, M1, γl) = (1500, 3000, 100) (right three columns)

Table R3: STL-10-LT with (N1, M1, γl) = (150, 100K, 10) (left one column) and CIFAR-100-LT with (N1, M1, γl) = (150, 300, 10) (right three columns)

Questions 2 (Upper-E):

Upper-E denotes CPE directly use GT membership to select a specific expert to produce pseudo-labels and make predictions, while our method utilizes the integration of the logits from the three experts to compute the loss in Eq. (4), which may push different experts learn better. Moreover, as shown in Eq. (3), the final prediction of our method is the soft ensemble of multiple experts, which further brings better performance.

To provide empirical validation, we replaced our DEA module with the GT membership and conduct experiment on CIFAR-10-LT. The experimental results in Table R4 demonstrate that: i) The design motivation for employing the DEA module to orchestrate expert collaborations through learned membership relationships is empirically well-founded, ii) The MFF module can helps experts toward more optimal direction by fusing different depth features.

Table R4: CIFAR-10-LT with (N_1, M_1, γ_l) = (1500, 3000, 200)

References:

We appreciate the reviewer's suggestion and confirm that the highlighted literature is relevant to our study, and we will include citations in the revised manuscript.

Properties of different layers (Question & Weakness1):

In deep networks, shallow layers capture local patterns while deep layers learn global semantics. For long-tailed learning, since head and tail classes may share similar local patterns, shallow features exhibit balanced discriminability across classes. Meanwhile, deep layers predominantly encode head class semantics due to their overwhelming sample dominance, thus biasing predictions toward head classes. This is empirically supported by Table 2: shallow features are relatively balanced although less discriminative, and deep features improve the discriminative ability but are less balanced.

Our backbone network employs WideResNet-28-2, which consists of three convolutional block groups. We use the outputs of these three block groups (the 10th, 19th, and 28th layers) to represent shallow, middle, and deep features respectively. While our primary analysis focuses on features from these three specified layers, the phenomenon observed in Table 2 is general applicability. To verify this, we conducted extended experiments using alternative intermediate layers (the 6th, 15th, and 24th layers), with supplementary results presented in the floowing table, which aligns with Table 2's observations, suggesting that the selection of different layers does not significantly impact the revealed observations.

Table R1

Computational overhead (Weakness 2):

While our proposed modules (three experts, MFF, and DEA) introduce a controlled parameter increase of 13.3% (1.5M → 1.7M), this design achieves strategically balanced efficiency-performance trade-offs. Experimental results on CIFAR-10-LT demonstrate: a 6.4% increase in epoch time (234.5s → 249.5s), a 1.6s increased in inference time for evaluating 10,000 samples (7.1s → 8.8s), and a substantial accuracy gain of +3.5% absolute improvement (71.9% → 74.4%), collectively indicating significant performance enhancement with modest computational overhead.