Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model

2. Tabular dataset:
   • Forest: A structured tabular dataset, used to investigate the impact of unseen class data in non-image modalities.

Q4: Some typos should be checked. For example, in Eq. (1), it should be argmin. In Line 431, it should be Table 5.

A4: Thank you for pointing out the typos. We have carefully reviewed the manuscript to correct these errors. Specifically, we have updated Eq.(1) to use "argmin" as suggested, and we have corrected the reference to "Table 5" in Line 431. We appreciate your attention to these details and have ensured that the manuscript is free from such typographical errors in the revised version.

Q5: Are the experimental results in tables the average accuracies with multiple seeds?

A5: Thank you for your question. Yes, the experimental results presented in the tables are the average accuracies obtained from multiple runs with different random seeds. Specifically, we conducted the experiments using seeds 0, 1, and 2, and the reported results are the average of these runs. This approach helps ensure the robustness and reliability of the results.

Q6: Is $C_{ib}$ the ratio between unseen and seen data?

A6: Thank you for your question. $C_{ib}$ is not the ratio between unseen and seen data. $C_{ib}$, or the imbalance factor, controls the distribution of samples within the unseen classes themselves. Specifically, it determines how the sample size for each class varies, creating different degrees of class imbalance within the unseen classes.

As mentioned earlier, when $C_{ib}$ is small, the earlier unseen classes will have fewer samples, and later classes will have more, leading to higher class imbalance. When $C_{ib}$ is larger, the sample sizes across the unseen classes become more balanced.

It is important to note that $C_{ib}$ does not directly refer to the ratio between unseen and seen data but instead to the imbalance within the unseen classes. We have enhanced the description of $C_{ib}$ in the revised manuscript to make this clearer.

Q7: Is each entry in Table 6 the mean of several C, i.e. the mean of each row?

A7: Thank you for your question. $F_{avg}$ represents the mean of each row, while $A_{avg}$ represents the mean of each column. We have enhanced the description of each entry in Table 6 in the revised manuscript to make this clearer.

We have updated the experimental results mentioned above in the Appendix of the revised manuscript.

Q3: The standard deviations of the experimental results should also be included.

A3: Thank you for your suggestion. We agree that including standard deviations is important to provide a more comprehensive evaluation of the experimental results. We have added the standard deviations and the updated results are shown in the table below:

3. Text dataset:
   • AGNews: A widely-used text dataset in natural language processing, included to assess how unseen class data affects semi-supervised models in text-based tasks.

Compared to tabular data, text modality data is relatively more affected by unseen classes. We attribute this to the greater difficulty in feature representation and the higher complexity of text-based tasks. Consistent with previous experimental findings, UASD is relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

These additional datasets aim to validate the effectiveness of our experimental setup in various domains, showcasing its practicality. We have incorporated the above experimental results and analysis into the revised manuscript.

• Letter: Another tabular dataset included to further explore the influence of unseen class data on semi-supervised models.

Based on the results from the two tables above, it can be observed that compared to image modality data, tabular data is less affected by unseen classes, and semi-supervised models demonstrate greater robustness on tabular data. We attribute this primarily to the simplicity of tabular data structure and the more effective feature representation. Moreover, similar to CIFAR, ImageNet, and CUB, ICT and UASD are relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

Q2: Only CIFAR datasets were used for experiments. Since previous SSL benchmarks provide more datasets from different modalities, such as text or tabular data, it is beneficial to consider more text and tabular datasets.

A2: In response to your concern, we would like to clarify that, in addition to CIFAR-10 and CIFAR-100, we have conducted experiments on a broader range of datasets to enhance the credibility and generalizability of our results. These include:

1. Image dataset:
   • ImageNet-100: A widely-used benchmark dataset providing a large-scale and realistic evaluation, used to analyze the influence of unseen class data on semi-supervised models.

Based on the results above, we can see that under Imagenet-100, the influence of the addition of unseen classes on the SSL methods is smaller compared to CIFAR-10 and CIFAR-100. Most SSL methods exhibit similar properties across CIFAR-10, CIFAR-100, and Imagenet-100. For example, ICT and UASD demonstrate robustness to the addition of unseen classes across all three datasets. SoftMatch and FixMatch are relatively sensitive to the addition of unseen classes across all three datasets.

Additionally, similar to Tables 1 and 2 in the original submission, we also calculated five metrics to measure the degree to which semi-supervised learning models are affected by the addition of unseen classes on Imagenet-100, from both a global robustness and a local robustness perspective.

• CUB: A fine-grained classification dataset that introduces additional diversity, enabling the study of the effects of unseen class data in specific scenarios.

Due to time and resource constraints, we only conducted experiments for r=0, r=0.5, and r=1. The results are shown in the table above. Experimental observations are similar to CIFAR10 and CIFAR100, where the performance of most methods decreases as unseen classes are added (e.g., FreeMatch and FlexMatch), while some methods exhibit robustness to the addition of unseen classes (e.g., UASD and ICT).

Q1: My main concern is the practicality of the framework. The main purpose of the paper is to test the influence of increasing numbers of unseen class data on the classification performance on seen class data. The proposed metrics are mainly one-order statistics of the classification performance, which show the change in performance with more unseen class data. However, the main interest of SSL with unseen class data in real-world applications may not be focused on one-order statistics. In real-world applications, we only care about the performance of a certain number of unseen class data instead of the performance change with more data, although many papers provide the figures with different numbers of unseen class data. Therefore, although the framework is valid for different sizes of unseen class data, the practicability for real applications may be limited.

A1: Thank you for your insightful comments, and I completely understand your concern regarding the practicality of the framework. I agree that in real-world applications, the primary focus is often not on the change in performance as more unseen class data is introduced, but rather on the performance with a specific number of unseen class data.

Indeed, fixing the size of the unlabeled set is a common approach in practical applications. In many real-world scenarios, we collect unlabeled data without knowing the exact proportion of unseen classes (i.e., the value of $r$. In such cases, the previous framework is indeed valuable.

However, we also believe that evaluating semi-supervised models under a single $r$ value is not sufficient for understanding the robustness of the models. Our motivation for evaluating the impact of unseen classes in unlabeled data across different values of $r$ is to provide a more comprehensive understanding of how these factors influence model performance under varied conditions. We aim to offer a fair and standard evaluation across multiple scenarios, which is important for understanding the broader applicability of semi-supervised learning models.

In the revised version, we have clarified that while both frameworks are valuable, they cater to different use cases: one for real-world applications where the size of the unlabeled set is fixed and another for assessing the general robustness of models under varying proportions of unseen class data.

Dear Reviewer wdK8,

Thank you for your time and effort in reviewing our work. We have carefully considered your detailed comments and questions, and we have tried to address all your concerns accordingly.

As the deadline for revising manuscript is approaching, could you please go over our responses? If you find our responses satisfactory, we hope you could consider adjusting your initial rating. Please feel free to share any additional comments you may have.

Thank you!

Authors

I have carefully read the author's response.

These responses have dispelled my doubts.

I will increase my score to 6.

Q10: As mentioned in line 269-299, there are three experiments (generated by three random seeds) for each kind of dataset construction. So, the experiment results given in the paper are still with high randomness. So, it is difficult to accept these conclusions derived from these experiment results. In order to dispel this doubt, the authors should strengthen the experiment from the following aspects(a. The number of datasets should be increased, e.g., 10 or more. b. The diversity of datasets should also be increased, rather than being limited to image datasets. c. For each dataset construction (labeled and unlabeled samples setting), more experiments should be carried out, e.g., 10 times or more.d. Some statistical significance tests should be introduced.)

A10: In the revised manuscript, we have addressed the reviewers' concerns as follows:

a. We expanded the number of datasets to include more datasets across various modalities, including image, text, and tabular data, to ensure a more comprehensive evaluation.

b. To increase diversity, we incorporated datasets from different domains rather than limiting the experiments to image datasets.

c. For each dataset construction, we conducted experiments over 10 repetitions to ensure reliability and robustness of the results.

d. We introduced statistical significance test (p-value) to rigorously validate the experimental findings, further enhancing the credibility of the analysis.

1. Image dataset:
   • ImageNet-100: A widely-used benchmark dataset providing a large-scale and realistic evaluation, used to analyze the influence of unseen class data on semi-supervised models.

Based on the results above, we can see that under Imagenet-100, the influence of the addition of unseen classes on the SSL methods is smaller compared to CIFAR-10 and CIFAR-100. Most SSL methods exhibit similar properties across CIFAR-10, CIFAR-100, and Imagenet-100. For example, ICT and UASD demonstrate robustness to the addition of unseen classes across all three datasets. SoftMatch and FixMatch are relatively sensitive to the addition of unseen classes across all three datasets.

Q5: In line 072, ”unseen classes in ---> “unseen classes in.

A5: Thank you for pointing out the typographical issue. We have corrected the sentence in line 072 from "unseen classes in ---> unseen classes in." to ensure clarity and consistency in the revised manuscript. We appreciate your attention to detail.

Q6: In line 141, Fig.2(b) ---> FIg.2(a).

A6: Thank you for identifying the error. We have revised the reference in line 141 from "Fig.2(b)" to "Fig.2(a)" in the updated manuscript. We appreciate your careful review and feedback.

Q7: In line 480-481, the table’s title should be above the table.

A7: Thank you for pointing this out. We have adjusted the placement of the table's title in lines 480–481 to ensure it appears above the table, as per standard formatting conventions. We appreciate your thorough review.

Q8: In the experiment, how are the integrals in formulas (1) and (2) calculated? It seems that the integral operation should be the accumulation operation.

A8: Thank you for your question. Yes, the integrals in formulas (1) and (2) are computed as accumulation operations based on empirical values. We have provided further explanation of it in the revised manuscript.

Q9: In line 190, how can we obtain the “the expected accuracy”? In my opinion, it should be “the mean of ....”.

A9: Thank you for pointing this out. You are correct; the term "expected accuracy" in line 190 refers to "the mean of accuracy values across different $r$ values." We have revised the text accordingly to ensure clarity.

Q3: To facilitate readers to intuitively assess the degree of impact and facilitate cross dataset comparisons, the evaluation indicators in formulas (1)-(5) should to be normalized if possible.

A3: Thank you very much for your valuable suggestion. In our experiments across different datasets and factors, we have ensured consistent comparisons based on the unified definitions of formulas (1)-(5). This ensures that the evaluation metrics remain comparable across all experiments. As an exception, in Table 7 in the appendix, due to the small magnitude of the data, we scaled the values by a factor of 10 to better present the experimental results. This adjustment only affects the display of the values and does not alter the relative relationships between the metrics or the comparisons of model performance.

Q4: More experiment details should be recorded, such as the names of seen and unseen classes, the number of labeled samples for each class, and the number of unlabeled samples for each class. If there is not enough space, put it in the appendix. Furthermore, the authors should provide a link to their code or dataset splits, if possible, to further enhance reproducibility.

A4: Thank you for your suggestion. To enhance reproducibility, we have provided an anonymous link to our code at [https://anonymous.4open.science/r/RE-SSL-F034/README.md], which is based on [1] and [2].
We also include the following experimental details to address your concerns:

• Regarding the number of unseen-class examples:
  For CIFAR-10, we select the first 5 classes as seen classes and the last 5 classes as unseen classes. From the training set of known classes (which contains 25,000 samples), we randomly select 100 samples to form the labeled set $D_L$. We then randomly select $r_s$ $\times$ 25,000 samples from the remaining samples of the training set of known classes and put them into $D_U^S$. From the training set of unknown classes (which contains 25,000 samples), we randomly select $r_u$ $\times$ 25,000 samples and put them into $D_U^U$. $D_U^S$ and $D_U^U$ together form the unlabeled set $D_U$.

  For CIFAR-100, we select the first 50 classes as seen classes and the last 50 classes as unseen classes. Other settings are the same as CIFAR-10.

• Regarding the number of unseen-class categories:
  We fix the number of unseen-class examples by setting $r = 0.2$. Unlike the first setup, instead of randomly selecting 25,000 samples from the entire training set of unknown classes, we randomly select 25,000 samples from the last $C_n$ classes of the unknown classes and put them into $D_U^U$.

• Regarding the indices of unseen classes:
  We fix the number of unseen-class examples by setting $r = 0.2$. Unlike the first and second setups, we randomly select $r$ $\times$ 25,000 samples from the $C_i$-th class of the unknown classes and put them into $D_U^U$.

• Regarding the degrees of nearness of unseen classes:
  Similar to the third setup, instead of randomly selecting $r$ $\times$ 25,000 samples from the $C_i$-th class of the unknown classes, we randomly select $r$ $\times$ 25,000 samples from the $C_i$-th class of the external MNIST dataset and put them into $D_U^U$.

• Regarding the label distribution in unseen classes:
  We determine the number of samples to be adopted from each unknown class to obtain $D_U^U$ based on $C_{ib}$.

We have incorporated these details into the appendix of the revised manuscript to improve clarity and reproducibility.
References:
[1] Jia, Lin-Han, et al. "LAMDA-SSL: Semi-supervised learning in python." arXiv preprint arXiv:2208.04610 (2022).
[2] Jia, Lin-Han, et al. "A Benchmark on Robust Semi-Supervised Learning in Open Environments." The Twelfth International Conference on Learning Representations. 2023.

Q1: Since $r$ (or $r_u$) is recorded in the tables in the experiment section, there is a precise definition for $r$. To reduce ambiguity, the formal definition of $r$ should be given based on the symbol system in Section 2.

A1: Thank you for your feedback and for pointing out the need for a precise definition of $r_u$ to reduce ambiguity. Here is the formal definition based on the symbol system in Section 2: In our setup:

• $D_B^U$ represents the total set of unseen-class data. $D_U^U$ is sampled from $D_B^U$.
• $r_u$ is the ratios used to control the sampling from $D_B^U$.

Formal Definition of $r_u$:
$r_u$ denotes the ratio of selected unseen-class data ($|D_U^U|$) to the total unseen-class data ($|D_B^U|$): $r_u = \frac{|D_U^U|}{|D_B^U|},$ where $|D_U^U|$ is the size of the sampled unseen-class data, and $|D_B^U|$ is the size of the total set of unseen-class data.

$r_u$ is used to control the degree to which unseen-class data is introduced into the unlabeled set $D_U$, thereby allowing us to evaluate the influence of unseen classes on the model's performance. A higher $r_u$ indicates a larger proportion of unseen-class data, while a lower $r_u$ reflects a scenario with fewer unseen-class samples.

We have incorporated the above in the revised manuscript.

Q2: It is necessary to provide a table or brief description for each indicator, showing its range, interpretation of positive/negative values, and how it relates to model performance.

A2: Thank you for your constructive suggestion. We agree that providing a table or a concise description for each indicator would help clarify their interpretation and relation to model performance. Below is a detailed description for each indicator:

We have incorporated the above table into Section C.1 of the revised manuscript to provide a concise summary of each indicator, as it effectively clarifies their range, interpretation, and significance for evaluating model performance.

3. Text dataset:
   • AGNews: A widely-used text dataset in natural language processing, included to assess how unseen class data affects semi-supervised models in text-based tasks.

Compared to tabular data, text modality data is relatively more affected by unseen classes. We attribute this to the greater difficulty in feature representation and the higher complexity of text-based tasks. Consistent with previous experimental findings, UASD is relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

These additional datasets aim to validate the effectiveness of our experimental setup in various domains, showcasing its practicality. We have incorporated the above experimental results and analysis into the revised manuscript.

• Letter: Another tabular dataset included to further explore the influence of unseen class data on semi-supervised models.

Based on the results from the two tables above, it can be observed that compared to image modality data, tabular data is less affected by unseen classes, and semi-supervised models demonstrate greater robustness on tabular data. We attribute this primarily to the simplicity of tabular data structure and the more effective feature representation. Moreover, similar to CIFAR, ImageNet, and CUB, ICT and UASD are relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

• CUB: A fine-grained classification dataset that introduces additional diversity, enabling the study of the effects of unseen class data in specific scenarios.

Due to time and resource constraints, we only conducted experiments for r=0, r=0.5, and r=1. The results are shown in the table above. Experimental observations are similar to CIFAR10 and CIFAR100, where the performance of most methods decreases as unseen classes are added (e.g., FreeMatch and FlexMatch), while some methods exhibit robustness to the addition of unseen classes (e.g., UASD and ICT).

2. Tabular dataset:
   • Forest: A structured tabular dataset, used to investigate the impact of unseen class data in non-image modalities.

Q2: There are five evaluation metrics. Which one is the most important?

A2: Thank you for your insightful question. All five proposed metrics are meaningful and contribute to a comprehensive evaluation of robustness, as they assess different aspects of model performance in the presence of unseen classes.

• Global Robustness Metrics:

  • $R_{slope}$ (Eq. 1): Measures the overall trend of accuracy changes across $r$, capturing the global robustness of the model as unseen classes are added.
  • GM (Global Magnitude) (Eq. 2): Reflects the overall influence of unseen classes by integrating the magnitude of accuracy deviations from the average performance ($\bar{ACC}$).
  • $P_{AD \geq 0}$ (Eq. 5): Represents the probability that adding unseen classes does not degrade performance, offering a probabilistic global robustness perspective.

• Local Robustness Metrics:

  • WAD (Worst-case Adjacent Discrepancy) (Eq. 3): Captures the maximum accuracy drop between two adjacent $r$ values, quantifying the local worst-case impact of unseen classes.
  • BAD (Best-case Adjacent Discrepancy) (Eq. 4): Highlights the best-case accuracy improvement or minimal drop between adjacent $r$ values, reflecting the local best-case performance changes.

Since these metrics evaluate robustness from complementary perspectives—global trends, local variations, and probabilistic considerations—they collectively provide a holistic view of model performance under varying conditions. Therefore, rather than prioritizing one metric, we recommend considering all five in combination to fully understand the robustness of semi-supervised models to unseen classes.

We hope this explanation clarifies the importance of each metric and their role in the evaluation framework. Thank you for your question!

3. Text dataset:
   • AGNews: A widely-used text dataset in natural language processing, included to assess how unseen class data affects semi-supervised models in text-based tasks.

Compared to tabular data, text modality data is relatively more affected by unseen classes. We attribute this to the greater difficulty in feature representation and the higher complexity of text-based tasks. Consistent with previous experimental findings, UASD is relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

These additional datasets aim to validate the effectiveness of our experimental setup in various domains, showcasing its practicality. We have incorporated the above experimental results and analysis into the Appendix of revised manuscript.

• Letter: Another tabular dataset included to further explore the influence of unseen class data on semi-supervised models.

Based on the results from the two tables above, it can be observed that compared to image modality data, tabular data is less affected by unseen classes, and semi-supervised models demonstrate greater robustness on tabular data. We attribute this primarily to the simplicity of tabular data structure and the more effective feature representation. Moreover, similar to CIFAR, ImageNet, and CUB, ICT and UASD are relatively less affected by unseen classes compared to other semi-supervised methods, demonstrating stronger robustness against unseen classes.

2. Tabular dataset:
   • Forest: A structured tabular dataset, used to investigate the impact of unseen class data in non-image modalities.

Q1: The main contribution of this paper lies in proposing a more reasonable experimental setup; however, its major issue is the credibility of the experiments. The experiments are conducted solely on CIFAR-10 and CIFAR-100, which makes the results unconvincing. While I understand that past studies has primarily been conducted on these two datasets, given that this paper aims to make innovations from an experimental perspective, it should propose more practical experimental setups and conduct experiments on more realistic datasets.

A1: Thank you for your valuable feedback and for emphasizing the importance of validating our approach on diverse and realistic datasets.

In response to your concern, we would like to clarify that, in addition to CIFAR-10 and CIFAR-100, we have conducted experiments on a broader range of datasets to enhance the credibility and generalizability of our results. These include:

1. Image dataset:
   • ImageNet-100: A widely-used benchmark dataset providing a large-scale and realistic evaluation, used to analyze the influence of unseen class data on semi-supervised models.

Based on the results above, we can see that under Imagenet-100, the influence of the addition of unseen classes on the SSL methods is smaller compared to CIFAR-10 and CIFAR-100. Most SSL methods exhibit similar properties across CIFAR-10, CIFAR-100, and Imagenet-100. For example, ICT and UASD demonstrate robustness to the addition of unseen classes across all three datasets. SoftMatch and FixMatch are relatively sensitive to the addition of unseen classes across all three datasets.

Additionally, similar to Tables 1 and 2 in the original submission, we also calculated five metrics to measure the degree to which semi-supervised learning models are affected by the addition of unseen classes on Imagenet-100, from both a global robustness and a local robustness perspective.

• CUB: A fine-grained classification dataset that introduces additional diversity, enabling the study of the effects of unseen class data in specific scenarios.

Due to time and resource constraints, we only conducted experiments for r=0, r=0.5, and r=1. The results are shown in the table above. Experimental observations are similar to CIFAR10 and CIFAR100, where the performance of most methods decreases as unseen classes are added (e.g., FreeMatch and FlexMatch), while some methods exhibit robustness to the addition of unseen classes (e.g., UASD and ICT).

Dear Reviewer 244R,

Thank you for your time and effort in reviewing our work. We have carefully considered your detailed comments and questions, and we have tried to address all your concerns accordingly.

As the deadline for revising manuscript is approaching, could you please go over our responses? If you find our responses satisfactory, we hope you could consider adjusting your initial rating. Please feel free to share any additional comments you may have.

Thank you!

Authors

Dear Reviewer 244R,

Thank you for your recognition and recommendation for our work. We are pleased that our response has successfully addressed all your comments. If there are remaining issues or questions regarding our paper, we would be more than happy to address them to further clarify our contributions.

Moreover, we gently remind you that in ICLR's scoring system, a score of 6 indicates a "marginally above the acceptance threshold", while a score of 8 represents "accept". If you believe our work merits acceptance, we hope you might consider giving us an "accept", as it would greatly support our work.

Best regards,

Authors

Q1: In the proposed five metrics, some formulas are not clearly defined. For instance, the meaning of $\hat{ACC}$ in Eq. 1 and $\bar{ACC}$ in Eq. 2 is not explicitly explained. Providing clearer definitions for these terms would enhance the understanding of the metrics and their implications in the context of the research.

A1: Thank you for your insightful comment regarding the clarity of the definitions for $\hat{ACC}$ in Eq. 1 and $\bar{ACC}$ in Eq. 2.

To clarify:

1. $\hat{ACC}$ in Eq. 1 represents the empirical accuracy corresponding to $r$, which is the observed performance of the model on the test set for a given $r$ value.
2. $\bar{ACC}$ in Eq. 2 represents the average accuracy of the model across all values of $r$, providing a summary of the model's overall performance. It serves as a reference point for comparing the deviations (in terms of $|\hat{ACC} - \bar{ACC}|$ at different values of $r$.

We acknowledge that these terms were not explicitly explained in the original submission, and we have ensured they are clearly defined in the revised manuscript to enhance the clarity and accessibility of the work.

We appreciate your constructive feedback and hope this explanation resolves the issue.

Q2: In the tables, it would be helpful to use bold text or dashed lines to highlight specific data points, emphasizing the advantages of certain algorithms under particular settings.

A2: Thank you for your valuable suggestion regarding the presentation of data in the tables.

We agree that emphasizing specific data points using bold text or dashed lines can significantly enhance readability and make the advantages of certain algorithms under particular settings more apparent.

Specifically, we have used bold text to highlight the best-performing results for each metric and setting in the revised manuscript.

Dear Reviewer Mtt1,

Thank you for your time and effort in reviewing our work. We have carefully considered your detailed comments and questions, and we have tried to address all your concerns accordingly.

As the deadline for revising manuscript is approaching, could you please go over our responses? If you find our responses satisfactory, we hope you could consider adjusting your initial rating. Please feel free to share any additional comments you may have.

Thank you!

Authors