CONTROL: A Contrastive Learning Framework for Open World Semi-Supervised Learning

Thanks to your valuable feedback. We sincerely appreciate your constructive comments.

R 4.1 In Sec. 1 - paragraph 3, why is the use of BCE loss a meaningful categorization?

We figure out that the similarity function is honed by directly computing the similarity of all sample pairs and their logits counterparts in other methods. However, not all the similarity of sample pairs can efficiently guide the subsequent learning process since unseen class samples are prone to be mislabeled as seen classes in initial training phases. That's why BCE-based is important and could be regarded as a meaningful categorization.

R 4.2 paragraph 4, the paper claims that using MSE instead of BCE lacks "flexibility." ...

In the original version we did not elucidate this issue clearly, in fact, it is the difference in the way sample pairs are selected that defines the different categories of methods, and the BCE-based methods are based on the nearest-neighbor feature selection which is more flexible. So we say that the BCE-based methods are flexible, not that the BCE is superior to the MSE.

R 4.3 In Sec. 1 - In Sec. 1 - paragraph 5, the paper suddenly discusses "seen-unseen pairs." Why is this concept important, and how is it related to the problem? This relationship is not explained.

Because these methods utilizes pairwise clustering in order to give the model the ability to classify seen and unseen classes, the quality of the sample pairs is very important, so we have to pay attention to the seen-unseen sample pairs.

R 4.4 In Sec. 2 - paragraph 2, the paper claims that previous works do not consider optimization at the feature level. However, they also jointly optimize the feature and classifier.

You may refer to our answer to reviewer fiFv R 3.1.

R 4.5 In Sec. 2 - paragraph 3, the paper claims that OpenCon is incapable of sustaining "continuous optimization" of representations. What is continuous optimization, and why can't OpenCon achieve this?

We give a detailed explanation in our latest PDF.

R 4.6 In Sec. 3.2, why should one consider BCE-based methods? BCE-based methods are just one specific approach by Cao et al. and some follow-ups. Eq. (1) does not represent all OWSSL methods.

In open world semi-supervised learning problem, high quality chosen samples are utilized to give the model the ability to classify both seen and unseen classes. The strategy for selecting the samples is critical. And BCE-based methods are more suitable for choosing high quality samples since these methods can choose samples pairs in a more flexible way as explained before.

R 4.6 Aside from the unclear overall logic, the sentences are significantly unpolished. Many grammatical errors make it difficult to grasp the meaning. There are also typos and inconsistent use of terminologies. For example, "L" in the abstract, uncapitalized "all" in Table 1, unformatted "seen/unseen" in Sec. 3.1, and "OWSSL" is not defined in Sec. 3.2.

We have made the appropriate changes in the paper.

R 4.7 The motivation if the method is (1) enhance the separability of known class features and (2) prevent the collapse of unknown class features. The paper employs SupCon (and a NN variant) for (1) and SelfCon for (2).

You may refer to our answer to reviewer fiFv R 3.1.

R 4.8 Missing important baselines such as GCD , OpenCon and RoPAWS.

We have compared Opencon in Table 1, where the combination of NACH and control outperforms Opencon. we have added GCD to the comparison methods. RoPAWS is not an OWSSL algorithm, and its goal is not to classify both seen and unseen classes, but rather to better ensure the correctness of seen categories when unlabeled data has unseen classes, so we will not discuss it.

The results of additional experiments are as follows， we evaluate all the methods on CIFAR-100 dataset.

We can see that, the combination of NACH and our proposed CONTROL still gives the best results among all the compared methods.

R 4.9 Does the proposed method also improve non-OW SSL benchmarks and other OWSSL methods?

We illustrate this with Fixmatch, a representative deep semi-supervised learning algorithm with 250 labeled data on cifar10 dataset using wideresnet-28-2 as backbone, in conjunction with L_simALL and L_supSeen in our proposed framework, with the following results:

We can see that even doing feature-level optimization based on the semi-supervised learning algorithm in the traditional setup results in a performance gain, but they are far less than in OWSS setup, which demonstrates the reasonableness of our proposed combination of CONTROL and OWSSL algorithms.

After discussing with other reviewers, I think that the rebuttal did not fully address the initial concerns:

• Presentation: The overall research insight is still not effectively conveyed in the manuscript. A major revision may be necessary to improve clarity.
• Novelty: If I understand correctly, the rebuttal claims that the technical contribution is in the analysis of previous approaches, and the proposed solution (SupCon, NN) straightforwardly arises from this analysis. This may also relate to Reviewer 1Kmz's narration concern; the delivery of this claim could be clarified further.
• Experiments: I believe the experimental results are okay, although combining additional objectives should enhance performance in general. Nevertheless, the combination of OpenCon and CONTROL in R2.2 is not convincing enough.

Thanks to your valuable feedback. We sincerely appreciate your constructive comments.

R 3.1 Novelty Concerns: The proposed method essentially adds three types of contrastive losses to the existing OWSSL loss. Of these, the supervised contrastive loss is merely a standard contrastive loss. Additionally, both the NN contrastive loss and the SimAll contrastive loss that utilizes augmented images as positive keys lack distinctive features compared to existing research.

Existing methods (e.g., BCE based methods) consider the open world semi-supervised learning problem from the logits level optimization, but we point out through theoretical analysis that when there is the phenomenon of seen-unseen pairs, the optimal classifiers learned through logits optimization are not the same as the ideal state (see Eqn. 4), but optimization based on comparative learning at the feature level ensures that even in the case of mismatches, we can still theoretically show that the classifiers we learn are the same as the ideal case (see Eqn. 5). We point out this key issue and validate our viewpoint with extensive experiments. To the best of our knowledge, no theoretical proof of the importance of feature-level optimization for OWSSL has been proposed and given before. All the three losses are proposed in the feature-level optimization which have theoretically guaranteed to get a performance gain.

R 3.2 Overall Paper Completeness: The completeness of the paper is generally lacking. The proof provided in Section 4.1 for motivation is not rigorous, with some intermediate steps omitted. And there are three hyperparameters related to the loss. However, the paper does not provide details on the values chosen for these parameters, nor does it discuss their sensitivity or robustness through experiments.

About the proof provided in Section 4.1, we illustrate the importance of feature-level optimization by whether the optimal classifier learned in the presence of mispairing is consistent with the ideal optimal classifier, Eq. 3 illustrates inconsistency at logits level and Eq. 4 illustrates consistency at feature level.

As mentioned before all the three losses are proposed in the feature-level optimization which have theoretically guaranteed to get a performance gain.

About the choice of parameters, the results of additional stripping experiments are as follows， we evaluate our methods on CIFAR-100 dataset with different parameters.

The values 0.5 0.1 0.3 that we report in the paper are optimal for the parameter combinations.

R 3.3 Reproducibility Concerns: The absence of a reproducibility statement raises doubts about the possibility of replicating the claimed state-of-the-art performance.

All the results are reproducible, and we'll make our code publicly available upon acceptance.

R 3.4 In Section 3.1, shouldn't C_seen be defined as C_L instead of C_seen = C_L cap C_U

Yes, the two are equivalent in our setup, and we've made changes for better understanding by the reader.

Thanks to your valuable feedback. We sincerely appreciate your constructive comments.

R2.1 Narration. Can the author elaborate more on high-level discussion on the motivation of the method and how it differentiates from previous work? This can be regarded as a revision of the introduction section but can be brief just for discussion now.

We have rewritten the introduction as you suggested.

Existing methods (e.g., BCE based methods) consider the open world semi-supervised learning problem from the logits level optimization, but we point out through theoretical analysis that when there is the phenomenon of seen-unseen pairs, the optimal classifiers learned through logits optimization are not the same as the ideal state (see Eqn. 4), but optimization based on comparative learning at the feature level ensures that even in the case of mismatches, we can still theoretically show that the classifiers we learn are the same as the ideal case (see Eqn. 5).

R2.2 A small-scale additional experiment. It would greatly enhance the claim of contributions in this paper if the author could provide a small-scale experiment on a dataset where CONTROl is combined with OpenCon.

We test the performance of CONTROL on the CIFAR100 dataset using the opencon training results as a backbone and not using the opencon training results as a backbone, respectively.

As we can see, OPENCON achieves good results, but as a backbone it does not further improve performance when combined with our proposed framework. We can see the advantages of combining CONTORL with existing semi-supervised learning algorithms from this perspective.

Thanks for your valuable feedback. We sincerely appreciate your constructive comments.

R 1.1 The whole Section 3 missed a lot of details, lacking the basic problem setting, and key components of relied models, which makes the whole method hard to understand.

We have reorganized section 3 according to your suggestions.

R 1.2 Fig 1 and Fig 4 should be further polished. There are also overlaps between those two figures.

Figure 4 uses the same example as Figure 1, and we do so to better illustrate whether each part of our proposed loss function should correspond to labeled or unlabeled data.

R 1.3 Page 2: what is a seen-unseen pair, Figure 2 is also hard to understand.

In the new version we have further elaborated seen-unseen pairs.

R 1.4 It should be good to include some real-world results except the two datasets.

Imagenet is widely adopted in SSL studies, such as the paper of ORCA and NACH. The data set is enough to analyze the effectiveness of different SSL methods.

R 1.5 Will the additional objectives significantly increase the training time, regarding the speed for convergence?

We just add our designed contrastive learning loss to the original model, which is not computationally expensive, so it does not significantly increase the training time. All the experiments were done based on ORCA on the cifar10 dataset with the same device.

We only increased the running time by 5.8 percent from the original algorithm, we gain 4.1 percent performance gain on unseen classes accuracy. It's well worth the time it takes.

R 1.6 Abaliton of L_supseen and L_simALL

About the proof provided in Section 4.1, we illustrate the importance of feature-level optimization by whether the optimal classifier learned in the presence of mispairing is consistent with the ideal optimal classifier, Eq. 3 illustrates inconsistency at logits level and Eq. 4 illustrates consistency at feature level.

All the three losses are proposed in the feature-level optimization which have theoretically guaranteed to get a performance gain.

All experiments were done based on NACH on the cifar10 dataset.

As we can see, LSimAll and LSupSeen can each gain performance improvements over existing algorithms.