Partial Optimal Transport for Open-set Semi-supervised Learning

Thanks for your valuable comments, which help us improve the manuscript.

W1:The clarity of writing.

We will reinforce the connection between MSF and OT in the introduction and provide an explanation on how the MSF is computed. Additionally, in the introduction, we will add following descriptions about MSF to improve the presentation.

"MSF is calculated through the optimal transport plan of partial optimal transport, where the MSF for an unlabeled sample is determined by the sum of its mass transport to the labeled distribution. Samples with higher scores are more likely to be identified as ID samples."

W2: Many definitions and acronyms are used before being defined.

Q1. The OSR means open-set recognition. We will revise the presentation in an updated manuscript.

Q2. The formal definitions of $\mathcal{L}$ and $\mathcal{U}$ are shown in Equation 8. We should have made it appear earlier in Sec 4.1.

Q3. Yes. We should say "The feature of each sample is of d-dimensions".

Q4. Yes. All inequalities in Eq.7 mean point-wise less than or equal to.

Q5. The definitions of L_x and L_u can be found in Section A.1. We will add corresponding descriptions in Algo3.

Q6. The number (50/100/400) denotes the number of labeled samples for each class. We will explain it in the head of Table 5.

Q7. Yes, it is a typo.

W3: The distribution in Eq.8 is not mathematically valid.

We acknowledge that defining $\mathcal{U}$ as a distribution is inappropriate; rather, it is the k-fold of a distribution. In our paper, focusing on modeling OOD detection in OSSL, this definition remains valid within our technical framework. In the revised version, the terminology will be modified to use "K-fold unlabeled distribution" instead of "unlabeled distribution" to describe $\mathcal{U}$.

Thanks for your thoughtful and constructive feedback.

W1: The novelty of the method is limited and the paper overclaims novelty.

We acknowledge that barycentric projection is a concept in the OT literature, and we recognize that the functionality of the proposed mass score function could be mathematically interpretated through barycentric projection.

Having clarified this, we would like to emphasize that we did not claim the ownership of the concept. Our primary contribution lies in modeling the OOD detection problem under the OSSL setting as a POT problem, and in training it within the SSL setting in an end-to-end manner. The Mass Score Function (MSF) is just one facet of the contributions of this paper, aiding in assessing the likelihood of a test input being an OOD sample.

In summary, our claim is that the proposal of the problem modeling and its associated techniques are novel in OOD detection under OSSL settings. We value your suggestions, and we agree that utilizing the concept of barycentric projection for explaining the proposed score function would enhance mathematical rigor, clarify intuitions, and improve presentation.

W2: The issue regarding the parameter k.

In the ablation study, we have reported the result of the performance on different k. Our findings reveal the robustness of the hyperparameter k in our algorithm. Consequently, we maintain a consistent setting of k=2 across all experiments in Table 1 to Table 3, consistently achieving excellent results despite varying proportions of ID samples. To further assess the robustness of parameter k, additional experiments were conducted with k set to 5 and k set to 10.

The results demonstrate that the value of k has good robustness over a wider range.

W3: Implementation details Ablation study on batch size.

In the revised version, we will include a description of the experiment settings.

Our experimental setup is shown below：

The batch-size B = 64.

The relative size of batch-size for unlabeled data µ = 2 in all experiments for fair comparison.

Iterations per epoch are 1024 and the total number of epochs is 512.

Optimizer: SGD with nesterov momentum = 0.9.

Learning rate η = 0.03 and use cosine annealing learning rate schedules.

The batch size we use is 64 and the relative size of batch size for unlabeled data µ is 2, which is a relatively small batch size in semi-supervised learning.

We compare our method with other SSL and OSSL methods on smaller batch size on CIFAR100 dataset.

For a detailed analysis of the experimental results, please refer to W4.

W4: Label missing problem in minibatch training.

The situation you've highlighted is indeed a reality. In small batch sizes, $\mathcal{L}$ and $\mathcal{U}$ may not sufficiently cover all classes, posing challenges for OOD detection. However, semi-supervised learning has undergone several iterations, and the occasional subpar detection performance in certain mini-batches has minimal impact on the overall performance.

Notably, with a batch size of 32, our method shows a slight decline in OOD detection performance but outperforms FixMatch in close-set accuracy. Conversely, with a batch size of 16, there is a noticeable drop in OOD detection, yet our method resiliently maintains commendable close-set accuracy.

W5：Relevant references are missing.

In Related Work section, we will incorporate a discussion on relevant papers.

References [2-3] address Open Set Domain Adaptation problems using POT. Reference [2] uses the mean cost of transport to control the proportion of POT. Reference [3] providing an approximate estimation of the transport ratio in POT. Compared to existing POT-based methods [2-3], our method stands out by not necessitating prior knowledge about the ratio of POT. Reference [1] proposed a novel pseudo-label based approach to tackle SSL in an open-world setting. TRSSL aims to assign labels for outliers at test time, which is a different semi-supervised setting.

Q1: The formula of score function.

Thanks for pointing it out. We agree it should be $**T**^\top \mathbf{1}_{n}$. We will address them in the revised version.

Q2: The use of cosine distance.

In our experiment, we normalize the vector before calculating the cosine distance. However, our method is also applicable when the OT cost is represented by the Euclidean distance. We conducted experiments on CIFAR10 using Euclidean distances, with 50 labeled samples for each class.

The results indicate that our proposed method is a universal approach, effective across different metrics.

We would like to express our sincere appreciation for your valuable comments on our manuscript. We have carefully considered each of your suggestions and provided detailed responses. As the deadline for the rebuttal is approaching, we want to ensure if our responses adequately address your concerns. We are open to any further discussion.

Thank you again for your time and consideration.

Thanks for your insightful comments, which help us improve the manuscript.

W1:Difference with other OSSL methods.

Our approach focuses on directly utilizing the distribution information. Specifically, we use POT between labeled and unlabeled samples to train a binary OOD classifier, and we treat OSSL as a multi-task learning problem.

In contrast, T2T and OpenMatch build their OOD classifiers using b (where b is the number of classes) one-vs-all classifiers, relying on labeled data and incorporating unlabeled data to improve the classifier performance. However, these methods exhibit a noticeable performance decline, especially when dealing with a large number of classification categories.

Similar to our methodology, MTCF employs a binary OOD classifier but utilizes a curriculum learning framework for model training. MTCF applies Otsu thresholding to select unlabeled samples, potentially leading to error accumulation that may impact the overall model performance.

We will add corresponding discussions to the manuscript in a revised version.

W2: Why POT is more effective at detecting OOD is not adequately provided.

In the third paragraph of Sec 4.3, we explored the effectiveness of POT. To complement our methodology, we incorporate the following details into the Methods section.

In the OSSL context, we propose that ID samples in the unlabeled dataset and labeled samples are drawn from a latent ID distribution, while OOD samples in the unlabeled dataset are drawn from a latent OOD distribution. In this setting, the optimal transport cost tends to be low between ID samples in unlabeled dataset and labeled samples.

The cost of transporting unit mass between ID samples (including labeled samples and ID samples in unlabeled dataset) is small. In POT, only a fraction of the mass in the unlabeled distribution can be transported. To minimize the overall transport cost, ID samples in unlabeled dataset obtain larger transport mass. This characteristic aids in identifying OOD samples based on the transport mass of each individual sample.

W3: Discussion about relevant references.

Reference [1] proposes a partial optimal transport method for positive-unlabeled learning. The proportion of positive samples $\pi$ in the unlabeled set is necessary, which is often unavailable in real open-set scenario. References [2-3] address Open Set Domain Adaptation problems using partial optimal transport. Reference [2] uses mean cost of transport to control the proportion of partial optimal transport, which is not robust in real world. Reference [3] providing an approximate estimation of the transport ratio in partial optimal transport.

Strengths of POT for OSSL: As outlined in W1, our method presents a simpler and more effective approach to detect outliers in OSSL using POT.

Special designs for OSSL: Existing POT-based methods [1-3], rely on the estimation of the transport ratio. However, in the context of OSSL, it is impractical to estimate the transport ratio for each mini-batch. To address this challenge, we introduce redundant mass for partial optimal transport as a robust parameter. This approach ensures that there is no need to estimate the transport ratio in every iteration, enhancing the stability and efficiency of OOD detection.

W4: Why Fixmatch algorithm yields better results.

In the OSSL scenario, while FixMatch may not generate accurate pseudo-labels for OOD unlabeled samples, it nonetheless retains the capability to generate high-quality pseudo-labels for ID unlabeled samples. Consequently, the close-set accuracy remains minimally affected by the presence of OOD samples.

W5: Lack of many experiment details.

In the revised version, we will include a description of the experiment settings.

Following the same parameter settings of original paper, we reproduce our baselines using their official codes. For a fair comparison, we unified the parameters of semi-supervised learning as follows.

Each experiment is done with a single NVIDIA A100 GPU.

The batch-size B = 64.

The relative size of batch-size for unlabeled data µ = 2 in all experiments for fair comparison.

Iterations per epoch are 1024 and the total number of epochs is 512.

Optimizer: SGD with nesterov momentum = 0.9.

Learning rate η = 0.03 and use cosine annealing learning rate schedules.

W6: Baseline on Imagenet30.

We supplemented additional experiments and obtained the following results.

Compared to OSSL methods, POT achieves the SOTA performance on the Imagenet30 dataset in terms of inlier accuracy and AUROC.

W7-W9: Issues in the paper.

Thank you very much for pointing out the issues. We will address them in the revised version.