SemiReward: A General Reward Model for Semi-supervised Learning

W1&W3: The method's complexity, involving multiple models and training processes and the design of the reward model and hyper-parameter selection, raises concerns about the method's practical feasibility.

R: From the complexity perspective, our algorithmic flow might be complex at first glance, but it is actually quite clear, concise, and user-friendly. We discuss the complexity of SemiReward from five aspects (for W1). As for the complexity of user usage, the most essential component in the inference process is the Rewarder, as described in the pseudocode (provided in PART 3) and Figure 3. It is a pluggable module to assess the quality of pseudo labels, subsequently filtering them based on predicted reward scores, which can be treated as previous pseudo-label selection strategies and utilized in various pseudo-label methods. As for the two-stage training process, it can be regarded as another line of self-teaching process involving two models (the Rewarder and the Generator). The role of the Generator is similar to the teacher model in the self-training pipeline to auxiliarily optimize the Rewarder. And, these two stages are actually the same process but use different input datasets, i.e., the second stage further utilizes the selected reliable pseudo labels $\hat{\mathcal{D}}_{U}$ in additional to the first stage. As for the complexity of the model itself, our Rewarder is relatively lightweight, and its architecture is easy to explain (as discussed in Sec. 3.1). The relevant parameters and computational costs of our model are provided in Table 5 of the main text as follows:

(view PART 2 to continue)

Furthermore, as for the practical usage of SemiReward, the results in Table A9 of the appendix indicate that the inclusion of our model significantly enhances the overall training speed of the student model. Note that the numbers in parentheses represent the acceleration factor. This indirectly underscores the lightweight and fast-convergence nature of our model (for W3).

Table: Training speedup times on nine SSL classification datasets with CV, NLP, and Audio modalities in various label settings.

Lastly, from the perspective of tunable hyperparameters, our model actually incorporates fewer hyperparameters compared to current state-of-the-art methods such as FreeMatch and SoftMatch. We have fixed the network architectures of the Rewarders and Generator and only tuned three hyperparameters for the Rewarder training. Meanwhile, ablation and analysis experiments (e.g., Sec. 4.3 and Appendix B) have demonstrated the hyper-parameter robustness of our SemiReward, whereas manually designed methods often require extensive tuning for different task types. The comparison table of tunable hyperparameters is provided for reference (for W3):

(view PART 3 to continue)

For instance, as demonstrated in the experiments outlined in Table A6 of the appendix, our model exhibits robustness within a certain range, as indicated by start_timing. Therefore, the user can apply the provided empirical hyperparameters to use SemiReward without hesitation (for W3).

W2: Offer clarity-enhancing pseudo codes.

R: Thanks for your suggestions on improving the readability of our manuscript, and we provide the relevant pseudocode below. It has been incorporated into the appendix of the revision:

# SR_Train Stage 1 
iteration < T:
	# set SemiReward data loader
	for x_l,y_l in loader:
		x_r,y_r,B_R = x_l,y_l,B_l
  # load data in B_R size batch, x_r is labeled data and y_r is ground truth label
	for x_r,y_r,B_R in loader:
		feat(x_r) = f_s.feat(x_r) # get feature
		y_f,r = G(feat(x_r)),S(feat(x_r),y_r) # get fake label and reward
		S = cosine_similarity_n(y_r,y_f)) #get label similarity as targte
	# calculate loss
		L_R += MSE(r,S)
		L_G += MSE(r,1)
	L_aux = (L_R+L_G)/B_R
	# adam update
  L_aux.backward
	update(G,R)
# SR_Train Stage 2 
iteration >= T:
	# set SemiReward data loader
	for x_u,x_l in loader:
		x_r = cat(x_u,x_l)
		# get pseudolabel y_p, Pseudolabel method can see in Pseudo Label algorithm (https://arxiv.org/abs/1908.02983)
		y_p = Pseudolabel(f_s(x_r))
		r = R(y_p,x_r) # calculate reward for each pseudolabel in N
		y_k = Topk(y_p,r) # select top k reward in N by reward
		B_R = (B_l+B_u)*k/N # get loader batch size B_R
  # load data in B_R size batch, x_r is unlabeled data
	for x_r,y_k,B_R in sr_dataloader: 
		feat(x_r) = f_s.feat(x_r) # get feature
		y_f,r = G(feat(x_r)),S(feat(x_r),y_k) # get fake label and reward
		S = cosine_similarity_n(y_k,y_f)) #get label similarity as target
	# calculate loss
		L_R,L_G = MSE(r,S),MSE(r,1)
	L_aux = (L_R+L_G)/B_R
	# adam update
  L_aux.backward
	update(G,R)
# SR_Inference
iteration > T:
	for x_u,x_l,y_l in loader:
		# get pseudolabel y_p, Pseudolabel method can see in Pseudo Label algorithm (https://arxiv.org/abs/1908.02983)
		y_p = Pseudolabel(f_s(x_u))
		feat(x_u) = f_s.feat(x_u) # get feature
		r = R(feat(x_u),y_p) # evaluate score
		mask_r = where(r>r.mean,1,0) # get threshold by T = r.mean
		L_u = CrossEntropy(y_p,f_s(x_u))*mask # filter label
		L_l = CrossEntropy(y_l,f_s(x_l))
	# calculate loss
	L =  L_u/B_U+L_l/B_L+L_aux # total loss
	# adamW update
  L.backward
	update(f_s)

Dear reviewer,

We appreciate your valuable feedback. In our response, we added pseudocode-related content (more specific has been updated in the manuscript). At the same time, we have explained through discussions from three angles that our Rewarder is actually not complicated and does not reflect additional hyperparameters, and is very easy to follow due to its end-to-end and plug-and-play features.

As you may know, unlike previous years, this year's discussion period only lasts until November 22nd, and we are getting closer to that deadline. We hope to discuss this with you in the meantime and will be happy to provide more information based on your feedback or further questions.

If you are satisfied with our response, please consider updating your score. If you need any clarification, please feel free to contact us.

Thank you for your reply and efforts in going through our response! Since the rebuttal period is approaching the end, feel free to discuss them with us if you have any other questions. We are glad to further improve the quality of our manuscript. Thanks again for your help!

Best regards,

Authors

W1: Lack of experiments with samples of different labeling levels (missing experiments).

R: In several settings of CIFAR-100, we have augmented the relevant tasks. As illustrated in the table, the backbone for the student model is set as pretrained VIT:

W2: Lack of large-scale benchmarks like ImageNet (missing benchmarks).

R: Following FlexMatch and SoftMatch, we have provided results for ImageNet with 100 labels per class in Table 3 of the main text. Additionally, we have supplemented the data results for 1% and 10% labeled datasets (i.e., 13 and 128 labels per class) following CoMatch, as presented below. We find that applying the proposed SemiReward (+SR) upon FixMatch and CoMatch can achieve around 1.3% performance gains, and CoMatch+SR outperforms the existing SOTA method, SimMatch (CVPR’2022). Note that all these experiments train the student model from scratch.

W3: Marginal gains in NLP and CV datasets.

R: Due to the relatively antiquated nature and the performance saturation on the STL-10 and AG News datasets, our approach did not achieve quite large gains on the average of multiple trials with different random seeds. But we can find out the fast-convergence and lightweight characteristics of our SemiReward. Consequently, we have supplemented our study with datasets from the CV and NLP domains with more difficult experimental settings (e.g., more class numbers and less labeled data) that exhibit remarkably superior performances.

Q1: Lack of explanation of the percentage of labels in the experimental setup.

R: The configuration of settings is guided by the approach outlined in the USB paper. Specific details and settings information can be found in Appendix Table A1. For your convenience in reference, we present them below:

Dear reviewer,

We appreciate your valuable feedback. In our response, we supplemented the relevant experiments you mentioned and clarified the settings of the relevant data sets for our experiments. Several new sets of CV and NLP data sets illustrate the effectiveness of our gain. At the same time, the actual number of different labels has also been added. If you have more suggestions, please feel free to make them.

As you may know, unlike previous years, this year's discussion period only lasts until November 22nd, and we are getting closer to that deadline. We hope to discuss this with you in the meantime and will be happy to provide more information based on your feedback or further questions.

If you are satisfied with our response, please consider updating your score. If you need any clarification, please feel free to contact us.

W1: Lack of clarity on the validity of the rewarder design and inference strategy.

R: Thanks for your concerns, which are the most challenging problems we tackled with the Rewarder. As you have considered, directly predicting pseudo-labels with the algorithmic flow of the Rewarder will fall back to the basic pseudo-labeling algorithm, where the teacher model generates pseudo-labels and confident scores (can be regarded as the quality in Thresholding methods). It cannot provide a credible evaluation of generated pseudo-labels, which are coupled with the student training and affected by the confirmation bias. Therefore, our Rewarder serves as the pseudo-label selection method that evaluates the quality of the given pseudo-label of unlabeled data. As introduced in Sec. 3.1, our Rewarder is learned to compute the reward score, and we will select the relatively high-quality pseudo-labels with reward scores above the average in a mini-batch. We have shown the proposed reward score is calibrated for pseudo-label evaluation and is easy to fit by the lightweight Rewarder. Meanwhile, whether we can filter out high-quality depends on both the student & teacher models and the Rewarder. At the early stage of the student training, the generated pseudo-labels might not be reliable due to the weak student & teacher models. It will cost impractical computational overhead if you try to generate high-quality pseudo-labels by maximizing the reward score. When the student & teacher models are knowledgeable during the middle training period, the high-quality pseudo-labels will be discriminated by the Rewarder since it is online optimized. Moreover, you can refer to the General Response for the summary of our algorithm.

W2: Introduce new hyperparameters in rewarder filtering in semi-supervised training.

R: In the context of $\tau$, it is important to note that it does not represent a new hard threshold hyperparameter; rather, it serves as a scoring threshold marker for the Rewarder. Our inference process entails scoring multiple candidate pseudo-labels for a given sample using the Rewarder and subsequently filtering them. This involves scoring pseudo-label candidates collectively, computing individual rewards for each, and then calculating the average reward score. The average reward score, denoted by the symbol $\tau$, serves as the filtering threshold. During this process, pseudo-labels with scores below the average reward (i.e., the threshold $\tau$) are excluded. As this threshold is the mean of a set of pseudo-label candidate lists, it remains controllable and adaptable to changes in the training process. This adaptability implies that we do not introduce additional hyperparameters through hard threshold methods.

W3: Ambiguity in the definition of fake labels.

R: Throughout the training process, please refer to the general response. The primary function of the Generator is to generate a fake label distribution, aiding the Rewarder in learning high-quality mapping relationships. Importantly, the goal of generating fake labels is not to produce a set of high-quality pseudo-labels for the Rewarder to learn. Such a learning distribution tends to exhibit confirmation bias, which is why the pseudo-label results directly generated by the student model are not employed. The purpose of learning this mapping, which decouples the training processes of the student model and the Rewarder, is to mitigate the aforementioned this confirmation bias. In finer detail, by inputting a pair of fake label and sample, the Rewarder assigns a score, and the training objective involves calculating the cosine similarity between the fake label of a data sample and its ground-truth label. Throughout the entire training process, in the initial stage, the Rewarder learns from the sample and its corresponding fake label as input. The learning objective is the difference between the true label of the sample and the fake label (reduction is applied here), normalized using cosine similarity (as depicted in Figure 4 of the main text, where the calibration curve explains the rationale behind using cosine similarity). The Generator generates a fake label distribution with random quality. These fake labels are not incorporated into the training process of the student model but serve to assist in the training of the Rewarder. This approach enables the Rewarder to comprehensively explore various scenarios of the score mapping, enhancing its ability to evaluate pseudo-labels effectively.

W4: Potential negative impact of mixing current designs with Student model with generator.

R: The proposed training pipeline effectively severs the interdependence of the student and rewarder training processes, a measure taken to mitigate confirmation bias. Through the generator generating multiple sets of fake labels with varying qualities and computing their respective scores, the rewarder initializes its mapping from pseudo-labels and unlabeled data to reward scores. In the first stage, these candidate training samples aid the Rewarder in its initial convergence, as elaborated in W3 response and the general response. Moving to the second stage, the rewarder necessitates a more diverse set of samples for further optimization, having already undergone substantial training on labeled samples. Once again, we employ samples and their corresponding pseudo-labels as input. However, in this stage, the learning objective shifts to the disparity between the sample's high-quality pseudo-labels (utilized as true labels) and the pseudo-labels generated by the generator. This approach facilitates a more extensive exploration of additional unlabeled samples to refine the rewarder's optimization.

Dear reviewer,

We appreciate your valuable feedback. In our response, we have tried our best to answer your questions about the Rewarder evaluation method and fake label. We have restated the problem through the general response and elaborated more specifically on the part where you gave us a weakness. If you have more questions, please inform us freely.

As you may know, unlike previous years, this year's discussion period only lasts until November 22nd, and we are getting closer to that deadline. We hope to discuss this with you in the meantime and will be happy to provide more information based on your feedback or further questions.

If you are satisfied with our response, please consider updating your score. If you need any clarification, please feel free to contact us.

Thanks for your insightful comment. We answer your concerns from two aspects, i.e., the necessity of decoupled two-stage training and the possibility of further optimizing pseudo labels with the Rewarder. We also added relevant discussion in Appendix B.7 of the latest revision for your reference.

Why is the rewarder not trained together with the student model?

Here, the first type of training process is to optimize the student and the Rewarder together without the Generator, where the teacher model generates candidate pseudo labels for the student and the Rewarder. We call the first type coupled training and the proposed two-stage training decoupled training. The following table shows the categories of the input, output results, and functions of the relevant models. There are two reasons for decoupling the training process of the student and the Rewarder. Firstly, the Rewarder requires diverse pseudo-labels as the training data to fit the ground truth reward scores rather than deterministic high-performance labels. Secondly, the student and the Rewarder might suffer from confirmation bias. Therefore, if we directly utilize pseudo labels generated by the teacher model as the couple training, the Rewarder might be unstable during the early training period and cannot learn more information in the second stage.

We conduct comparison experiments of the two training processes on CIFAR-100 (400 labels) based on FlexMatch in the following table (report top-1 accuracy). When using the coupled training of the student and the Rewarder, FlexMatch+SR yields worse performance than the baseline (82.12 vs. 82.20). Meanwhile, the proposed two-stage training decouples the student and the Rewarder by the Generator (aiming to maximize the reward score) and achieves a great trade-off between performance gains and speedup. As shown in Figure A2(a), in coupled training, the selected pseudo-labels are occasionally of slightly higher quality, but are unstable and affected by the student model, while decoupled training stably produces high-quality pseudo-labels.

Why is SemiReward unable to maximize the reward score to generate better pseudo labels?

Currently, the reward score output by the rewarder only evaluates the pseudo label. Thus, we can obtain the pseudo labels with a gradient ascent trick (GA) as you suggested: Given a mini-batch of selected pseudo labels, we modify them towards more high-quality pseudo labels (or fake labels) by maximizing the reward scores with a step of gradient ascent in the inference process of the Rewarder. In the coupled training process, the Rewarder can directly affect the pseudo labels generated by the teacher model. However, in the decoupled training, the pseudo label and reward score have no gradient backpropagation relationship with each other, while fake labels generated by the Generator are related to the reward scores. Therefore, we add these gradient-ascent enhanced fake labels to the selected pseudo labels for the student training.

We also implement the gradient ascent trick on CIFAR-100 in the table above. When using the coupled training, FlexMatch+SR (GA) can only obtain a limited performance gain and speedup over the baseline. Applying the gradient ascent to the decoupled training will yield a little performance gain with more extra computational costs and cause unstable training. As shown in Figure A2 (b), the quality of fake labels is relatively random and diverse, and it is difficult to steadily obtain high-quality labels through the gradient ascent and directly apply to the student training. Therefore, we chose the proposed decoupled training without GA as the final version.

In conclusion, on the one hand, the first coupled training process is sub-optimal for SemiReward, and we avoid the existence of confirmation bias through decoupling method training. On the other hand, if we try to enhance the pseudo labels with the gradient ascent trick, we need a more complex loss design and violate the original intention of making a fast-convergence and plug-and-play method. The construction idea of SemiRewarder can be more widely used in other fields. Thanks again for your efforts and valuable feedback! We are looking forward to your further response.