Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics

Thank you very much for your constructive comments! We have carefully studied them and revised the paper accordingly.

Weakness 1: More theoretical analysis behind the observations should be presented, which could make the paper more convincing.

Response: We appreciate your insightful feedback. Due to limited time during the rebuttal, we would like to propose a preliminary theoretical idea as a potential direction for addressing this concern.

Our idea is to use the local elasticity hypothesis to prove that adding 2-phase labels can promote feature separability. Local elasticity is the core concept introduced in [1], and it is defined as follows: when the model is updated at sample $x$ via SGD, the predicted change  $|f(x', w^+) - f(x', w)|$ of another feature vector $x'$ is positively correlated with the similarity between $x$ and  $x'$. Specifically, local elasticity describes a phenomenon that if $x$ and $x'$ belong to the same class (similar), the predicted change is significant; if they belong to different classes (dissimilar), the change is small.

Reference [2] utilizes the local elasticity assumption to derive conditions for feature separability. This study describes the temporal evolution of features for two classes of samples, denoted as $X_{i}^1(t)$ and $X_{j}^2(t)$, in a binary classification context. Furthermore, Theorem 2.1 in [2] indicates that:

Given the feature vectors $X_{i}^1​(t)$ and $X_{j}^2​(t)$ for $i, j \in [n]$, as $t \to \infty$ and large $n$,

1. if $\alpha>\beta$, they are asymptotically separable with probability tending to one,
2. if $\alpha\leq\beta$, they are asymptotically separable with probability tending to zero.

Our two-phasic metric is designed to identify samples that exhibit a two-stage characteristic in training dynamics. As illustrated in Figure 4, the 2-phasic samples tend to lie closer to the decision boundary. Intuitively, incorporating these samples may help reduce the inter-class effect $\beta$, thereby promoting stricter feature separability. In the future, we plan to study the local elasticity about 2-phase samples, estimate the intra-class effect $\alpha$ and inter-class effect $\beta$ before and after adding 2-phase samples. This analysis will allow us to verify our hypothesis and provide a theoretical foundation for the effectiveness of our method.

[1] He H, Su W J. The local elasticity of neural networks. In International Conference on Learning Representations, 2020.

[2] Zhang J, Wang H, Su W. Imitating deep learning dynamics via locally elastic stochastic differential equations. Advances in Neural Information Processing Systems, 2021.

Weakness 2: The additional overhead during the computation of spatial and temporal measures makes the proposed method less efficient, especially when the size of dataset is large. The authors could analyze the computational efficiency of the method from an experimental view.

Response: Thank you for pointing out this concern. We have conducted an empirical analysis of the computational efficiency of our method. Specifically, on the Cora dataset using GCN as the backbone, the measured time costs are as follows:

• Training dynamics recording time: 0.11625 seconds
• Two-phasic metric computation time: 0.00253 seconds
• Total training time: 2.24 seconds

These results indicate that the combined overhead of recording training dynamics and computing the two-phasic metric accounts for approximately 5% of the total training time. Therefore, we consider this additional computational cost to be acceptable in practice.

Weakness 3: The authors could perform ablation study regarding the design 2-phasic metric and analyze effectiveness of the spatial measure and the temporal measur.

Response: Thank you for the valuable suggestion. In response, we conducted an ablation study using GCN as the backbone on the Cora dataset to evaluate the contributions of the temporal and spatial characteristics in the 2-phasic metric. The experimental results are summarized in Table A. The results demonstrate that both the temporal and spatial components contribute positively to the overall performance, with the temporal features having a more significant impact. We will include the results and corresponding discussion in the final revised version of the paper.

Table A: Ablation study on the temporal and spatial characteristics of the two-phasic metric

Other Comments Or Suggestions: In the equation of Line 241, there is a typo in the average change of other categories.

Response: Thank you for pointing out the error. We have corrected $\neg \Delta_{b i}^{(i)}$ to $\Delta_{\neg b i}^{(i)}$ in the formula on Line 241.

Thank you very much for your thorough and constructive feedback! Below, we present our responses to each of your concerns and questions.

Weaknesses: The method introduces hyperparameters that must be carefully selected to ensure the effectiveness of the proposed approach.

Response: We agree with you that the sensitivity of hyperparameter in 2-phasic metric is crucial for its practical applicability. As shown in Figure 6, although the model’s performance exhibits fluctuations with certain hyperparameter variations, the overall performance remains relatively stable. This indicates that some hyperparameters have low sensitivity and that there may be correlations among them. Therefore, in future work, we plan to reduce the number of hyperparameters by employing hyperparameter fusion strategies, aiming to simplify the deployment of the 2-phasic metric while retaining its performance advantages.

Other Comments Or Suggestions: A more detailed discussion on the theoretical guarantees of the proposed metric would enhance its reliability for practical use.

Response: We appreciate your insightful feedback. Due to limited time during the rebuttal, we propose a preliminary theoretical idea as a potential direction for addressing this concern.

Our idea is to use the local elasticity hypothesis to prove that adding 2-phase labels can promote feature separability. Local elasticity is introduced: when the model is updated at sample $x$ via SGD, the predicted change  $|f(x', w^+) - f(x', w)|$ of another feature vector $x'$ is positively correlated with the similarity between $x$ and  $x'$ [1].

Reference [2] utilizes the local elasticity assumption to derive conditions for feature separability. This study describes the temporal evolution of features for two classes of samples, denoted as $X_{i}^1(t)$ and $X_{j}^2(t)$. Furthermore, Theorem 2.1 in [2] indicates that:

Given the feature vectors $X_{i}^1​(t)$,$X_{j}^2​(t)$ for $i, j \in [n]$, as $t \to \infty$ and large $n$,

1. if $\alpha>\beta$, they are asymptotically separable with probability tending to one,
2. if $\alpha\leq\beta$, they are asymptotically separable with probability tending to zero.

As illustrated in Figure 4, our 2-phasic samples tend to lie closer to the decision boundary. Intuitively, incorporating these samples may help reduce the inter-class effect $\beta$, thereby promoting stricter feature separability. In the future, we plan to study the local elasticity about 2-phase samples, estimate the intra-class effect $\alpha$ and inter-class effect $\beta$ before and after adding 2-phase samples. This analysis will allow us to verify our hypothesis and provide a theoretical foundation for the effectiveness of our method.

[1] The local elasticity of neural networks. ICLR, 2020.

[2] Imitating deep learning dynamics via locally elastic stochastic differential equations. NeurIPS, 2021.

Q1: In pseudo-labeling tasks, what metrics have been proposed by existing works to identify effective samples?

Response: Pseudo-label selection metrics mainly fall into two categories: confidence-based and uncertainty-based. Confidence-based methods typically use the maximum softmax score as the selection criterion [1]. Enhancements include FlexMatch [3], which applies dynamic class-wise thresholds for more flexible filtering, and SoftMatch [4], which weights samples by confidence to balance pseudo-label quality and quantity.

Uncertainty-based metrics prioritize samples with low prediction uncertainty. A representative approach is Monte Carlo dropout [5], which estimates uncertainty via the variance of predictions under multiple dropout runs. Another method leverages training stationarity (or time consistency), selecting labels with stable predictions over time [7]. Additionally, recent work proposes learning a metric via a neural network that models the relationship between label embeddings and feature embeddings to assess pseudo-label quality [8].

[1] Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. AAAI, 2021.

[2] Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 2021.

[3] Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. ICLR, 2022.

[4] Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ICLR, 2016.

[5] Temporal self-ensembling teacher for semi-supervised object detection. IEEE Transactions on Multimedia, 2021.

[6] Semireward: A general reward model for semi-supervised learning. ICLR, 2024.

Q2: How do you construct an effective memory bank to support the calculation of the proposed metric?

Response:
In our experiments, we adopted a uniform sampling strategy to construct the memory bank. Specifically, model predictions were sampled and stored at fixed training steps. With only 50 recorded training dynamics, we achieved satisfactory accuracy in experiments on image datasets.

Thank you very much for your thorough and constructive feedback! Below, we present our responses to each of your concerns and questions.

Weakness 1: In the Booster test experiments, the authors adopted a three-stage protocol. This design is not commonly seen in general pseudo-labeling approaches. The authors should provide a detailed analysis of this hyper-parameter setting to validate its robustness.

Response: The three-stage training protocol was designed to validate whether the proposed two-phase labels can effectively enhance existing pseudo-labeling methods. It is worth emphasizing that the primary contribution of this paper lies in the identification of the two-phase labels, which serve as a complementary enhancement to baseline pseudo-labels. During Stage 2, we use the baseline pseudo-labeling method while recording training dynamics. These recorded dynamics were subsequently utilized to calculate the 2-phasic metric used to identify pseudo labels in Stage 3.

We determine the length of Stage 2 based on two robust principles:

1. In graph datasets, the transition from Stage 2 to Stage 3 is determined by model convergence criteria. Specifically, Stage 2 concludes when additional training iterations using the baseline pseudo-labeling method no longer yield performance improvements, at which point Stage 3 is initiated.
2. In image datasets, a fixed schedule is adopted, where Stage 3 begins after recording four epochs of training dynamics in Stage 2, followed by six training epochs in Stage 3.

Weakness 2: As shown in Figure 4, on the Cora dataset, two-phase labels are significantly closer to the decision boundaries compared to high-confidence labels. However, this phenomenon is not as evident in the CIFAR-100 image dataset. The authors should further analyze and clarify the reasons behind this discrepancy.

Response: Thank you for highlighting this critical observation. We find that this phenomenon is primarily attributed to the use of pre-trained models in image datasets. Specifically, pre-trained models learn diverse feature representations from large-scale pre-training datasets, which may cause samples from the same class to be scattered across multiple clusters in the latent space. This dispersion effect reduces the observable distinction between high-confidence pseudo-labels and the decision boundaries.

To validate this point, we conducted an additional controlled experiment using a ResNet-18 model trained on CIFAR-100, replacing the original pre-trained Vision Transformer (ViT) backbone. This modification eliminates the representational biases introduced by pre-training. Our empirical results demonstrate that, under this setting, high-confidence pseudo-labels exhibit clearer separation from the decision boundaries. We will include these findings and their analysis in the camera-ready version.

Other Comments Or Suggestions: The SoftMatch method achieves a balance between the quality and quantity of pseudo-labeled samples by weighting them according to their confidence, leading to significant performance improvements in pseudo-labeling tasks. Could this idea also be applied to the selection of two-phase labels to further enhance their effectiveness?

Response: Thank you for this insightful suggestion. The SoftMatch method effectively balances pseudo-label quality and quantity by weighting samples based on their confidence scores. Inspired by this motivation, we adopt a similar strategy in our approach.

Specifically, we first calculate a normalized 2-phasic metric to the [0,1] range by $\phi=Normalize(-1*2-phasic)$. We then calculated the mean $\hat{\mu}$ and variance $\hat{\sigma}$ of $\phi$ and assigned loss weights $\lambda$ to samples using the formula in SoftMatch

$ \lambda(\mathbf{p})= \\begin{cases} \phi(\mathbf{p})* \exp\left(-\frac{(\phi(\mathbf{p}) - \hat{\mu})^2}{2\hat{\sigma}^2}\right), & \text{if } \phi(\mathbf{p}) < \hat{\mu}, \\\ \phi(\mathbf{p}), & \text{otherwise}. \\end{cases} \ $

As shown in Table A, experimental results show that integrating the SoftMatch weight into our 2-phasic metric led to a slight decrease in classification accuracy. We are conducting further analysis to investigate the underlying causes of this phenomenon.

Table A: Applying SoftMatch-inspired weighting to our 2-phasic metric

Thank you so much for your detailed and constructive comments! We have carefully studied them and revised the paper accordingly.

Q1: The paper identifies limitations, including parameter sensitivity and computational overhead. Could you discuss any ongoing or planned future work to address these issues?

Response: Thank you for raising these important points. Below is our response to these limitations.

1. Hyperparameter determination

As shown in Figure 6, although the model’s performance exhibits fluctuations with certain hyperparameter variations, the overall performance remains relatively stable. This indicates that some hyperparameters have low sensitivity and that there may be correlations among them. Therefore, in future work, we plan to reduce the number of hyperparameters by employing hyperparameter fusion strategies, aiming to simplify the deployment of the 2-phasic metric while retaining its performance advantages.

2. Computational Overhead

Calculating the 2-phasic metric requires recording training dynamics but does not introduce significant computational overhead. Specifically, it introduces an additional space complexity of O(N|T|) and time complexity of O(NC|T|), where N is the number of unlabeled samples, C is the class count, and |T| is the number of recorded training dynamics. Empirical validation (e.g., C=100 and |T|=50 for CIFAR-100) shows that this complexity is not too high in real-world datasets.

Moreover, the complexity limitations can be effectively mitigated through two strategies:

1. Parallelization: The recording of training dynamics and the computation of the 2-phasic metric are decoupled from the backpropagation process of the neural network. This allows for parallel execution, effectively hiding the additional computational latency within the training loop.

2. Efficient Sampling for Memory Bank: To further optimize memory usage, we propose an adaptive sampling strategy for memory bank construction. Rather than storing the training dynamics at all epochs, we selectively retain a representative subset by an adaptive sampling strategy based on the recorded training dynamics.

Q2: Have you ever considered evaluating 2-phasic metric across various types of neural networks, such as RNNs and Transformers?

Response: The principle of the 2-phasic metric is to capture the transition of neural network learning patterns from simple to complex. The universal law of neural networks learning from easy to hard has been well-established [1][2], endowing the 2-phasic metric with broad applicability across various neural network architectures. In our experiments, we used GCN and ViT as backbones for node classification and image classification tasks, respectively. The ViT represents a Transformer-based architecture. In the appendix, we additionally validated the effectiveness of the 2-phasic metric using GAT as the backbone for node classification. Moving forward, we plan to apply the 2-phasic metric to diverse data types and additional backbone architectures to further verify its generalizability.

[1] Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233–242, 2017.

[2] Siddiqui, S. A., Rajkumar, N., Maharaj, T., Krueger, D., and Hooker, S. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. In The Eleventh International Conference on Learning Representations,2023.

Other Comments Or Suggestions:

1. Ensure figure references are accurate and well-positioned.

2. Enhance captions and explanations for figures and tables to better guide readers

Thank you for your valuable suggestions. In response, we have made the following revisions:

• We carefully reviewed all figure and table references, including the main paper and the Appendix.
• We examined the captions of all figures and tables and revised some of them, such as the caption of Table 2.