Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Dear Reviewer JsM2,

We sincerely appreciate your thoughtful feedback. We will address each of your concerns in detail.

Weakness #1: Why selecting pseudo-labels using energy score can ensure better model calibration

Since the confirmation bias is induced by self-training, using confidence selection may result in overconfident but wrong pseudo-labels and hurt the calibration[1,2].

In contrast, the energy score leverages the probability density of the predictions, exhibiting reduced vulnerability to overconfidence [3]. Thus, the energy function can alleviate confirmation bias caused by overconfidence, thereby obtaining a reliable data subset on which the model is calibrated.

We also compare the calibration metric ECE value [4] between confidence threshold and energy score in Figure 1 of the paper, and the results confirm our idea.

Weakness #2: The suggestion to demonstrate the reliable and smoothed pseudo labels of the unlabeled data for better understanding

We demonstrate the generation process of reliable pseudo-label and smoothed pseudo-label in the attached PDF.

To generalize the proposed supervised learning framework to LTSSL, the main challenge is unknown $\mathbb{P}_{u}(Y = y \mid \boldsymbol{x}^u)$, where $\boldsymbol{x}^u$ denote sample in unlabeled dataset.

To address this issue, we first approximate $\mathbb{P}_{u}(Y = y \mid \boldsymbol{x}^u)$ using the output of the calibrated and integrated classifier as expressed in Eq. (15). Additionally, we use energy score to filter out reliable unlabeled data, ensuring the model's calibration on this data subset, which constitutes the reliable pseudo-labels subset.

Furthermore, we can also estimate the unknown $\mathbb{P}_{u}(Y = y \mid \boldsymbol{x}^u)$ by leveraging the smoothness assumption. Specifically, we construct smoothed pseudo-labels by propagating labels from nearby samples using Gaussian kernel density estimation.

We will include more detailed textual and graphical demonstrations in our next version of the paper.

Reference

[1] Mishra, S., Murugesan, B., Ayed, I. B., Pedersoli, M., & Dolz, J. (2024). Do not trust what you trust: Miscalibration in Semi-supervised Learning. arXiv preprint arXiv:2403.15567.

[2] Loh, C., Dangovski, R., Sudalairaj, S., Han, S., Han, L., Karlinsky, L., ... & Srivastava, A. (2022). On the Importance of Calibration in Semi-supervised Learning. arXiv preprint arXiv:2210.04783.

[3] Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. Advances in neural information processing systems, 33, 21464-21475.

[4] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.

Dear Reviewer WTLw,

We sincerely appreciate the reviewer for thoughtful feedback. We address your concerns one by one.

Weakness #1: How the proposed method solves the unlabelled data for diverse label distributions of unlabeled data

Having an accurate estimation of unlabeled data prior $\widehat{\boldsymbol{\pi}}_u$ helps solve LTSSL like supervised LTL, which can simply achieve balanced training by logit adjustment. We propose to use EMA estimation of dual-branches by selecting data subset from mini-batch via energy score.

Accurate estimation of $\widehat{\boldsymbol{\pi}}_u$ requires calibrated model [1]. Methods like ACR and CPE, using confidence-based sample selection, producing overconfident but incorrect pseudo-labels [2]. In contrast, energy score leverages prediction probability density, mitigating the overconfidence problem [3]. Thus, energy selection is used for a reliable unlabeled data subset, on which model is calibrated, enabling accurate estimation of $\widehat{\boldsymbol{\pi}}_u$.

In Figure 1 of our paper, we empirically compared the estimation error of confidence and energy score.

Weakness #2: How to interpret "Furthermore, these methods primarily focus on correcting model outputs without delving into the role of representation learning in improving performance"

Recent LTSSL methods focus on adjusting the classifier output to overcome the long-tail effect. Methods like DASO, ACR, and CPE improve pseudo-label quality through techniques such as classifier output ensembling or logits scaling, ignoring the representation learning.

In contrast, our method delves into the long-tail effect in representation learning. CCL introduces balanced FixMatch for classifier training and enhances representation learning with reliable and smoothed pseudo-labels, as described in Eq. (17) and Eq. (18).

Weakness #3: Relationship between reliable pseudo-labels and smoothed pseudo-labels

The main challenge in generalizing our supervised learning framework to LTSSL is the unknown of $\mathbb{P}_{u}(Y = y \mid \boldsymbol{x}^u)$ for unlabeled samples.

We first approximate $\mathbb{P}_{u}(Y = y \mid \boldsymbol{x}^u)$ using integrated classifier output in Eq. (15). We use energy score to filter out reliable unlabeled samples and their pseudo-labels, ensuring that model is calibrated on this subset. Learning from reliable pseudo-labels enhances feature discrimination.

However, self-training can introduce confirmation bias and negatively impact the representation learning. Thus we propose smoothed pseudo-labels for $\boldsymbol{x}^u$ by propagating labels from nearby samples, leveraging the smoothness assumption in the feature space without classifier information. The purpose of $\mathcal{L}_{\mathrm{spl}}$ is to align the smoothed pseudo-labels of $\mathcal{A}\_{w}(\boldsymbol{x}^u)$ and $\mathcal{A}\_{s}(\boldsymbol{x}^u)$, serving as "consistency regularization" in SSL.

Weakness #4: The time and space complexity

Denote feature space dimension $D$, batch size $B$ and number of classes $C$, the time and space complexity of CCL are as follows.

Time Complexity

For $\widehat{\mathcal{L}}\_{\mathrm{rpl}}$:

1. Calculating kernel similarities: $\mathbb{R}^{B \times D} \times \mathbb{R}^{D \times B}$, with complexity $\mathcal{O}(B^{2}D)$.
2. Calculating $\mathbb{E}\_{\boldsymbol{x}^{\prime} \sim \mathbb{P}(\cdot| Y=y)}\left[\kappa\left(\boldsymbol{z_x}, \boldsymbol{z_{x^{\prime}}}\right)\right]$ using Gaussian kernels: $\mathbb{R}^{B \times B} \times \mathbb{R}^{B \times C}$, with complexity $\mathcal{O}(B^{2}C)$.

Total: $\mathcal{O}(B^{2}D+ B^{2}C)$.

For $\widehat{\mathcal{L}}\_{\mathrm{spl}}$:

1. Calculating kernel similarities: $\mathbb{R}^{B \times D} \times \mathbb{R}^{D \times B}$, with complexity $\mathcal{O}(B^2 D)$.
2. Calculating $\widehat{\mathbb{P}}(Y \left.=y \mid \boldsymbol{x}^u ; \mathcal{B}^l\right)$ based on Eq. (19): $\mathbb{R}^{B \times B} \times \mathbb{R}^{B \times C}$, with complexity $\mathcal{O}(B^2 C)$.
3. Inverse matrix $(\boldsymbol{I} - \beta \boldsymbol{G})^{-1}$ in Eq. (21): $\mathcal{O}(B^3)$.

Total: $\mathcal{O}(B^2 D + B^2 C + B^3)$.

Space Complexity

For $\widehat{\mathcal{L}}\_{\mathrm{rpl}}$:

1. Pairwise kernel similarity : $\mathcal{O}(B^2)$.
2. Calculating $\mathbb{E}\_{\boldsymbol{x}^{\prime} \sim \mathbb{P}(\cdot| Y=y)}\left[\kappa\left(\boldsymbol{z_x}, \boldsymbol{z_{x^{\prime}}}\right)\right]$ using Gaussian kernels: $\mathcal{O}(BC)$.

Total: $\mathcal{O}(B^2 + BC)$.

For $\widehat{\mathcal{L}}\_{\mathrm{spl}}$:

1. Computing $\widehat{\mathbb{P}}(Y = y \mid \boldsymbol{x}^u ; \mathcal{B}^l)$: $\mathcal{O}(BC)$.
2. Pairwise kernel similarity: $\mathcal{O}(B^2)$.

Total: $\mathcal{O}(B^2 + BC)$.

In Table 2 and Table 3 of the global response (PDF) of our rebuttal, we empirically report the averaged minibatch training time with a single 3090 GPU and the GPU memory usage.

Generally, $B=64$, $D=256$, and $C=100$. Compared to the # of model parameters, CCL adds negligible overhead relative to the neural network's computational cost. In Table 2 of our global response (PDF), we should note that the training time of CCL is comparable to existing state-of-the-art method ACR.

Thank you for the valuable feedback again. We will address all the issues mentioned in the next version of the paper.

Reference

[1] Garg, S., Wu, Y., Balakrishnan, S., & Lipton, Z. (2020). A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33, 3290-3300.

[2] Loh, C., Dangovski, R., Sudalairaj, S., Han, S., Han, L., Karlinsky, L., ... & Srivastava, A. (2022). On the Importance of Calibration in Semi-supervised Learning. arXiv preprint arXiv:2210.04783.

[3] Liu, W., Wang, X., Owens, J., & Li, Y. (2020). Energy-based out-of-distribution detection. Advances in neural information processing systems, 33, 21464-21475.

Dear Reviewer biXa,

We sincerely appreciate the reviewer's thoughtful feedback. We will address your concerns individually.

Weakness #1: How does the proposed method address the issue of having mini-batches with no samples from certain classes during training

For our proposed method CCL, to avoid none class samples in each min-batch during training, we maintain a learnable class prototype for each class to facilitate the contrastive learning and address the aforementioned issue. The design of the learnable class prototypes is achieved through a nonlinear mapping of the classifier's weights.

Weakness #2: The comparison of training time overhead between the proposed method and ACR.

Thank you for the suggestion for considering the time overhead of the proposed method! We evaluated the average time taken to process a minibatch with a single 3090 GPU.

Table 1: Average batch time of each algorithm

Since CCL additionally considers representation learning compared to ACR, it applies corresponding data augmentations to labeled data as well, resulting in an overhead approximately 1.5 times that of ACR. However, as seen from the Table 1, the execution speed is nearly identical.

Dear Reviewer 1xLE,

We sincerely appreciate your thoughtful feedback. We will respond to each of your concerns individually.

Weakness #1: Misunderstanding between $\widehat{\mathcal{L}}_{\mathrm{cls}}$ in Eq. (22) and $\widehat{\mathbb{P}}^{\mathrm{cls}}\left(Y=y \mid \boldsymbol{x}^u\right)$ in Eq. (15)

To clarify, $\widehat{\mathcal{L}}_{\mathrm{cls}}$ represents the loss function for the classification part of the dual-branches $f_s(\cdot)$ and $f_b(\cdot)$.

However, $\widehat{\mathbb{P}}^{\mathrm{cls}}\left(Y=y \mid \boldsymbol{x}^u\right)$ denotes the pseudo-labels for training dual-branches $f_s(\cdot)$ and $f_b(\cdot)$, where we propose to fuse the predictions of balanced and standard branches by Eq. (15) to obtain $\widehat{\mathbb{P}}^{\mathrm{cls}}\left(Y=y \mid \boldsymbol{x}^u\right)$.