Unsupervised Transfer Learning via Adversarial Contrastive Training

We thank you for your thorough review of our manuscript and for your constructive suggestions. Our point-by-point responses to your comments are given below.

C1 Improve the presentation.

Thank you for your constructive suggestion. Please see the response to C4 of Reviewer f4DL.

C2 The role of the matrix $G$ in the proposed method and its theoretical analysis.

Thank you for your valuable feedback. The matrix $G$ plays a pivotal role in both the theoretical analysis and the algorithmic design. Specifically:

• Theoretical Perspective: The matrix $G$ ensures that the sample-level regularization term in line 180 remains unbiased, whereas the sample-level regularization term in eq. (2) is biased. As discussed in Section 3.2, the biased sample-level loss introduces challenges for the error analysis. In contrast, the unbiased sample-level loss as eq. (8) simplifies the analysis, facilitating a more tractable approach to proving the theoretical guarantees.
• Practical Perspective: Experimental results in Table 1 highlight that ACT significantly improves downstream classification accuracy compared to two biased self-supervised learning methods: Barlow Twins and Hao Chen et al. (2022). This demonstrates the practical benefits of incorporating matrix $G$, especially in terms of enhancing the performance of the model.

C3 The computational costs of updating $G$.

As outlined in step 8 of Algorithm 1, the update of matrix $G$ involves only two encoder evaluations and a summation over the samples in a mini-batch. The computational cost of this step is relatively minimal compared to the cost of training the encoder, which not only requires encoder evaluations but also involves backpropagation and gradient descent updates.

C4 The activation function and architectures of the networks.

To maintain a fair comparison, we use the ReLU activation function and ResNet18 as the backbone, as done in BT and Hao Chen (2022). However, we appreciate your suggestion and will consider exploring the use of GeLU and transformer-based architectures, in future work involving more complex tasks.

C5 Hyperparameter $\gamma$ of ACT.

We would like to clarify that ACT as eq. (4) does not include a hyperparameter $\gamma$. We presume you mean $\lambda$. If it is, we conduct ablation studies across a range of values for the regularization parameter $\lambda$. The experimental results, as shown in https://anonymous.4open.science/r/RE1-FFCE, demonstrate that ACT is robust to the selection of $\lambda$

C6 The experiments appear insufficient.

We appreciate your feedback.

• Comparison with BT: Experimental results in Table 1 demonstrate that ACT significantly outperforms Barlow Twins in downstream classification accuracy. Additionally, the results in Table 2 show that ACT achieves SOTA performance when compared to mainstream baselines.
• Transfer Learning: We include experiments for transfer learning from CIFAR100 to CIFAR10, which are presented in https://anonymous.4open.science/r/RE4-8B12.
• Ablation Studies: We conduct ablation studies across a variety of augmentation methods and a range of regularization parameter $\lambda$. The results are provided as https://anonymous.4open.science/r/RE1-FFCE and https://anonymous.4open.science/r/RE5-98C8.

C7 ACT exhibits lower performance than BYOL in linear evaluation.

Thank you for your insightful comment. (1) While ACT demonstrates lower performance than BYOL on the Tiny ImageNet dataset in linear evaluation, it still outperforms other methods. Additionally, as shown in Table 2, ACT consistently outperforms BYOL on the CIFAR-10 and CIFAR-100, which highlights its effectiveness. (2) The observed performance gap on Tiny ImageNet could be influenced by the unique characteristics of the dataset. Our theoretical analysis indicates that ACT's performance is dependent on properties of dataset, and this variability may explain the differing performance on Tiny ImageNet.

C8 ACT bears similarities to an exponential moving average.

Thank you for your comment. However, to the best of our knowledge, the update of $G$ has limited relevance to EMA used in BYOL. Specifically, EMA in BYOL involves online and target neural networks. In contrast, ACT employs a single network.

C9 BYOL works even without batch statistics.

Thank you for your comment. We would like to clarify that the paper you referenced, BYOL works even without batch statistics, has limited relevance to ACT. In BYOL, batch normalization (BN) plays a critical role in preventing collapse. The paper you mentioned shows that using group normalization can achieve competitive performance compared to BYOL with BN. However, rather than relying on BN, ACT prevents collapse through an explicit regularization term. This distinction means that the impact of the paper you referenced in ACT is not as critical as it is in BYOL.

We thank you for your thorough review of our manuscript and for your constructive suggestions. Our point-by-point responses to your comments are given below. Thank you for pointing out these typos. We correct them in the revised manuscript.

C1 The dimension of ACT.

• We appreciate the suggestion and agree that evaluating ACT with a 512-dimensional representation would ensure fairness in comparison. We implement the proposed ACT method using the same 512-dimensional representation as the baseline methods, which outperforms other methods. Further, we conducted ablation studies across a range of dimensions of ACT. Our experiments suggest that ACT is robust to dimension choices within a reasonable range. The experimental results are shown as https://anonymous.4open.science/r/RE3-1437.
• Choosing the appropriate dimension for representation learning may vary depending on the dataset, task complexity, and computational and memory constraints. We recommend performing cross-validation (CV) over a range of dimensions. Additionally, dimension selection may be influenced by resource constraints, as lower-dimensional representations can offer faster inference and reduced memory usage.

C2 Improve the presentation of the technical parts.

Thank you for your constructive suggestion. Please see the response to C4 of Reviewer f4DL.

C3 Notations $\mathbb{P}\_{s}(\cdot)$ and $\hat{f}\_{n_{s}}$.

• $\mathbb{P}\_{s}(\cdot)$ denotes the probability distribution of the source data, while $\mathbb{P}\_{s}(k)(\cdot)$ denotes the probability distribution of the source data that categorized into the $k$-th latent class $C\_{s}(k)$. Both of them are defined on the domain $\mathcal{X}$.
• $\hat{f}\_{n_{s}}$ denotes the ACT estimator defined as eq. (4). The subscript $n_{s}$ is the number of unlabeled training samples in the source domain. We use this subscript to emphasize that this estimator depends on the training dataset, and we show that the error of this estimator converges as $n_{s}$ increases.

C4 The derivation of line 305.

Thank you for your suggestion. The last term of inequality in line 305 used the fact that $\hat{f}$ is the minimizer of the empirical risk over the hypothesis class. For the sake of clarity of the presentation, we provide the complete derivation:

We define the minimizer of the biased sample-level loss as

where $\widehat{\mathcal{R}}(\cdot)$ is defined as (2). We then consider the expected population risk of this estimator. For each $\bar{f}\in\mathcal{F}$, it holds that

$

\mathcal{L}(\hat{f}_{n_{s}}^{\mathrm{bias}}) &=\{\mathcal{L}(\hat{f}_{n_{s}}^{\mathrm{bias}})-\widehat{\mathcal{L}}(\hat{f}_{n_{s}}^{\mathrm{bias}})\}+\{\widehat{\mathcal{L}}(\hat{f}_{n_{s}}^{\mathrm{bias}}) -\mathcal{L}(\bar{f})\}+\{\mathcal{L}(\bar{f})-\mathcal{L}(f^{*})\}+\mathcal{L}(f^{*}) \\ &\leq\{\mathcal{L}(\hat{f}_{n_{s}}^{\mathrm{bias}})-\widehat{\mathcal{L}}(\hat{f}_{n_{s}}^{\mathrm{bias}})\}+\{\widehat{\mathcal{L}}(\bar{f})-\mathcal{L}(\bar{f})\}+\{\mathcal{L}(\bar{f})-\mathcal{L}(f^{*})\}+\mathcal{L}(f^{*}) \\ &\leq 2\sup_{f\in\mathcal{F}}|\mathcal{L}(f)-\widehat{\mathcal{L}}(f)|+\{\mathcal{L}(\bar{f})-\mathcal{L}(f^{*})\}+\mathcal{L}(f^{*}),

$

where the first inequality follows from the fact that $\hat{f}\_{n_{s}}^{\mathrm{bias}}$ minimizes the empirical risk $\widehat{\mathcal{L}}(\cdot)$ over the hypothesis class $\mathcal{F}$, and thus $\widehat{\mathcal{L}}(\hat{f}\_{n_{s}}^{\mathrm{bias}})\leq\widehat{\mathcal{L}}(\bar{f})$.

Standard techniques in empirical process theory can be applied to estimate the first term in unbiased situations. However, the biased nature of $\widehat{\mathcal{L}}(\cdot)$ complicates the process of bounding the first term.

C5 (1) The on-the-fly estimation is not the same on each batch. (2) Parameterize $G$ using a network.

Thank you for your constructive suggestion.

• We agree that the true $G^{\*}$ should be consistent across batches, as it is determined by the population risk rather than by the batch-specific empirical risk. However, as indicated by our theoretical analysis, the on-the-fly estimation of $G^{\*}$ differ only slightly between batches, provided that the batch size is sufficiently large.
• You raise an interesting point about parameterizing $G$ using a network. Since the solution to the inner maximization problem in Eq. (4) has a closed-form solution, as shown in step 4/8 of Alg. 1, we prefer to rely on the closed-form solution. However, we appreciate your suggestion and will consider exploring this idea in more complex tasks in future work.

We thank you for your thorough review of our manuscript and for your constructive suggestions.

C1 Examples of Assumption 3.8.

We exemplify the source/target distributions as Gaussian mixtures of the same componential variances. Then $\epsilon_{1}$ is the maximum distance between the means of the source and target distributions for each latent class. $\epsilon_{2}$ is the maximum distance between the mixture weights of the source and target distributions. Thus, Assumption 3.8 not only requires that the source and target distributions for each latent class are close in terms of their means, but also that their mixture weights are similar.

C2 Explanations of Assumption 3.7.

The concept of $(\sigma_{s},\delta_{s})$-augmentation is introduced to quantify the concentration of augmented data. We now provide a step-by-step explanation:

• Augmentation distance: for a given augmentation set $\mathcal{A}$, the augmentation distance between two samples $x_{1}$ and $x_{2}$ are defined as: $\\|x_{1}-x_{2}\\|\_{\mathcal{A}}:=\min\_{x_{1}^{\prime}\in\mathcal{A}(x_{1}),x_{2}^{\prime}\in\mathcal{A}(x_{2})}\|x_{1}^{\prime}-x_{2}^{\prime}\|$. Since augmentations can capture semantic meanings of the original sample through various views, this distance reflects the maximal semantic similarity between the two samples.
• $\sigma_{s}$-main-part of the latent class: for a latent class $C_{s}(k)$, the $\sigma_{s}$-main-part is defined as $\widetilde{C}\_{s}(k)\subseteq C\_{s}(k)$ satisfying $\mathbb{P}\_{s}\\{x\in\widetilde{C}\_{s}(k)\\}\geq\sigma_{s}\mathbb{P}\_{s}\\{x\in C_{s}(k)\\}$. The parameter $\sigma_{s}$ quantifies the concentration of the distribution $\mathbb{P}\_{s}(k)$ of this latent class. Specifically, for fixed $\widetilde{C}\_{s}(k)$ and $C\_{s}(k)$, a larger value of $\sigma_{s}$ indicates a higher concentration of $\mathbb{P}\_{s}(k)$.
• Augmentation diameter of the $\sigma_{s}$-main-part: the parameter $\delta_{s}$ is defined as the diameter of the $\sigma_{s}$-main-part in augmentation distance, that is, $\sup\_{x_{1},x_{2}\in\widetilde{C}\_{s}(k)}\\|x_{1}-x_{2}\\|\_{\mathcal{A}}$. For a fixed distribution $\mathbb{P}\_{s}$ and a fixed parameter $\sigma_{s}$, the smaller value of the diameter $\delta_{s}$ means a higher concentration of the augmented distribution, as well as greater similarity between augmented data samples.
• Summary: The concentration of the augmented distribution, as measured by parameters $(\sigma_{s},\delta_{s})$, depends on both $\mathbb{P}\_{s}(k)$ and $\mathcal{A}$. Specifically, for a fixed $\mathcal{A}$, a smaller value of $\sigma_{s}$ and a higher concentration of $\mathbb{P}\_{s}(k)$ result in a smaller $\widetilde{C}\_{s}(k)$, leading to a smaller value of $\delta_{s}$. Additionally, for a fixed distribution $\mathbb{P}\_{s}(k)$, a smaller value of $\sigma_{s}$ and a larger $\mathcal{A}$ lead to smaller $\\|x_{1}-x_{2}\\|\_{\mathcal{A}}$ for each pair $(x_{1},x_{2})$, resulting in a smaller value of $\delta_{s}$.

Example: Suppose the samples in the $k$-th latent class follows the uniform distribution on $[0,R]$, i.e., $C_{s}(k)=[0,R]$ and $\mathbb{P}\_{s}(k)=\mathsf{unif}(0,R)$. For each $\sigma_{s}\in(0,1]$, we can find a $\sigma_{s}$-main-part of $C\_{s}(k)$ as $\widetilde{C}\_{s}(k)=[0,\sigma_{s}R]$. Further, we define $\mathcal{A}(x)=\{x^{\prime}\in\mathbb{R}:|x-x^{\prime}|\leq r\}$ for each $x\in\mathcal{X}$. Then the augmentation diameter $\delta_{s}$ of the $\sigma_{s}$-main-part is given as $\sup\_{x_{1},x_{2}\in\widetilde{C}\_{s}(k)}\\|x_{1}^{\prime}-x_{2}^{\prime}\\|\_{\mathcal{A}}=\max\\{\sigma_{s}R-2r,0\\}=:\delta_{s}.$ The parameters $\sigma_{s}$, $\delta_{s}$, $r$ and $R$ are interrelated by this equality. Note that the parameter $R$ reflects the concentration of the distribution $\mathbb{P}\_{s}(k)$ within the latent class. A smaller value of $R$ indicates a higher concentration of $\mathbb{P}\_{s}(k)$, which in turn leads to a smaller value of the augmentation diameter $\delta_{s}$. Additionally, a larger augmentation set, i.e., a larger value of $r$, results in a smaller value of the augmentation diameter $\delta_{s}$.

C3 The role of $\sigma_{s}$ in the performance guarantee.

Thanks for your suggestion, we have added the interpretation for $\sigma_s$ in revised manuscript.

• The probability of the inequality Line 343 is directly determined by $\sigma_s$. The closer $\sigma_s$ is to 1, the larger the probability of this event.
• The technical effect of $\sigma_s$ in Assumption 3.7. is to convert the condition $\psi > 0$ (Line 1646) into a probabilistic form, ensuring that this condition can be definitively satisfied.

Due to space constraints, we are unable to address the questions regarding unbiasedness and supervised transfer learning in this rebuttal. We kindly ask that you evaluate our existing responses for now, and we will be glad to provide the remaining explanations once we receive your feedback.

We thank you for your thorough review of our manuscript and for your constructive suggestions. Our point-by-point responses to your comments are given below. Thank you for pointing out these typos. We correct them in the revised manuscript.

C1 Additional experiments: (1) comparisons with SOTA methods, and (2) transfer learning.

Thank you for your constructive suggestion.

• We have re-implemented SwAV, and the corresponding experimental results are presented in https://anonymous.4open.science/r/RE6-DDAF. Due to time constraints, we were unable to include additional comparisons with DINO. Alternatively, we kindly ask you to refer to the benchmark results in CIFAR10 available at https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html. In summary, ACT outperforms both SwAV and DINO, demonstrating its strong competitive performance.
• The transferability of ACT: Please see the response to C6 of reviewer L3h2.

C2 The results for re-implementations are lower than expected.

Thank you for your comment.

• Our re-implementation of Barlow Twins is based on the official implementation available at https://github.com/facebookresearch/barlowtwins. Unfortunately, the authors of ``HaoChen 2022'' do not provided their implementation, so we implement it according to their paper.
• To ensure a fair comparison, the hyperparameter choices for ACT and all other methods, including Barlow Twins and ``HaoChen 2022'', is almost aligned with the settings used in [1].
• The official Barlow Twins implementation's README provides a link to a Barlow Twins experiment on CIFAR-10 as https://github.com/IgorSusmelj/barlowtwins. The experimental results reported in this linked implementation closely align with ours, further supporting the fairness of our re-implementation.

[1] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for Self-Supervised Representation Learning. (2021)

C3 Additional alignment objective.

We agree that the the diagonal part of the cross-correlation matrix serves as a alignment term under certain conditions. As shown by Lemma 4.1 of [1], the alignment risk is then related to the diagonal part of the cross-correlation matrix as:

It is crucial that the right-hand side of the inequality is consistent with the alignment term in the loss function of Barlow Twins, provided that $\mathbb{E}\_{x}\mathbb{E}\_{x_{1}\in\mathcal{A}(x)}[f_{i}(x_{1})^{2}]=1$ for each $i\in\\{1,\ldots,d\\}$. However, this condition does not hold generally.

• From a theoretical perspective, the cross-correlation loss alone, as discussed previously, is insufficient for learning representations invariant to augmentations, as previously discussed. To address this, we introduce an additional explicit alignment term in the loss, as also suggested by [2,3].
• From a practical perspective, we fully agree with the necessity of distinguishing the effect of alignment from the de-biased operation. We conduct an ablation study comparing ACT with and without the explicit alignment term as https://anonymous.4open.science/r/RE2-7B34. Our results indicate that the explicit alignment term can slightly improves ACT's performance.

[1] Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the Generalization of Contrastive Self-Supervised Learning. (2023)

[2] Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations. (2022)

[3] Jeff Z. HaoChen, and Tengyu Ma. A Theoretical Study of Inductive Biases in Contrastive Learning. (2023)

C4 Assumption 3.5.

We appreciate your question. As outlined in the section ``Image transformations details'', the data augmentations used in our experiments -- crops, horizontal mirroring, brightness adjustment, grayscaling, and Gaussian blurring -- are linear operations, and thus Lipschitz continuous. Specifically, the Lipschitz constants of crops horizontal mirroring, and grayscaling are less than and equal to 1, and the Lipschitz constant of brightness adjustment depends on the adjustment factor. The Lipschitz constant of Gaussian blurring relies on the the kernel size and the variance of Gaussian. Consequently, all Lipschitz constants can be explicitly calculated, but due to character limitations, we had to place these details in the additional appendix of the revised manuscript.

C5 Assumption 3.7.

See the response to C4 of Reviewer etBC for a detailed discussion on Assumption 3.7.

We thank you for your thorough review of our manuscript and for your constructive suggestions. Our point-by-point responses to your comments are given below.

C1 Lack of practical implementation.

We add a detailed PyTorch-type pseudo-code in the appendix of the revised version.

C2 The choice of the augmentation strategy.

Thank you for your comment.

• We include ablation studies across a variety of data augmentation methods. The experimental results are shown as https://anonymous.4open.science/r/RE5-98C8. The experimental results show that ACT is generally robust to the choice of augmentation strategy.
• Selecting the appropriate augmentation strategy for ACT may vary depending on the dataset and the complexity of the task. To empirically choose the best approach, we recommend using cross-validation (CV). However, this method can be computationally expensive. Fortunately, as our ablation studies demonstrate, ACT shows a relatively low sensitivity to the choice of augmentation method.

C3 The choice of the regularization parameter.

The regularization parameter $\lambda$ in ACT balances the alignment term $\mathcal{L}_{\mathrm{align}}(\cdot)$ and the regularization term $\mathcal{R}(\cdot)$.

From a theoretical perspective, our theoretical analysis suggests that $\lambda=\mathcal{O}(1)$. Specifically,

• In Lemma A.4, we demonstrate that the alignment factor $R\_{t}(\varepsilon,f)$ can be bounded by the alignment term $\mathcal{L}\_{\mathrm{align}}(\cdot)$, while the divergence factor $\max_{i\neq j}|\mu_{t}(t)^{\top}\mu_{t}(j)|$ is bounded by the regularization term $\mathcal{R}(\cdot)$.
• Based on the definition of the population risk $\mathcal{L}(f)=\mathcal{L}\_{\mathrm{align}}(f)+\lambda\mathcal{R}(f)$, we find

$

\mathcal{L}_{\mathrm{align}}(f)\leq\mathcal{L}(f) \quad\text{and}\quad \mathcal{R}(f)\leq\lambda^{-1}\mathcal{L}(f)\lesssim\mathcal{L}(f),

$

where we used $\lambda=\mathcal{O}(1)$. This allows us to bound both the alignment factor $R_{t}(\varepsilon,f)$ and the divergence factor $\max_{i\neq j}|\mu_{t}(t)^{\top}\mu_{t}(j)|$ in terms of the population risk $\mathcal{L}(f)$, which leads to the conclusion in Theorem A.5.

From a practical perspective, we include ablation studies across a range of regularization parameter $\lambda$. The experimental results are shown as https://anonymous.4open.science/r/RE1-FFCE. The experimental results show that ACT is robust to the selection of $\lambda$. For more complex tasks, we recommend performing cross-validation (CV) over a range of regularization parameter.

C4 Presentation suggestions.

Thank you for your constructive suggestion. To improve the clarity and readability of the technical sections, we add a comprehensive table of notations in the appendix. Additionally, we include a proof sketch in the appendix to offer an overview of the key steps in the proofs. We also provide more detailed explanations and broken down the steps more explicitly in the proofs of theoretical results to ensure better understanding.

C5 The necessity of the approximation error analysis.

Thank you for your insightful comment. As shown in eq. (8) of the manuscript, the excess risk can be decomposed into the approximation error and the statistical error, where the latter is bounded by the Rademacher complexity. The approximation error reflects the capacity of the deep neural network class to approximate the target function, and it generally decreases as the scale of the network class increases. On the other hand, the Rademacher complexity increases with the scale of the network class. This creates a trade-off between the approximation error and the statistical error, suggesting that the network class should be chosen with an appropriate scale that depends on both the number of samples and the complexity of the task. This theoretical insight is consistent with the findings in experimental practice.

However, if the approximation error is ignored, the excess risk is only bounded by the Rademacher complexity, which would imply that the network class should be as small as possible. Such an theoretical result clearly contradicts practical applications, where smaller and simpler network classes often struggle to capture the underlying patterns in complex tasks, particularly when dealing with large datasets.