Understanding the Role of Equivariance in Self-supervised Learning

We thank Reviewer PgKg for appreciating our writing and theoretical contributions. Below we address each of your concerns.

---

Q1. The experiments are conducted on few datasets which is not very convincing.

A1. Following your suggestion, we further validate our findings on ResNet-50 and CIFAR-100. The results all agree with our findings in the main paper, that 1) better rotation prediction brings better test error, and 2) model equivariance contributes to better test classification. Notably, model equivariance contributes even more on CIFAR-100.

Results on CIFAR-100.

Results on ResNet-50 (crop+flip) on CIFAR-100.

---

Q2. There exist some typos in the paper, such as the “xIn” in line 352.

A2. Thanks for pointing out. We will fix them in the revision.

---

Q3. It is better for the authors to provide a design enhancement example and conduct experiments to prove the effectiveness of the example, as it only explains the success of previous E-SSL works.

A3. We note that the key focus of this work is to establish the first theoretical understanding of E-SSL, instead of devising a new variant. The former may have a larger significance because it is more fundamental to this field. Driven by our theoretical insights, we also first study combining model equivariance with training equivariance for E-SSL, and it improves RotNet by 25.22% on CIFAR-10 and 36.26% accuracy on CIFAR-100. Remarkably, we are the first to attain comparable performance by predicting augmentations alone (while previous methods rely on merging with contrastive learning). This illustrates that our analysis can open doors for new paths for E-SSL designs.

---

Q4. Is 89.49% mentioned in line 279 the ACC of the SimCLR framework during testing?

A4. Indeed, we made a typo here. Later, when merging with the model equivariance (Sec 5.3), its performance (82.26%) can be close to SimCLR. We will fix this confusion in the revision.

---

Q5. In the paragraph starting from line 328, it is believed that the action space of the instance discrimination task is N, but I think it is actually a combination of many binary classification tasks with an action space of 2. So I think contrastive learning is not a special case to equivariant tasks.

A5. We understand that contrastive learning (CL) can be understood as many binary classification tasks between positives and negatives. Nevertheless, this understanding defines infinitely many binary tasks and every task has been seen only once during training, a setting rarely studied in the ML literature. The deviation from common learning tasks makes it hard for us to have a solid understanding of how contrastive learning actually behaves.

Instead, it is a common and natural way to understand CL as instance classification (IC). In fact, in the seminal work by Wu et al. [1] (prior to InfoNCE) that first proposed contrastive learning (their loss is the same as InfoNCE), they do regard CL as a non-parametric approach to IC, as the title suggested. We refer to [1] for a detailed explanation of this connection.

We further note that for a very large number of classes like IC, it is common to subsample the classes in the denominator of CE loss. A well-known example is the negative sampling technique proposed in word2vec [2] for learning on a very large vocabulary. In instance classification, existing works also subsample the negative classes [1,3], leading to a mini-batch number of negative samples during training.

Hope this elaboration address your concern! We will clarify this part in the revision.

Ref:

[1] Wu, Zhirong, et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR. 2018.

[2] Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." NeurIPS 2013.

[3] Cao, Yue, et al. "Parametric instance classification for unsupervised visual feature learning." NeurIPS 2020.

---

Q6. Principle 2 states that it is best to be category related, does it mean that category label information needs to be known during pre-training stage in SSL?

A6. No. For SSL, the labels are not supposed to be known during pretraining. What we are discussing in Principle 2 is that the choice of the augmentation type (eg the rotation or the flip) should be related to the class labels, in the sense that extracting class-relevant information from the input is helpful for predicting these transformations (no class label required) during E-SSL pretraining. We will make it clear in the revision.

---

Q7. Why can predictive patches like MAE be considered as an equivariant SSL task?

A7. In this paper, we consider a general notion of equivariant SSL, in the sense that the representations is learned to be aware of the pretext, eg aug parameters. MAE augments by masking patches at random positions $p$ (the aug parameter in MAE). Then, MAE takes the position $p$ as input, and learns to reconstruct images at these locations, thus its representations can adapt to the input position variables $p$, in contrast with contrastive learning whose representation is global and invariant to augmentations. Thus, we regard MAE more as an equivariant method. We will explain it in the revision.

---

We hope this clarifies your questions. If you would find it satisfactory, we respectfully hope that you may consider re-evaluating our work based on the revisions.

We sincerely thank Reviewer TYZN for a critical reading of our work. We carefully examine the intuition you propose, and find that it actually fits well into our theory.

---

Q1. Discussion on the intuitive understanding of why E-SSL works and how it fits into our theory.

A1. We resonate with your insight that for rotation prediction to be effective, the class/object itself must exhibit a rotational bias (i.e., it is a necessary condition). Nevertheless, this intuition lacks a solid theoretical explanation. We find it naturally fits as a piece of our explaining-away framework, where it can have a rigorous explanation.

To extend our current analysis, besides the causal diagram in Fig 2, we further consider the inherent rotation angle $\bar A$ and an edge $\bar A\to \bar X$, i.e., the original object is also generated from its intrinsic angle. Notice that now there is a new collider structure $C\to \bar X\leftarrow \bar A$, which means that $I(\bar A;C|\bar X)>0$, which rigorously explains why class information $C$ is helpful for predicting the intrinsic rotation $\bar A$ from the original object $\bar X$.

Since the intrinsic rotation $\bar A$ is unknown, RotNet further applies a (known) random rotation $A$ to $\bar X$ and get $X$, and the rotated image has an angle $A'=\bar A+A$. Apparently, figuring out $\bar A$ is directly helpful for predicting the random $A$ (the E-SSL target). Formally, we have a collider structure $\bar A\to X\leftarrow A$ and have a positive synergy $I(A;\bar A|X)>0$.

Thus, our explaining-away analysis justifies that 1) knowing $C$ is also helpful for predicting $\bar A$, and 2) knowing $\bar A$ is helpful for predicting $A$ as well. In this way, introducing $\bar A$ does not invalidate our previous theory but provide more fine-grained analysis on the way of how class information helps rotation prediction.

Thank you again for this valuable perspective. We will definitely incorporate it and acknowledge your insights.

---

Q2. Theorem 2 is tied to an over-simplified mixing model ($X=A+\lambda C$).

A2. We note that Theorem 1 already gives us a general characterization that is agnostic of the data distribution or the model class. Therefore, our framework and analysis does apply to general real-world scenarios. Due to its generality, it is also hard to obtain a very fine-grained quantitative analysis. Thus, the purpose of Thm 2 is to gain more quantitative insights with a linear model. Note that linear data assumptions are also commonly adopted in SSL theory, eg [1]. We leave more complex cases if left for future work.

Ref:

[1] Wen, Zixin, and Yuanzhi Li. "Toward understanding the feature learning process of self-supervised contrastive learning." ICML. PMLR, 2021.

---

Q3. Rigor:

Notation: notations should be consistent throughout the paper (e.g., line 250 and 256, line 236/237 and line 240) and assumptions should be made clear (e.g., eq (3) and line 293, lambda=1 is not stated)

Thanks. We will fix it.

Principle III: refers to theoretical results - "style features may also explain the equivariant target A" - , which have not been explicitly shown (in section 4.1 or in the appendix)

Indeed, we didn’t elaborate the style features here. Similar to the intrinsic rotation variable $\bar A$, the feature variable $S$ also constitutes the object $\bar X$ (with a causal edge $S\to \bar X$). Due to the explaining-away effect in the collider structure $S\to X\leftarrow A$, $S$ can also explain and help predict $A$ with $I(S;A|X)>0$. Therefore, style features, as an easy-to-learn shortcut, can lead the model to learn fewer class features. We will elaborate this relationship rigorously in the revision.

Contrastive learning vs. Instance discrimination

We note that contrastive learning can be seen as a minibatch approximation to instance classification by random subsampling a few instances (e.g., 2048) at each time. In expectation, the objective is still distinguishing the positive instance from all the other negative instances in the dataset. This negative class sampling technique is widely used in standard classification tasks, e.g., negative sampling in word2vec [1]. Thus, we think it is plausible to regard contrastive learning essentially as an instance classification task.

References:

[1] Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality. NeurIPS 2013.

---

Q4. Reference for claims and Typos.

A4. Thanks. We will add the references and fix the typos.

---

Q5. Can you please clarify what is meant by "global" vs. "local" transformations (principle II)?

A5. Here, local changes refer to those that only modify pixel values but do not modify the pixel’s positions (eg color inversion). Therefore, to predict these operations, the NNs only need to look at each pixel’s values without relying on global context. Instead, global operations change pixel positions (e.g., rotation, flip), and in these cases, figuring out the global content can facilitate the prediction of these global changes. We will clarify it in the revision.

---

Q6. Can authors rigorously justify the extension of the explaining-away principle from X to C?

A6. Of course. In fact, according to probabilistic graphical models, as long as there is a collider structure between the variables, there will be an explaining-away effect. In the causal structure $C\to \bar X\to X$, we can also omit the intermediate node $\bar X$ (which also holds since $X$ is causally dependent on $C$) and write the collider structure $C\to X\leftarrow A$, which implies the explaining-away effect with $I(A;C|X)>0$.

---

Thank you for your careful reading and for sharing your insights. If you would find our explanations satisfactory, we respectfully ask you to consider re-evaluating our work based on the revisions. We are also happy to address any remaining concerns during the discussion stage.

We thank Reviewer yP9X for acknowledging our contributions to theoretical understandings. Below, we further address your concerns on the empirical side.

---

Q1. Experiments are conducted using smaller-scale models.

A1. Thank for your advice! The experiments are mainly designed to validate our theoretical insights, and as shown in previous works, E-SSL methods generalize well across different model sizes. Following your suggestions, we further validate our findings on a larger model ResNet-50 and a more complex dataset CIFAR-100.

As shown below, the results agree well with our findings in the main paper, that 1) better rotation accuracy under a larger model brings better test classification accuracy; and 2) model equivariance brings significant gains. Notably, the improvement of model equivariance is even more significant on CIFAR-100, considering that CIFAR-100 has 100 classes whose accuracy is harder to improve.

Results on CIFAR-100 (akin to Table 2 on CIFAR-10).

Results on ResNet-50 (with crop+flip) on CIFAR-100.

---

Q2. Could the principles 1,2,3 be applied to other domains beyond vision-based SSL?

A2. Yes! Our principles are based on a theoretical analysis of the general methodology of E-SSL, not tailored to vision-based SSL. In other domains (text, graph, speech, time series etc), there are also many similar SSL methods that can be understood as E-SSL in the general sense (see e.g., the discussion of MAE and BERT in Sec 5.1). Taking the BERT model in NLP as example, our theory can understand its success from these the three principles:

• Principle I: First, BERT learns representations by predicting the masked words (task $A$), and it is shown that a large masking ratio is important for learning useful features (for task $C$). As analysed in Principle I, this is also because we need to corrupt the input enough in order for the high-semantic class-related features to be utilized for masked prediction.
• Principle II: Meanwhile, class information (high-level text semantics) is indeed helpful for masked prediction, thus demonstrating the importance of class relevance (Principle II).
• Principle III: At last, masking prediction as a token-level dense prediction task helps prevent the shortcut features (Principle III).

Thus, the insights of our principles are general and not limited to a certain modality. As the first theoretical work in the field of E-SSL, it can help explain many existing E-SSL-like approaches as well as inspire new designs in multiple domains.

---

Hope the explanation above address your concerns! Please let us know if there is more to clarify.

The computation of I(A;C|X) requires knowledge of both the pretraining label $A$ and the class label $C$. When both are available, we can compute it following its decomposition $I(A;C|X)=H(A|X)-H(A|X,C)$. Estimating entropy in high-dimensional space is generally hard, and a common approach is to leverage variational estimates for the two entropy terms $H(A|X)$ and $H(A|X,C)$ using neural networks.

Variational estimates of entropy. Take $H(A|X)$ as an example. We can learn a NN classifier $P_\theta(A|X)$ optimized by minimizing the cross-entropy $L(\theta)=-E_{P_d(X,Y)} P_{\theta}(A|X)$. The following inequality shows that CE loss is the variational upper bound of $H(A|X)$. When $\theta$ is minimized at the minimum with $P_\theta(A|X)=P_d(A|X)$, we have $L(\theta)=H(A|X)$ as the perfect estimate. Since NNs are generally expressive approximators, we believe that the (converged) CE loss can serve as a good estimate for $H(A|X)$.

$L(\theta)=H(A|X)+KL(P_d(A|X)\|P_\theta(A|X))\geq H(A|X)$

Variational estimates of $I(A;C|X)$. Combing these two, we notice that a good estimate for $I(A;C|X)$ is the difference between the cross entropy losses of the predictor $P_\theta(A|X)$ and $P_\theta(A|X,C)$, which is exactly what we studied in the verification experiment in Sec 4.1. In other words, the gap between the CE losses (or similarly the accuracies) between “rotation” and “rotation+cls”, can serve as a quantitative measure of the synergy effect $I(A;C|X)$. A more computationally friendly way is to select only a small subset of samples and to train it shortly. As shown in Figure 3, the gap is already exhibited at the early stage of training.

We thank Reviewer a2cS for acknowledging our theoretical contributions. Below we address your main concerns.

---

Q1. Comparing augmentation aware methods (e.g. RotNet) and truly equivariant methods (eg CARE).

While this distinction may not affect the general ideas and theoretical arguments, it may impact empirical results and practical considerations

A1. Thank you for your insightful question. For a controlled study on the role of true equivariance, we add the equivariant objective proposed in CARE [1], which is known to enforce true equivariance when it attains an optimum. The results are shown below.

We find that although the two have similar rotation loss, training with the equivariance regularization leads to further improvement in test accuracy (57.32% → 64.50%), which is quite similar to our findings on the model equivariance. As discussed in Sec 5.3, these pieces of evidence suggest that enforcing feature equivariance (either through model architecture or feature regularization) does have considerable benefits compared to vanilla predictive loss. This is a valuable insight and we will add it in the revision. Thanks for bringing it up!

---

Q2. Quantitative estimate of I(A;C|X).

A2. Indeed, in our analysis, the mutual information I(A;C|X) is the key factor that upper bounds the utility of class features during pretraining and thus determines the downstream transferability of E-SSL. In fact, the performance gap between “rotation” and “rotation+cls” shown in the controlled experiments in Sec 4.1 can serve as a surrogate metric for the MI as a variational estimate.

Due to the limit of space, we put the detailed derivation in a following comment that you may read if interested. Please let us know if there is more to clarify.

---

Q3. In experiments such as figure 1, what is the training loss/method considered?

A3. For a fair comparison, we follow the standard RotNet training setting and adopt CE loss for predicting discrete variables. We adopt the same training hyperparameters for all settings. More details are described in Appendix B.1. We will make it more clear in revision.

---

Q4. Theorem 1 puts forward an argument which seems key to the success or not of equivariant SSL (when used without an invariant objective), "How much information about C do you need to know to predict A". Bringing this point (and class relevance in general) forward in section 3.1 would make the motivation from figure 1 clearer.

A4. Thank you for your suggestion. During our writing, we felt that although important, it was hard to bring out the “how C helps to predict A” perspective earlier in Sec 3.1 without explaining the explaining-away effect (that appears later in Sec 4.1). Though it may sound natural after understanding the explaining-away effect, before that, people might wonder, “why do you care about how C helps to predict A” (since this retrospective understanding is also new). Bearing this in mind, we did not want to introduce puzzles here. We will try to re-organize it for a better balance. Thanks again for sharing your thoughts!

---

Thank you again for your insightful questions, which helps make this work more complete. Please do not hesitate to let us know if you have additional concerns.