Self-supervised Transformation Learning for Equivariant Representations

Thank you for taking the time to thoroughly review our manuscript.

In response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.

We would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.

Sincere thanks in advance for your time and efforts.

---

[W1] Comparison to SEN and SIE

In response to the reviewer's comments on the absence of experimental comparisons with existing equivariant self-supervised learning (SSL) methods, we have conducted comparisons with the SIE[1] and SEN[2] methods.

As in the SIE paper, our implementation of the SEN method adopts the InfoNCE loss instead of the triplet loss to minimize the need for extensive hyperparameter tuning. Additionally, SIE employs a strategy where the representation dimension is divided to separately address invariant and equivariant representations. For consistency, all methods, including SEN and SIE, were trained using SimCLR as the base model.

Table A1 in the rebuttal PDF presents the results, demonstrating that our method performs competitively against both SEN and SIE, supporting the robustness and effectiveness of our approach.

[W2] Various Seeds

We appreciate the reviewer’s feedback and have addressed the concern regarding the robustness of our results. In response, we conducted additional experiments using different random seeds for pretraining to ensure that the observed improvements were statistically significant. The results of these additional experiments are presented in Table A1 in the rebuttal PDF, which illustrates that our method, STL, consistently outperforms baseline approaches across various downstream tasks, achieving improvements ranging from an average of 3% to nearly 10%. These results demonstrate that STL significantly enhances performance compared to existing methods, confirming the robustness and generalizability of our approach.

[W3] STL with AugMix

We have indeed conducted experiments applying AugMix to SimCLR, and the results are documented in Table A1 in the rebuttal PDF. Our findings indicate that while incorporating AugMix with SimCLR yields a performance improvement compared to using standard augmentations, the model using STL with AugMix demonstrates superior performance. This suggests that our STL approach effectively leverages the AugMix augmentations to enhance representation learning beyond the capabilities of SimCLR with AugMix.

[W4] Detailed Comments

Thank you for bringing the typo on line 40 to our attention. We will correct the error in the final manuscript. We appreciate your meticulous review.

[Q] Complementary Relation

Table 5 in the main manuscript illustrates the performance of three models: the Only Equivariance model(employing $\\mathcal{L}\_\\text{inv}$ and $\\mathcal{L}\_\\text{equi}$), the Only Transformation model (employing $\\mathcal{L}\_\\text{inv}$ and $\\mathcal{L}\_\\text{trans}$), and the STL model (incorporating $\\mathcal{L}\_\\text{inv}$, $\\mathcal{L}\_\\text{equi}$, and $\\mathcal{L}\_\\text{trans}$). The results include the average accuracy in downstream classification tasks, and the regression and classification outcomes for transformation prediction tasks.

The Only Equivariance model significantly improves downstream task performance over SimCLR, but its transformation prediction capabilities are limited. Conversely, the Only Transformation model excels in transformation prediction compared to SimCLR but shows limited improvement in downstream tasks. In contrast, the STL model, which integrates both $\\mathcal{L}\_\\text{equi}$ and $\\mathcal{L}\_\\text{trans}$, demonstrates enhanced performance in both downstream tasks and transformation prediction, empirically validating their complementary nature.

The mechanism of this complementarity lies in the ability of the transformation representations, learned through $\\mathcal{L}\_\\text{trans}$, to facilitate the effective learning of equivariant transformations in the representation space. Specifically, the transformation representations allow $\\mathcal{L}\_\\text{equi}$ to ensure that transformations in image space correspond accurately to transformations in representation space. This correspondence is demonstrated by the improved performance metrics, such as MRR, H@k, and PRE, for STL's equivariant transformations compared to the Only Equivariance model, as evidenced by the results in Table A2 in the rebuttal PDF.

---

References

1. Garrido, Quentin, Laurent Najman, and Yann Lecun. "Self-supervised learning of split invariant equivariant representations.", ICML, 2023. 2, Park, Jung Yeon, et al. "Learning symmetric embeddings for equivariant world models.", ICML, 2022.

Thank you for taking the time to thoroughly review our manuscript.

In response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.

We would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.

Sincere thanks in advance for your time and efforts.

---

[W1-1] Comparison to SIE and SEN

In response to the reviewer's comments on the lack of comparisons with existing equivariant SSL methods, we compared our approach with SIE [1] and SEN [2]. Following the SIE paper, we used InfoNCE loss for SEN to reduce hyperparameter tuning. SIE splits representation dimensions for invariant and equivariant features. All methods, including SEN and SIE, used SimCLR as the base model. Table A1 in the rebuttal PDF shows our method is competitive with SEN and SIE, demonstrating its robustness and effectiveness.

[W1-2] Evaluation on 3DIEBench

The 3DIEBench dataset, while valuable in equivariance learning by pre-applying transformation with transformation label, poses a challenge in evaluating with STL due to its static nature. As transformations in 3DIEBench are pre-defined and fixed, we cannot re-apply the same transformation to different samples during training as STL requires dynamic application of transformation during training to learn the interdependency of each transformation. To utilize 3DIEBench effectively, we would not only need full access to the simulator used for generating the 3DIEBench dataset but also real-time generation to apply the pair-wise transformation during training.

[W2] Related Work Recommendation

Thank you for highlighting the missing references. We will incorporate these references in the final version.

[W3] Computational Costs

To empirically evaluate computational costs, we conducted an experiment using ResNet-50 with a batch size of 256. We utilize a single NVIDIA 3090 GPU, measuring the average training time per iteration over 1000 iterations following a 1000-iteration warm-up phase. As presented in Table A6 in the rebuttal PDF, the additional computational cost due to our network design is approximately 1.1 times that of SimCLR. In comparison to EquiMod, the computational cost remains nearly equivalent.

[W4, Q4] STL Extension on Various Base Models

STL extension on BYOL

In our STL extension to BYOL, we utilize the dissimilarity loss function intrinsic to BYOL to define the invariant, equivariant, and transformation losses. The dissimilarity loss in BYOL is: $\\mathcal{L}\_\\text{BYOL} (y, y^+;g, q, \theta, \xi) = \|\overline{q}\_\theta (g_\theta (y)) - \overline{g}_\xi (y^+) \|_2^2$

STL extension on SimSiam

For the SimSiam model, the dissimilarity loss is defined as follows: $\\mathcal{L}\_\\text{SimSiam} (y, y^+; g, h) = \frac{1}{2} \\mathcal{D}(h(g(y)), \\text{stopgrad}(g(y^+))) + \frac{1}{2} \\mathcal{D}(h(g(y^+)), \\text{stopgrad}(g(y)))$ where $\\mathcal{D}(a, b) = -\frac{a}{\\|a\\|\_2} \\cdot \frac{b}{\\|b\\|\_2}$

STL extension on BarlowTwins

In the case of BarlowTwins, the dissimilarity loss is given by: $\\mathcal{L}\_\\text{BarlowTwins} (Y=\\{y\_i\\}\_i, Y^+=\\{y^+\_i\\}\_i; g, \\lambda) = \\sum\_i (1 - \\mathcal{C}\_{ii})^2 + \\lambda \\sum\_i \\sum\_{j \\neq i} \\mathcal{C}\_{ij}^2$

where $\\mathcal{C}\_{ij} = \frac{\\sum\_b g(y\_b)\_i g(y^+\_b)\_j}{\\sqrt{\\sum\_b (g(y\_b)\_i)^2} \\sqrt{\\sum\_b (g(y^+\_b)\_i)^2}}$ and $g(y)\_i$ is the i-th dimension of $g(y)$.

[Q1] Dimensional collapse caused by multiple infoNCE losses

Throughout our experiments, we did not observe dimensional collapse, likely due to the design and implementation of STL. Each loss function is crafted to enhance discrimination between samples, reducing collapse risk. Additionally, using a non-linear projector and appropriate batch size helped maintain dimensionality in learned representations. To ensure proper batch size for transformations, we used the aligned batch configuration proposed in this paper, preserving batch complexity by matching the number of transformation types to sample types.

[Q2] Ablation study on auxiliary transformation backbone network

In Table A5 in the rebuttal PDF, we present empirical results that demonstrate the impact of varying the number of layers in the auxiliary transformation backbone on the average accuracy across different downstream classification tasks. Our findings indicate that change in the number of layers in result in only marginal variations in performance. This suggests the size of hypernetwork is not a primary factor influencing the accuracy in these tasks.

[Q3] Convergence Speed

In Figure A2 in the rebuttal PDF, we have provided the linear evaluation results across different $\\mathcal{L}\_\\text{equi}$ : $\\mathcal{L}\_\\text{trans}$ weight ratios while fixing $\\mathcal{L}\_\\text{inv}$ to 1. Our observations indicate that there is no significant difference in convergence speed attributable to the varying ratios.

[Q5] Transformation representation

Table A4 in the rebuttal PDF shows the results of downstream classification and transformation prediction tasks on STL, based on different weights of equivariant and transformation losses. The best performance for both tasks is observed with weight ratios of 1:0.2 or 1:0.5. When the transformation loss ratio is reduced to 1:0.1, performance in the transformation prediction task decreases, indicating insufficient learning of transformation representations. Increasing the ratio to 1:1 prioritizes transformation learning but disrupts the balance with other components, leading to performance drops in both tasks. At a higher ratio of 1:2, transformation prediction improves significantly, but classification performance declines sharply. While a higher ratio benefits transformation learning, maintaining a lower ratio enhances the generalizability of image representations.

Thank you for taking the time to thoroughly review our manuscript. In response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments. We would greatly appreciate it if you could review our responses to your comments and the submitted PDF. Sincere thanks in advance for your time and efforts.

---

[W1] Related Work Recommendation

As mentioned in the review, the suggested related work [1] is similar to STL in that [1], too, utilizes the difference in the pair of images, and we will update the reference accordingly. The main difference between [1] and STL lies in utilizing the transformation group. As [1] utilizes the transformation group information for selecting the appropriate loss term utilized for training. Acknowledging this limitation [1] suggested a way to bypass this limitation, however, as STL does not require the transformation group in addition to the transformation label, STL could be considered as a more general methodology for learning representations with pairs of images.

[W2] Transformation Equivariance

You are correct in noting that our metric is designed to measure the proximity of the transformed representation to the target representation relative to the original. The intention was to assess how well the equivariant transformation aligns with its image space counterpart compared to an identity transformation. However, we recognize that comparing solely to the identity transformation does not fully capture the nuances of equivariance across a variety of transformations. Therefore, we employed suggested metrics (MRR, H@k, and PRE) using a pretrained STL-10 model to extend our analysis to include metrics that evaluate the relative alignment of transformations in the image space. We tested on STL-10 test data subjected to transformations like individual transformations such as cropping and color jitter, as well as the standard combination of transformations used during training. Our results, detailed in Table A2 in the rebuttal PDF, show that our method surpasses existing techniques in most metrics, except for crop H@5 and PRE. This improvement suggests that the equivariant transformations learned by our approach more accurately reflect actual transformations in the image space compared to prior methods.

[W3] In-domain Evaluation on STL-10 and ImageNet100

To address the concern regarding the in-domain performance evaluation, we have included the results of the linear evaluation conducted on the pretraining dataset in Table A3 in the rebuttal PDF. Our findings demonstrate that, except for the SEN method, which focuses solely on equivariant learning, all other approaches, including STL, effectively maintain the performance level of the base invariant learning model within the in-domain setting. This evidence underscores the robustness of our approach in preserving performance both in-domain and out-of-domain.

[Q1] Interdependencies between Transformations

While it is true that individual augmentations can be applied independently in the image space, STL focuses on the interdependencies between transformations in the representation space, particularly concerning equivariant transformations. Due to the length limitation, please refer to the global response for further discussion.

[Q2] Leveraging Complex Augmentation

From "[...] transformation labels limits the performance gain [...]", we highlight that while complex augmentation like AugMix significantly enhance performance (shown in Table 1 in the main manuscript denoted as STL with AugMix), prior equivariance learning methods cannot leverage such complex augmentations due to inaccessbility of the corresonding transformation labels thereby bounding their performance gain.

[Q3] InfoNCE with Batch Information

Equation 2 was simplified to focus on the core concept. However, as we acknowledge that the InfoNCE loss requires consideration of the full batch, we will update Equation 10 to incorporate batch interactions as follows:

$$\mathcal{L}_\text{InfoNCE} (y, y^+; g, \tau, \{y_i\}_i) = -\log \frac{\exp\left(\text{sim}\left(g(y), g(y^+)\right)/\tau\right)}{\sum{y_i \neq y} \exp\left(\text{sim}\left(g(y), g(y_i)\right)/\tau\right)}$$

This reflects the necessary batch-level computation. Similarly, Equations 11-13 should be conditioned on the batch samples, aligning the formulation with the InfoNCE criterion’s requirements which will updated in the final version accordingly.

[Q4] The order of the transformations

In our implementation for AugSelf and Equimod, the order of transformations, including Color Jitter, is fixed rather than randomized. As mentioned in your review, this consistency must hold to prevent any association of different transformations with the same labels, thereby supporting effective learning and representation alignment.

[Q5] Analysis on individual transformations

We have conducted a detailed analysis of individual transformations to examine their equivariance properties in Table 5 in the main manuscript. Furthermore, we analyzed the similarity between equivariant transformations and their corresponding transformations using metrics such as MRR, Hit@k, and PRE. For this analysis, we employed an STL-10 pre-trained model, transforming the STL-10 test data using crop, color jitter, and combinations of augmentations. Table A2 in the rebuttal PDF shows that our method surpasses existing approaches in all metrics except for H@5 and PRE in the crop transformation. This superiority demonstrates that equivariant transformations learned through STL effectively capture and reflect the actual transformations applied.

---

References

1. Shakerinava et al., "Structuring representations using group invariants.", NIPS 2022.
2. Garridov et al., "Self-supervised learning of split invariant equivariant representations.", ICML 2023.

Thank you for taking the time to thoroughly review our manuscript.

In response to your detailed feedback, we have gone to great lengths to address and accommodate every single one of your comments.

We would greatly appreciate it if you could review our responses to your comments and the submitted rebuttal PDF.

Sincere thanks in advance for your time and efforts.

---

[W1] In-domain Evaluation on STL-10 and ImageNet100

To address the concern regarding the in-domain performance evaluation, we have included the results of the linear evaluation conducted on the pretraining dataset in Table A3 in the rebuttal PDF. Our findings demonstrate that, except for the SEN method which focuses solely on equivariant learning, all other approaches including the proposed method (STL) effectively maintain the performance level of the base invariant learning model within the in-domain setting. This evidence underscores the robustness of our approach in preserving performance both in-domain and out-of-domain.

[W2] Comparison to Equivariant Contrastive Learning (E-SSL)

From the suggested baselines, E-SSL[1] and Prelax[2], we conducted experiments on E-SSL as the code is publicly available while Prelax does not. For the comparison with E-SSL, we re-implemented it by applying crop and color transformations to the original image and predicting their parameters. The results including E-SSL can be seen in Table A1 in the rebuttal PDF.

[W3] Intra-relationship between Transformations

Figure 1 (c), in the main manuscript, demonstrates the inter-relationship between different transformation types. To further explore the intra-relationship within transformation representations, we applied various augmentations, specifically focusing on different aspects of cropping and color adjustments. For cropping, we considered variations in the box's center position and scale. For color transformations, we varied parameters such as brightness, contrast, saturation, and hue.

We employed UMAP visualizations to represent these transformations in the transformation representation space, as shown in Figure A3 in the rebuttal PDF. The results reveal that in the space, crops align according to the movement of the center position of the box along the x-axis, while color transformations align according to the degree of parameter adjustment. This arrangement suggests that the representations are sensitive to the specific parameters of the transformations.

We believe these findings support the view that transformation representations are organized in a manner that reflects their inherent properties, thereby capturing both inter- and intra-relationships effectively. This structured representation allows for a more nuanced understanding of transformation effects, supporting our hypothesis on transformation sensitivity and contributing to the broader discourse on representation learning.

---

References

1. Dangovski, Rumen, et al. "Equivariant contrastive learning.", ICLR, 2022.
2. Wang, Yifei, et al. "Residual relaxation for multi-view representation learning.", NIPS, 2021.