Self-Supervised Detection of Perfect and Partial Input-Dependent Symmetries

(4) Substituting the equivalence class... In the Appendix I read...

+-> The symmetry level error depends on various factors such as the quality of the embeddings in $L_{inv}$, or the extent of class imbalance within the data. For example, if a class is significantly underrepresented in the dataset – say, with only 20 samples – the predictions for the symmetry level of this class are likely to be less reliable. We do not think that it is possible to give a bound for this error, as it depends on several difficult-to-quantify factors, such as class imbalance in unsupervised settings or the quality of the embeddings.

(5) What do the up-down arrows...

+-> The up-and-down arrows aim to represent supervised learning between the prediction and the pseudo-label. We included the suggested changes in Figure 3 for clearer understanding. Please refer to the new manuscript.

(6) About results in Table 2...

+-> The effectiveness of symmetry standardization becomes more pronounced with the increased presence of rotational symmetries in the dataset. Non-equivariant models face significant challenges when dealing with data exhibiting extensive symmetries. Symmetry standardization, by collapsing elements into their centers of symmetry, effectively simplifies the modeling task for these non-equivariant models. Hence, the benefits of symmetry standardization are most apparent in scenarios with abundant symmetries. We included in the revised manuscript an extended discussion and comparison with the invariance achieved through IE-AEs.

---

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

(3.1) The estimation of the symmetry…

+-> For multimodal distributions there is indeed no center of symmetry, but several, and the reasoning in Appendix B.1 seems to suggest that minimizing the $L_2$ objective would likely lead to capturing the median point of the multi-modal distribution as a canonical representation. This could, in principle, be enough to generate pseudolabels to estimate several sub-centers of symmetry with a well designed pseudolabel estimation method. Although we did not get to research the multi-modal case, we believe that with the recently added extension of our method to unimodal, symmetric distributions and cyclic groups, our work sets a solid and flexible theoretical framework for detecting many types of symmetry distributions, which in principle could include multi-modal ones.

Regarding the experiment about non-overlapping rotation intervals per-class, note that the unsupervised nature of the problem makes the experiment that you propose precisely the same as the ones presented in our paper. To understand this, let’s say that instead of rotating the MNIST class 3 in [-20º, 20º] and the class 4 in [-40º,40º], we rotated the class 3 in [80º, 120º] instead. Because the data is unsupervised, the network is not aware of what a ‘straight’ digit 3 is. However, our method realizes that digits of such a form have rotational symmetries around some center of symmetry, which in this case is the ‘straight’ digit 3 rotated in 100º. Therefore, even though we have rotated the class 3 in [80º, 120º] instead of in [-20º, 20º], our method will still detect that the level of symmetry of the input is described by a [-20º, 20º] whose center of symmetry is on the 100º orientation. In other words, different non-overlapping per-class intervals will be transformed into overlapping ones due to the ability of our method to find centers of symmetry i.e. to establish a natural reference point in 0º. This means that, for instance, the MNISTMultiple experiment is equivalent (or even exactly the same) as the one you propose. We hope that this explanation answers the question.

(3.2) As far as I understand...

+-> Indeed, the rotations predicted are with respect to a canonical representation, but we believe it is necessary to establish such a reference point when detecting levels of symmetry. This task inherently involves quantifying how much an input deviates from a state of non-symmetry, which we define as the identity element of the group (e.g. a 0º rotation in SO(2)) or, more generally, a meaningful canonical representation. Establishing such a reference point is crucial, even in scenarios with multimodal distributions, in which the natural choice for canonical representations would be the center of symmetry of each sub-mode within the multimodal distribution.

(3.3) How limiting is...

+-> In practice, equivalence classes could be approximated by using large enough discretizations of SO(2), but this approach would be computationally unfeasible. This approach also has the disadvantage that, under some distance measures, the computed class would be degenerate in many cases. For instance, if in this hypothetical computation of equivalence classes, we decide to compare elements of the dataset pixel-by-pixel in order to discern if they are in the same class, we would need rotated versions of each input to be exactly the same at each pixel value. We could of course define an acceptable threshold beyond which we include an element in an equivalence class, but this still would not solve the need for lots of compute for a large-enough discretization of SO(2) in large datasets and/or images. In contrast, if instead of comparing the images pixel-by-pixel, we consider comparing the cosine similarity of the embeddings in the latent space, it is more likely that our set will capture the symmetries of the data without running into degenerate sets. In summary, a purely equivalence-class based method could be approached, but we believe that using the G-invariant latent space to estimate the equivalence classes is much more elegant and robust. Rather than limiting, our assumption makes our model more flexible in realistic scenarios, as bypassing the equivalence class calculation makes the method considerably cheaper, and its consideration is grounded in a very realistic assumption. The main limitation about this shift is that if the assumption turns out not to be true in a particular case, then the set $N(x)$ set will not be representative of the symmetries of $x$. However, it is incredibly difficult to come up with scenarios, both realistic and synthetic, in which similar inputs don't share symmetries.

We thank the reviewer for their appreciation of our paper, careful reading and constructive feedback towards improving our manuscript. We address the reviewer’s points below (numbered in order) and have updated our manuscript accordingly:

(1) Although the spirit of this work... In a similar vein... The work lacks an ablation study...

+-> We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We have been on a time crunch to extend our method to other distributions and groups, but we believe that our method can be extendable to other groups such as O(2) and to 3D rotations, and we are working on having these experiments on more realistic datasets for the camera-ready version. We also thank the reviewer for suggesting to test our model on data that does not strictly follow a uniform distribution, which resulted in new results as reflected on the updated manuscript. Lastly, the use of the NT-Xent loss has been removed from our method, as the improvements have proved to be marginal or non-existent after further hyperparameter optimization of the rest of the parts of the method. Therefore, no ablation study on this loss and the consistency loss is needed anymore, as both are no longer part of the method. Regarding CIFAR10, While the method promises to be applicable to other datasets such as the ones tested in the IE-AE work, in datasets with high intra-class variability e.g. CIFAR10, objects belonging to the same class may not share a G-invariant representation that connects them through a group action. In such cases, the standard IE-AE is unable to capture this relationship, and consequently, our method is also limited, as it relies on IE-AEs. We have included this discussion in the limitations section of our method (Section 6).

(2) There are some unanswered questions… In the context of IE-AEs, the treatment of inherently symmetrical data can be predicted through the theory underlying these models. An IE-AE determines group transformations by using the function $\psi$ to rotate the G-invariant canonical representation back to the original input's orientation. For an input that possesses inherent full rotational invariance, any rotation applied to its canonical representation will align with the original input's orientation, therefore any prediction by $\psi$ is valid as a solution. This is because the input itself is invariant under all rotations in the group G. Thus, the predicted transformation by $\psi$ in such cases can be the identity (0º) or any rotation at all, as the input's inherent symmetry makes every rotated version identical to the original. This prevents an IE-AE based method to discern whether a shape is inherently invariant to all rotations.

Note that when asking 'What is the level of rotational symmetry of this inherently invariant input in the dataset?' there are multiple valid answers as per the previous reasoning: the symmetry level could be described as zero, or encompassing the full SO(2) group, or actually any other distribution within SO(2), as any of the choices maintain the given input invariant in the dataset. In the case of our IE-AE, the $L_2$ term will push $\psi$ towards a degenerate symmetry distribution in 0º for this kind of inputs.

(3) -> Understanding the functioning of $L_2$ and how it leads to centers of symmetry in the canonical representation is important to address the raised concerns that follow. At first sight, minimizing $L_2$ in combination with the IE-AE loss seems to only ensure that the canonical representation of an equivalence class is a member of that class, as opposed to the standard IE-AE approach where the canonical representation is chosen arbitrarily. However, it can be proved that the element of the class that minimizes $L_2$ is in fact the center of symmetry of the class. This derivation is included in the updated manuscript (see Appendix B.1). Now let's connect this reasoning to your questions.

6 In section 3.2, why does minimizing...

+-> This is an excellent question. It can be somewhat tricky to understand why minimizing our proposed objective function $L_2 = d(\psi(x), e)$ results in fulfilling the constraint $\psi(c_{[x]})=e$. At first sight, minimizing $L_2$ in combination with the IE-AE loss seems to only ensure that the canonical representation of an equivalence class is a member of that class, as opposed to the standard IE-AE approach where the canonical representation is chosen arbitrarily. However, it can be proved that the element of the class that minimizes $L_2$ is indeed the center of symmetry. This derivation is included in the updated manuscript (see Appendix B.1). As the new derivation shows, the objective $L_2$ is able to create the right conditions for symmetry level prediction, for any unimodal, symmetric distribution. Our method is not limited to this class of distributions only, as the same minimization objective can also lead to predict cyclic symmetry levels (as presented in the revised manuscript) and possibly other groups and distributions. We are working on having these experiments for the camera-ready version.

---

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

We thank the reviewer for their careful reading and constructive feedback towards improving our manuscript. We address the reviewer’s points below (numbered in order) and have updated our manuscript accordingly:

1 While the proposed method seems general enough...

+-> We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We hope that this update shows that our method is general enough to be compared with other methods such as the standard IE-AE. We recognise that we have been on a time crunch to extend our method to other distributions and groups, but we believe that our method can be further extendable to other groups such as O(2) and to 3D rotations, and we are working on having these experiments for the camera-ready version.

2 It is not clear whether or to what extend different levels...

+-> The comparison between cars and planes intended to serve as an intuitive example of two objects that can appear with different symmetry levels in the same dataset, but we indeed did not quantify this difference in CIFAR10. The purpose of this is to offer an intuitive example of two objects that would usually appear with different symmetries: cars, showing very little rotational symmetries and planes, which when photographed from below, show full rotational symmetries.

3 While experiments...

+-> In Section 5, we presented some of the practical applications of detecting levels of symmetry: building out-of-distribution symmetry classifiers and using symmetry standardization as a pre-processing tool to improve the generalization and performance of non-equivariant models. The benefits of learning the levels of symmetry were further motivated by the works presented in the Introduction section.

4 Since the proposed method builds on invariant-equivariant autoencoder...

+-> Comparison of the IE-AE with our method is included now in the updated manuscript for the symmetry standardization experiments (Section 5). Note that a comparison with the IE-AE for the out-of-distribution symmetry experiment can not be performed, as this type of practical application can only be obtained by jointly using the $\Theta$ network and the modified IE-AE. In effect, a regular IE-AE can not be used to build out-of-distribution symmetry detectors, as it lacks the ability to predict an input-dependent symmetry distribution to compare each input to. Lastly, the figures of the symmetry level predictions in Section 5 have been modified for improved readability, as suggested.

5 Suppose a symmetry is outside the symmetry boundary...

+-> As mentioned in our introduction, prior research suggests that full equivariance can overly constrain models, leading to reduced performance when the symmetry group defined for G-equivariance does not align with the actual symmetries present in the data. However, models exhibiting full equivariance demonstrate enhanced robustness when encountering test data with symmetry distributions that differ from the training data (e.g. an SO(2)-equivariant network trained on MNIST will have higher accuracy in RotMNIST during inference than a regular CNN trained on the same dataset). Conversely, these fully equivariant models may underperform compared to partially equivariant models, when exposed to test data that falls within the symmetry constraints of the training data (which is usually the case). In such cases, partially equivariant models are desired. This presents a trade-off: generalization to out-of-distribution symmetries versus optimized performance for in-distribution symmetries. Interestingly, combining our out-of-distribution symmetry detectors with both fully and partially equivariant models could be a promising approach. This might enable a model to adaptively balance between full and partial equivariance, leveraging the benefits of both for improved generalization and performance. We hope that this paragraph answers your question, and we thank you for raising an interesting point that poses a valuable addition to our manuscript and future work.

We thank the reviewer for their careful reading and raised concerns. We address the reviewer’s points below (numbered in order) and have updated our manuscript accordingly:

1 Method section writing could be simplified...

+-> We recognise that it can be challenging to follow the definitions without a running example. Some of the most crucial terms defined in the method section now show a running example, in order to make the presented results easier to understand (please refer to the updated method section in the manuscript, new changes in the manuscript are coloured in red).

2 The novelty in the method is lacking...

+-> The method builds on top of the IE-AE and modifies it to provide it with new, desirable properties that are necessary to learn the level of symmetry of each input, as stated in Proposition 3.1. Partial G-CNNs layers are not used in any part of the method, but rather motivate both the need for determining the level of symmetry, as well as the approach of modelling the level of rotational symmetry as a single-parameter estimation of uniform rotations centered at the identity. This nuance is now made clearer in the updated manuscript (Section 2.3). Similarly, the use of contrastive learning is optional, does not contribute towards the novelty of the proposed method, and has been removed in the revised manuscript, as we found the improvements to be marginal after further hyperparameter optimization of all the proposed components.

3 Motivation for applying contrastive learning is unclear... What is the benefit of applying contrastive learning?

+-> The motivation behind contrastive learning is that minimizing a contrastive objective gives greater separation in the latent space, which is useful to create higher quality neighbors and improve the accuracy of the proposed method. However, the application of contrastive learning is not necessary for the method to be able to calculate the level of symmetries. Importantly, the use of the NT-Xent loss has been removed from our method, as pointed out in the previous response.

4 First and third rows in Fig.4 are same...

+-> First and third rows are indeed inputs, while the rows underneath represent each input’s reorientation into their canonical representation. This figure is now modified to clearly define and name each row.

---

We appreciate the reviewer’s feedback and have made comprehensive revisions to our manuscript. We believe these updates significantly enhance the clarity and understanding of our method, and hope that the reviewer finds the revised sections reflective of these improvements and more aligned with the objectives of our work. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

5 The way of calculating nearest neighbors... How are nearest-neighbors…

+-> The methodology for calculating nearest neighbors is indeed crucial for estimating the symmetries in the equivalence classes and is briefly mentioned in the manuscript. The explanation in the manuscript states: ``Let $N_{k,d}{=}N_k: X \rightarrow P(X)$ be a function that maps each input $x$ to the set $N_{k,d}(x)\subset X$ of $k$-neighbors around $x$ in the $G$-invariant latent space $Z_\mathrm{inv}$ as measured by some distance metric $d$." To put this idea in more detail: the set $N_k(x)$ consists of the $k$ elements $y$ of the dataset whose G-invariant embeddings $\eta(y)$ are nearest to the G-invariant embedding of the input $\eta(x)$, based on the chosen distance metric d. For instance, to compute $N_2(x)$, we first calculate the distance between every $\eta(y)$ in $\eta(X)$ and the embedding $\eta(x)$. Then, we select the two elements $y\in X$ that have the smallest distances in $\eta(X)$ to $\eta(x)$. We hope this answers the question, and have updated the manuscript to include a more explicit explanation of this computation (Section 3.4).

6 Generating the pseudo-labels: Why is the estimator...

+-> The decision to use the method of moments estimator for generating pseudo-labels is motivated by its robustness to outliers, compared to other potential estimators. Let's consider the alternative of a maximum likelihood estimator (MLE) for the distribution of rotation angles. The MLE is the maximum angle observed in the nearest neighbors set $N_k(x)$. However, this approach is particularly sensitive to outliers. For example, if any element in $N_k(x)$ has an erroneously predicted high rotation angle, the pseudo-label for $x$ would be disproportionately influenced by this outlier. In contrast, the method of moments estimator in the absolute-valued distribution, which we employ, calculates the pseudo-label as two times the mean of all angles in $N_k(x)$. This averaging effect mitigates the impact of any occasional anomalous angles, as these outliers are diluted when calculating the mean. The previous reasoning is now included in the updated manuscript (see Appendix C).

7 Have you tried to demonstrate the method on more realistic data?

+-> While the method promises to be applicable to other datasets such as the ones tested in the IE-AE work, in datasets with high intra-class variability e.g. CIFAR10, objects belonging to the same class may not share a G-invariant representation that connects them through a group action. In such cases, the standard IE-AE is unable to capture this relationship, and consequently, our method is also limited, as it relies on IE-AEs. We have included this discussion in the limitations section of our method (Section 6).

---

We hope that these responses clarify your questions and concerns. Please let us know if you have any follow-up / additional questions.

Best regards,

The Authors

We thank the reviewer for their careful reading and constructive feedback towards improving our manuscript. We address the reviewer’s points below (numbered in order) and have updated our manuscript accordingly:

1 The assumption of pure rotation..

+->We updated our manuscript showing how our proposed framework can be applied to different distributions such as Gaussian or cyclic, and more generally, arbitrary (unimodal) symmetric distributions. We hope that this update shows that our method is general enough to be compared with other methods such as the standard IE-AE.

2 This limited setting is very...

+-> In our work, we followed experimental settings set by previous works such as the IE-AE, which used controlled symmetries in datasets like RotMNIST, Tetris Shape dataset with random SO(3) rotations, and Shapenet with random 90º rotations. Indeed, we recognize the importance of applying our method to more complex datasets such as the ones in this work, and the truth is that time constraints have limited the extent of our method in this initial phase, both regarding its extension to other distributions and experiments on more datasets. We believe that our method can be extendable to distributions in other groups such as O(2) and 3D rotations, and we are working on having these experiments for the camera-ready version.

3 There is not much of a discussion as to how you could drop...

+-> As mentioned in the first response, it is indeed not necessary to have a uniform distribution in the data. The main assumption of the method is the existence of centers of symmetry, which enables our model to detect, more generally, arbitrary unimodal, symmetric distributions. The assumption can be further relaxed to detect the symmetry level of other distributions such as cyclic distributions (as shown in the updated manuscript) and possibly, other distributions and groups.

4 The need for (and the quality of) the self-supervision and...

+-> The $\Theta$ network is designed for generalization to new datasets (meaning generalization to test datasets, if we are understanding you correclty) without requiring retraining or pre-training on these datasets. Regarding the reasoning about relying on pseudo-labels for predictions during testing: If we were to rely solely on pseudo-labels as predictions during the test phase, we would indeed have to re-run the pre-training phase on the test dataset. This approach contradicts the standard testing methodology, where the model should generalize to new data without training again. Additionally, using pseudo-labels as predictions would be problematic for smaller test datasets or individual input samples. For instance, if we only have a small set of 200 test samples with several classes in it, using these for neighbor computations would likely yield poor-quality neighbors. More critically, predicting the symmetry level of single inputs would be impractical, as the input would be its own neighbor, rendering the approach ineffective. In contrast, in our approach, once the $\Theta$ network is trained, it is capable of inferring the symmetry level of new, unseen inputs independently. The IE-AE's role is primarily during the training phase, where it aids in estimating the symmetry levels that are subsequently used to train the $\Theta$ network. After this training phase, the IE-AE becomes redundant and can be discarded. The trained $\Theta$ network is what we use for direct inference on new data, which is how we obtained the symmetry level prediction results for the experiments shown. In summary, the design of the $\Theta$ network is such that it eliminates the need for recalculating or relying on pseudo-labels during the testing phase, addressing in this way the concerns of efficiency and practicality. We updated the manuscript to include this nuance in our explanation (see Section 3. All changes in the revised manuscript are coloured in red).