Centroid- and Orientation-aware Feature Learning

We appreciate the time you've dedicated to reviewing our paper. Please find below our detailed response to your comments:

1. Improvement of our paper (W1)

• We apologise for the omission of this proof since translational equivariance in CNN is a well-known concept in the community, and the proof is similar to the Theorem 3.2. We have now added the proof of theorem 3.1 in the Appendix.

• Related to ‘training and inference performance’: We apologise for the confusion. It refers to the training and inference time. We have changed it in the paper.

• Related to L_m loss: We highlight that

Although image moments capture the centroid and orientation features with a high degree of accuracy, they can still have a degree of subtle inaccuracy. Therefore, we utilize the image moments as guiding principles during the initial stages of learning.

2. Problem statement and motivation of our paper (W2)

Thank you for the comments. We apologize if the motivation seems disconnected from the experimental validation. We have changed the first paragraph of the introduction to clarify our problem as follows:

‘The distinct 2D projections that arise from the rotation of a 3D object can be seen as unique categories of in-plane rotation. Each category corresponds to a specific orientation in the 3D space. This principle is widely applied in fields such as computer graphics, computer vision, and related areas where the understanding and manipulation of 3D objects in a 2D images are essential. In this context, robust learning of centroids and orientations of objects in images, along with other features, is important, especially when objects are captured with 3D shifts and rotations such as protein, and galaxy studies.’

Therefore, we demonstrate that the capability of the model to learn key features, including centroids and orientations in the experimental validation. Table 3 and 4 show that DAE and beta-VAE have poor performance in factorizing two datasets, XYRCS and dSprites. On the other hand, the disentanglement scores of the proposed model on these datasets demonstrate that the model effectively learn all features including centroids and orientations.

3. Optimal latent dimension for TARGET-VAE and IRL-INR (W3, Q2)

Target-VAE and IRL-INR conducted their experiments with large latent dimension, pursuing two objectives: 1) learning centroids and orientation, and 2) reconstructing the original data. Therefore, the two methods do not address the disentanglement scores in their paper, while they included the centroids and the orientation correlation. Therefore, the latent dimensions chosen in their respective papers are optimal for reconstructing the original data but may not effectively factorize all features. In contrast, our goal is not only to learn centroids and orientation but also to independently learn the other features while successfully reconstructing the original data. To show this, we chose a smaller latent dimension because modularity (or independence) and compactness are important in disentanglement properties. Figure 2, 4, 5, and 7 along with Table 3 and 4 demonstrate the capability of the model to factorize all the features while reconstructing the original data.

4. Objective of our paper (Q1)

We pursue disentanglement of all features. DAE already shows an ability to disentangle features. However, it fails factorizing all features when the data has a rotational feature. Therefore, we introduce a new architecture outlined in the paper with $L_{moment}$ to explicitly learn centroids and rotations, leading to fully factorized features.

5. Contributions of our paper (W1)

Our contribution is not limited on $L_{moment}$. We:

• Introduce a translational and rotational equivariant encoder.
• Propose an architecture that eliminates computationally expensive decoder, such as spatial or hypernetwork decoders.
• Drastically reduce computational costs while achieving superior disentanglement scores.

6. Further discussion

Despite detailed problem statement, clear motivation and challenges around robustly learning centroid and orientations, especially in real images, theoretical framework, and extensive experimental results (using appropriate datasets), we received a score of three. We would greatly appreciate more detailed feedback that led to this score on these specific areas. Understanding these details will help us improve and refine our paper. Your insights are invaluable, and we are eager to make the necessary updates based on your feedback.

Thank you for your response, and we appreciate that your views, and glad we added some clarity. Please find below our detailed response to your additional comments:

1. Comments on ‘Each category corresponds to in the 3D space." -- this is not a proper sentence, nor is it clear’:

Thank you for your feedback, and we apologize for the missing word. The original sentence was: ‘The distinct 2D projections that arise from the rotation of a 3D object can be seen as unique categories of in-plane rotation. Each category corresponds to a specific orientation in the 3D space.’

We have revised all the response. We truly appreciate your feedback.

2. Introduce a translational and rotational equivariant encoder. (Q1)

We would like to emphasize that the primary objective of DAE is to disentangle all features. However, as illustrated in Table 3 and 4, and Figure 9 and 10, DAE has poor performance in factorizing all features when the data has a rotational feature. Additionally, partial modifications to DAE may negatively impact its overall performance. This forms the motivation for the proposed work, CODAE, where we introduce two aspects, which we believe underpins the novelty: (i) a translational and rotational equivariant encoder, and (ii) a loss function based on image moments. We are in agreement that neither the translational and rotational equivariant encoder nor image moments alone can improve the disentanglement scores, for the reasons of:

• The translational and rotational equivariant encoder, designed following the Theorem 3.2, has the limitation in practice due to the impossibility of integration on S1. The computations are performed on a subset of S1, limiting rotational equivariance.
• While image moments capture the centroid and orientation features with a high degree of accuracy, they can still have a degree of subtle inaccuracy.

Therefore, we use the image moments as guiding principles during the initial stages of learning. Our evaluation results show that the combination of a translational and rotational equivariant encoder with image moments significantly improves disentanglement scores, particularly when the data involves rotational features. Figure 5 illustrates that the capability of the proposed architecture to learn semantic representations of objects when the objects of interest are positioned and oriented arbitrarily within the image.

We will add these as part of the modified version of the manuscript to improve the clarity.

3. Propose an architecture that eliminates computationally expensive decoder, such as spatial or hypernetwork decoders. (Q2)

While we understand the rationale for the suggestion, we politely would like to disagree with the reviewer’s comments mainly because different architectures rely on its own decoder design. More specifically:

• The contribution of Spatial-VAE lies in its spatial decoder.
• The contribution of Target-VAE is its specialized encoder, built on top of the spatial decoder from Spatial-VAE.
• The contribution of IRL-INR is its hypernetwork decoder.

As such, each model's distinctive feature is embedded in its decoder design. As such, it is almost impossible to rely on the same decoder unless we extensively modify their respective designs, distorting their purposes, which would be very unfair comparison. Given this, our method of comparison is fair retaining their design in the referenced papers [1, 2,3]. However, we used the same decoder for CODAE, DAE and $\beta$-VAE, which is valid and permissible.

4. Drastically reduce computational costs while achieving superior disentanglement scores. (Q3)

Continuing from the previous response, each model has its own encoder or decoder model (as intended than modified or distorted) to learn semantic representations of objects when the objects of interest are positioned and oriented arbitrarily within the image. Our proposal in the paper involves designing a translational and rotational equivariant encoder while avoiding spatial or hypernetwork decoders. We believe this approach provides a clear rationale for reducing computational costs.

[1] Explicitly disentangling image content from translation and rotation with spatial-vae, Neurips 2019

[2] Unsupervised object representation learning using translation and rotation group equivariant vae, Neurips 2022

[3] Rotation and translation invariant representation learning with implicit neural representations, ICML 2023

We appreciate the time you've dedicated to reviewing our paper. Please find below our detailed response to your comments:

1. Ablation study (W)

We conduct the ablation study, and the results indicate that the disentanglement scores increase with the inclusion of the $L_{moment}$. We will include the ablation study table in the paper and post it separately in the response soon.

2. Contributions of our paper (W)

While the concept of moment loss is widely recognized in the community, this paper introduces the first method to incorporate image moments into the learning mechanism - which is not common. In addition, our contribution is not limited on $L_{moment}$. We:

• Introduce a translational and rotational equivariant encoder.
• Propose an architecture that eliminates computationally expensive decoder, such as spatial or hypernetwork decoders.
• Drastically reduce computational costs while achieving superior disentanglement scores.

3. Additional experiments (W)

Thank you for the comments. We have conducted additional experiments and new results will be posted separately soon.

4. GF-Score (W)

Thank you for the comments. We have added the scores in the paper and will post it separately in the response soon.

5. Image moments for other factors such as shape and color

We do not compute moments for other factors such as shape and color. Based on disentanglement literature (beta-VAE and DAE), these features can be learned. However, the table 3 and 4 show that these models struggle to learn features like shape and color when objects are captured with orientations. Therefore, the primary object of this paper is to guide the network to learn centroids and orientations using image moments, leading to the proposed model to effectively learn other features as part of $z_{inv}$.

6. Further discussion

Despite detailed problem statement, clear motivation and challenges around robustly learning centroid and orientations, especially in real images, theoretical framework, and extensive experimental results (using appropriate datasets), we received a score of three. We would greatly appreciate more detailed feedback that led to this score on these specific areas. Understanding these details will help us improve and refine our paper. Your insights are invaluable, and we are eager to make the necessary updates based on your feedback.

Thank you for your comments and questions.

1. All the experiments are conducted on toy datasets such as MNIST digits which contain very simple 2D objects and almost uniform background region (W).

First of all, we would like to politely highlight that the statement “all the experiments are conducted on toy datasets such as MNIST digits which contain very simple 2D objects and almost uniform background region.” Undermined all the efforts and work presented in this paper. In contrary to the statement, the evaluation includes two real world datasets, such as EMPIAR-10029 (protein) and Galaxy-Zoo datasets, where objects and background are complicated, and hence do not have a uniform background. Figures 4 and 5 clearly show that our proposed model learns key features of complicated datasets including centroids and orientations. Secondly, aside from Table 1, the disentanglement scores in Table 3 and 4 are far better than those presented in the literature and those achieved by the baselines presented in the paper. As such, we find the dismissal not only disheartening but also expects an unrealistic accuracy on these tasks. Please see our detailed responses below:

• However, all the experiments are conducted on toy datasets such as MNIST digits which contain very simple 2D objects and almost uniform background region.

Please see the opening response above. Evaluations include rather complex EMPIAR and GalaxyZoo datasets. In particular the EMPIAR (10029) dataset includes 3D rotation.

• How is the performance of the proposed method on natural image datasets, such as CIFAR or ImageNet…..

The primary objective of the proposed work is to design a method that learns key features including centroid and orientation. Our selection of datasets is based on the literature and on existing work on disentanglement methods, but now includes additional 3D datasets as requested by the Reviewer 8vxB (3D datasets, such as 3D airplanes, 3D teapots, 3D face). As such, including CIFAR or ImageNet is beyond the scope of disentanglement studies.

2. Problem statement and motivation of our paper

We agree that we could have phrased the problem statement much more clearly, which, based on the comments, rephrased as follows: ‘The distinct 2D projections that arise from the rotation of a 3D object can be seen as unique categories of in-plane rotation. Each category corresponds to a specific orientation in the 3D space. This principle is widely applied in fields such as computer graphics, computer vision, and related areas where the understanding and manipulation of 3D objects in 2D images are essential. In this context, robust learning of centroids and orientations of objects in images, along with other features, is important, especially when objects are captured with 3D shifts and rotations such as protein, and galaxy studies.’

3. Further discussion

We feel that the score of three (3) is purely based on misunderstanding of the difficulty of the challenges around robust learning of the centroids and orientations (please see [1, 2, 3]), our evaluation process (such as missing the fact that we included real-world and complex datasets) or overlooking the comparison in Table 3 and 4. As such, we feel that the lower scores are not justified. We believe additional datasets, and more extensive evaluation, and our responses above clear these issues. We will be happy to receive more detailed feedback that led to this score on these specific areas.

[1] Explicitly disentangling image content from translation and rotation with spatial-vae, Neurips 2019

[2] Unsupervised object representation learning using translation and rotation group equivariant vae, Neurips 2022

[3] Rotation and translation invariant representation learning with implicit neural representations, ICML 2023