3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction

We thank reviewer f5v8 for constructive comments and suggestions.

[W1. Clarifying contributions]

Please see our general response above

[W2. Claims of structural guarantee of 3D rotational equivariance]

The statement "structurally guarantee 3D rotational equivariance" is indeed misleading. While the network can use components like spherical convolutions and Wigner-D matrix predictions to maintain equivariance, perfect 3D rotational equivariance cannot be guaranteed with 2D image inputs, as they inherently lack full 3D structural information. The network approximates 3D equivariance using the spherical mapper to handle 3D rotations effectively, but absolute equivariance is not achievable with 2D data. We will clarify that the network aims to approximate 3D rotational equivariance as closely as possible given these constraints.

[W3. symmetry, other losses, $w_l$]

Please see our response above

[W4. Detailed configurations of Table 3 ablation studies]

Table 3 in Section 5.5 demonstrates the effectiveness of our proposed Wigner-D matrix compared to other rotation representations in the spatial domain (Euler angles, Quaternion, axis-angle, rotation matrix). For all cases, we retained the backbone networks and SO(3) equivariant layers. The only modifications were the output prediction dimension size and the ground-truth rotation representation.

[W5-1. Clarification on the complexity of 3D orientation and translation]

We acknowledge that the original statement might be confusing. To clarify, we can represent the 3D translation and orientation:

• 3D translation: movement along the x, y, and z axes. (3 dims)
• 3D orientation: involves rotations around these axes, represented by Euler angles (3 dims), quaternions (4 dims), or rotation matrices (9 dims).

We will update our manuscript to improve clarity: 3D orientation is more complex due to rotational symmetries and the non-linear nature of rotations. Unlike translation, rotations present challenges such as gimbal lock and the need for continuous representation without singularities.

[W5-2. Overstatement of SO(3) equivariance]

The claim "SO(3)-equivariance is crucial for accurate pose estimation" might be overstated. While it can enhance performance and robustness, it is not the only way to achieve competitive results. As shown by the non-equivariant baselines in Tables 1 and 2, alternative approaches also deliver strong performance. We will revise the manuscript to reflect this nuanced perspective.

[W6. Attribution of images in figure 1]

We apologize for the oversight in attributing images. We will revise the figure to clearly credit all original sources.

[Q1. Evaluation of model with higher resolution grid and gradient ascent]

Table R2 shows the impact of changing the grid resolution $Q$ on performance. The higher resolution (18.87M) of the SO(3) grid improves the performance at a lower threshold at inference time.

Table R5 shows the evaluation results using gradient ascent on the pose distribution. While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.

[Q2. Combining high precision and distribution modeling]

Our method currently struggles to model object symmetries and uncertainty. However, Table R1 demonstrates that combining our approach with a distribution loss based on cross-entropy [48, 34] is effective for symmetric object modeling, and a classification-regression framework [38] can be an alternative solution for distribution modeling.

[Q3. Adjusting frequency levels and mitigating high-frequency instability]

Increasing the frequency level at the network's output is challenging due to the need to define the maximum frequency level of spherical harmonics at the model initialization. Adjusting the final output frequency can help mitigate performance drops at high frequencies caused by non-linearity. Using gating functions on Wigner-D coefficients in Equiformer (Liao and Smidt, ICLR 2023) could address instability issues.

[Q4. Clarification on spatial domain parametrization in spherical CNNs]

The term "inadequate" refers to the practical challenges and inefficiencies of using spatial domain parametrizations directly within spherical CNNs. These challenges arise because spherical CNNs naturally operate in the frequency domain, where rotations are represented and manipulated more efficiently using spherical harmonics and Wigner-D matrices.

Although conversion between the frequency and spatial domains is possible (by inverse Fourier transform), maintaining consistent parametrization in the frequency domain is more straightforward for the SO(3)-equivariance with spherical CNNs and proves to be more effective, as demonstrated in Table 3 of the main paper.

[L1. Clarifying the loss of spatial Information in spherical harmonics projection]

In Section F, the statement about significant errors refers to the loss of spatial information due to truncating the frequency level of spherical harmonics for efficiency. "3D data" refers to data points on the 2-sphere. These errors, not explicitly detailed in the experimental results, arise from truncating higher frequency components, leading to a loss of spatial detail. We will update the manuscript to clarify this point and provide a more precise explanation.

[L2. Clarifying computational costs in the limitation section]

In Section F, we note that while our method achieves the best inference time (Table A6), there is room for improvement in space complexity. The computational cost, due to the complexity of Wigner-D coefficients and convolutions in SO(3)-equivariant networks, remains high. These operations require handling high-dimensional representations and specialized mathematical processes. Therefore, reducing memory usage and computational load is a target for future enhancement.

Thank you for the response. I appreciate all the new results presented in the PDF, and the decision to revise the background/related work sections.

It is exciting to see the proposed MSE loss can be combined with the Cross-Entropy loss from I2S to achieve both high precision and accurate distributions.

Regarding R3, what exactly is the takeaway from this plot? That the model puts more weight on higher-frequencies early on in training? Is this something that could be addressed with a weighted MSE loss to boost performance in the very low data regime?

Regarding R4, it would be nice to understand the pitfalls of the MSE loss over Wigner-D coefficients. For instance, could the gradient of the MSE loss push the model away from the ground truth rotation in some cases? This doesn't need to be addressed here but it would be interesting to study in the future.

Regarding R5 and R2, it is interesting that there is a sweet spot in terms of how precise the model can be. Do you think this is due to the limited precision of the truncated Fourier series?

Given the proposed revisions and the new results, I updated my rating.

We thank reviewer HCET for constructive comments and suggestions.

[W1. Clarifying our contributions compared to I2S [34]]

Please see our general response above.

[W2-a. Conversion from Euler angles to Wigner-D during training (L245)]

The conversion from Euler angles to Wigner-D matrices involves a mathematical transformation using the $ZYZ$ sequence of rotations and the corresponding Wigner-D matrix elements. The model outputs are indeed the coefficients associated with the Wigner-D matrices. The loss function in Eq. (5) compares these predicted coefficients directly with the ground-truth coefficients, facilitating the direct regression of 3D rotations in the frequency domain.

Additionally, unlike Image2Sphere [34], we do not perform the inverse Fourier transformation (iFT) of the output of the networks during training. Instead, we train on the output Wigner-D coefficients in the frequency domain, and we do not apply the iFT even at test time.

We hope this explanation clarifies the process and resolves any confusion. Thank you again for your valuable feedback.

[W2-b. Computing the similarity with the SO(3) grid during inference (L263)]

The similarity calculation in L263 is different from an inverse SO(3) Fourier transform. It is a process to find candidate Euler angles from the predicted Wigner-D coefficients. The predefined SO(3) HEALPix grid includes a set of predefined Wigner-D coefficients corresponding 1:1 to Euler angles. We calculate the similarity between this grid and the predicted Wigner-D coefficients to obtain a discretized distribution. This probability distribution provides multiple hypotheses for the pose (i.e., Euler angles) predictions of the input image. We then use argmax or gradient ascent to determine the final predicted Euler angles at inference, as shown in the results of Table R5. This method is similar to the approaches used in [48, 34].

While an inverse SO(3) Fourier transform could convert the predicted Wigner-D coefficients to Euler angles, it has high time complexity, as it requires separate calculations for each frequency level. The SO(3) iFT method produces a single value, making distribution modeling difficult.

In contrast, our design choice of querying the predefined HEALPix SO(3) grid enables distribution modeling by calculating simple vector-matrix multiplication. This allows us effective symmetric object modeling by combining the existing cross-entropy loss [48, 34], as demonstrated in Table A1 and Figure A1.

[W3, L1. Evaluation on SYMSOL]

Please see our general response above.

[W4. Clarification on paragraph L113 "Rotation representation in frequency domain"]

The paragraph "Rotation representation in frequency domain" describes how 3D rotations are represented in the frequency domain using spherical harmonics and the Wigner-D matrix.

The Wigner-D rotation representation in L113-125 is not limited to a specific case of 3D rotations but can be converted from any 3D rotation representation, such as Euler angles, quaternions, and 3D rotation matrices. Our SO(3) equivariant network predicts the Wigner-D representation in the frequency domain instead of predicting rotations in the spatial domain (Euler angles, quaternions, etc.).

To address the confusion, we clarify that the $U$ in Equation (1) and the $D$ in Equation (2) are indeed the same. We will update the manuscript to use consistent notation in Equations (1) and (2). We apologize for any confusion caused by this inconsistency.

[Q1. Clarification of the spherical notation in L212]

In Section 4.3, $\mathcal{S}$ is a spherical harmonics function defined in the frequency domain. $N$ denotes the total number of spherical harmonics in the frequency domain. $p$ denotes the number of points on the sphere in the spatial domain. We will update the manuscript to clearly indicate that $\mathcal{S}$ is defined in the frequency domain and ensure the distinction between $N$ (number of spherical harmonics) and $p$ (number of points on the sphere) is clear.

[Q2. Typos]

We will revise our manuscript according to your findings of typos. Thank you for your feedback.

Thank you for your thorough comment and insightful feedback.

[Reply to W1]

We will ensure that the main text is updated to reflect the clarifications provided in the rebuttal, as multiple reviewers have raised concerns about the clarity of the method section.

---

[Reply to W2-a]

We apologize for any lack of clarity in our original submission regarding the "mathematical transformation”. Specifically, we map the Euler angles, which represent an element of $SO(3)$ , to a set of coefficients corresponding to the Wigner-D matrix elements. These coefficients are critical in capturing the rotational properties in the frequency domain.

Mapping from SO(3) to Wigner-D Coefficients: As you correctly pointed out, each Wigner-D matrix element provides a fixed map from $SO(3)$ to $\mathbb{C}$. Our method involves predicting these Wigner-D coefficients directly, bypassing the complexities and potential pitfalls of spatial domain parametrizations.

To achieve this, we utilize an equivariant network to predict the Wigner-D matrix coefficients in the frequency domain directly from the input image features. The prediction process does not explicitly involve an inverse $SO(3)$ Fourier transform of an impulse function centered on the ground truth rotation. Instead, our network learns to map the input features to the Wigner-D coefficients that represent the corresponding $SO(3)$ rotation, leveraging the equivariant properties of the network to ensure rotational consistency.

The output of our network is a vector of Wigner-D coefficients that directly encode the rotation in the frequency domain. These coefficients are then converted back to a rotation matrix or another suitable representation (e.g., Euler angles) as needed for evaluation or further processing.

Clarification of the GT Transformation: We would like to gently remind you that Appendix A (and B.2) contains the detailed conversion equations between the Euler angles and the Wigner-D matrix, including the small Wigner-d matrix. We will further revise the transformation process in our final manuscript, elaborating on how the conversion between Euler angles and Wigner-D matrices is handled in our approach.

---

[Reply to W2-b]

Thank you for your detailed feedback on the computational efficiency of our SO(3) grid generation and similarity computation for SO(3) distributions. We would like to clarify and justify our approach.

Clarification on the Inference Procedure and Similarity Calculation: The similarity calculation described in L263 is designed to identify candidate Euler angles from the predicted Wigner-D coefficients by comparing them against a predefined SO(3) HEALPix grid at inference.

You are correct that our approach involves precomputing the inverse SO(3) Fourier transforms to generate the set of predefined Wigner-D coefficients. This approach indeed requires significant space complexity, with a high time complexity of $O(B^6)$ naively to initialize the predefined SO(3) grid. Consequently, our inference method is actually slower and uses more significantly more memory than just computing the IFT of the predicted coefficients.

However, after the predefined SO(3) grid is initialized, both the training and testing phases benefit from this precomputation, leading to reduced computation times during these phases. Additionally, for repeated runs, the grid can be stored and reloaded as needed, further optimizing execution time.

We apologize for the inaccurate expression: "The SO(3) iFT method produces a single value, making distribution modeling difficult”. What we intended to convey was that the inverse SO(3) Fourier transform results in a specific mapping, not a single value.

We hope this explanation addresses your concerns and clarifies our design choices. Our approach aims to balance computational feasibility with the ability to accurately model complex pose distributions in SO(3) space.

---

[Reply to W3]

Thank you for acknowledging the addition of the SYMSOL experiments. As you noted, the proposed MSE loss and the previously used likelihood loss work complementarily. However, our model does not surpass the performance of I-PDF [48] on SYMSOL, and we agree that future research should focus on further improving symmetry modeling to address these challenges.

We thank reviewer HnJh for constructive comments and suggestions.

[W1.1, W1.2, Q1, Q2 clarifying the contributions]

Please see our general response above.

[W1.3, W4.4, Q3. Continuity of rotations, Sensitivity analysis of the SO(3) HEALPix discretization]

We carefully claim that our learning method focuses on continuous rotations. We directly learn the Wigner-D coefficients, which are derived from 3D rotations (Euler angles), without any discretization during the training phase. The use of the SO(3) HEALPix grid during inference serves two purposes:

1. To convert SO(3) rotations from the frequency domain to the spatial domain, and
2. To address pose ambiguity by providing multiple solutions.

As a result, we obtain a distribution with very sharp modality. By taking the argmax of this distribution, we achieve sufficient precision in 3D orientation estimation, specifically around 1.5 degrees.

Table R2 shows the effect of changing $Q$ on performance is. For the ModelNet10 benchmark, we achieve similar results in the common target metrics such as Acc@15 and Acc@30 even with a lower size of SO(3) discretization grid ($Q$=4.6K). With our model's choice of $Q$=2.36M, we obtain high scores even on low-threshold evaluation metrics like Acc@3. Comparative experiments at lower thresholds are provided in appendix Table A1.

Therefore, the continuity of rotations in our method ensures that we can predict an accurate, more precise pose. This is a significant advantage over other methods that suffer from precision loss due to discretization, as our approach maintains high accuracy across varying levels of discretization.

[W1.4 The claims of parametric model]

Our statement about the insufficient expressivity of parametric models (L82) refers to the potential lack of flexibility due to the dependency on predefined prior models, compared to non-parametric models. We acknowledge that this claim was not directly validated in our work. To provide a more accurate representation, we will revise our claim to reflect that.

[W2.1. Justification of the spherical mapper]

The spherical mapper maintains the geometric structure of the image when projecting onto the S2 sphere, as detailed in [34]. This method involves lifting the 2D image onto the sphere and converting spherical points using spherical harmonics. Table R6 shows that the spherical mapper outperforms simple Fourier transforms on 2D feature maps.

Using depth information from methods like DepthAnythingv2 for 3D lifting is a good idea and can enhance geometric accuracy. Additionally, centroid ray regression has been explored in research such as [70]. However, incorporating external depth modules increases computational costs and broadens our research scope, so we consider this for future work.

[W2.2 Non-linearites for equivariant layers]

The FFT-based approximate non-linearity [19] and equivariant non-linearity for tensor field networks [51,65] mentioned in the text provide non-linearity based on Fourier kernels, avoiding the need for FFT and iFFT steps. However, these methods are not included in our quantitative evaluation as their code is not publicly available. Testing these approaches could potentially improve performance or efficiency when combined with our Fourier-based SO(3) equivariant network.

[W2.3, W.4.3, Q4 Design choice of MSE]

Please see general response above.

[W.2.4, W4.2, W4.3, Q5. Evaluation on SYMSOL]

Please see general response above.

[W2.4 Discretised distribution on SO(3). (argmax vs. clustering)]

Table R5 shows the evaluation results using gradient ascent on the pose distribution. While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.

[W3.1 Clarification of Wigner-D representation, ]

We will clarify the Wigner-D matrix representation in the main paper. While detailed equations are provided in Sections A.2, A.2.2, and B.2 of the appendix, we will include a concise explanation in the main text.

Specifically, the Wigner-D representation is implemented in a flattened form at different frequency levels. For example, the matrix coefficients in a frequency level $l$ is represented as a flattened vector of size $(2l+1)*(2l+1)$. We use a maximum frequency level of 6, resulting in a vector of size 455 (i.e., $1∗1+3∗3+5∗5+...+13∗13=455$) for the Wigner-D coefficients. This will be clearly explained to directly connect the task to the representation in our final manuscript.

[W3.1 Notation of $f$ in eq (4) and (12)]

We will clarify the notation to differentiate the two $f$ functions in Equations (4) and (12). These functions are indeed different. Equation (4) represents the general form of a group equivariant network. Equation (12) describes the rotation of spherical harmonics using the rotated coefficients of the Wigner-D matrix. We will update the notation to clearly distinguish between these two functions and ensure the connection between the theory and its specific instantiation is clearly defined.

[W3-2. Derive the weights $w$]

Please see our response above.

[W4.1. Fair comparison of the backbones in Table 1]

Thank you for the suggestion. We include a comparison in the appendix (Table A1) that aligns the sizes of the backbone networks. Some networks, like Inception-v3, are not directly comparable, so we group representative methods for fair comparison based on similar backbones. These results show that our method outperforms existing methods, even on finer metrics like Acc@3 and Acc@5, demonstrating its effectiveness.

[W5. Typos and formatting errors]

We will conduct a thorough proofreading to correct typos, syntactical, and grammatical errors. Additionally, we will adjust the positioning of footnote 2 to ensure it appears at the correct location where it is cited in the text. Thank you for your helpful suggestions.

We thank reviewer iMp8 for constructive comments and suggestions.

[W1. Comparison of pose visualization]

We present a comparison of pose visualizations in Figure R2. This visualization method is the same to those used in Figures A2 and A3 in the appendix.

We compare to the I2S [34] baseline. The left side of Figure R2 shows results from ModelNet10-SO(3), while the right side presents on PASCAL3D+. The numbers next to "Err" above the input images represent the error in degrees between the model's predicted pose and the ground truth (GT) pose.

These results demonstrate that our model provides more accurate and precise pose estimations, even in cases where the I2S baseline fails. Additionally, on the PASCAL3D+ benchmark, which includes objects captured in real-world scenarios, our model consistently shows correct pose estimations, particularly in challenging situations where the I2S baseline struggles.

[Q1. Evaluation on in-the-wild datasets]

The PASCAL3D+ dataset is an in-the-wild dataset captured in typical real-world environments, which we evaluated in Table 2 and Figure R2. As far as we know, PASCAL3D+ is the primary in-the-wild dataset used for 3D orientation estimation. We have not found any other in-the-wild datasets that are commonly used for this task.

[Q2. Handling occlusion]

Yes, our method is capable of handling occlusions.

While our SO(3) equivariant estimator can offer some robustness to occlusions due to its rotational invariance and localized feature extraction, handling occlusions effectively often requires additional techniques and strategies. Our network can extract local features in a manner that is robust to 3D rotations, allowing it to identify visible parts of objects even when other parts are occluded. Additionally, the network's ability to recognize objects regardless of their orientation helps in identifying partially visible objects.

The results in Table R1 for the SYMSOL II dataset, which includes self-occlusion scenarios in symmetric solids, demonstrate that our model jointly trained with $\mathcal{L}_{\text{dist}}$ [48, 34] handle occlusions better than the I2S [34] baseline. Our model improves accuracy by directly predicting a single clear pose, unlike the baseline model.

Figure R1 shows the SYMSOL II visualization results, specifically in examples 4, 5, and 6. When provided with occluded marks of symmetric objects, our model effectively uses these cues to estimate a single pose.

[Q3. Transformer instead of convolution]

We choose a convolution-based design, because using transformers instead of convolutions results in a slight performance drop.

Table R3 presents the results when transformers are used as the backbone network instead of convolutions. We trained the model using a Vision Transformer (ViT) pre-trained backbone pre-trained by the geometric task of cross-view masked image modeling [A].

Although the ViT is heavier and requires longer training time (1.4x), its performance actually declines. This suggests that convolutional backbones may still be more effective for 3D orientation estimation tasks.

[A] CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion (Weinzaepfel et al., NeurIPS 2022)

First, thank you for taking the time to read and consider our rebuttal.

---

Reply to R1 regarding pose visualization comparison:

We would like to clarify that the visualization of pose distribution using the surface of the 2-sphere with the Mollweide projection [A] conveys the same information as visualizations that use bounding boxes or the $x$, $y$, $z$ axes. For example, the pose distributions shown on the spheres in Figures R2, A2, and A3 can be converted into single pose values that can be directly plotted onto the object, as demonstrated in Figure 6 of [B].

[A] Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold (Murphy et al., ICML 2021)
[B] A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation (Yin et al., ICLR 2023)

However, we believe that the visualization method on the 2-sphere using the Mollweide projection [A], as used in Figures R2, A2, and A3, offers the additional benefit of enabling multi-hypothesis plotting in 3D pose space. This allows for deeper insights into symmetry and pose ambiguity modeling.

---

Reply to Q1 regarding in-the-wild datasets (OOD scenarios):

Evaluating out-of-distribution (OOD) performance is generally not the primary focus in the context of this 3D orientation estimation task. However, we have conducted OOD generalization experiments using our proposed method by training the model on different datasets, between ModelNet10 and PASCAL3D+. The results are presented below:

Table: Cross-dataset evaluation for validating out-of-distribution generalization on ModelNet10-SO(3) and PASCAL3D+ datasets.

As the results indicate, the model does not perform well when evaluated on out-of-distribution dataset. Nevertheless, we recognize this as an important area for future research, and we appreciate your suggestion to explore this further.