$SE(3)$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation

W1. The authors introduced a new equivariant nonlinearity inspired by [4], but the motivation and benefits are not clearly demonstrated. What is the distinctive advantage of this new nonlinearity, compared to existing SE(3)-equivariant nonlinearities?

Our nonlinearity, unlike norm and Gate nonlinearities, can change the direction of the feature, by reducing/keeping the projection of the tensor to an equivariance space. In the meanwhile, compared with the nonlinearity method that can change the direction of the feature by applying Fourier and Inverse Fourier transforms [1, 2], ours is more computationally efficient.

[1] Maurice, et al. "General E(2)-equivariant Steerable CNNs." NeurIPS 2019

[2] Adrien, et al. "A Functional Approach to Rotation Equivariant Non-linearities for Tensor Field Networks." CVPR 2021

W2. The number of parameters was not fixed during the ablation experiments regarding the maximum spherical harmonics degrees. A recent study [5] claimed that the reported increase in performance due to incorporating higher-type irreps in various works could actually be due to the increased number of parameters. It is essential to control the number of parameters to be similar between the ablated models and the proposed model.

Thank you for the great suggestion. We agree that this is an essential variable to be controlled in these experiments. Thus, we include here additional ablation experiments on the ScanNet benchmark with increasing the numbers of parameters, by using the highest orders of spherical harmonic embeddings to match the parameter count of our model. These were conducted using a short training schedule of 100K training steps, and the results are shown in Table 1 in the rebuttal PDF. We observe a similar trend as reported in our submission (Table 3), showing that these improvements do not come from the increased model complexity. We will include these additional results in the camera-ready version of the paper.

Q1. According to the equations in Appendix F, different types of irreps do not mix in the self-attention layer. They also do not mix in the proposed nonlinearities in Appendix F. It seems like, in the proposed method, each of the irreps (except for type-0) can only indirectly modulate other irreps of different types via attention. Am I correct?

You are correct. When generating key, query, and value features, we do not mix different feature types. While calculating the attention matrix, as shown in Appendix F, we use all types of features, which makes them indirectly modulate other irreps of different types.

Q2. Subtracting the mean of the center is not stable under the addition or removal of camera points. Is it possible to use relative positional encoding, similar to rotary embedding, to achieve translational equivariance without relying on centroid subtraction?

We thank the reviewer for pointing that out. When there are two cameras (which is the primary setting explored in our experiments), the relative position is exactly twice our current translation (the translation after subtracting the centroid). We use spherical harmonics for our current translational positional embedding, which is equivariant to $SO(3)$ while rotary embedding is not equivariant to $SO(3)$. For multiple cameras, introducing a relative positional encoding would be a good idea, however, this approach introduces the problem of permutation equivariance of the camera order in the Perceiver IO architecture (the index of the camera impacts the index of relative translation). We believe there is a more elegant solution to the translation equivariance problem, and hope to look into it in future work.

L1. Spherical harmonics can only encode the orientation of and not the length. Therefore, typical SE(3)-equivariant networks address this by incorporating additional length encoding. However, in this paper, the distance information is discarded.

As stated in Section 3.3.1, we incorporate an additional radial component $||r||^l$ into the original “order-l” spherical harmonics of the corresponding degree. Despite this adaptation, these functions retain their fundamental characteristics and are still referred to as spherical harmonics. We provide an introduction and discussion of spherical harmonics in Appendix A.4, as well as the design of incorporating the invariant length.

L2. Using irreps features inevitably introduces a band-limit. Increasing this band-limit is difficult because the feature dimension increases quadratically, which is also discussed by the authors.

Thanks for the deep comment. Yes, the dimension of the spherical harmonics is a limitation in our approach. That's why we address it in the decoder by predicting an equivariant frame followed by a conventional decoder, which enables higher frequency embedding for the query.

L3. The authors also mentioned that higher-degree spherical harmonics caused instability in training. However, this might be due to the choice of equivariant nonlinearities. Liao et al. [3] reported that certain nonlinearities cause instability in training.

Thank you for the insightful remark. We have indeed observed the same behavior stated in [3] about the instability of $S^2$ activation, which treats the features as the Fourier coefficients for $S^2$, processing scalars, and high-order features together. They addressed this behavior by separating the scalars and higher-order features. On the other hand, our approach separates different orders of features and processes them independently (as mentioned in Q1). Replacing our nonlinearity with norm linearity did not stabilize training, suggesting that this instability is not solely due to the choice of nonlinearity.

W1. Since the equivariance consists of two parts, namely translation and rotation, what would be the quantitative impact of removing these two parts respectively?

Thank you for the valuable suggestion. We conducted an ablation study where we individually integrated only rotation equivariance and translation equivariance into the model. These were conducted using a short training schedule of 100K training steps due to the time limit. The results, displayed in Table 1 of the rebuttal PDF, show that models without translation or rotation equivariance perform worse than our complete model.

Q1/2. The task setting of implicit depth estimation seems to be very compatible with the existing sparse view NeRF/GS methods. DeFiNe can synthesize novel view images, can EPIO do the same?

Thank you for pointing that out, this is a great direction that we would like to explore in follow-up works. EPIO and DeFiNe were designed for “single-query” predictions, without relying on volumetric rendering (like NeRF) or explicit 3D structures (like 3DGS). Single-query novel view synthesis is challenging, especially in the generalizable setting, and DeFiNe itself does not report results in this task, but rather shows that by jointly learning novel view synthesis and depth estimation it can marginally improve depth estimation. Our proposed EPIO architecture could be extended to novel view synthesis, especially because it accepts traditional non-equivariant decoders (i.e., from DeFiNe), and we indeed show results on a toy novel depth synthesis experiment in Appendix O.2. Unfortunately, extending EPIO for an additional task and retraining the model is impractical in the rebuttal period, but we will aim to provide initial results (similar to the novel depth synthesis experiments) in the camera-ready version.

Having said that, there is a follow-up work to DeFiNe (DeLiRa) [1] which explores the use of Perceiver IO for volumetric rendering, focusing on novel view synthesis with depth estimation guidance. We believe EPIO could be used in this setting as well, and that would be a very interesting extension.

[1] Guizilini et al. “DeLiRa: Self-Supervised Depth, Light, and Radiance Fields”, ICCV 2023.

W1. Many important details from Sections 3.4 to 3.6 are placed in the appendix, making the main paper not self-contained.

We appreciate and thank the reviewer for the valuable feedback. We decided to organize the paper this way not only due to limited space but also because we want readers to have a clear understanding of our main proposed components, especially those without a prior equivariance background. Thus, we leave standard components (e.g., equivariant linear and normalization layers in appendix A.3) in the appendix, where readers unfamiliar with the field can get a better understanding through visualization and more detailed analysis and proofs.

Having said that, we agree that a more balanced structure might be better, and will move more technical details and visualization from the appendix E to the main paper for the camera-ready version.

W2. Sections 3.4 to 3.6 are organized into fragmented components, where the holistic process of the Perceiver IO is missing.

Thank you for the suggestion. We decided to introduce fundamental attention layers in Section 3.4 before discussing the encoder in Section 3.5 and the decoder in Section 3.6 because attention mechanisms are essential components of both modules (encoder and decoder). By first explaining these fundamental operations, we aim to help readers build an understanding going from the basics to the overall more complex structures.

If this approach is causing confusion, we propose to instead first briefly give a whole overview of our holistic method with the illustration of Figure 2, then present the input design of the encoder first, followed by an introduction of the encoder, the “canonicalization” of the encoder output to serve as input to the decoder, the design of the decoder, and finally the prediction process. We believe this new order would create a better flow that approximates more closely the holistic nature of the Perceiver IO architecture, from input to output.

W3. It is better to only briefly discuss components that are equivariant itself, such as attention, and discuss only how they made the input to the attention equivariant.

Thank you for the suggestion. The reason we provide an overview of the attention mechanism and compare our module structure with previous work is to lay emphasis on the equivariant operation of proposed module and help readers without a relevant background to have a better understanding of why the conventional operation is not equivariant and ours is.

However, to avoid any potential confusion, we will move more details from Section 3.4 (especially the descriptions of previous works) to the appendix for reference, so the focus is on our contributions. We will also move Appendix I to Section 3.5.1 to provide more information about the input construction for the encoder.

W4. Why is only rotation sampled and encoded when constructing the latent array in Section 3.5.2 and Figure 4, while the inputs have the encoded camera translation?

The translations of the two cameras are opposites because we subtract the mean of the translation to achieve translational invariance. As discussed in Section 3.5.2, we need to average the PE (positional encodings) of the two cameras to obtain the latent array. Since averaging the PE of two opposite translations might cause some zero values, we discard the PE for the translation of the latent array construction. Note that we keep the spherical harmonics PE for translation in inputs, as a way to preserve translational information.

W5. In Section 3.6, only reverse rotation is applied to the latents after several self-attention transformation blocks, while the translation is omitted.

As stated in Section 3.3. 2, after we subtract the cameras’ central position, our model becomes translationally invariant. Because the hidden features are now rotationally equivariant and translationally invariant, we apply the reverse rotation to the latent array to make it rotationally invariant. Note that, when we provide query cameras for the decoder, their position is also extracted from the center, which makes the whole model translationally invariant.

W6. The authors indicate that the Fourier encoding is not equivariant but use it for the output query.

As stated in L334-339, a limitation in the use of spherical harmonics (SH) is that dimensionality grows linearly (2x) with increasing orders, which constrains the highest frequency that can be used in the positional encodings. Therefore, we learn instead an equivariant frame of reference designed to make the input to the decoder invariant, which enables us to use traditional decoders without SH, which can then reach higher frequencies. It is important to note that, even though we use traditional Fourier positional encodings for the decoder, the model is still equivariant, since the input to the decoder is invariant by design. The theoretical proof that guarantees an invariant input for the decoder is provided in Appendix J. There is an ablation study in Table 3, where “EquiDecoder” indicates the use of an equivariant decoder with equivariant positional encoding for the query, our design is better than the equivariant decoder, which supports the effectiveness of our design.

W7. Many illustrations (Figures 8-11) in the appendix are confusing.

Thank you for pointing that out. We have enhanced these figures and included the updated versions in the rebuttal PDF (Figure 1,2,3,and 4). Any additional feedback would be highly appreciated, so we can further improve the quality of our submission.

Dear Reviewer KQ34

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best wishes

AC