Grounding Continuous Representations in Geometry: Equivariant Neural Fields

Regarding 3DShape2Vecset [5], we indeed observed a difference between reconstruction performance on occupancy of shapes (ENF obtains 0.929 IOU as measured over the largest 16 classes, where 3DShape2Vecset obtains 0.967 IOU measured over the largest 7 classes). We feel this can at least partially be explained by the difference in latent and model size; 3DShape2Vecset uses a set of 512 latents of dimension 512 (512^2 parameters) to represent a single shape, whereas our model as well as the Functa baseline only has ~800 degrees of freedom. We were unable to reproduce these results from the available code however, and since we’re approaching NeF representations for data representations more broadly – not focussing on shape data specifically –, we left this model out of our comparison. In our response to reviewer 4oVU, we do show downstream results when ablating over the pose information in our latent set parameterization, i.e. when using a set of pose-free latents as conditioning variable–a parameterization comparable to generalizing 3DShape2Vecset to arbitrary signal data. Results for CIFAR classification show the clear benefit of having a geometrically grounded localized latent set (82.1% -> 47.9% accuracy).

Unclear segmentation results We acknowledge the reviewer's concern that the choice of segmentation on the Shapenet-part dataset is questionable due to the global alignment of the dataset. We do think that these results, showing performance on-par with non-equivariant NeF-based baseline methods, allow for an assessment of performance of our method in the setting when the dataset does not exhibit global symmetries, i.e. it shows ENF performs on par also with these equivariance constraints. Comparing this setting with more classical point cloud specific methods shows that modality agnostic NeF-based representations only perform marginally worse. We investigate these results more in-depth in Appx. D, but since other reviewers also identified the segmentation results as a possible point of confusion for the reader, we believe that replacing this experiment in the main text with a generative modeling experiment will more effectively highlight the advantages of our method. We modify the method section by moving Fig. 8 to the appendix and summarizing the segmentation results more concisely in the main body.

The reviewer also pointed out that it is unclear whether the point-cloud methods used reconstruction as a pretext phase, this is not the case, the results for point cloud specific methods as reported in [6] are trained only on the segmentation task.

Additional comments

• The reviewer argues that figure 8 is uninformative without any comparison to other methods, we do agree with this and will move the figure to the appendix.

• The reviewer argues that our k-nearest neighbors approach to make the attention operation more efficient would restrict the smoothness of the resulting function. In practice, we find no training instabilities resulting from our parameterization, we think attributable to our use of the Gaussian window. The Gaussian window forces the attention coefficients to zero as distance grows and hence gradients for such latents naturally vanish, making the KNN approximation functionally equivalent.

• The reviewer argues that the steerability property for CNFs has also been defined in [7, 8]. We thank the reviewer for bringing these papers to our attention and we will refer to them in the main text. Especially [7] provides an interesting perspective on the notion of steerability in shape spaces. Though similar, we argue that our viewpoint on steerability in CNFs through bi-invariance constraints on the binary function that parameterizes the CNF is a useful and novel one, in that it allows for simple equivariant NF implementations. However, steerability is indeed a widely used concept in deep learning e.g. [7, 8, 12, 13, 14]. In fact, the definition of our steerability constraint via bi-invariants in Lemma 1 of our manuscript is similar to proofs in [9, 10] which show that 2 argument kernels should be bi-invariant to be equivariant. After analyzing our manuscript again we do agree that in these sections references to the denoted works should be added, and we do so in the revision we attach to this response.

We thank the reviewer for their thorough assessment of our manuscript and his appreciation of the simple and intuitive proposed solution. Moreover, we appreciate the thoughtful questions and points of discussion raised by the reviewer, which we feel will strengthen the work. Below we will elaborate on the questions and address the weaknesses.

Lack of motivation for using CNF latent encodings in downstream tasks The reviewer notes that we do not explicitly discuss the motivation for using Neural Field (NeF) based representations. Building on an increasingly large body of work utilizing Neural Fields as representations for a host of tasks (see e.g. [11] for a broad overview of use-cases in 2D and 3D reconstruction, generative modeling, compression, robotics, forecasting), we indeed realize we left this motivation mostly implicit in the current version of the manuscript. We agree with the reviewer that explicitly adding this motivation strengthens the manuscript, and dedicate some space to this in the introduction of the revision we attach to this response (ln 040 -> 044). We agree with the advantages stated by the reviewer and will further elaborate below.

The first advantage of NeF-based representations results from its discretization/resolution agnostic nature; a NeF-representation is not tied to a grid and as such is able to seamlessly transfer over on different discretizations / samplings of the same underlying data, a principle corroborated by the findings in our experiments on zero-shot resolution transfer and robustness to sparsity on the OMBRIA dataset.

Another advantage of NeF-based representations is that they are applicable to a range of spatial data modalities and geometries; as long as there is coordinate-signal data available, it is possible to fit a NeF-based representation to this data. As a result, this unifies models applicable to these different modalities and geometries that classically require their own specific engineering efforts. We show this in our experiments; we use the same downstream architecture for forecasting over spherical data as we use to classify image data. We feel this transferability is a desirable property compared to designing specific architectures for tasks and data types as is the case for classical grid-based or grid-free (point cloud) representations, since this in turn allows for the transfer of modeling principles between modalities and geometries.

A third, somewhat adjacent, advantage is that these continuous representations scale better by increasing resolution size; NeF-based representations are shown to scale with signal complexity rather than discretization resolution, which is shown in figure 2 of [3].

For these reasons, we feel the pursuit of NeF-based continuous signal representations is worthwhile. We amend our introduction to better reflect this.

Lack of motivation for using local CNF latent encodings The reviewer points out that we are harsh in our description of the use of global latents in e.g. Functa, denoting them as a limitation. We would like to point out that we do not necessarily oppose the use of global latents, or find them inherently limiting, only that through the use of a single global latent no explicit geometric information (e.g. position, orientation, relative position) on features in the signal is retained to be leveraged by downstream models, but instead these features are necessarily represented implicitly, limiting performance (as shown in our experiments) on tasks that require fine-grained reasoning (classification, segmentation). The reviewer highlights that there is an argument to be made for certain cases where this global latent is desirable, e.g. the notion of latent-space interpolation would be quite complex with our proposed local set-latent, but is very natural when representing a signal with a single latent. In settings with globally aligned data, this is naturally true. We do see the need for a nuanced representation of our method in contrast to previous works, and rewrote the intro section (ln 074 -> 079) to indicate the specific sort of tasks we think are limited by implicit representation of geometric information, and indicate the tradeoff between ease of downstream use (global latents) and performance (equivariant local latents).

Geometry separation The reviewer notes that our argument regarding the absence of the ability to separate pose and context information within the latent space of NeF literature is confusing, particularly in light of works [7, 8]. These works also introduce a steerability constraint, similar to ours, and utilize invariant and equivariant features. Although these works both include an experiment involving the use of an equivariant encoder-decoder architecture in implicit shape representation tasks, replacing the encoding in occupancy networks [9], neither of these works position themself as part of NeF literature (i.e. general continuous data representations). We acknowledge that the phrasing we used in our rebuttal was confusing and that indeed both of these works introduce (implicitly) the notion of a steerable implicit latent representation. We would like to stress that this phrasing appears only in our response to the reviewer and was not included in the manuscript itself. We change this phrasing in our response.

[1] LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., & Jackel, L. (1989). Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2.

[2] Wu, W., Qi, Z., & Fuxin, L. (2019). Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition (pp. 9621-9630).

[3] Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M., & Hoogendoorn, M. (2021). Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611.

[4] Cohen, T. S., Geiger, M., Köhler, J., & Welling, M. (2018). Spherical cnns. arXiv preprint arXiv:1801.10130.

[5] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2020). Message passing neural networks. Machine learning meets quantum physics, 199-214.

[6] Satorras, V. G., Hoogeboom, E., & Welling, M. (2021, July). E (n) equivariant graph neural networks. In International conference on machine learning (pp. 9323-9332). PMLR.

[9] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019). Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4460-4470).

Generative modelling on ENFs The reviewer raises the question whether we considered applying our methodology in the generative context. This is a very interesting possible application of ENFs, and as such, we follow [1], and train a diffusion model. With this rebuttal we also add an extra experiment on generative modeling over ENF latent spaces in Tab. 7, Fig. 9. We train a diffusion model, parameterized as a Diffusion Transformer (DiT-B)--a natural choice due to the set-structure of our ENF latent space–on the latents that we obtain from pretraining the ENF on an image reconstruction objective. We first obtain a set of latents for each image, and subsequently use these images in a denoising objective, and then train a DiT-B on a denoising diffusion objective on this latent space, where the forward diffusion kernel is given by:

We thank the reviewer for their comprehensive evaluation of our manuscript. We appreciate the kind words about the clarity and novelty of the work and their excitement about using equivariant neural fields as general backbone for downstream tasks.

Baseline comparisons to other equivariant methods The reviewer raised a valid weakness regarding not comparing to other equivariant baselines. We pose a general framework for acquiring continuous latent representations via CNF encodings for different data modalities and geometries. Therefore, we only compared methods which are applicable to different types of data modalities and geometries as well. Functa [1] is the original paper proposing this learning over arbitrary functasets and, as far as we know, no equivariant works exist in this line of research. There is a line of recent work that explores the use of deep weight-space methods (also referred to in our response to Rev. 4oVU) for amongst other tasks, learning over Neural Field representations, but we argue that due to the widely different scope and applicability of Conditional Neural Fields and weight-space methods (weight-space methods can be applied to general neural network architectures for tasks like model characteristic prediction [2], Functa / Conditional Neural Fields are generally applied to represent spatial signal data), this comparison is not sensible; highest-performing weight-space methods generally achieve around 45%-65% test set accuracy on CIFAR10 classification with very specific use of augmentations, and as such we argue that comparison to these approaches is ineffectual and might be confusing (see elaboration under ‘Comparison to weight-space methods’ in response to reviewer 4oVU). The current work is mainly interested in adding explicit geometry in the latent-spaces of CNFs encodings, and hence Functa is our main point of reference.

The reviewer also raised some questions which we will answer below.

Application to non-equivariant settings The reviewer raises the question whether there are scenarios where enforcing equivariance might not be beneficial. It could be argued that equivariance constraints limit the expressivity of the framework when applied to tasks that lack the same symmetries. However, we like to argue that adding the constraints, which enable weight-sharing, could still be beneficial in such a scenario. For instance, when analyzing the classification results of CIFAR-10 or ShapeNet16 there could be observed that even though these datasets do not exhibit exact symmetries within the data, translation equivariance -- via relative positions between latent points and sampled coordinates -- outperforms the same model using absolute positions. This suggests that weight-sharing over patches, derived from these relative relationships, leads to better performance. However, you could also observe from the same experiment that restricting it further (to SO(2) equivariance) did harm the performance a bit. So we posit that restricting the model too much could still be harmful when the symmetry is not contained in the data.

Weakness: comparable performance to baselines on ShapeNet segmentation The reviewer points out that our segmentation results are weaker compared to traditional point cloud segmentation baselines. Noting that other reviewers (9x4v, AKXe) have also highlighted this issue, we intend to move this experiment to the appendix and replace it with a generative modeling experiment in the main text. We initially included the segmentation experiment because it lacks symmetries, thereby demonstrating that our method performs comparably even in non-symmetric settings. As detailed in Appendix D.2, we discovered that without conditioning on the acquired latents and using only the class embedding, ENF achieved class and instance mIoU scores of 64.3 and 69.2, respectively. This indicates that many points in this dataset can be correctly segmented based solely on their absolute positions. However, we believe that the generative modeling experiment more effectively highlights the strengths of our method. We will retain the segmentation experiment in the appendix for reference.

We would like to thank the reviewer again for their valuable suggestions, and invite the reviewer to discuss if any concerns remain.

[1] Dupont, E., Kim, H., Eslami, S. M., Rezende, D., & Rosenbaum, D. (2022). From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204.

[2] Yin, Y., Kirchmeyer, M., Franceschi, J. Y., Rakotomamonjy, A., & Gallinari, P. (2022). Continuous pde dynamics forecasting with implicit neural representations. arXiv preprint arXiv:2209.14855.

[3] Knigge, D. M., Wessels, D. R., Valperga, R., Papa, S., Sonke, J. J., Gavves, E., & Bekkers, E. J. (2024). Space-Time Continuous PDE Forecasting using Equivariant Neural Fields. arXiv preprint arXiv:2406.06660.

[4] Zhou, A., Yang, K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., ... & Finn, C. (2024). Permutation equivariant neural functionals. Advances in neural information processing systems, 36.

[5] Papa, S., Valperga, R., Knigge, D., Kofinas, M., Lippe, P., Sonke, J. J., & Gavves, E. (2024). How to Train Neural Field Representations: A Comprehensive Study and Benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22616-22625).

[6] Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., ... & Finn, C. (2024). Neural functional transformers. Advances in neural information processing systems, 36.

[7] Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., & Maron, H. (2023, July). Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning (pp. 25790-25816). PMLR.

[8] De Luigi, L., Cardace, A., Spezialetti, R., Ramirez, P. Z., Salti, S., & Di Stefano, L. (2023). Deep learning on implicit neural representations of shapes. arXiv preprint arXiv:2302.05438.

[9] Zhou, A., Yang, K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., ... & Finn, C. (2024). Permutation equivariant neural functionals. Advances in neural information processing systems, 36.

[10] Ramirez, P. Z., De Luigi, L., Sirocchi, D., Cardace, A., Spezialetti, R., Ballerini, F., ... & Di Stefano, L. (2024). Deep Learning on Object-centric 3D Neural Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence.

We thank the reviewer for taking the time to evaluate our manuscript thoroughly and contributing to its improvement. We address each of the reviewers' concerns separately below. We hope to continue the discussion if any concerns remain.

Difference in architecture compared to Functa baseline The reviewer highlights the contrast between the proposed ENF architecture and the Functa [1] architecture used as baseline - built on top of SIREN [2]. We assert that comparing Functa with our attention-based architecture is a reasonable and relevant comparison, as it remains the most prominent work on CNF-based signal representations, making it an essential point of reference for our proposed model. Functa introduces a specific framework for parameterizing signals through layer-wise MLP shift modulations, parameterized by a latent vector. This latent vector may then be used by simple MLP-based downstream models. Because this single vector constitutes the conditioning variable in Functa, it is not possible to use a cross-attention operation in combination with the original Functa architecture, and in fact one of our primary contributions lies in proposing a method for utilizing point clouds as conditioning variables in CNFs. To enable additional comparison, we provide specific parameter counts, inference time and memory complexity of our model compared to Functa in our response to Rev AKXe, showing drastically improved parameter efficiency of our model which we attribute to the inclusion of locality and weight-tying as inductive biases.

We do agree (also noted by Rev. AKXe ) that an additional ablation over the specific geometric conditioning that we propose may contribute to better interpretability of the experimental results, and as such we perform an additional experiment where we use a geometry-free latent conditioning set, i.e. the latents have no position and as a result no locality (this approach can be seen as an extension of 3DShape2VecSet from shape to arbitrary signal data). We achieve this by making the “bi-invariant” $\mathbf{a}$ (and as a result $\mathbf{q}$) only a function of $x$. We find that this implementation of the framework –keeping all hyperparameters identical to the setup we used for CIFAR classification – leads to highly unstable training that saturates around 22 reconstruction PSNR on the test set, likely attributable to the fact that now any update to one of the latent codes affects the output of the NEF globally, leading to a much more complex optimization landscape. This highlights another advantage of either having a single global latent, or using locality as inductive bias; optimization of single or locally responsible latents seems to lead to a simpler optimization landscape compared to optimizing a set of global latents. We apply a simple transformer with 4 layers, 256 hidden dim and 4 heads as a downstream classifier (without positional encoding, since the latents don’t have positional attributes in this setting). We train for 500 epochs with Adam using a learning rate of 1e-4, after which the train loss has converged, obtaining 0.98 train and 0.43 test set accuracy. We observed overfitting early into training. Utilizing early stopping, best performance was achieved after just 5 epochs, yielding a test set accuracy of 0.47. These observations are in line with the outcome of our other experiments; geometry-grounded latents are more informative for downstream tasks. The table below is added to Appx. D.

What downstream Functa architecture is used We recognize the need for reproducibility, and added more info on the Functa baseline used in Appx. C.2 to include details on the downstream architectures we used. We’ve edited the main body of the text to refer to this section more clearly. We tried keeping close to the original setup used in [1], and so for the downstream architectures we use a 3-layer 1024 hidden dim residual MLP, slightly larger in parameter count compared to the 3-layer 256 hidden dim PONITA MPNN that we use in all our ENF experiments (2.1M vs. 1.7M). Like in our ENF experiments, the same architecture is used in classification, segmentation and forecasting, only the output head is changed.

We thank the reviewer for their thorough assessment of our work, and are glad to see that the reviewer appreciates the clarity of our presentation, as well as the novelty of the proposed method and rigor of our experimental validation.

The reviewer raises a number of valid concerns, which we address below.

High time complexity The reviewer rightfully raises concerns regarding computational complexity of our method. Indeed, as noted in the manuscript, the original formulation of the ENF architecture is computationally expensive due to the “global” computation of attention coefficients. However, using the kNN approximation we’re able to drastically lower computational complexity; computational complexity for the original method scales quadratically with the number of latents N and input coordinates M; O(N*M), whereas when applying the kNN approximation, we can keep K fixed to a relatively small number (e.g. k=4, leading to O(4M) complexity). We recognize the need for quantitative evaluation of the computational efficiency of our method, and as such add an ablation to Appx. D. Below we show the reduction in runtime that the kNN approximation yields, as well as a comparison with Functa. These results show that ENF with kNN approximation uses similar number of FLOPs and memory compared to Functa, but is significantly (>10x) faster in runtime, which we attribute to the shallow nature of our ENF model as compared to the relatively deep SIREN used in Functa. Additionally, we investigate the impact that the kNN approximation has on performance below (also added to Appx. D.), and show that impact to both reconstruction and downstream performance are negligible.

Lack of ablation studies The reviewer highlights that the manuscript would benefit from adding ablation studies on the Gaussian spatial windowing (GSW) and the k-nearest neighbors (kNN) approximation. We do agree with this and provide the ablation study in the table below. We update this rebuttal, now that we've finalised running these experiment on CIFAR-10 classification. We keep the hyperparameters identical to the ones listed for Tab. 1 / Sec. 4.2, but notably do not train the downstream model on augmented CIFAR-10, and train the ENF backbone for only 10 epochs.

As can be seen, although reconstruction performance is relatively unaffected by ablating over the kNN and GSW (i.e. when removing them from the model), the downstream performance is significantly lower when not using GSW. This is in line with our expectations, as (like explained in Sec. 3.1) there is nothing enforcing locality of the latents. Using only the kNN slightly improves performance, possibly since it adds some measure of locality back into the latent space, but it seems the smoothness in locality enforced by the GSW is vital for downstream performance. The Gaussian spatial windows significantly improve the performance on CIFAR-10 classification and reconstruction, since it allows for weight-sharing between latents.

Suboptimal Segmentation Performance We acknowledge the reviewer's observation that our segmentation results are weaker compared to traditional point cloud segmentation baselines. Since other reviewers (9x4v, 4oVU) also identified this as a weakness, we choose to de-emphasize this experiment to the appendix and in turn, follow [1] in adding an experiment showcasing generative modelling capabilities of the ENF framework in the main text (see also our response to Rev. yjcq). The segmentation experiment was initially included because it showcases versatility of Neural Fields in varying datatypes and modalities. We find that these findings– due to the global alignment of ShapeNet data– actually demonstrate that our method performs comparable to baselines in settings without global symmetries. As discussed in Appendix D.2, we found that without any conditioning on the acquired latents and using only the class embedding, ENF achieved class and instance mIoU scores of 64.3 and 69.2, respectively. This indicates that many points in this dataset can be correctly segmented purely based on their absolute positions. However, we believe the generative modeling experiment more effectively highlights the advantages of our method. Nonetheless, we will retain the segmentation experiment in the appendix for reference, as they indeed showcase that equivariance as inductive bias has limited benefit in settings without global symmetries.

We are happy to continue the discussion when the concerns are not completely addressed yet.

EDIT: We updated the results for the ablation over KNN/GSW in the above table.

Dear reviewer, thanks again for helping us to improve the manuscript, your suggestions and feedback were valuable!!