From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields

We would like to thank the reviewer for their time and insightful comments. Below, we address the reviewer’s concerns in detail.

We agree with the reviewer about the importance of the suggested experiments/ablations. We are still working on the suggested experiments, and due to time constraints we will present the results in the upcoming days during the discussion phase.

The essential structure of NeoMLP boils down to a transformer with input tokens, latent tokens, and output tokens all together. This kind of architecture has been used widely, and in particular has been formalized generally in Set Transformer and Perceiver IO (former has a different goal, latter is quite relevant). The paper proposes to view such a transformer layer as an MLP, with the input, latent and output tokens corresponding to input, hidden and output nodes. While an interesting perspective, especially in the context of neural fields where MLP architectures are often used, nothing is done with this perspective: it is not used to imbue any inductive biases or any architectural decisions.
The paper further tries to avoid making this connection to general transformers as much as possible, which feels very misleading
A lot of the standard transformer parts of the model, e.g. the reason for attention and for positional encoding, are explained as if the ideas are new rather than it is the standard when using a transformer. 
I think the main issue with this paper is the writing/story. Why not be clear and up front that the architecture is essentially just a transformer? The connection to MLPs/message passing can be made, but what does the paper gain from such a story? 

It is not our intention to claim that we invent the ideas for attention, and we certainly not avoid making the connection to general transformers. On the contrary, we think we are very upfront about NeoMLP being a transformer architecture, as well as about reusing the existing attention mechanisms. As an example, in lines 272-273 we mention that “RFFs allow our architecture to use a standard transformer” and in lines 308-309 we mention that “NeoMLP is a transformer architecture”. Furthermore, we often mention in the manuscript that we employ self attention (or leverage in the title), to highlight that we are using an existing module from the deep learning literature. Our novelty lies in using self-attention for neural fields instead of the previously used cross-attention. Set Transformer and Perceiver IO are both attention/transformer architectures, but they have a lot of differences with our method. Importantly, neither the Set Transformer nor the Perceiver IO are neural field architectures; they operate on sets of points instead of being a (neural) function of individual points. Futheremore, they are based on cross-attention operations that encode the input set of points in a latent set, while NeoMLP is only employing self-attention. In the updated manuscript, we will include these works in our related work, alongside a discussion with the similarities and the differences between these works. We consider the connection with MLPs and message passing important, because it organically led us to conceive this architecture. In retrospect, a self-attention based neural field might seem obvious, yet we are the first to propose it. As such, we think that establishing such a connection is an integral part of our story.

The auto-decoding framework from [29] is also very standard and there are many works that explore it. A lot of section 2.3 is explaining autodecoding as if the ideas are new (e.g. fitting and finetuning stages). v-sets and v-reps seems to be needlessly defined as well. 

We apologize for the confusion. We do not intend to claim that fitting and finetuning are our ideas, since they are standard practice in auto-decoding neural fields. We will update the manuscript to make it clear that these steps exist in auto-decoding neural fields. We define the terms v-sets and v-reps to differentiate them from existing neural field representations like functasets, and be more explicit when we refer specifically to v-sets.

Compared to alternatives like sinusoidal activations [40], RFFs allow our architecture to use a standard transformer" why? Could still have a initial SIREN layer, which would work just as well (as long as omega_0 is set sufficiently).

Many state-of-the-art neural fields address the spectral biases using non-standard activation functions, including sine waves, Gabor wavelets etc. On the other hand, a standard transformer is using ReLU/GeLU/SiLU activations. Using sine wave activations within Transformers is not common practice, and to the best of our knowledge, it has not been studied or tested extensively to provide any performance guarantees. An initial Siren layer could be used before the transformer, but that is functionally equivalent to Random Fourier Features (with trainable parameters), as shown in Benbarka et al. [1].

It is nice that ablations on RFF are done, but these are the least surprising component: their benefit is not a surprise at all as spectral bias is well studied and [42] proposes RFF specifically for this issue.

The study of Tancik et al. [42] focuses on standard MLPs and employs tools from the Neural Tangent Kernel literature to demonstrate the spectral biases of MLPs. While it might seem obvious, we postulate that spectral biases would be present in NeoMLP as well if left unattended, and thus, we performed this ablation study. Future work can employ similar NTK tools to theoretically study the spectral biases in NeoMLP.

References

[1] Benbarka et al. Seeing Implicit Neural Representations as Fourier Series, WACV 2022.

We would like to thank the reviewer for their time and insightful comments. We agree with the reviewer about the importance of the suggested experiments/ablations. We are still working on the suggested experiments, and due to time constraints we will present the results in the upcoming days during the discussion phase.

Could the author report results on a higher-resolution 3D dataset (e.g., a full ShapeNet split or large-scale NeRF scenes) to demonstrate ν-reps at a realistic scale?

We thank the reviewer for the suggestion. We consider the ShapeNet split that we use already a high-resolution dataset, with points clouds in the order of 100,000 points, on which the signed distance function is calculated.

We would like to thank the reviewer for their time and insightful comments. Below, we address the reviewer’s concerns in detail.

This paper misses many related works, considering that using a transformer for a conditional neural field or generalizable INR is already well-explored [NewRef-1,2,3]. The experiments also need to compare the performance with these prior studies showing decent performance on downstream tasks.

We thank the reviewer for suggesting these related works, which we will include in the updated version of our manuscript. Generalizable INR [NewRef-2] and Locality-aware INR [NewRef-3] differ substantially from our method because they are encoder-based neural fields with encoders that operate on patches, in contrast to NeoMLP which is an auto-decoding neural field. Overall, auto-decoding methods inherit the fundamental advantages of neural fields: they are resolution independent, they scale gracefully with the signal complexity instead of the signal size, and they are modality independent, i.e. they do not make assumptions about the observations. Encoder-based neural fields, on the other hand, lose those advantages because of the encoder, which must be specialized for each modality and/or resolution. Furthermore, they use transformers to encode the signal by processing a set of patches at once, while NeoMLP is a transformer that processes individual coordinates (practically batches of individual coordinates).

The motivation of this study is ambiguous and over-interpreted/claimed. For instances, the authors claim that the lack of Connectivism in MLP is the main motivation of NeoMLP. However, the architecture converges to a conventional transformer, which is commonly used in this field, and there is lack of motivations for the design of NeoMLP compared with the existing works on conditional neural fields or generalizable INRs. Also, the architecture design is not new concept but well known as Perceiver-IO [NewRef-4].

Connectionism inspired us to perceive a unified architecture that integrates inputs and latents in a native fashion. Our architecture converges to a conventional transformer because transformers are ubiquitous set neural networks in modern deep learning, i.e. neural networks that operate on sets/fully-connected graphs, with numerous papers supporting their expressivity and performance, both theoretically and practically. Since we are operating on a fully-connected graph of input, output, and latent tokens, a transformer is the most logical approach. Perceiver IO is of course an attention/transformer architecture, but it has a lot of differences with our method. Importantly, the Perceiver IO is not a neural field architecture; it operates on sets of points instead of being a (neural) function of individual points. Furthermore, it is based on cross-attention operations that encode the input set of points in a latent set, while NeoMLP is only employing self-attention.

What is the main differences of NeoMLP from the existing studies using Transformer in terms of Connectivism?

Existing neural fields that use transformers either use cross-attention to encode individual coordinates [Wessels et al.], or cross-attention to encode the whole signal from a set of observations [NewRef-4], or a standard self-attention transformer with latent tokens that functions as an encoder in an encoder-based neural field [NewRef-2,3], encoding the whole signal from a set of observations/patches. Instead, NeoMLP is a self-attention based Transformer that encodes individual coordinates, and functions as an auto-decoding neural field.

The computational costs to decode each data instance compared with other methods.

We refer the reviewer to appendix C, which contains a detailed discussion of the computational complexity of NeoMLP compared to the other baselines, both for the high-resolution signal fitting experiments and for fitting v-sets. In summary, NeoMLP has a higher computational complexity compared to the baselines, but it can actually fit high resolution signals faster, and it does so while having a smaller memory footprint, since it can make use of small batch sizes. For fitting v-sets, NeoMLP consistently exhibits lower runtimes for the fitting stage, while Functa is much faster during the finetuning stage, which can be attributed to the meta-learning employed for finetuning, and the highly efficient JAX implementation.

We would like to thank the reviewer for their time and insightful comments. We agree with the reviewer about the importance of the suggested experiments/ablations. We are still working on the suggested experiments, and due to time constraints we will present the results in the upcoming days during the discussion phase.

I would greatly appreciate it if the authors could address the questions and weaknesses raised above.