On the Optimality of Activations in Implicit Neural Representations

My key concern is that the paper is hard to read, especially for many potential users of this work (e.g. NeRF practitioners). When the paper involves this much theory, it is particularly important for the presentation to be clear and to offer graphics / intuition when possible....

We thank the reviewer for their concern and we understand their sentiment that the paper is dense. One of the main reason we included some of theory in its abstract form in the main paper is that a key point of this paper was to develop a theoretical framework that can be used by theoreticians and practitioners in the field. In present day machine learning there is a deep divide between practise and theory and part of our goal in this paper was to bridge this gap. Therefore, we thought it reasonable to include some of the theory in main paper and back it up with experiments towards the end, showing that our theory can be used to better understand empirical observations about INRs. We have added some more explanations, references and details to various aspects of the proofs of the main theorems in the appendix in the revised version, thereby making these parts of the paper much more readable.

what happens if you try to fit a function in L^2(R) but not in V(F) with a weak Riesz basis vs. a strong basis?

We carry out this procedure in the experiments section. We consider general $L^2$ signals and compare fitting them with neural networks admitting activations such as sinc, Gaussian, sine, wavelets and ReLU. In all cases sinc does better showing that the condition that it generates a Riesz basis leads to its ability to fit a general $L^2$ signal much better than the others.

under def 3.1 I don't understand the step "the upper inequality in .....

Sorry, this was a typo. It should read $s \in V(F)$. Thank you for pointing this out.

Almost all modern radiance field parameterizations with INRs use some form of explicit spatial representation (Instant NGP, ZipNeRF, TensoRF, DVGO, factor fields, and many more) usually with just a single hidden layer in the MLP decoder. since theorem 3.10 only requires a single hidden layer, it would be interesting to examine whether sinc activations yield improvements in those kinds of models as well

This is an interesting suggestion. Most of the above mentioned methods use shallow networks with RFF encodings over a spatial grid. Since sinc activations can be used as a better drop in replacement for RFF-INRs, it should be possible to get better or on-par results with sinc-INRs. However, we would also like to note that in these settings, the INRs only encode a small volume in a partitioned space, resulting in low resolution signals. Therefore, the advantage we gain from using a more capable sinc-INR might be not significantly evident.

Minor typos / notational problems...

Thank you! All of these have been fixed.

In eqn 5, it is not clear what argument the central dot is supposed to represent or where it comes from

The central dot is just a placement for the variable of the function $s$ i.e. it can be read as $s(x - \tau)$ where $x$ is the function variable.

F with bar + hat + tilde is excessive. and what is bar?

We thank the reviewer for bringing this up. The bar notation is a typo and should not be there in equation (6), we have corrected this. In general, notation in functional analysis and harmonic analysis has conventions that have gone through for decades although many of them are not ideal. It is always the case that one uses hat to denote the Fourier transform and tilde to denote a distribution, test function, associated to a given function. As we are combining both of these, the notation can seem rather cumbersome, though it retains the key aspect that these operators, taking an associated test function and then applying a Fourier transform, are function dependent. If we replace the notation with something completely different we fear it will cause confusion amongst those who work in this space and this is why we dicided to stick with the notation.

Thank you for your response. We have added an explanation of what the partition of unity condition is doing and how it is bridging the gap between $V(F)$ and $L^2(\mathbb{R})$. The explanation is given in the appendix in section A.2. Can you please have a look at the explanation and let us know if that is helpful?

Here we give a summary of the key steps:

Let us assume that we are given a function $g \in L^2(\mathbb{R}) - V(F)$, that is $g$ is a square integrable function that does not reside in the space $V(F)$. A natural question that arises is, can we we still use elements in the space $V(F)$ to approximate $g$? Mathematically, what this question is asking is if we are given a very small $\epsilon > 0$ can we find a function $G \in V(F)$ such that

$\vert\vert G - g \vert\vert_{L^2} < \epsilon ?$

This is precisely where the partition of unity condition comes in. Mathematically, the reason the partition of unity condition is able to bridge the gap between $V(F)$ and $L^2(\mathbb{R})$ is as follows. We can write $g = g - G + G$ for any function $G \in V(F)$. The question now is does there exist a $G \in V(F)$ that makes the quantity $g - G$ very small in the $L^2$-norm? In other words, given a very small $\epsilon > 0$ can we make $g - G$ smaller than $\epsilon$ in the $L^2$-norm.

The way to answer this question is to first note that there is a simple way to try to construct such a $G$. Namely, project $g$ onto the subspace $V(F)$. Let us denote this projection by $\mathcal{P}(g)$ and note that $\mathcal{P}(g) \in V(F)$. We can then look at the difference $g - \mathcal{P}(g)$ and ask can it be made very small? In general this technique does not work and the projection $\mathcal{P}(g)$ won't be close to $g$ in the $L^2$-norm.

However, there is another projection. Namely, we can project $g$ onto the $\Omega$-scaled signal space $V_{\Omega}(F)$ for $\Omega > 0$ and form $\mathcal{P}_{\Omega}(g)$, which lies in the scaled signal space

$V_{{\Omega}}(F)$

The partition of unity condition is precisely the condition that allows one to deduce that there exists a $\Omega > 0$ such that the difference
$g - \mathcal{P}_{\Omega}(g)$ can be made very small.

Thus the second condition from the Riesz basis definition, the partition of unity condition, is telling us how to approximate functions outside of $V(F)$ using the scaled translates $\{F(\frac{x}{\Omega}-k)\}$ and the scaled signal spaces $V_{\Omega}(F)$. It says that we cannot necessarily perfectly reconstruct a function outside of $V(F)$ but we can reconstruct it up to a very small error using the $\Omega$-scaled signal space $V_{\Omega}(F)$. The mathematical proof of how it does this is given in sec. A.1.1 of the appendix.

Another way to state what we are saying is that the partition of unity condition bridges the gap between $V(F)$ and $L^2(\mathbb{R})$ via the scaled signal spaces $V_{\Omega}(F)$ telling us that reconstruction is possible only in $V_{\Omega}(F)$ for some $\Omega > 0$.

We hope that helps your understanding. Please let us know if there is anything else we should add. Furthermore, thank you so much in taking the time to read our work. It is greatly appreciated.

We thank the reviewer for the valuable suggestions. Please find our answers below.

Not a major weakness but the use of orthogonal polynomials (such as Hermite) has been proposed...

This is a very reasonable question from the reviewer and we thank them for asking it. It is true that practitioners in ML have found that Hermite polynomials do perform well as activations. However, this has predominately been shown on the empirical front. Our goal in this paper was to establish a theoretical framework that offers conditions that the practising machine learner should seek when designing a new activation. We found that for signal reconstruction in $L^2$ a key condition is the one of being able to generate a Riesz basis. We should say the Hermite polynomials do not generate a Riesz basis as they do not live in $L^2$ however they do form an orthonomal basis in the Gaussian measure space $L^2(\sigma)$, where $\sigma$ denotes the Gaussian measure. It would be great if the reveiwer could provide a reference for where the Hermite polynomials are being used for signal reconstruction. In the setting of implicit neural representations, they are generally not used as they underperform when compared to other activations.

While it is clear that the main selling point of the paper is the technical analysis, the experimental presentation can be more thoroughly presented (even if it is short). Fig. 1 would have us believe that for the dataset described, the proposed method yields the best overall performance. Very little additional discussion is provided. Nerf experiments are presented yet there is no discussion of whether the changes can be dropped into current implementations commonly used by practitioners.

Thank you for the suggestion to improve our paper. We have added a more clear explanation to the experiment corresponding to Figure 1. Also, we used the original settings proposed by [1] for NeRF experiments. However, we would like to point out that $\mathrm{sinc}$-INRs should be able to be used as a better drop in replacement for any INR, as we have verified across various tasks.

[1] - Mildenhall, Ben, et al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis." European Conference on Computer Vision. Cham: Springer International Publishing, 2020.

Dynamical systems experiments are nice but its not obvious why the vanderpol oscillator dynamics would be more challenging than those in Fig. 1.

We agree that a more theoretically involved analysis can be conducted to investigate the intricate connections between the dynamics of the systems and the performance of the INRs that are used to encode them. However, we believe that precise modeling of this connection would be out of the scope of this paper.

Thank you very much your response and your two comments. We greatly appreciate you taking the time to give us feedback.

1. We see your point of view and agree there is merit in what you are saying. Our theory provides a sufficient condition to chose an optimal activation yet the theory does not provide necessary conditions. This is in itself an open problem and as of yet we feel trying to show the theory is necessary will require a lot of work and we hope to take this up in a future project. However, in order to give credence to our theory, we did test it among various different types of experiments. For these experiments we purposefully chose 4 different types of functions. We chose a sine function because sine and cosine form a Fourier basis for periodic functions (note a cosine is just a shifted sine), we chose a wavelet function because wavelet functions can generate a basis for frequency space, we chose a Gaussian function because Gaussians generate a weak Riesz basis and finally we chose a sinc function because such a function generates a Riesz basis. Thus each of the functions we chose is related to a type of basis used in signal processing. We wanted to then use the experiments as a means of testing how well each of these bases do with the hindsight that a sinc function should do extremely well as our theory says that a sufficient condition for a function to perform well as an activation for INRs is if it generates a Riesz basis. Finding that the other functions did not perform as well as a sinc does not necessarily prove that the Riesz basis condition for an activation is necessary for optimal performance but it does suggest that as a sufficient condition it is extremely useful for the practising machine learner. We completely agree that this does not rule out another basis function such as Hermite functions doing equally as well yet we decided to test on sine, wavelet, Gaussian, and sinc because these are the most popular ones used within the INR community.

2. Thank you for the point regarding the experiments, we certainly understand what you are saying. The main reason for choosing the classical NeRF architecture was simply one of choice. Most newer NeRF architectures are enrichments of the classical NeRF architecture for a particular problem or as a means to enhance the ability of the classical NeRF. Therefore, since our approach requires focusing on the activation we felt that it be best if we went with the classical NeRF architecture from 3+ years ago as this has been used by several practitioners in the field and those using more modern architectures will easily be able to drop in our activation into their architecture as most of the newer NeRF style architectures build upon the classical one. Further, our goal was to present a more broad overview of results showing that our theory is useful not only in vision contexts but dynamical ones too. So we thought we would go with famous architectures people know about in each one.

We thank the reviewer for carefully going through our paper and providing suggestions to improve our current version. We have addressed all your comments. Please find detailed answers below. Please note that the Theorem and proposition numbers may have changed in the revised version as we have now moved related works to the end of the paper as you suggested.

There are countless typos in the paper, including undefined notation, which makes parsing the contents quite challenging. For example, I believe that in the paragraph below Eq 5, T is undefined, and I think it should be 1/Omega instead. Similarly, there's an inequality.....

Thank you! You are right $T$ is incorrect and it should actually be $\Omega$ not $1/\Omega$. The inequality in thm. 3.7 has also been corrected.

This is an even more serious problem in the appendix, where it appears that the content wasn't even re-read once, as there are gross typos (including some mathematical ones) in every second to third sentence....

1. We have gotten rid of Appendix A as it was not needed in the main paper.
2. Appendix B is now referencing sec. 3.3 for the proofs correctly.
3. There was a typo with the notation $\overline{\hat{\widetilde{F}}}$. There should be no bar over this. We have fixed this.

The notation should be optimized significantly.....

We understand the reviewers sentiment with the notation. However, we would like to clarify this is the notation that is generally used in sampling theory employing functional analytic methods. Our reason for choosing this notation is to stick with convention. However there is a mistake with the notation in equation (6). There should be no bar on the Fourier transform of the test function. We have removed this.

In general, the standard convention is $\widetilde{F}$ is used to denote a test function (in the sense of distributions) associated to $F$, and a $\hat{F}$ has always been used to denote the Fourier transform. Unfortunately, these conventions are normally fixed in the field of functional analysis. It happens to be the case that we need to use both these operations and hence that is why we had to go with that notation. Nonetheless, if the reviewer likes we are happy to replace the notation $\widehat{\widetilde{F}}$ with a completely different notation but this will then end up loosing how this function is connected to $F$.

Sections 4.3.1 and 5 are extremely terse and poorly explained. Several concepts are not defined or referenced (e.g., spectral derivatives), and a reader like me, who's unfamiliar with dynamical systems, will get completely lost.

Thank you for the suggestion. We have rewritten the above sections in more detail now. To facilitate the space constraints, we have moved some qualitative figures to appendix.

The Related Works section is far too technical at the beginning of the paper. I suggest the authors move it to the end of the paper and also simplify its content.

Thank you. We have addressed this and moved the related works to the end of the paper now.

As a more minor point, please use ...

We have changed $sin$ to $\sin$ and sinc to $\mathrm{sinc}$. Thank you for the suggestion.

Furthermore, I found the proofs of Prop 3.3, Thm 3.7, and Lemma 3.8 in Appendix B to be much too terse...

We presume you mean the proofs of Prop. 3.3, Thm. 3.4 and Th. 3.10. We greatly appreciate the reviewer taking the time to go through the proofs of these theorems. It is by no means easy as it uses deep mathematics. To clarify aspect of the proofs, we have added more details explaining several of the steps along the way and given clear references where needed. Hopefully now it is much more accessible. Thank you for pointing this out.

To define a notion of optimality of an activation function for a given class of functions, maybe the authors derive bounds on the growth of $n(\epsilon)$ in Thms 3.4 and 3.10?

By an activation being optimal we mean this in regards to whether it is a Riesz basis or not. This is certainly not a unique condition, there could be various different activations that form a Riesz basis but amongst the ones used by practitioners, namely, ReLU, tanh, sine, Gaussian, sinc, wavelets, we find that sinc is the only one that can generate a Riesz basis, making it an optimal activation to use from our theorem 3.10. We thank the reviewer for their comment, we have included a paragraph in Remark 2.11 (see revised version of our paper) clearly explaining this.

Understanding the growth rate of $n(\epsilon)$ is a very interesting problem, though at this point is out of the scope of this work. The growth of $n(\epsilon)$ will allow us to prove an effective version of theorem 3.10, allowing one to understand exactly how much overparameterization a network needs in order to fit such signals. We hope to carry out such a study in a future project.

While the universal approximation theorems hold for 2-layer -INRs, it is unclear to me whether sinc retains its advantage in deeper architectures. Could the authors comment on this?

This is a very interesting question from the reviewer. What makes the situation for general depth difficult is that when you add depth you are composing activation functions. For example for a 3-layer network, employing $F$ as activation, the structure of the network becomes $W_3F(W_2F(W_1x + b_1) + b_2) + b_3$, where $W_i$, $b_i$ denote the weights and biases of each layer. In order to apply our theory to this situation, we would need to understand whether the composition of $F$ with itself can form a Riesz basis. In general, if $F$ forms a Riesz basis then there are no conditions guaranteeing that $F \circ F$ will satisfy the conditions of a Riesz basis. As far as we know there are no conditions guaranteeing that $F \circ F$ is a Riesz basis when $F$ is. Understanding how to generalize the theory to deep networks is something we are considering for future work.

It would be interesting to investigate whether gradient descent can recover the 2-layer INR weight settings that the authors use for their constructions in Thms 3.4 and 3.10.

This is also a very interesting question from the reviewer. Understanding whether gradient descent can actually converge to the weights that allow for thms. 3.4 and 3.10 to be valid is an important question, though we feel a theoretical proof showing this is out of scope for this work. We do thank the reviewer for bringing this up and we hope to take it up in a future project. Empirically it seems to be the case that gradient descent/Adam is converging to the optimum weights as we find from our experiments that sinc INRs are often giving the best PSNRs.

Dear reviewer, we made the changes you had asked for and uploaded a revision of our paper on the 15th of November. We were wondering if you had the chance to review our changes and if you are happy with it. It would be great to get some feedback as to whether the changes we made were all done appropriately. We thank you for taking the time to review our paper and greatly appreciate your feedback.

We would like to thank the reviewer for their valuable comments for improving our paper. Please find our answers below.

Clarification of Notation: Regarding the mathematical notation, it would enhance clarity to emphasize the argument 'x' in Equation 3...

Thank you for the suggestion. We have made this change. Please see the revised version of our paper.

How does increasing the depth of the neural network affect the theoretical guarantees associated with Riesz basis functions? Could you elucidate on potential failure modes of the sinc activation function? Are there particular scenarios where traditional activation functions might outperform sinc?

This is a very interesting question from the reviewer. What makes the situation for general depth difficult is that when you add depth you are composing activation functions. For example for a 3-layer network, employing $F$ as activation, the structure of the network becomes $W_3F(W_2F(W_1x + b_1) + b_2) + b_3$, where $W_i$, $b_i$ denote the weights and biases of each layer. In order to apply our theory to this situation, we would need to understand whether the composition of $F$ with itself can form a Riesz basis. In general, if $F$ forms a Riesz basis then there are no conditions guaranteeing that $F \circ F$ will satisfy the conditions of a Riesz basis. As far as we know there are no conditions guaranteeing that $F \circ F$ is a Riesz basis when $F$ is. Understanding how to generalize the theory to deep networks is something we are considering for future work.

Our theory implies the sinc activation is optimal for fitting signals in $L^2$. However, for more general function spaces such as $L^1$ the sinc activation would not be a good choice, as there is no $L^1$ sampling theorem for sinc. Thus for target signals that lie in $L^1$ but not $L^2$ sinc would not be guaranteed to provide a good choice for interpolation. We remark that almost all signals used in practise lie in $L^2$ so this is not a real disadvantage.

In general, activations such as sinc are useful has they have a bandwidth hyperparameter that allows sinc networks to fit high frequency target signals. This is why they are able to overcome spectral bias. However, in contexts where one is trying to fit a very low frequency signal, such as image classification, sinc will not add much value and this is generally why you don't see activation such as sinc, Gaussian or sine being used in image classification.

Considering that state-of-the-art INRs often employ positional encoding (which by themselves can be deemed as an activation function), how does the sinc activation function interact with common positional encodings like random Fourier features[1] or sinusoidal encodings[2]?

We have compared image reconstruction performance against RFF positional encodings in Figure 1. As shown, sinc activations outperform positional encodings by a significant margin. It has also shown in [1] and [2] that Gaussian and Wavelet activations also outperform RFF positional encoding despite the fact that practitioners often tend to use RFF encodings in INRs out of habit, although strong empirical evidence suggests against it. We tried using sinc activations in conjunction with the RFF positional embeddings, but observed that this setting deteriorates the performance of the RFF embeddings.

[1] - Ramasinghe, Sameera, and Simon Lucey. "Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] - Saragadam, Vishwanath, et al. "Wire: Wavelet implicit neural representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Moreover, is there a positional encoding strategy that effectively incorporates the sinc function?

This is an excellent question. Recent research, notably [3], has provided compelling evidence that the effectiveness of positional encodings need not be exclusively tied to a Fourier perspective. They demonstrate that non-Fourier embedding functions, such as shifted Gaussian functions, can be effectively utilized for positional encoding. These functions are characterized by having a sufficiently high Lipschitz constant and the ability to generate high-rank embedding matrices, attributes that are shown to achieve results comparable to Random Fourier Feature (RFF) encodings.

Building on this, research in [4] further confirms that shifted Gaussian functions with spatially varying variances can surpass the performance of RFF encodings. Given that sinc functions also exhibit these desirable properties, they can be feasibly employed as shifted basis functions for high-frequency signal encoding.

To explore this, we developed a sinc-based positional embedding layer. For a 2D coordinate $(x_1,x_2)$, each dimension is embedded using sinc functions:

where $t_1, \dots, t_N$ are equidistant samples in $[0,1]$. Then, these embeddings are concatenated to create the final embedding as,

In a comparative study using the DIV2K dataset for image reconstruction, our sinc-based positional embedding layer demonstrated superior performance to an RFF-based layer, as shown in the table below. The shown metrics are averaged values over the dataset:

This result indicates that sinc-based positional embeddings offer a promising alternative to RFF encodings. We have included a short discussion of sinc positional embeddings in the supplementary material and anticipate this will stimulate further research in this area.

[3] - Zheng, Jianqiao, Sameera Ramasinghe, and Simon Lucey. "Rethinking positional encoding." arXiv preprint arXiv:2107.02561 (2021).

[4] - Ramasinghe, Sameera, and Simon Lucey. "A learnable radial basis positional embedding for coordinate-MLPs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.