SPDER: Semiperiodic Damping-Enabled Object Representation

We thank the reviewer for their thoughtful feedback on our submission. Please find our answers and responses below.

> I think that additional experiments could be used as there are other coordinate-based network architectures which claim improvements over positional encoding and SIREN architectures. […] I believe a better fit would be [1] or [2], which show that simple networks with different activations, structures, or positional embeddings can improve on the FF/SIREN architectures.

Following the reviewer’s suggestion, we ran additional experiments applying SPDER activations to the models from both of the suggested baselines: GaborNet from MFN [1] and Spline Positional Encoding [2]. Then, we evaluate these new models on the same image dataset as the original MFN paper [1].

PSNRS

To highlight the difference, SPDER beats the best GaborNet loss in 800 steps and the best Spline loss in 300 steps. SPDER’s representation is lossless because the loss is so small it doesn’t show up on an 8-bit scale (not true for the others, see Appendix A.13). The 5-layer SPDER (which we tested in our paper) has significantly higher PSNRs than the others in very few training steps, meaning it is also very efficient.

We will be sure to include these results in the main paper.

> I'm not sure why Instant-NGP is chosen here, as it is a hybrid implicit-explicit representation and is very different from the other coordinate-based networks.

Its paper says it is “state-of-the-art quality on a variety of tasks” related to image representation. We wanted to show our simple MLP can beat this complicated setup which requires a heavy GPU setup.

Also stated in Appendix A.8, our method could also be combined with a hierarchical one like Instant-NGP, as it is more efficient.

Comparisons on radiance fields and SDFs

We observed that SPDER works for signals that can be well approximated in a low dimensional representation over a frequency domain. For example, an image can be well approximated by using the discrete cosine transform (JPEG). Similar methods exist for audio and video. To the extent of our knowledge, domains like SDFs or radiance fields do not exhibit a similar property. We hypothesize that this fact renders SPDER ineffective in these domains (mentioned at the bottom of page 4). This is likely an instance of the “no free lunch” theorem; no single architecture can perform the best on all modalities.

We anticipate there can be future work on projecting these modalities into a vector space where SPDER’s activation functions can perform effectively. We believe this is reasonable as current methods for SDFs and radiance fields are generally divided into two parts: 1) preprocessing/conditioning 2) MLP. Our current implementation only uses the latter.

The main claim of our paper is that existing models are prone to the inductive bias of not being able to encode all frequencies, which we show our simple MLP overcomes. This is due to the “damping factor” in the nonlinearity (i.e. the square root in $\sin(x) * \sqrt{|x|})$ which allows the network to remember where inputs came from. This is most evident in the image case.

We thank the reviewer for their thoughtful feedback on our submission. Please find our answers and responses below.

> The method itself seems very simple to code (although no code was provided, and no training details given). Overall the paper has the strengths of simplicity and should be very easy to use in practice.

The code was provided in the supplementary material, along with the appendix. Training details are included in Appendix A.9 (Reproducibility).

> There is a statement buried in page 4 that says: "For example, they are less successful at novel view synthesis, where the input direction is polar.". However, no examples of performance on novel view synthesis was provided. Can the authors comment on this?

We observed that SPDER works for signals that can be well approximated in a low dimensional representation over a frequency domain. For example, an image can be well approximated by using the discrete cosine transform (JPEG). Similar methods exist for audio and video. To the extent of our knowledge, domains like SDFs or radiance fields do not exhibit a similar property. We hypothesize that this fact renders SPDER ineffective in these domains. This is likely an instance of the “no free lunch” theorem; no single architecture can perform the best on all modalities.

The main claim of our paper is that existing models are prone to the inductive bias of not being able to encode all frequencies, which we show our simple MLP overcomes. This is due to the “damping factor” in the nonlinearity (i.e. the square root in $\sin(x) * \sqrt{|x|})$ which allows the network to remember where inputs came from. This is most evident in the image case.

> I am concerned that this method tends to overfit heavily and easily to a given particular input image/signal, and evidence for generalization was very limited to (as far as I can tell) a single video example. How sensitive is the model to noise in the input, for example? This can be important for real-world applications.

The implicit neural representations should “overfit” to the signal or the object it is being trained on, as the goal is essentially to learn a mapping from pixel position to pixel value for a single object. The desired generalization is between the pixels in an image or between frames in a video. Our experiments demonstrate this generalization effect in the image superresolution task (Section 4.3, Appendix A.5), audio interpolation (Section 4.4, Appendix A.6), and video interpolation (Section 4.5, Appendix A.7).

Regarding video examples, we report on two videos (not one). They are “Big Buck Bunny” and “Bikes”, which are from skvideo and are standard in video computer vision research (Appendix A.7).

> I am not an expert in this space, but one question I have is whether the baselines are strong. SIREN is the most commonly used baseline. Maybe the authors can provide some perspective in this regard.

We compared SPDER to the most popular coordinate-based MLP models we found in the existing literature. Most of these models are simple neural networks. We also compared our work to Instant-NGP (Section 4.2 Image Representation), which uses a much more complicated setup and is state-of-the-art for many INR tasks.

> Lastly, I would recommend the authors, in the spirit of academic inquiry, to tone down the language a little bit.

We will change the wording accordingly.

We thank the reviewer for their thoughtful feedback on our submission. Please find our answers and responses below.

> Do not prove the derivatives of different variants of proposed method.

We prove how activation functions of the form $\sin(x) * \delta(x)$ for any sublinear $\delta(x)$ suffice for our task in Appendix A.1 (Proof of Lipschitz Bound).

> Can you evaluate the proposed method on more complex images? Consider to perform formal evaluations on popular benchmarks, such as CelebA, UHDSR4K and Vimeo-90k to demonstrate its superiority.

We report our results on the DIV2K dataset (“DIVerse 2K resolution high-quality images”—commonly used for super-resolution and representing complex images) in Table 1. In fact, it is typical for the signal representation literature to evaluate only a few samples [1, 2, 3]. Our sample size is 800, which is far more than the largest one [3] with N=32.

[1] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020. [2] Fathony et al., Multiplicative Filter Networks, ICLR 2021. [3] Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020