FreSh: Frequency Shifting for Accelerated Neural Representation Learning

The experiment in Appendix A seems for demonstrating that weights of embedding layer could not be trained sufficiently by gradient descent. However, why is the optimization algorithm SGD? Meanwhile, is the index L2 norm （Eq. 8） used by the paper (720) meaningful? This experiment should be illustrated more clearly.

Thank you for pointing that out. In the experiment in Appendix A, we used the FreSh codebase. As such, we actually used Adam as the optimizer, not SGD. The reference to SGD in the text was intended as a general term for variants of the original SGD algorithm. We clarified this in the revised version of the paper.

We have updated the text to better clarify the experimental setup and provide a more detailed explanation of the experiment, including how the frequency is measured. The L2 norm is used as a measure of vector length in this context.

(...) Fresh might shift some special spectrum distribution that enjoys the fastest learning speeds of MLPs towards the target signals. So the detailed description of this special spectrum distribution or the validation of the existence of this special spectrum might be an interesting problem. This point will not affect my scoring. If authors could try to find this special spectrum and do more exploration about this problem, I think that it will further deepen our understanding of deep learning.

FreSh operates under the assumption that the best spectrum for efficient learning is the one most similar to the spectrum of the target signal. Based on our results, this appears to be a reasonable heuristic. However, we acknowledge that it may not be the only mechanism influencing model performance, especially in the case of high-frequency models like Wire. Given the complexity of this topic, we hope to explore this further and provide a more comprehensive explanation in a follow-up work.

Thank you for your feedback. We address the particular comments below.

(...) spectral bias can be overcame by special initialization and training dynamics adjustment, such as reparameterization and batch normalization. Moreover, as stated in the paper that Fresh is a simple initialization technique in line 024, Fresh should compare to the previous initialization method like “From Activation to Initialization: Scaling Insights for Optimizing Neural Fields”. Thank you for referring us to the insightful paper, “From Activation to Initialization: Scaling Insights for Optimizing Neural Fields” (FAI). We have updated the related work section to include this reference.

We note that the initialization method proposed in FAI can be used in combination with FreSh. To verify this, we trained Siren on one image from each dataset used in our paper, using both the Siren initialization and the one proposed in FAI. Our results show that FreSh improves performance similarly to FAI initialization. Importantly, both methods can be used together, yielding even better results.

We believe this simplified experiment demonstrates the potential of combining orthogonal INR improvement methods. A full experiment (including all images and multiple seeds) will be included in the paper at a later stage.

There lacks a reliable theoretical or experimental link between the key observation (line 019) and Fresh (line 241，Table 2). More concretely, there need an explanation that the adjustment of embedding layer hyperparameters could affect the whole model's spectrum. A more quantitative expression would be better. The illustration in Fig. 1 seems to be vague. Clarification of this point would certainly raise my opinion of the paper.

We updated the appendix with examples illustrating how changes to the frequency parameters of INRs affect their spectra (Figure 8, page 19). Additionally, we performed an experiment where we initialized 5 different Siren models with varying values of $\omega_0$ and computed the average frequency, weighted by the normalized frequency vector. This experiment demonstrates how increasing hyperparameter values increases the average frequency of a model (see table below).

Unfortunately, we find that explaining the frequency spectrum in detail would require more space than is available in Section 1 / Fig. 1. Instead, we provide a visual explanation of the spectrum in Figure 2 (which we revised) and have added a reference in the caption of that figure to additional spectrum examples (Figure 8, page 19).

Some experimental results show that Fresh could not provide a better result even worse than baseline models, such as results with time input in Table 5 and in Table 6. Intuitively, Fresh should not obtain worse results. It might show that there are several unclear relationships. Solving the weakness 2. might help this issue.

Regarding the video experiments (Table 5), FreSh improves results when time is not used as an input. We suspect that due to the different characteristics of video signals along the time direction, it requires different treatment than spatial dimensions. Please refer to our discussion of the time input in our general comment.

In the NeRF experiments (Table 6), FreSh does indeed sometimes result in worse performance than the baseline. As this is the first work aiming to perform frequency matching, our method is not yet perfect and occasionally selects suboptimal configurations. We plan to prepare a follow-up work to further analyze the relationship between frequency and performance, particularly in high-frequency models like Wire and NeRF, with the goal of improving FreSh based on these new insights.

Paper seems to have a lot of overlap with SPDER https://arxiv.org/pdf/2306.15242 (especially in theory, and that the untrained model output aligns with end reconstruction) and should compare the baseline against it.

Thank you for highlighting the SPDER paper, which, like our work, emphasizes the importance of spectral bias. However, SPDER’s approach is distinct from ours. SPDER addresses spectral bias by designing an architecture optimized for learning various frequency components, similar to Siren (Sitzmann et al., 2020), WIRE (Saragadam et al., 2023), and Finer (Liu et al., 2024). In contrast, FreSh enhances existing architectures by configuring their frequency-controlling hyperparameters, making it complementary to architectural innovations like SPDER.

Given SPDER's promising results, we attempted to test it as a baseline. However, we identified limitations in its applicability, particularly to high-resolution images (see experiment below) and tasks like NeRF (see SPDER paper).

Experiment: Re-implementing SPDER in the FreSh codebase

We replicated a simplified version of Table 3 from our paper, testing on one image per dataset under the same training settings. We tested SPDER, Siren ($\omega_0=30$) and Siren + FreSh:

In our setting, SPDER underperformed, which we don’t believe warrants using it as a baseline for our study. We speculate the lower performance of SPDER can be attributed to the following:

PSNR calculation bug in SPDER
SPDER calculates PSNR as

$\text{PSNR}=20 \log_{10}(\frac {\text{MAX}_I} {\text{MSE}})$

instead of the correct

$\text{PSNR}=20 \log_{10}(\frac {\text{MAX}_I} {\sqrt{\text{MSE}}})$

See SPDER GitHub for the relevant code. $\text{MAX}_I$ is the maximum possible pixel value.

Overparameterization dependency of SPDER
SPDER’s original experiments involved low-resolution images and large networks (5 hidden layers). In contrast, our setup with high-resolution images and smaller networks (3 hidden layers) led to a performance degradation, suggesting that SPDER relies on relatively high parameter counts to perform well.

Evaluation protocol
SPDER evaluated on the entire image, whereas we tested on unseen pixels.

Other baselines should be (Ramasinghe & Lucey, 2022) which suggests Gaussian activations. It is mentioned in the paper but not reported on to the best of my knowledge.

We report the results of WIRE (Saragadam et al., 2023), which is a direct generalization of Gaussian activations and has been shown to perform better.

Instant NGP (HashGrid?) may also be used in the image baseline, as it excels in high-resolution images but is not included here for that.

Thank you for the suggestion. Our experiments are set up in different repositories, so we were unable to quickly add InstantNGP as an image baseline. However, we have updated the paper to include InstantNGP for the video experiment (see table below). We observe that InstantNGP does not generalize well to unseen pixels, which aligns with findings from Mihajlovic et al., 2023.

Overall, SIREN seems to be a very weak baseline model. I understand that tuning the omega_0 parameter can improve results, but I don’t believe this to be as significant of a contribution on top of the existing model.

Although Siren was proposed a few years ago, we believe it remains an important baseline model, still used in recent works like ResFields (Mihajlovic et al., 2023) and SPDER, and achieving state-of-the-art results on video (Mihajlovic et al., 2023), which we further improve.

Often, when used as a baseline, the $\mathbf{\omega_0}$ parameter of Siren is not fine-tuned, leading to suboptimal performance. We consider this an important issue, as it complicates fair comparisons with novel architectures. Including Siren in our work aims to encourage researchers to compare against well-tuned Siren models, increasing confidence in new methods.

Furthermore, we report improvements for models beyond Siren, such as Fourier Features, Finer, and InstantNGP.

Thank you for your feedback. We address the particular comments below.

The experimental evidence is extremely weak in my opinion and does not demonstrate a significant increase that would warrant other researchers’ usage of this method. Most results show improvements of less than 1dB, which is essentially identical to the naked eye.

We agree with the reviewer that the absolute differences are small. This is due to our decision to train until convergence (addressed in the next paragraph). To demonstrate that these differences are significant, we repeat experiments three times and report standard errors. For instance, in the image experiments, the standard error is very small (approximately 0.03), meaning the absolute differences are large relative to the standard error.

The results in Table 1 are concerning. After 15,000 steps the difference between the Base Siren model and using Fresh have virtually no difference (PSNRs of 31.18 and 31.81).

The differences are significant due to the small standard errors. We chose long training times to allow models to converge, ensuring a fair comparison. Shorter training times would artificially inflate differences, but could lead to misleading results.

Our training length aligns with standard practices in the field; for example, Siren (Sitzmann et al., 2020) uses 10,000 steps for image experiments. Similarly, works like WIRE (Saragadam et al., 2023), Finer (Liu et al., 2024), and Fourier Features (Tancik et al., 2020) train INRs for several thousand steps.

Table 3 - Why does FreSH have no improvement on Finer and Finer with k=0 and most samples? If it is only applicable to the SIREN and Fourier features model it would be way too limited to warrant an accept as these are several years old baselines.

While FreSh does not improve Finer in the image setting, Finer with k=0 shows improvements for most datasets. Additionally, FreSh achieves state-of-the-art results on video (table 5) and provides multiple improvements over baselines in the NeRF task (Siren, Fourier, Finer with k=0, Hashgrid). Furthermore, Siren and Fourier features remain important baselines whose performance is often underreported due to suboptimal embeding configurations - an issue FreSh directly addresses.

Moreover in Table 6, overall the results are not impressive as most PSNRs increase by < 1dB (i.e. Finer has virtually no change)

The results are significant due to the small standard errors. The absolute differences appear small because we train models to convergence, ensuring a fair comparison.

Table 8, why does Fresh hurt the results when you include time? It seems as expected input for video.

Time is always incorporated via the weight modifications in ResFields. However, we observe a performance decrease when it is also provided as an input to the embedding layer. Please see our general comment for a detailed discussion of this phenomenon.

I thank the authors for their diligence in the responses. I am impressed with the depth of their justifications and am willing to update my score to recommend acceptance.

However, I will note:

• The PSNR changes are concerningly low, and I am not convinced by the argument that the standard errors in the change being small compared to the mean change justifies the method's use. (In fact, this would imply that one can say with confidence that the benefits are minor). As a suggestion, the authors should add a convincing reason in the abstract/intro as why other researchers should opt to use this (i.e. it will speed up training by x% on average for high resolution images). But just my personal take.

Thank you for your feedback and listing all our typos (they are fixed). We address the particular comments below.

I am curious if you can say more about these models (NeRF and Wire) which use very high frequencies. What purpose do these high frequency components serve if they are not reflected in the training data? Why don’t these models perform poorly as FreSh would predict?

Our hypothesis is that these models (NeRF and Wire) perform well despite not having a “reasonable” spectrum due to mechanisms other than frequency alignment. As far as we know, it’s not well explained in the literature. As they don’t depend on frequency alignment, FreSh cannot predict the performance of these models accurately.

For further discussion on high-frequency models, please refer to our general comment.

You mention that your FreSh measurements are noisy due to random initialization, so you average across 10 initializations. [1] find that SIREN’s random initialization of the first layer has a significant effect on the final PSNR (Appendix A.6). I would be interested to see if your method could be further improved by using FreSh to select not only the hyperparameters, but also the random initialization to start from.

We find that the noise in the Wasserstein distance measurement primarily arises from the random sampling of images from the target signal in video and NeRF experiments, rather than from weight randomness. Additionally, we observe that the variation in final PSNR is negligibly small in our setting (with 3 hidden layers and 256 hidden neurons). If we were to use networks with higher variability in final PSNR (such as deeper and narrower MLPs, as noted in the paper cited by the reviewer), the impact of random initialization would be more pronounced. In such a setting, it could be possible to choose the model initialisation with FreSh, e.g. by choosing the one with the lowest Wasserstein distance.

In the appendix you say that there is high variability in the model configurations selected by FreSh, even within a single dataset. But how much of this variability is random vs. actually important for performance? I would be curious to see how per-image FreSh compares to a dataset-wide FreSh: how much of the advantage of FreSh comes from optimizing for each individual image, and how much can be achieved just by making an appropriate hyperparameter choice for each dataset.

Due to the repeated measurement of the Wasserstein distance, the selection of embedding hyperparameters is highly consistent and not random. The question of how much of FreSh's advantage comes from optimizing for each individual image versus selecting an appropriate hyperparameter value for the entire dataset is intriguing. However, it is not clear which specific hyperparameter value from those proposed by FreSh should be chosen for a dataset-wide approach, and the answer would likely depend on the specific strategy.

Thank you for your feedback. We fixed the minor presentation comments and addressed the rest below. We redesigned Figure 2 to improve its clarity.

The description of a vector spectrum by summing the DFT appears a little non-standard (L285-7). Could the authors provide a reference for this approach (potentially outside the INR literature?)

To the best of our knowledge, there is no standard method for converting a 2-dimensional DFT into a vector. Our goal was to simplify the 2D DFT into a representation that ignores signal directionality (note that the frequency spectrum representation remains the same for a transposed image), since randomly initialized frequency embeddings do not make this distinction. Given that transposing the image is equivalent to transposing its Fourier representation, we find that summing along the diagonals works well for our purposes. Additionally, collapsing the 2D DFT into a 1D vector allows us to use an explicit formula for the Wasserstein distance, rather than relying on an approximation.

If the reviewer feels it would improve clarity, we are open to renaming the "spectrum vector" to "frequency vector" in the final version.

In Appendix C it is mentioned that the for image regression the same image is resampled 10 times due to randomness of the model output. The same network should be deterministic in output - does the randomness occur due to re-sampling the network with reinitialisation?

Yes, the randomness arises from using a different weight initialization in each iteration. We updated the FreSh algorithm in the appendix to clarify that model weights are reinitialized for each output calculation.

The paper mentions that other activations are addressed L131. Have experiments on non-periodic functions (e.g. Gaussian) been performed?

Both Finer (Liu et al., 2024) and Wire (Saragadam et al., 2023) utilize non-periodic activations, with Wire being a generalization of Gaussian activations.

Could the authors provide details on the compute required for Fresh on more complex signals like NeRF (L414-6 notes that this is negligible for images).

The compute required for FreSh is similar across all modalities, including NeRF. FreSh consists of two main steps:

1. Initializing INR models (10 in our case) and performing inference.
2. Calculating the DFT, spectrum vectors, and Wasserstein distance.

Since step 1 always outputs images, the compute required for step 2 doesn't depend on modality. The compute for step 1 (10 initializations and inferences) is negligible compared to training, which involves thousands of inferences.

Table 5 records video results. It mentions 'Results for NeRF are provided for reference only as it is incompatible with FreSh'. Could the authors clarify this?

In this context, "NeRF" refers specifically to Positional Encoding, not the NeRF task. We have updated the manuscript to use "Positional Encoding" for clarity.

Could the authors clarify what is meant by 'direction-dependent frequency magnitudes' and its importance for video (L510)?

This refers to the qualitative differences in signal characteristics along different directions (e.g., temporal vs. horizontal). For example, in videos, scene transitions often cause drastic changes in pixel values between frames, a phenomenon not typically observed along the spatial dimensions. To clarify, we have added Figure 16 (page 24), which illustrates these changes. For additional discussion please refer to our general response.