ASMR: Activation-Sharing Multi-Resolution Coordinate Networks for Efficient Inference

Thank you for the valuable comments and feedback.

Comparison to Grid-based Methods

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [3] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset.

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where direct comparison is not applicable. The main focus of our work is not to beat the state-of-the-art INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs without sacrificing their original reconstruction capabilities.

Clarification on Figure 2

We apologize for the unclear description and appreciate you pointing this out. The left and right figures are meant to show different sets of KiloNeRF and LoE model configurations. The intention of the left figure was to complement Table 1 (analytical relationship between MAC, #Params, depth) with empirical data. The model configurations are chosen such that all the parameter counts are within the same range across the 4 models and the correlations between MAC and #Params can be clearly demonstrated. On the other hand, the right figure is an extension of Table 2 (ultra-low MAC fitting results) showing the full MAC-PSNR curves of the models when having 3 to 6 layers. In our revised manuscript, we have separated the two figures and positioned them alongside the tables that they are meant to complement. Additional model configurations of the ultra-low MAC experiments are added to Appendix B accordingly.

Decoupling Between MACs and Depth

By width, we are referring to the number of hidden units in a network, which is independent of the choice of basis. If you are referring width to the number of unique inputs (coordinates) that each layer of the network needs to infer, then you are correct in that "doubling the size of the basis" would halve the number of layers and increase the width (number of unique inputs) by 2. However, as provided in the original manuscript, the derivation of ASMR's inference MAC is as follows:

$

\text{MAC} = \sum_{i=1}^LB^iM_i \leq M\sum_{i=1}^LB^i &= MB\frac{B^L - 1}{B - 1} \approx \frac{MB}{B-1}B^L = \frac{MB}{B-1}B^{\log_B N} = \frac{B}{B-1}MN \leq 2MN

$

where the MAC is bounded above by $2MN$ regardless of the choice of basis since $\frac{B}{B-1} < 2$ for all $B$. Hence, asymptotically, ASMR does indeed decouple its inference cost from the network depth. In fact, you may see from the right figure of Figure 2 in the original manuscript that the MAC of ASMR is not strictly but approximately constant in a practical setting.

Additional Video Results

We have tested ASMR on the UVG dataset, which is commonly used as a benchmark for testing video representations such as in the works of [1] and [2]. The results are consistent with our findings on the cat video presented in the original manuscript. Please refer to our General Response which contains additional experimental results on audio and 3D shapes.

Writing Issues

We have made the following changes (highlighted in red) in the revised manuscript:

• We fixed the typo and specified that a uniform base of 2 is used in the caption of Figure 1.
• We replaced the notations of base and cumulative base in Figure 1 so that they are in line with those defined in Section 3.1.
• We fixed the labeling issue of Equation 1.

References

[1] Adversarial Generation of Continuous Images, CVPR 2021

[2] Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, ICLR 2022

[3] Implicit Neural Representations with Levels-of-Experts, NeurIPS 2022

Thank the authors for the experiments on actual inference speed and clarification on how the MACs were profiled. I am more convinced now of the claims in the work regarding the computational cost. So I increased the soundness score by 1 and and confidence score by 1.

It's quite hard to raise the score from 6 to 8 and sadly due to ICLR's grading scale, it's not possible to give a 7. But looking at the additional experiments on audio and 3D shape and the reply to reviewer zbDF, I am inclined to a score slightly higher than 6. Although the results with video don't seem to be too promising (when the number of parameters is controlled). It is also worth detecting the reason behind this phenomenon.

Thank you for your comments. We appreciate your positive feedback and recognition of our work.

Latency

To demonstrate the improved inference speed of our ASMR method on resource-limited devices, we measured the rendering latency of a 512 $\times$ 768 image (same as Kodak) on various hardware platforms. These platforms range from high-end to low-end CPUs, including a server-level AMD 7413 2.65GHz 24-Core CPU, a desktop 3.1 GHz Dual-Core Intel Core i5 CPU, and a Quad-core Cortex-A72 (ARM v8) 1.8GHz CPU of Raspberry Pi 4 model B. The reported latency is an average across 10 runs. The instantiation of each model is the same as the one mentioned in Section 4.3 of the manuscript, with parameter counts of around 130K. It can be observed that ASMR consistently speeds up SIREN by $>3\times$ in terms of latency, and the speed-up becomes more significant when tested on devices with reduced computation power (viz. $>4\times$ on Raspberry Pi). Processes that are too computationally intensive and result in hanging or being automatically killed due to excessive RAM usage are indicated by (Failed).

Other Data Modalities

To further demonstrate ASMR's ability to generalize across multiple data modalities, we have tested on the Librispeed dataset (Audio), UVG dataset (Video), and ShapeNet dataset (3D shapes) and included the results in the general response. Consistent with our findings in the original manuscript, our new results show that ASMR successfully reduces the inference cost of a SIREN model without sacrificing (or sometimes brings an even better) reconstruction quality. In particular, we found that ASMR performs better with low-dimension data such as audio, with an exceptional improvement over vanilla SIREN. For more details, please kindly refer to the general response at the top.

MACs Calculation

DeepSpeed Profiler is sufficient in calculating the MACs of all the models except for ASMR. To obtain the per-sample MAC, we divide the MAC outputted by the get_model_profile() function by the size of the input. For ASMR, we first obtained the MACs of each individual layer and modulator using DeepSpeed Profiler, then manually amortized the per-layer inference cost across the grid by the resolution of the layer.

Clarification on Figure 1

Your understanding is correct, and this is exactly why ASMR could reduce the inference cost of SIREN by orders of magnitude. ASMR does not only achieve activation-sharing on the INR backbone but also on the modulators. We would like to highlight that even though the modulations also exhibit a repeated pattern, their pattern is actually different from that of the previous-layer activation (as indicated by different colors in Figure 1). The summation of both results in activations that are unique to each coordinate. Intuitively, one may think of modulations as a set of phase shifts to the sine activations generated by the SIREN backbone. ASMR does so efficiently by finding a common set of phase shifts that can be applied repeatedly to generate a unique combination of frequency supports representing the data.

As for the superior representation performance of ASMR over SIREN, we argue that it is because SIREN suffers from spectral bias while ASMR does not. As illustrated in Appendix G, SIREN learns the lower frequency information first, then progressively adds higher frequency details. For data with abrupt changes in value (i.e. very high frequency) such as non-analytical 3D shapes, it may even fail to converge and get stuck with a "smooth'' representation. On the other hand, by injecting coordinates that repeat themselves (i.e. periodic) in progressively higher frequencies, ASMR could tackle a much wider band of frequencies from the start of the training. Its discretized nature also permits it to learn any sort of data as long as it is within its grid resolution.

Writing Issues

We have made the following changes (highlighted in red) in the revised manuscript:

• We replaced "inference bandwidth" with "inference throughput"
• We fixed the labeling issue of Equation 1.

Other Modulation Methods

We acknowledge that there exists a wide variety of methods for implementing modulation and have revised our manuscript by including the missing references (highlighted in red). However, we would like to highlight the motivation behind choosing bias modulation over other methods: its effectiveness and simplicity in terms of implementation and inference cost. We aim to introduce ASMR as a straightforward addition to a SIREN model, which can significantly reduce its inference cost without adding much complexity. Compared to advanced methods like variational modulation in [3,4] or low-rank modulation in [1], bias modulation is the best fit for these characteristics. Intuitively, bias modulation used in ASMR can be interpreted as phase shifts that describe the variation of decomposed coordinates at each hierarchical level. The above ablation study shows that bias-only modulation can effectively model such variation and even outperform the more complex low-rank modulation technique, which incurs more parameters.

Comparison to Modality Specific INRs

We fully agree that each modality has its own state-of-the-art (SOTA) that excels in efficiency and quality. However, we argue that most of these SOTAs are usually overly modality-specific in the sense that their tricks are specialized to excel only in the tasks that they are intended for. For instance, [1,2] take special measures for the time dimension and grid-based methods fail to learn global latent representations. In contrast, general backbone models like SIREN are applicable to all modalities and downstream tasks. Hence, we see great values in proposing ASMR as a simple, modularized addition to SIREN that reduces its inference cost while inheriting its generalization capabilities. The modularity of ASMR also makes it orthogonal to other INR techniques such that they could be potentially combined for specific modalities or tasks.

Comparison to Instant-NGP and Intention of Section 5.2

We are aware of grid-based INRs such as Instant-NGP (Megapixel image, SDF, NeRF) and NVP (Video) that have demonstrated excellent training and inference efficiencies. However, just like [5] stated, grid-based methods come with the trade-off of losing the ability to learn global latent representations of data. Intuitively this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent. This takes away the INR's ability to learn abstracted features of the data and hence jeopardizes INRs' potential to be an implicit representation for downstream tasks or an encoder for an entire dataset. Indeed, it was our intention to include Section 5.2 for empirically demonstrating this on a meta-learning task to encode an entire CIFAR-10 dataset.

As such, we believe that grid-based INRs and purely implicit INRs should be seen as complementary representations where a direct comparison is not applicable. The main focus of our work is not to beat the SOTA INRs for specific modalities, but rather to propose a generic technique for reducing the inference cost of purely implicit INRs without sacrificing their original reconstruction capabilities.

References

[1] Adversarial Generation of Continuous Images, CVPR 2021

[2] Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, ICLR 2022

[3] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023

[4] Versatile Neural Processes for Learning Implicit Neural Representations, ICLR 2023

[5] Implicit Neural Representations with Levels-of-Experts, NeurIPS 2022

Other Data Modalities: Audio and 3D Shapes

To demonstrate the versatility of ASMR in handling different modalities, we present additional experimental results on audio and 3D shape datasets (please kindly refer to the general response at the top). We found that ASMR performs exceptionally well in particular on low-dimensional data like audio. Not only does it exhibit low MACs behavior, but it also leads to faster convergence and remarkably better results vs vanilla SIREN.

It is a valid concern that ASMR's discrete nature may make it difficult to extend to 3D data. However, we argue that as long as the data is discretizable (which is the case for almost any data including SDFs and neural fields), ASMR could effectively become a low-inference cost counterpart to a vanilla SIREN model. For instance, we trained an ASMR on 3D occupancy grids of random objects in ShapeNet, and found that matches the quality of SIREN while operating at a significantly lower inference cost (for more details, please refer to the general response at the top).

Additional Video Experiments

We present additional results for the video-fitting task on the UVG dataset, which is a commonly used benchmark for testing video representations. The results are consistent with our findings on the cat video presented in the original manuscript. For details please refer to our general response at the top.

Thank you for the comments and suggestions.

Comparison to Multi-Scale INR [1]

In particular, we appreciate reviewer zbDF for drawing the relation between ASMR and [1] for the similarity in ideas. In this regard, we would like to emphasize the major improvements of ASMR over [1]. To demonstrate these improvements, we conduct a thorough ablation study on the Cameraman dataset, contrasting ASMR and [1]. Specifically, we analyze the following key differences: (1) Multi-Scale Coordinates vs. Coordinate Decomposition, and (2) FMM vs. Bias-only Modulation. Each model consists of 9 layers with a hidden size of 64 and uses a base 2 partition. All of them are trained for 10K iterations.

We believe that there are two major differences between ASMR and [1]

1. Multi-Scale Coordinates vs. Coordinate Decomposition

ASMR is motivated by the observation that data often exhibits both hierarchical AND periodic patterns. As a result, ASMR chooses to decompose coordinates like a change of basis operation. This is contrary to [1] which simply uses progressively finer coordinates. As a consequence, ASMR only injects $B_i^d$ unique coordinates into a layer $i$, where $B_i$ is the basis of partition of layer $i$ (assume $B_i$ is uniform across dimensions) and $d$ is the dimension of the data, while multi-scale INR injects $C_i^d$ unique coordinates, where $C_i$ is the resolution (or equivalently cumulative basis) of layer $i$. Note that $C_i$ grows with each layer while $B_i$ is a predefined constant and $B_i \ll C_i$ in later layers. Even though activation-sharing is also employed in [1], the inference cost of an INR with multi-scale coordinates is not truly decoupled from its depth, as the modulation layers need to operate on increasing resolution (i.e. increasing $C_i$). Hence, when the full resolution is large, multi-scale INR will incur a noticeable overhead for calculating modulation while ASMR's forward pass of the modulator maintains a negligible constant, regardless of the choice of modulation as shown in the table above. In addition to reducing the number of MACs, the use of coordinate decomposition in ASMR introduces an additional inductive bias that takes into account of underlying periodic structure of data, alleviating the spectral bias that is often encountered by SIREN as illustrated in Appendix G. Our ablation study shown above demonstrates that ASMR's coordinate decomposition significantly enhances performance compared to multi-scale coordinates as well as reducing MACs.

2. FMM vs. Bias-only Modulation

Although ASMR and [1] both use modulation techniques, they have different motivations and objectives. The Factorized Multiplicative Modulation (FMM) proposed in [1] does not rely on positional information and only uses parameters predicted by the hypernetwork, which represents a specific style or image context. On the other hand, the hierarchical modulation proposed in ASMR is designed to easily incorporate decomposed coordinates using bias-only modulation that results in minimal overhead. One can implement FMM using multi-scale coordinates, similar to ASMR, where a modulator is used in each layer to generate positional-dependent modulation parameters. However, we argue that the multi-scale coordinate representation proposed in [1] is not efficient when combined with FMM. For instance, for an image with a resolution of $512^2$ (Cameraman dataset), the modulator of FMM in the final layer needs to operate on the full resolution. This means that the final FMM modulator has to modulate the corresponding weight matrix for each of the $512^2$ coordinates, introducing a significant overhead. In contrast, the modulator in ASMR only needs to operate on a small constant number of bases ($2^2$ when the base is 2), followed by an efficient upsampling operation. As shown in the above ablation study, AMSR, using the combination of coordinate decomposition and bias-only modulations, achieves the lowest MACs and the best reconstruction quality, significantly outperforming [1].

References

[1] Adversarial Generation of Continuous Images, CVPR 2021

Sorry for the late reply. Thank you very much for the reply. However, I still have the following concerns. I believe most of them can be address by editing.

First, comparing with modality-specific INRs is for addressing and discussing the limitations of the method and does not hurt the paper's contribution. Rather, it is good for the readers, and it is a benefit to the paper.

• Furthermore, I do agree that this work can be used for 3D-SDF. But is it applicable for 3D rendering? If it is not, discussing it in the limitation section will be great (since 3D rendering is one of the most important applications in INRs). I don't think this is not a negative point or hurts the paper's contribution. Addressing it in the limitation or discussion section will be much better and even strengthen the paper.

Second, I still believe that we need a comparison with Instant-NGP or give a better reason that we need efficient MLP-based INRs (I have tried to address this reason in the paragraph below).

• It is hard to understand the intention of the following sentence, “Intuitively, this is because features learned by grid-based methods are highly concentrated on the explicit grid instead of the implicit network, which is overly position-dependent.” Why is this a negative thing about grid-based INRs, and why should we use MLP-based INRs instead?
• Also, the performance of meta-learning on Instance-NGP may be affected by the meta-learning method and setup. Furthermore, I still think the comparison on the meta-learning scenario does not give information to understand the benefit of this work. If one can compare instant-NGP on the meta-learning scenario, why do the authors not compare it with the main experiment (e.g., PSNR, MAC).

For this paper, I think it is better to highlight the benefits of MLP-based INRs rather than show the meta-learning experiments. For instance, there are several works that use MLP-based INRs for downstream tasks [1,2,3,4,5]. I think the authors should claim this is the reason why the community should consider efficient MLP-based INRs (therefore, we don't need to compare with Grid-based INRs, since it is non-trivial to use Grad-based INRs for downstream tasks + MLP-based INRs need more efficiency for better downstream tasks).

[1] From data to functa: Your data point is a function and you can treat it like one
[2] COIN++: Neural Compression Across Modalities
[3] Modality-Agnostic Variational Compression of Implicit Neural Representations
[4] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion
[5] Locality-Aware Generalizable Implicit Neural Representation

Third, (sorry for the late additional question, this does not affect the score), I think there are several aspects of efficiency, including throughput, FLOPs, and training GPU days. I think the authors have well addressed the MAC (and throughput or Latency during the rebuttal), additionally, considering other metrics will be interesting as well. I think adding these metrics will be great if the paper gets accepted. Also, I think adding the Latency result in the paper would be much better.

Finally, I believe this paper should include the results of multi-scale INR in the main paper and did not highlight enough. Since the main concept of the paper is to share the activation (as the method itself consists of this terminology), the paper should highlight that they are not the first to use this idea (only related work shows it). In the introduction, the reader still may get confused that this paper is the first to argue this concept as the activation sharing has no reference and is highlighted as bold.

I think publishing this paper has good benefits for the community, but I feel that the paper should improve the paper by addressing the following.

• discussing the limitations, highlighting the prior works, and why efficiency for MLP-based INR is needed even though we have efficient Grid-based INRs.

We would like to express our sincere appreciation for your detailed review posted previously. From both our own perspective and that of reviewer 24bm and rQpt, the additional experiments and comparisons to baselines have significantly improved our work.

As the discussion deadline is approaching, we kindly request your feedback on whether the follow-up response adequately addresses your concerns. Your experienced comments on the subject matter would be invaluable to our work's future explorations.

Your timely response is greatly appreciated.

Thanks.