Neural Experts: Mixture of Experts for Implicit Neural Representations

W1: Thank you for bringing these important works to our attention, we will address them in our paper.

W2: We compare with [4,new] on their dataset (300 test images from LSUN bedrooms). We use the same learning rate and same measure (PSNR after 300 steps for SIREN). Note that these have not converged at 300 iterations (see also Fig. 5 in [4,new]), hence for the experiments in our paper we report after 30k steps (also done for the last row in the table).

W3: While keeping the number of overall parameters fairly constant, we experiment with how to allocate parameters between the components. The first column is width x layers for the input encoding, num experts x (width x layers) for the experts themselves, width x layers for the manager encoding, width x layers for the manager and the second column is % parameters for input encoding, % parameters for the experts, % parameters for the manager. We can see that as long as the majority of the parameters is not for the manager (one third of the total params or less) our neural experts SIREN model performs similarly.

Q1: We now experiment with changing the number of experts vs number of layers within the experts. Like the previous table, we keep the parameters as similar as possible (~366k) by changing the layer width of experts. The results show that less layers and somewhat high number of layers is best, with 6 experts and 1 layer performing the best out of all the configurations we tried. Our chosen configuration of 4 experts and 2 layers is somewhat consistent with this rule of thumb.

Q2: It is not possible to use our method for NeRFs, for the reason the reviewer points out, see our discussion in the General Comments section above.

Q3: The parameters are similar 349584 for vanilla SIREN MOE vs 365968 for our neural experts SIREN (so 95.5% of the params of our model, or our model is a 1.05x increase in parameters).

I appreciate the authors' effort in providing such a detailed rebuttal.

One of my major concerns is the comparison of your paper with "MoEC: Mixture of Experts Implicit Neural Compression". It seems that their model is the vanilla MOE in your paper, right? Furthermore, I find that Random Pretraining for manager is very important to performance. When you conduct the vanilla MOE experiment, have you also applied the Random Pretraining for manager strategy? If not, I wonder whether the Random Pretraining for manager strategy can improve the performance of vanilla MOE.

One of the major weaknesses of this paper is the experimental evaluation ...

• vs other architectures: All the listed architectures are about changing the activation function. Our proposal is orthogonal to that as it can work with any activation function, and we have shown our method with two activation functions. We have added two more, Softplus+FF and FINER, in the updated Table 1 (see General Comments). We also explain why we cannot test our method on NeRFs in the General Comments section above.
• Dataset Size: Our experiment dataset sizes are similar to the papers you listed. In each of those, relevant experiments (Image,SDF,Audio) are done on similar or smaller sizes (only FINER's SDF experiments has more: 5 shapes to our 4 shapes). In fact many experiments and comparisons are done on as small as 1 instance. (FINER: 16 2D images and 5 SDF shapes, Gaussian activations: comparisons on 1-3 images per property, WIRE: 1 image, 1 3D shape, comparisons on 1 image per property). We compare on 24 images, 4 SDF shapes, and 3 audio samples.

The proposed MoE architecture also did not show improvements when used with softplus activation. Softplus activations alone is not expressive enough. We give new results with Softplus+FF (Fourier Features) in the updated Table 1 (see general comment).

This paper also did not include other metrics normally used to evaluate the tasks, such as SSIM and LPIPS for 2D image fitting (e.g. FINER [3]).

• We have added these metric in the updated Table 1 (see General Comments)

Q1. Can this work be combined with traditional ReLU MLPs with positional encoding or MLPs with activation functions other than sine? Yes, we give new results with Softplus+FF (Fourier Features) in the updated Table 1 (see general comment).

Q2. How does this work compare to other novel INR architectures (e.g. WIRE, FINER, etc)? As mentioned before, those are orthogonal contributions as those methods change the activation function, and we can use any activation function in our framework. See our new results with FINER in the updated Table 1.

W1)a: The limitation we are mainly interested in is learning capacity rather than computation time. For traditional INRs, as each coordinate needs to be processed by the whole network, all parameters have to contribute to the output of every point in the domain, making parameter optimization difficult. For our proposed network, we have subnetworks that are specialised for particular regions of the domain, and thus it is easier to optimise towards better solutions (as our results show). We will make this more clear in our revised version. As the reviewer mentions, saved computational time would be minimal (and highly dependent on hardware architecture), hence we have not explored this.

W1)b: We apologize for the poor wording regarding locality limitation. What we meant is that a desirable property of INRs is for features to change rapidly (to allow modelling of sharp boundaries). This can't be achieved in networks without explicit spectral bias (which we mention in lines 62-66). We will update the paper to clarify this. Our method gives an alternative way for handling sharp changes, i.e., by changing the expert, which is a more natural way to handle discontinuous signals. This is consistent with your observations in Figures 3c and 6.

W2): We will add this reference. Apart from the obvious difference (ours is an MoE so can learn different regions, while they have fixed regions in a grid structure), KiloNeRF's priority is post optimization rendering speed, hence their simple structure. Their architecture leads to insufficient performance when optimizing directly (see their Figure 3), so they rely on distillation to train. Our method on the other hand is an alternative for the original network that optimizes directly with better signal reconstruction performance, which is not the case for KiloNeRF. Furthermore we show improvement over grid based methods in Table 5.

W3): See general comment. We leave combining our MoE approach with hash based methods as future work. Also note that our updated table 1 has comparision with FINER from CVPR2024 (see general comment).

Q1): No, vanilla MoE was with Sine activations, we will change the name to Vanilla SIREN MoE to make it more clear

Q2): We have added the Chamfer distance now (see table in general comment).

Results with InstantNGP: We give results with InstantNGP in the updated Table 1 (put as a new general comment above). The results show that our Neural Experts SIREN and Neural Experts FINER achieve superior performance compared to InstantNGP (89.35 and 90.84 PSNR vs 84.56 PSNR) with an order of magnitude fewer parameters (366k vs 7.7M). In the InstantNGP experiment, we use their default architecture for image experiments (16 encoding levels, 2 parameters per level, maximum cache size of 2^19, and a 2 hidden layer decoder with 64 neurons). Note that most of InstantNGP's parameters are spatial parameters (hash encoding), specifically 99.86% of all parameters. This comparison between our Neural Experts SIREN/FINER and InstantNGP shows that the spatial parameters (parameters dedicated for specific spatial regions, so the experts and the hash encoding) are effective, but if used with patterns and heuristics (like InstantNGP or the other methods discussed in lines 70-76) many of the spatial parameters become redundant and underutilized quickly. This motivates the learning of the spatial regions to achieve a parameter efficient approach (our Neural Experts method).

Extending InstantNGP to include the MoE architecture is non-trivial as the hash encoding parameters, which make up over 99% of the total parameters, are used in the beginning of the network (to provide a specialized encoding to then go through a tiny MLP decoder). The obvious Neural Experts extension (which requires a shared input encoding), is to apply MoE to that tiny decoder, which we call Naive Neural Experts InstantNGP. Results are reported in the updated table 1, and we observe that the baseline InstantNGP performs better than this variant. However, the shared input hash encoding parameters are still over 99% of the total parameters. As shown in our response to Reviewer ekS9, the parameter allocation ratio between the input encoder, the experts and the manager plays an important role in the MoE's performance, with the best performing ratio being 14%, 55% and 31% respectively. So this naive result is to be expected. Constructing a more fair distribution of parameters would require a fundamental change to the InstantNGP backbone. Either we greatly increase the parameters of the expert and manager (which would make the total parameters another order of magnitude larger) or we need a way to split the hash encoding parameters into the experts. Both of these extensions are outside the scope of the current work.

Overall, these results show that learning regions for spatial parameters, as in our Neural Experts, achieve better performance with significantly fewer parameters.

W1: We have added results in the updated Table 1 in General Comments (see "Ours Neural Experts SIREN small (New)"). This version has a width of 68 in each layer (encoding, experts and manager) instead of 128, leading to 98,686 parameters. Note that it still outperforms the SIREN baseline, which has 99k parameters.

W2: See comment in General Comments.

W3: Yes, the convergence is unstable as we optimizing both the parameters and the regions. However our results show that even with unstable convergence we have a consistent performance increase from our baselines, and thus show the promise of this line of work.