Magnitude Invariant Parametrizations Improve Hypernetwork Learning

We are thankful for the detailed, careful, and thought-provoking constructive feedback and comments. Addressing the raised questions:

Weaknesses:

While the authors do empirically show that MIP benefits training, it is not clear whether the increased variance could also be controlled with, e.g., appropriately chosen (i.e., lower) learning rates and (i.e., higher) momentum, in the original parametrisation (which could attain similar performance, albeit slower).

This is an important point. In our experiments, we controlled for this by testing a range of learning rates and reporting the best converging models. We consistently found that MIP was less sensitive to hyperparameter choices than the default formulation, and that there were settings, like HyperMorph with momentum SGD, where we could not find any suitable hyperparameter setting (learning rate & momentum factor) that would allow the model to meaningfully train.

Questions:

The reviewer raises many interesting questions and suggests interesting areas of future work. Regrettably, we do not have the time or space to address many of these for this submission in sufficient detail, but we are thankful for the suggestions as they will inform future work.

It would be interesting to see whether the residual formulation of the hypernetwork closes meaningfully the gap between fully task specific parameters and the ones predicted by the hyper network, i.e., $\theta^0 + h(E_{L2}(\gamma);\omega)$. For example, what is the performance if on a new task one starts from $\theta^0$ and just optimizes for a specific number of steps on that task to get $\theta^\ast_t$? Is $\theta^\ast_t - \theta^0$ related to $h(E_{L2}(\gamma);\omega)$. Is the performance of $\theta^\ast_t$ similar to the performance of $h(E_{L2}(\gamma);\omega)$ ?

While not exactly the same as the setup described by the reviewer, we did perform experiments where we individually trained classical networks for each task $\theta_t$ for the HyperMorph and SSHN settings, to understand how classical networks converged for these settings. When comparing the task-specific networks to the performance of the hypernetwork weights for the same task, we found that hypernetwork weights performed no worse than task-specific weights.

Does the update given by the hypernetwork in this residual formulation need to be dense? Is the hypernetwork only adapting a few dimensions of $\theta^0$?

In our experiments, the contribution of the hypernetwork $\Delta\theta$ was not particularly sparse, but perhaps this is because the learning objective was not set up to promote this property in the first place. We believe that adding a sparsity constraint either to the loss, or structurally (such as with low rank decompositions) would be an interesting avenue for future experiments. We will include this discussion in the revised manuscript.

It is not clear why for non-scalar inputs $\gamma$, i.e., $\gamma \in \mathbb{R}^D$ with $D \geq 2$ a simple unit normalisation transformation, i.e., $\hat{\gamma} = 1/||\gamma||\gamma$ would not work for removing the dependence on the magnitude. It seems to me that in this case each $\hat{\gamma}$ would just correspond to a different point on the hypersphere and the output of the hypernetwork would not be independent of the input.

This is a good question that we should have addressed more directly. Dividing by the norm of the input produces information loss in the hypernetwork input space, even for multi-dimensional hypernetwork inputs. As an example, we can consider inputs $\gamma_A = [0.1, 0.2]$ and $\gamma_B = [0.4, 0.8]$, dividing them by their respective norms results in $\gamma_A / ||\gamma_A|| = \gamma_B / ||\gamma_B|| = [0.447,0.894]$. While $\gamma_A$ and $\gamma_B$ correspond to different points in the input space, they are mapped to the same constant norm representation. This is why our input encoding maps each dimension to a separate pair of coordinates.

Since hypernetworks often feature low dimensional input spaces, removing a degree of freedom in the input introduces substantial information loss.

We appreciate the constructive feedback and comments. Addressing the raised questions:

Weaknesses:

The discussion regarding why input/output proportionality results in unstable training is somewhat insufficient. It is a well-known fact that neural networks with piecewise linear activation functions, such as ReLU networks, exhibit this proportionality between inputs and outputs. However, if proportionality is the main reason, why ReLU networks are not well-known for having such problems for classical tasks such as regression/classification (not in hypernetworks)? Therefore, I think more explanations or discussions are needed in the text.

This is a natural question, that we should have addressed more directly. We will revise the manuscript to include further discussion about why the input/output proportionality issue is less important for regular neural networks. Most notably:

• Deep neural networks are often used to model high dimensional unstructured data such as image, text, or audio. In these settings, neural network inputs are most often standardized to have zero mean and unit variance, which reduces the effect of the proportionality issue. In contrast, many applications of hypernetworks have relatively low input dimensionality. For example, hypernetworks are often used with scalar or low dimensional inputs. The proportionality issue is exacerbated in hypernetwork problems that feature low dimensional or scalar input spaces.

• A similar issue has been demonstrated for some applications of regular networks, like NeRFs and implicit networks. In such cases, alternatives such as sinusoidal encodings and activation functions have been shown to help $1,2$ .

$1$ Implicit Neural Representations with Periodic Activation Functions - Sitzmann et al.

$2$ Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains - Tancik et al.

Questions:

I might miss the related information. Apologies if that is the case. Are there any results of combining additive output with BatchNorm/LayerNorm in F igure 5? It is interesting to see such results because it verifies that MIP has something that normalization layers cannot offer. If proportionality is the main reason, what are the theoretical reasons that MIP can solve the problem but normalization cannot?

The reviewer is correct, Figure 5 does not include results combining the additive output with BatchNorm/LayerNorm. In our experiments, adding normalization layers to MIP had little effect in per-epoch convergence, but it increased the per-epoch computational cost because of the additional overhead of the normalization layers.

More generally, the fundamental reason normalization layers are not well suited to deal with this issue is that they remove degrees of freedom by enforcing zero mean and unit variance in the input. Since hypernetworks often feature low dimensional input spaces, removing degrees of freedom in the input introduces substantial information loss. Enforcing constant magnitude inputs has similar shortcomings. This is why our input encoding maps each dimension to a separate pair of coordinates.

If the problem is proportionality, how about we use other activation functions such as ELU or GELU? Does that solve the problem? If not, can we still say the problem is proportionality?

We found that other activations like GELU, SiLU/Swish and Tanh, while not having the same proportionality property, still propagate the input magnitude enough for it to be detrimental in training. MIP improves training for all choices of non-linear activation. See section C.3 of the supplement.

We appreciate the constructive feedback and comments. Addressing the raised questions:

I am not really sure if the output encoding part of the framework fits well with the problem that the authors claim to solve. It is not clear how output encoding relates to the input and output proportionality problem. It also makes interpreting the experimental results where both input and output encoding are used harder. Intuitively, output encoding allows the model to learn the task even if the hypernetwork does nothing, so it is not clear if we improve hypernetwork training or just make the hypernetwork path less critical and rely on "classical" parameter learning which is typically more stable.

This is subtle, and we should have done a better job of explaining it, and will do so. Using the output encoding changes the learning problem for the hypernetwork. Rather than learning how to predict parameters, the output encoding provides a task-independent parameter initialization so that the hypernetwork only has to learn the delta for each task. Under the default formulation, the prior is that there is no commonality between tasks. With the output encoding, we change that prior so that there is some commonality.

In Figure 5 we include an ablation of input and output encoding, where we find that each component improves the learning process individually, and that best results are achieved when both are used jointly.

Another thing that is not clear to me is that it is not clear to me what is special about hypernetworks so that input normalization needs to work differently than "classical" networks. The scaling property would hold for a "classical" network as well for the appropriate activations. And yet Appendix C.5 claims that standard normalization techniques of "classical" networks would not work for hypernetworks. Appendix C5 seems to suggest that "classical" normalization techniques could make the output independent of the input. It is not clear to me why this is a problem only for hypernetworks and not for "classical" networks.

Our observation that "normalization techniques could make the output independent of the input" was referring to the specific case of scalar inputs. We should have said this, and we will revise the manuscript to reflect that.

Including normalization layers does make activations follow normalized distributions, but they introduce learning challenges in settings with low dimensional inputs. The reason why these challenges rarely arise with classical networks is that classical networks are rarely used with low dimensional or scalar input spaces. In contrast, hypernetworks are often used to predict parameters based on hyperparameters of interest, and frequently have low dimensional or scalar inputs.

Also it would be useful to clarify why one could not just divide the hypernetwork input by its norm, at least for the case of a multi-dimensional hypernetwork inputs. In my mind, if the problem was just the scaling of the output, this should have worked. The suggested methods go beyond just fixing the scale of the input vector as a whole so it is not very clear if the problem is indeed the input output proportionality or some more general feature scaling issue.

This is a good question that we should have addressed more directly. Dividing by the norm of the input produces information loss in the hypernetwork input space, even for multi-dimensional hypernetwork inputs. As an example, we can consider inputs $\gamma_A = [0.1, 0.2]$ and $\gamma_B = [0.4, 0.8]$, dividing them by their respective norms results in $\gamma_A / ||\gamma_A|| = \gamma_B / ||\gamma_B|| = [0.447,0.894]$. While $\gamma_A$ and $\gamma_B$ correspond to different points in the input space, they are mapped to the same constant norm representation. This is why our input encoding maps each dimension to a separate pair of coordinates.

We appreciate the constructive feedback and comments. Addressing the raised weaknesses:

The paper mainly focuses on fully connected layers and common activation, initialization choices, and optimizers (SGD with momentum and Adam) in its experiments, which may not encompass a broader spectrum of hypernetwork architectures or other types of networks.

We extensively tested our proposed parametrization across learning tasks, primary model architectures, optimizers, initialization, and non-linear activations, finding consistent improvements in all settings. That said, we agree with the reviewer that hypernetwork architectures and applications are diverse and varied. In our work, we tried to cover the settings that are most prevalent in the literature. Are there specific architectures that the reviewer would like us to try?

There's also a mention of unexplored territories like the effect of MIP on transfer learning and other less common architectures and optimizers, indicating a scope for broader empirical validation.

We agree that other applications are an interesting avenue for future work. Given that we see different (though not directionally different) results for Adam and SGD (which together dominate the literature), it would be interesting to see what happens with other optimizers that might become available.

Furthermore, the impact of MIP on real-world applications or larger-scale problems is not thoroughly explored, which might be crucial for the adoption of this technique in practical settings.

We agree that evaluation on real world data is important. We did show the benefits that MIP brings to practical medical imaging applications. The HyperMorph task of loss-regularization for medical image registration features a complex medical imaging task and solves a real world problem.