Implicit regularization of deep residual networks towards neural ODEs

Q 1: It seems to me that both fully-connected and convolutional neural networks would be the particular cases where the ODE parameter matrices A, V, W are considered sparse (some or most of the entries are fixed to zero and the rest are optimised according to Equation (5)). The experiment, depicted in Figure 3, suggests that results in Theorem 4 might be expanded to the more generic sparse scenarios. Might be good if the authors could confirm this observation.

We confirm that the reviewer is right. In Section 4.3, we describe a general setup where Theorem 4 holds, where it is only assumed that the residual connection $f$ is a Lipschitz-continuous $C^2$ function. This includes in particular the sparse version of the model considered in the paper. This is an interesting comment that we will incorporate in the next version of the paper.

Q2: If it happens that the weights are initialized randomly, for every finite depth of the neural network (as there’s finite time and finite depth), there could be still a Lipschitz-continuous function (ii) over that set (as locally Lipschitz function on a compact set is Lipschitz as per, e.g. https://faculty.etsu.edu/gardnerr/5510/cspace.pdf ). However, what’s appearing in Figure 3 appears to go beyond literal application of Theorem 4: not only the function is compact but also it appears that the Lipshitz constant is preserved to be small during the optimisation.

This is a very good point, thanks to the reviewer for raising it. It is true that for a fixed depth, it is always possible to construct a Lipschitz-continuous function that interpolates the weights, e.g., by piecewise-affine interpolation. Nevertheless, without further assumption, the Lipschitz constant of such a function depends on the depth $L$. On the contrary, as observed by the reviewer, in our results, the Lipschitz constant of the function that interpolates the weights is independent of the depth (see, e.g., Theorem 4 (ii) and Theorem 6 (ii)). Moreover, the proofs give a bound on this Lipschitz constant during training, even though we did not state it in the theorems to simplify the exposition. We will add a remark about this in the next version of the paper.

Q3: Another interesting aspect is that it would be intuitive to see smaller Lipshitz constant could be beneficial for out-of-distribution recognition. I wonder if the authors could clarify on this aspect?

Yes, this is a correct intuition. This is confirmed by a recent NeurIPS paper (Marion 2023, ref. below), where it is shown that the generalization error of deep residual networks / neural ODEs grows with the Lipschitz constant.

Generalization bounds for neural ordinary differential equations and deep residual networks, Marion, NeurIPS 2023.

To what extent would the proposed derivation generalise for higher-than-two layers of multilayer perceptrons?

As mentioned above, Section 4.3 presents generalizations of our results to an arbitrary residual connection $f$, which we assume to be a Lipschitz-continuous $C^2$ function. This includes in particular the case where $f$ is a deeper MLP. Our finite training time results are fully proven in this general case. Regarding the limit $T \to \infty$, we prove that convergence to a neural ODE holds for a general residual connection $f$ if it satisfies a Polyak-Lojasiewicz (PL) condition. Proving the PL condition in the case where the residual connection is a deeper MLP is a challenging mathematical question, which deserves future work.

Dear Authors,

I've gone through the responses to my review and to the ones by other reviewers. I think the questions are sufficiently answered, and my score remains unchanged.

Many thanks, The Reviewer

Thanks to the reviewer for the positive review. The reviewer is perfectly right in stating that the main novelty in the paper is to get rid of the linearity assumption and to present results in a very general setup.

The results are under the condition that the resnet is trained under clipped gradient flow, which is often not used in practice. However, I do understand that the main contribution of the paper is to characterize the behaviour of the model in the limit, so it shouldn’t affect the main contribution of the work.

This is a good point—thanks to the reviewer for raising it. Clipping is used in our approach to constraint the gradients to live in a ball. It is merely a technical assumption to avoid blow-up of the weights during training.

However, in any scenario where we know that the weights do not blow up, clipping is not required. A first example of such a scenario is the limit $T \to \infty$ in the paper. In this situation, a Polyak-Lojasiewicz condition holds, and, consequently, the weights remain bounded.

Another scenario is by using gradient flow with momentum (a setup actually closer to Adam, which is very used in practice) instead of vanilla gradient flow. One might then show a similar result to our Theorem 4, without clipping, because the gradient updates in the momentum case are bounded by construction.

We will add a comment about this in the next version of the paper.

For the experiments on cifar, the authors show the plot of a random coefficient and its value as one goes deeper into the layer. However, it would be more beneficial to see the difference in Frobenius norm of the weight matrices as well.

Thanks to the reviewer for the suggestion, we are currently working on the figure, and will add it to the next version of the paper. We will add a link to the figure in a follow-up comment if time allows before the end of the discussion period.

One of the key drawbacks is requiring weight-tying or weight sharing. While results with weight-tied networks don't look bad, still this would have been nice to take care of. Can the authors elaborate on where all and where exactly does it interfere with the proof strategy?

This is a good point, and the reviewer is right in pointing it out. We consider in the paper a weight-tied initialization in order for the network to satisfy an ODE-like structure at initialization. Such a structure is necessary for an ODE structure to hold after training. As stressed in Section 4.3 of the paper, the weight-tying assumption can be relaxed by taking a so-called “smooth initialization”, where the weights at initialization are a discretization of a smooth (potentially random) function.

It remains that this scheme does not encompass iid initializations (e.g., Gaussian or uniform), which is a classical scheme in initializing neural networks. In this case, the large-depth limit falls into a different regime, namely a stochastic different equation instead of an ordinary differential equation (see Marion et al., 2022, ref. below). It turns out that analyzing the convergence in this regime is challenging and deserves a paper by itself. We will make it clearer in the conclusion of the paper.

Scaling ResNets in the Large-depth Regime, Pierre Marion, Adeline Fermanian, Gérard Biau, Jean-Philippe Vert, arXiv:2206.06929, 2022.

The other issue is that the linear overparameterization $m > c_1 n$ would, in practice, be somewhat unreasonable and is thus a bit of a stretch. Maybe this is a norm in the literature, but do the authors have some ideas how to go around this?

We thank the reviewer for pointing out this important assumption. The assumption $m > c_1 n$ is required to show that our loss landscape satisfies a Polyak-Lojasiewicz (PL) condition, which is one of the main modern frameworks to show convergence of neural networks. The PL condition can be seen as an alternative to convexity, which is not satisfied in neural networks. Alleviating the assumption $m > c_1 n$ would require working under a weaker condition than PL to show global convergence. Unfortunately, we are not aware of such a weaker condition in the neural network literature at the moment.

Note however, as highlighted in Section 2, that the linear overparameterization condition is mild with respect to usual assumptions in the literature (polynomial overparameterization).

Would it be possible to extend the approach to incorporate layer normalization?

This is a very interesting suggestion, since normalizations are necessary for neural networks to work in practice. To clarify this point, we propose to add the following discussion to the next version of the paper.

First, conceptually, the $1/L$ normalization introduced in the paper (see, e.g., equation (1)) can be seen as an alternative to normalization (batch or layer). The connections between these three types of normalizations is still theoretically unclear. For instance, batch normalization has been shown empirically to be close to a $1/\sqrt{L}$ normalization (De and Smith, 2020, ref. below).

Second, empirically, the experiment with CIFAR-10 presented in Figure 3 includes batch normalization layers, and the implicit regularization towards a neural ODE still holds in this case.

Finally, mathematically, some of our results extend to a fairly general setting (see Section 4.3) where we only assume that the residual connection is a Lipschitz-continuous $\mathcal{C}^2$ function. The intuition suggests that this should include in particular the case where layer normalizations are added to the architecture, although this should clearly necessitate a rigorous and separate mathematical analysis.

Batch normalization biases residual blocks towards the identity function in deep networks, De and Smith, NeurIPS 2020.