On the Generalization and Approximation Capacities of Neural Controlled Differential Equations

We thank the reviewer for acknowledging our contributions and results. Let us first address the two mentioned weaknesses. We agree that the presentation of Theorem 4.1 can be improved and we will try to do so within the page constraints. In particular, as this has also been highlighted by Reviewer 1, we will make the constants explicit. Concerning the experimental part, we make two remarks. First, as the discretization gap $|D|$ vanishes at linear speed - i.e. $1/M$ when sampling $M$ points in a equidistant fashion - the effect of discretization is almost null when short time series are sampled at high frequency. We thus have chosen this low sampling regime to obtain a visible effect. Secondly, our article is of theoretical nature ; we have chosen to give an extensive presentation of our theoretical results, which are complex and require long developments, instead of a more developed experimental section. We are planning on conducting a follow-up work benchmarking various methods designed for supervised learning with irregular time series, which will be the empirical companion paper to this work. We also refer the reader to the growing body of applied works on NCDEs, which we extensively cite in the introduction.

We now answer the two questions raised by the reviewer. First, the underlying continuous path $x$ is supposed to be unknown and observed at a finite number of sampling times. It is continuous of bounded variation, which is required for NCDEs to be well defined (Assumptions 1 and 2). We make the simplifying assumption that the paths are Lipschitz, but this can be weakened to include paths paths of bounded variation with non linear modulus of continuity. We work in a fixed design setting, i.e. we consider our data to be fixed throughout the article. As highlighted in our response to Reviewer 2, this stands in contrast to settings in stochastic control theory, where one learns over the distribution of random paths (i.e. stochastic processes with parametrized vector fields). Secondly, we thank the reviewer for catching the typo on $|D|$, which we have fixed.

We warmly thank the reviewer for acknowledging the originality, quality and clarity of our work.

Relation to earlier work. We agree that our presentation of the related work by Kidger (2022) and Kidger (2020) is incomplete and that the abstract is misleading. The contributions of these two works can be summarized as follows. First, the authors proves the existence of solutions to neural CDEs. This result follows from the Picard-Lindelhöf theorem. Secondly, the authors of this article also prove a universal approximation property for neural CDEs i.e. that sufficiently wide NCDEs can approximate any continuous function mapping the space of paths to $\mathbb{R}$ (Theorem 3.1. in Kidger 2020, and Theorem C.25 in Kidger 2022). We have emphasised these contributions in the abstract and the introduction. In contrast to this, our approximation bound provides a precise decomposition of the approximation error in the setup where the function linking the path to the outcome is a CDE.

We also thank the reviewer for noticing typos, which we have fixed.

Implications of our work. A deeper discussion of practical implications is indeed needed, as suggested by the reviewer. We see three principal implications of our work.

1. Designing new architectures for NCDEs. Our paper offers performance guarantees and sheds light on the theoretical behaviour of NCDEs. We think that a promising application of our work lies in the design of new architectures for the neural vector field $\mathbf{G}_\psi$. In our work, as in most applications of NCDEs, this neural vector field is chosen as a simple MLP. However, a recent and rich literature has highlighted the benefits of more structured architectures (for instance through orthogonal weights - see Jia 2019). Our work could help to understand the theoretical implications of such extensions to NCDEs. On a sidenote, we would like to mention the recent article by Chiu et al. ("Exploiting Inductive Biases in Video Modeling through Neural CDEs", 2023), which was unpublished by submission date, and which takes a first step in this direction by using a U-Net as a neural vector field for learning from video data.

2. Quantifying the impact of sparse sampling. On an applied level, our results can serve as a guideline to practitioners that use irregular time series. For instance, from our personal experience, MDs often wonder whether they have collected enough longitudinal data in a clinical setting to conduct inference with NCDEs. Our results on the sampling bias may help to make an informed decision in such a case. Additionally, our results can help a practitioner facing a "decide now or sample more" dilemma, in which an agent needs to decide whether she defers a decision in order to acquire more information through time-consuming sampling (see for instance Schäfer, "TEASER: early and accurate time series classification", 2020, for a discussion of similar problems).

3. Links with Control Theory. The reviewer mentions control theory, which can be seen as an inversion of the setup of NCDEs as noted by Kidger 2022 (see Section 3.1.6.2). In control theory, the dynamics are fixed (known or unknown) and one tries to optimise the input path. In our setup, the path is fixed and the dynamics are optimised. We think, however, that our results can transfer to control theory: our theoretical framework can indeed accommodate many situations in which one learns with continuous time data. Consider, for instance, a setup with fixed dynamics in which one minimises a criterion by optimising over the space of paths by choosing the vector fields of a neural SDE (see Zhang, "Neural Stochastic Control", 2022 for a close setup). One could use our results on the continuity of the flow to bound the difference between two outputs - driven by two paths with different vector fields - by the difference between the two vector fields, and hence derive an upper bound on the covering number of such a class of predictors. To perform this extension, one would however need to leverage additional tools from stochastic calculus and rough analysis.

To satisfy page constraints,we have included these remarks in Appendix F.1 and include a reference to this part in the conclusion. If the reviewer sees other implications, we would be more than happy to include them as well.

We thank the reviewer for this thorough review of our paper and its theoretical part which raises interesting questions, and for acknowledging our contributions. Before turning to the core of the review, we agree that giving the explicit form of all constants will make Theorem 4.1. more clear, as it has also been noted by Reviewer 3. We have added references to the constants in Theorem 4.1.

Comparison with earlier work by Chen et al. (2020). We first address the comparison made with the earlier results of Chen et al. ("On Generalization Bounds of a Family of Recurrent Neural Networks", 2020 - we refer to AISTATS version from now on). We would first like to emphasise that the parameter $q$, which in our setting corresponds to the depth of the neural vector field applied at each update of the latent state, is completely absent from the aforementioned article: Chen et al. only consider shallow vector fields (i.e. "vanilla RNN" in their phrasing). Their neural vector field has no hidden layers but is only parameterized by matrices $W$ and $U$ which have resp. $d_h \times d_x$ and $d_h \times d_h$ parameters. We strongly encourage the reviewer to compare the definition of Chen et al.'s vector field (p.2, left column) with ours (Equation 1, page 4) to convince himself of the difference between our models.

With this clarification in mind, we argue that the first bounds obtained by Chen et al. and ours are of identical nature. Let us take a close look at Theorem 2 in Chen et al. (p. 4, left column): one can see that the complexity term scales as the square root of the number of parameters $d$, times some log factors. We obtain a similar result (see our Theorem 4.1 on page 6): the complexity term, in our bound, grows as the square root of the number of parameters times some log factors. As stressed above, the factor $q$ appears here since having $q$ layers in the neural vector field multiplies the number of parameters by $q$. This is not a surprise since both works use the same fundamental technique (Lipschitz-based complexity measures). In the last part of their paper, Chen et al. provide a slightly improved bound at the price of stronger restrictions on the parameter space. We leave an extension of these techniques to NCDEs to future work. We thank the reviewer for noticing this precise point, which we had missed. We have added a mathematical and precise comparison of the results in Appendix F.2.

Let me explain again what the issue is with the experiments :  the second and the third figures of Figure 4 are the only place where the true generalization error is plotted - but we have no idea from this plot what the theoretical bound is for these same experiments. The empirical Rademacher complexity should have been computable on the trained nets considered in these experiments.

The authors have themselves said the following in the added response at the bottom of page 36, "Our bounds are not depth independent since increasing the depth of the vector field Gψ worsen our generalization bounds which scale with the square root of the number of parameters."

This is the point I have been trying to make.

A generalization bound needs to be "non-trivial"  - that it needs to show independence from some parameter on which one would have naively expected the bound to scale with.

The whole point of generalization theory is to answer the fundamental question - "When are bigger models better?"

This point is simply not getting made here - not at all readable from the given bound.

Even if this is missing, the paper could have been acceptable had the experiment made the point of the theoretical bound being correlated to the true generalization error.

With both these key points missing I am unable to change my scores.