Efficient Training of Neural Stochastic Differential Equations by Matching Finite Dimensional Distributions

Hi,

Thank you for the review, and we're happy to address the raised concerns.

1. We're going to fix the template in the forthcoming revision.

2. Continuity and Markov Assumption

As mentioned in the response to reviewer SBGZ (link), the proof can be directly extended to cadlag Markov processes defined on an open interval $[0, T)$. The only modification needed in the proof is to construct the sequences to converge from the right in the last paragraph of the proof of Theorem 5. If you and the other reviewers deem it necessary, we are happy to extend the proof to the cadlag case. This allows our theory to cover jump processes.

Regarding the Markov assumption, we believe this can be partially addressed by augmenting the paths with more information or leveraging a hidden Markov model. However, these are beyond the scope of neural SDEs and may warrant a separate paper. If you feel it is necessary, we can add additional discussion on the relaxation of these assumptions.

We thank you again for your insights, and please let us know if there is anything unclear; we're more than happy to clarify!

Hi,

Thank you for the detailed and constructive review. We're happy to address the raised concerns.

1. Experiments are only carried out on 2-dimensional time series

We apologize for the confusion, but this is not the case. For each dataset, we model all the features jointly, i.e., the dimension is $2$ for the metal price and exchange rate dataset, $5$ for the stock indices dataset, $4$ for the energy price dataset, $3$ for the bonds dataset, and $16$ and $32$ for the Rough Bergomi Model data. We present the two-dimensional joint marginals only because plotting higher-dimensional distributions on paper is challenging.

2. Visualizations of the full generated time series

Thank you for pointing this out. We'll add them in the forthcoming revision.

3. Non-Euclidean real-world dataset

We're not familiar with this type of dataset that is suitable to be modeled as an SDE. We would appreciate any recommendations you might have.

4. Choice of kernels

We can include a study on the kernel choice in the appendix in the forthcoming revision.

5. Augmentation of a time-series dataset

We're not sure what this means; could you please elaborate? Do you mean using the generative method for data augmentation and applying it to a downstream task?

6. Random feature approximations to the kernels for high-dimensional generation

Could you please elaborate on this as well? Do you mean computing the kernels with random feature approximation? It seems like this is just another way to compute/approximate the kernel.

7. Sample complexity

We agree that this is an interesting aspect and will look into it. We'll try to integrate sample complexity analysis in the final revision by the end of the discussion period, but we cannot guarantee.

8. Confidence intervals

Repeating some models on higher-dimensional datasets or datasets with longer sequences can be computationally expensive, so we omitted the confidence intervals for consistency. The previous state-of-the-art Issa et al., 2023 in this area also didn't report error bars, so we believed this was acceptable. That being said, we're happy to include confidence intervals for some smaller datasets in the appendix in the forthcoming revision.

We thank you again for your insights, and please let us know if there is anything unclear; we're more than happy to clarify!

I misinterpreted your $X,Y$ taking value in $\mathcal E$. I thought you were saying that $\omega\rightarrow X(\omega,\cdot )$ is in $\mathcal E$. In any case, you do need the processes to be separable (continuity would suffice), i.e. the FDD is determined by a dense subset of $[0,T)$, to argue for the last part of Theorem 5. This is not explicitly stated in the assumption. I would suggest you just go with $X\in C_{\mathcal E}[0,T]$ or $D_{\mathcal E}[0,T]$ where $\mathcal E$ is a Euclidian space.

For the sensitivity, I am wondering if the scoring rule is smooth (and how smooth) in a change in the parameter of the neural SDE. This will affect the convergence rates of your algorithm. I think this is interesting especially for the volatility part, as we know that the KL is infinity if you perturb the volatility (the two laws are not absolutely continuous).

Hi,

Thank you for the detailed review and we're happy to address the raise concerns.

1. For the proof, it is important that we choose $\nu$ to be an equivalent measure to the Lebesgue measure $\mu$ on $[0, T] \times [0, T]$. So there exists a $\mu$-a.e. positive function $\lambda(t_1, t_2)$ such that $d\nu = \lambda d\mu$. (in case you need a proof, see https://math.stackexchange.com/questions/1393425/equivalent-finite-measures-if-and-only-if-strictly-positive-radon-nikodym-deriv)

Then $E_{(t_1, t_2) \sim \nu} S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)}) = E_{(t_1, t_2) \sim \nu} S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)})$ suggests $E_{(t_1, t_2) \sim \mu} \lambda(t_1, t_2) S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)}) = E_{(t_1, t_2) \sim \mu} \lambda(t_1, t_2) S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)}),$ which further implies $E_{(t_1, t_2) \sim \mu} \lambda(t_1, t_2) [ S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)}) - S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)})] = 0.$

Recall that by definition of the proper scoring rule $S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)}) \geq S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)})$ and $\lambda(t_1, t_2) > 0$ $\mu$-a.e. due to the equivalence of $\nu$ and $\mu$. This makes the integrand to be non-negative $\mu$-a.e., forcing $\lambda(t_1, t_2) [ S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)}) - S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)})] = 0$ $\mu$-a.e..

Again, due to the fact that $\lambda(t_1, t_2) > 0$ $\mu$-a.e., $S(P_{\pi_{t_1, t_2} (Y)}, P_{\pi_{t_1, t_2} (Y)}) - S(P_{\pi_{t_1, t_2} (X)}, P_{\pi_{t_1, t_2} (Y)}) = 0$ $\mu$-a.e..

I think two important points here are the equivalence of between $\mu$ and $\nu$ and that $s$ is a strictly proper scoring rule.

2. I'm checking where we need $\mathcal{E}$ to be Polish. The disintegration theorem we used (Theorem 8.5 of Kallenberg (2021)) only requires the value space to be Borel. We'd sincerely appreciate it you can help us to point it out.

3. For the other minor issues, we can fix them in the upcoming revision.

We thank you again for the insights and please let us know if you feel there is anything unclear; we're more than happy to clarify!

Hi,

We deeply appreciate the detailed reviews and are happy to address the concerns.

1. In Theorem 2, does the result hold for any scoring rule $s$, or does $s$ also need to be strictly proper?
Yes. $s$ has to be strictly proper. This is stated in the first paragraph of section 4.1: "Let $s$ be any strictly proper scoring rule defined on ...". We agree this is probably not sufficiently clear and will move this condition to the theorem statement in the forthcoming revision.

2. The complexity reduction claim
We apologize for this confusion. $D$ refers to the discretization step in the numerical integration of the SDE, i.e., for the SDE

we evaluate the integral

by numerical integration with $D$ discretization steps using the Euler-Maruyama method:

where $\Delta t = T/D$, $t_{k+1}=t_k + \Delta t$, and $\Delta W_k$ is the Wiener increment over the interval $[t_k, t_{k+1}]$.

$B$ in Algorithm 1 refers to the batch size of the SGD optimizer and is different from $D$.

The $O(D^2)$ complexity comes from the previous state-of-the-art Neural SDE training method proposed in Issa et al., 2023, which requires solving a PDE

(see (2) in the linked paper) and backpropagate the gradients through the PDE solver. The double integral is typically numerically approximated using a rectangular rule with $D$ discretization steps:

where $\Delta u = T/D$, $\Delta v = T/D$, and $u_i = i \Delta u$, $v_j = j \Delta v$ for $i, j = 1, \dots, D$. The double sum requires $O(D^2)$. Also, the method of Issa et al., 2023 does not have a better complexity on $B$ as their objective also involves a double sum over $B$ (see (4) in the linked paper, our $B$ is their $m$, the double integral occurs in their $k_{sig}$).

So overall, our method reduces the complexity from $O(D^2)$ to $O(D)$ (or, from $O(D^2 B^2)$ to $O(D B^2)$ if you prefer to also include $B$ ) as we don't need to solve the PDE with the double integral.

3. Not sufficient preliminary material on scoring rules or background on how these rules are used to measure divergence between two Markov processes.
Thank you for pointing out the lack of clarity. We're happy to add more explanation to the background of scoring rules in the forthcoming revision. We'll move the RBF kernel example earlier.

We thank you again for the insights and please let us know if you feel there is anything unclear; we're more than happy to clarify!