We thank the reviewer for their helpful comments and appreciate that they find our introduction to be excellent. We agree with the reviewer that our variational inference resembles earlier works such as Li et al. (2020) and is a direct application of Girsanov's theorem. However, to benefit from this strong, established result, we did need to show how a new SDE driven by BM can be obtained from an SDE driven by fBM. As the reviewer confirms, this is where the novelty lies (in addition to our novel scheme of weight optimization). Moreover, our variational inference framework paves the way to our video prediction application. We have incorporated all the minor suggestions.

We thank the reviewer for their constructive feedback and are glad to hear that they find our exposition and mathematical formulation clear and detailed. We respond to their questions and comments below.

$**On experimentation.**$ We thank the reviewer for bringing this up. We have now conducted an additional evaluation on the $**real video dataset of chaotic double pendulum**$, as suggested by the Reviewer J9Kf. Note that this dataset comprises of real video captures of a double pendulum in motion. We provide the details in our first response to J9Kf as well as in our $**revised Sec. 5 and Appendix**$. Overall, results on this dataset further reveal the merits of fractional noise in capturing the nuisances of real world.

Note that this new dataset and SM-MNIST contain different forms of nuisances and present different challenges to our stochastic model. First, SM-MNIST digits move with a constant velocity along a trajectory until they hit at wall at which point they bounce off with a random speed and direction. This sudden event intersperses the deterministic motion with moments of uncertainty, i.e. each time a digit hits a wall. This is the reason why a stochastic model fits better than an ODE and unlike BM, our noise can model the smooth and correlated trajectory simply by raising the Hurst index.

On the other hand, the double-pendulum dataset is actually governed by a set of coupled ordinary differential equations. However, despite being a simple physical system, it exhibits a rich dynamic behavior with a strong sensitivity to initial conditions and noises in the environment (motion of the air in the room, sound vibrations, vibration of the table due to coupling with the pendulum etc.). Combined with the chaotic nature of the system, this creates a major challenge for any model based upon smooth ODEs. Our model on the other hand heavy lifts this difficulty onto the (fractional) stochastic noise, leading to a more appropriate model. As shown (see our new Table in Sec. 5), our model outperforms the BM baseline also in this dataset.

$**Why latent video generation?**$ The main contribution of our work is a new differential equation, driven by the approximate fractional Brownian motion as well as a variational inference framework, whose drift, diffusion and control functions can be learned. Due to the stochastic nature of the real world, video perception becomes an appealing and important problem for showcasing the benefits of our method. Note that, we do not claim that our video architecture, despite being novel, brings an ultimate solution. Yet, it enables variational learning under our neural SDE, which is essential to validate our contributions. In principle, our stochastic differential equations can be applicable to various sorts of stochastic video perception methods. We thank the reviewer for mentioning this and leave further applications to a future study.

$**BM vs fBM, Markovian-ness and long range dependencies.**$ Brownian motion is a stochastic process with stationary, independent and normally-distributed increments. Due to this independence, the probability of each event depends only on the state attained and therefore correlations to further away past or future states are not considered, \ie it is a Markov (memory-less) process, $*what happens next depends only on the state of affairs now*$. This can be understood as the noise process (infintesimal increments) being $\delta$-correlated for BM. The increments of fBm (with Hurst index $H$), on the other hand, are not independent unless $H=1/2$ for which we recover BM. If $H<1/2$ then the increments of the process are negatively correlated. If $H>1/2$ then the increments of the process are positively correlated, shows $*long-range dependence*$ (LRD), usually related to the $*rate of decay in statistical dependence*$ of two points with increasing time interval. For fBM, the rate of decay follows a power law, indicating that the process is non-Markovian. Please see [1] for a simple yet rigorous proof of this.

In general, we agree that one may have long time dependencies for ordinary stochastic differential equations (without fractional noise). However, for larger classes of such models, e.g. the so called reversible diffusions, one obtains an exponentially fast relaxation of fluctuations under certain asymptotic conditions on potentials (well explained in the book by Pavliotis [2]). On the other hand, even simple linear systems driven by fBM, such as the the fOU process, inherit a non exponential, power law decay of stationary correlation functions from the driving process.

$**References**$

[1] Huy, Dang Phuoc. "A remark on non-Markov property of a fractional Brownian motion." Vietnam Journal of Mathematics 31.3 (2003).

[2] "Stochastic processes and Applications", by G.A. Pavliotis, Springer, Chapter 4.

We thank the reviewer for the nice summary of our work as well as their constructive feedback. We appreciate that the reviewer has a good grasp of the essence of our contributions. We address their key comments below, while we have incorporated all of their minor corrections.

$**On runtime of large video models.**$ As the reviewer rightously confirms, the computational cost is not a drawback specific to our method. Almost all similar pipelines require moderate GPU resources and training time. For example, on UCF101 dataset, a version of TGAN [1] took three days to train on an A100 GPU costing over $350 using Google Cloud Platform [2]. For KITTI dataset, Akan et al.[3] train on a Tesla V100 GPU for approximately 4-5 days. Similarly they trained SVG and SRVP on a Tesla T4 GPU taking approximately 2 days. Both of these GPUs are almost twice as fast as our RTX 4090. Please find a runtime analysis of our method against that of Tong et al. (2022) in our second response to Reviewer J9Kf.

$**Difference between a Type I and a Type II fBM.**$ Type I, also called Mandelbrot-Van Ness or 'standard' fBM is the most well known definition of fBM. Type II, also called Riemann-Liouville fBM, is historically most prevalent in econometric literature. As can be seen in the integral definitions (Eq. (3) and (4)), Type II omits the first integral from $-\infty$ to $0$ in the definition of Type I. So Type II is, in a sense, a simplification. Yet, to the best of our knowledge, its covariance has no simple analytical expression. On the other hand, the Type I covariance is the straightforward, well known Eq. (1).

As described by [5,6] the main difference is that Type I has stationary increments, and Type II has non-stationary increments. This means that Type II has a larger emphasis on the origin $t=0$, which might not be favourable for some applications. For example, if during training we sample a sequence from a video dataset at a random start point, this $t=0$ has no special or distinguished meaning and should not be treated differently by the driving fBM process. In other words, Type I ensures a shift in time has no effect on its increments. However, this is not the case for Type II. This difference is relevant for our framework, since the increments are driving the SDE. We thank the reviewer for bringing this up and have added our explanations to the appendix.

$**A finite KL term inside of the ELBO and Novikov's conditions.**$ It is seen in Equations (13) and (16) that the diffusion terms of prior and posterior have to be equal. This holds by construction. However, to derive an explicit finite KL term in our ELBO, we are required to invoke the Girsanov theorem, which necessitates a second technical condition, namely the Novikov condition. The reviewer is right that the control term $u(\cdot)$ learned via a neural network is assumed to satisfy Novikov's condition. In fact Chen et al. [7] combined with the appendix in https://openreview.net/attachment?id=wG12xUSqrI&name=supplementary_material (Lemma 8) indicates two sufficient conditions for Novikov to hold:

• sub-Gaussian tails for the initial samples ($X_0$ or $Z_0$), which is easy to enforce
• uniform Lipschitz continuity of the neural network

The second is challenging for us to guarantee. This is similar to earlier works performing stochastic variational inference, which either use an approximation argument [7] or assumes that Novikov’s condition holds at the outset [8,9,10]. In our work, we admit the latter.

$**References**$

[1] Saito, M. et al. "Temporal generative adversarial nets with singular value clipping." Proceedings of the IEEE international conference on computer vision. 2017.

[2] Gordon, C., and Parde, N. "Latent neural differential equations for video generation." NeurIPS 2020 Workshop on Pre-registration in Machine Learning. 2021.

[3] Akan, AK et al. "Stochastic video prediction with structure and motion." arXiv preprint arXiv:2203.10528, 2022.

[5] Lim, SC and Sithi, VM. Asymptotic properties of the fractional brownian motion of riemann-liouville type. Physics Letters A, 1995.

[6] Marinucci M. and Robinson, PM. Alternative forms of fractional brownian motion. Journal of statistical planning and inference, 1999.

[7] Chen, Sitan, et al. "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions." The Eleventh International Conference on Learning Representations. 2022.

[8] Li, Xuechen, et al. "Scalable gradients and variational inference for stochastic differential equations." Symposium on Advances in Approximate Bayesian Inference. PMLR, 2020.

[9] Wang, Benjie, et al. "Neural Structure Learning with Stochastic Differential Equations." arXiv preprint arXiv:2311.03309 (2023).

[10] Liu, X. et al. "Let us Build Bridges: Understanding and Extending Diffusion Generative Models." NeurIPS 2022 Workshop on Score-Based Methods. 2022.

$**On discretization and the choice of the solver.**$ We would like to clarify the distinction to Tong et al. (2022), which mainly differs from our work in two ways: (i) fractional Brownian motion (fBM) is approximated as a a Gaussian process (GP), (ii) only the Type II representation of fBM is used as as an integral over increments of the Wiener process. Tong et al. performs a finite time discretization of the Type II integral to obtain a first approximation of the increments of FBM. In a second step, this approximate GP is further approximated using a sparse GP approach based on a smaller set of pseudo or inducing points which are distributed over time. Conditioned on the inducing points, samples from the sparse GP are independent random variables at each discrete time point. Finally, this (conditioned) white noise process is further interpreted in terms of the Euler discretisation of an ordinary SDE leading to effective drift and diffusions. For the latter SDE, one can apply Girsanov's theorem and the corresponding ELBO (conditioned on the inducing points) to perform inference.

Note, that their current derivation of effective drift and diffusion relies on the Euler discretisation of SDE. For higher order SDE solvers, the approximation has to be adapted, which requires new derivations. As a main difference, in our paper, the approximation is not based on the discretisation in the time domain but of the discretisation of an integral representation (our equation (5)) over a spectrum of decay constants of Ornstein-Uhlenbeck (OU) processes (driven by the same Wiener noise). Since each OU process already represents a noise process with temporal correlations, we can expect that a linear combination of a small number of such processes can yield a good approximation of the covariance of fBM over some given time interval. Our approximation leads to a system of SDEs (without conditioning) for which the ELBO can be easily obtained. Since the time discretisation of the resulting SDE is performed $*after*$ the OU approximation, any SDE solver can be directly applied. With this flexibility, in our paper, we have chosen the second order Stratonovich–Milstein solver.

We thank the reviewer for their diligent and detailed review, and all the feedback. We are delighted to hear that they find our method to be a novel solution to a difficult problem, and our paper to be well-written. We have now significantly improved the paper in the light of the suggestions of the reviewer.

$**On clarity.**$ We thank the reviewer for the helpful indication of their understanding of Figure 4 and Section E. In the light of these, we have considerably improved the readability and clarity of our paper. We further address the questions below.

The static content vector $w$, in the explanation of the video modelling framework, should not be confused with the weights $\omega$ (Greek letter omega) in the fBM approximation. The approximation processes $Y_k$ are integrated jointly with $X$. The state of the SDE process is thus constructed as in Eq. (14): $Z = (X, Y_1, \dots, Y_K)$. The values of $\gamma$ are described in our Appendix for all experiments. For the video models, we only take one sample path to calculate the ELBO. For the bridge and time dependent Hurst experiments we take $16$ or $32$ samples.

$**On the prior and Moving MNIST.**$ In our video modelling experiments, both the prior model and posterior model are parameterized. The prior is as defined in Eq. (13). The posterior model is the prior model with the added control function $u(t)$, as defined in Eq. (16). Since only the control function receives information of the data sequence, the posterior is capable of reconstructing the sequence during training. When the model is trained, we use the prior as a generative model. Specifically, due to maximizing the ELBO, the prior will learn to model the dataset. For example, to condition on the first frames of a movie and generate predictions, we first use the posterior model on the given frames. At the time point where we start predicting, we use the prior. Practically, this simply means setting $u(t)=0$.

The Peak Signal-to-Noise Ratio (PSNR) metric is calculated frame-wise, averaged over time, by sampling $100$ videos conditioned on the first $5$ frames. The best PSNR is selected from these $100$ samples. This approach is adapted from SLRVP to allow direct comparison. This metric is representing how good the model captures the data. Please note that we have considerably improved the manuscript under the 'Latent video models' section and incorporated this explanation.

$**On empirical evaluations.**$ Following the reviewer's suggestion, we added extra experiments on a $**real-world dataset composed of videos of the chaotic motion of a two-link double pendulum**$, released by IBM under https://developer.ibm.com/exchanges/data/all/double-pendulum-chaotic/ [1]. We thank the reviewer for this to-the-point suggestion. We provide the results below as well as in our revised Sec. 5:

Even though this system is governed by ODEs, its chaotic nature poses a big challenge for any ODE-based method to faithfully capture its behaviour. Arguing that the hard-to-predict transitions can be modeled by stochastic noise, we evaluate our video model using both BM and fBM, where the latter attains a lower ELBO as a higher PSNR. We have $**revised Sec. 5**$ and incorporated qualitative results indicating the diversity of our samples. We hope that this new result clarifies some of the concerns and strengthens our empirical evaluation.

$**Comparisons to Tong et al. (2022).**$ Note that, Tong et al. (2022) only reports Hurst index estimation in their experiments. Following the same protocol, we have already compared our method against theirs in time-dependent Hurst index. This experiment ($**Fig. 3**$) not only extends our method to $* multifractional Brownian Motion*$, but also shows that our approach yields more correct Hurst estimates, especially in the extremes. This is further supplemented by our conceptual discussion in $**App. E.2**$. An important difference is that our work covers Type I, whereas Tong et al. (2022) does not.

$**Further applications.**$ Our experiments contain the applications of video prediction, which is an extrapolation task that predicts future frames given past frame. Interpolation is an easier problem and certainly possible within our framework. Note that, our bridge experiments already contain non-Markovian boundary value problems. Our versatile framework can be extended to other applications, which we leave for a future study.

[1] Asseman, Alexis, Tomasz Kornuta, and Ahmet Ozcan. "Learning beyond simulated physics." (2018).