Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs

We would like to thank Reviewer HuLD for identifying the simplicity of our model and our strong results. Furthermore, we appreciate their valuable contributions in posing important questions and seeking clarifications that contribute to the enhancement of our paper. In the following, we provide responses to the comments raised by Reviewer HuLD. Given the chance, we would be pleased to integrate the suggested modifications outlined below into a final revision.

... their model is, in fact, more similar to the dynamic linear model, or Kalman filter.

We agree with Reviewer HuLD that the dynamic linear model and Kalman filter are related to our approach. The main difference between these techniques and Koopman-based methods is the domain where the dynamics are investigated. For the dynamic linear model and Kalman filter it is the state space, whereas in Koopman methods it is the observable (function) space. In practice, given inputs from the state space $x_{1:T}$, the dynamic linear model and Kalman filter can be viewed as studying a dynamical system $A$ respecting the equation $x_t = A x_{t-1}$. In contrast, Koopman approaches transform states to the observables domain, and study the dynamics there. Namely, $\psi(x_t) = A \psi(x_{t-1})$, where $\psi$ is a nonlinear transformation. Our framework, KVAE, uses nonlinear encoder and decoder, and it employs linearity in the latent space. Therefore, we believe it aligns better and can be justified more profoundly from a Koopman perspective.

Also, one of the claims is that the model can take care of irregular time series, the capability is simply using a pre-existing model, neural controlled differential equations.

Indeed, we use NCDE to handle irregularly sampled time series information, but we do not state our model's ability to handle irregular time series as a contribution. The ability to handle irregular time series is simply a feature of our model. In retrospect, we believe that utilizing NCDE is a strength of our framework as it provides strong results and it could be potentially replaced by any other module that interpolates time series data.

... How do you deal with this ill-posedness?

Thank you for raising this great question. In our experience, we did not observe any issues during training or inference due to the mentioned ill-posedness. Furthermore, our ablation study in the revised version Section 5.3 and Table 4 considers a model whose prior is not learned ($\alpha=0$, no GRU). The results show a significant drop in discriminative scores in comparison to KVAE's performance. Thus, while jointly optimizing the prior and posterior may be ill-posed in theory, the strong performance we obtain outweighs this potential shortcoming. If given the opportunity, we would be happy to discuss this limitation in the final version.

What is $z_t$ and what is $y_t$? Are they different random variables? ...

Regrettably, there seems to be a typo in our notation. Thank you for pointing this out. We fixed this issue in the revised version, where we always use $z$ for the latent variable.

What is the probabilistic model for $p(x_{1:t} | z_{1:t})$? Is it a parametric distribution, e.g., normal distribution? How do you compute the log likelihood function?

We would like to thank the reviewer for pointing this out. Following a common choice in probabilistic modeling, we assume that $p$ is a Gaussian distribution whose mean is the output of the decoder. Under this assumption, the log likelihood becomes a simple MSE between $x_{1:T}$ and the output of the decoder, $\tilde{x}_{1:T}$.

A similar reasoning applies also for the $L_\text{pred}$ term. We added a new discussion in the revised version Appendix C, describing technical details related to the evaluation of the loss function.

I am not fully convinced with the argument on the difference between the Kalman filters ... But I think it is worth mentioning the similarity between the current model and the well-established classical approach.

Thank you for sharing this reference. We totally agree with the reviewer that our approach should be also discussed and related to the Kalman filter and the above mentioned work. In the final version, we will cite this work and discuss it in the background section.

Now, I am confused.

1. is $p(z_{1:T}) = p(\bar{z}_{1:T}) \prod_t \delta (z_t - A \bar{z}_t-1)$ a proper distribution? Does it normalize to one? Why does $\bar{z}$ not appear in the left-hand side? I don't understand how you can approximate a Dirac delta function by one.

2. I don't understand why the eigenvalues of $A$ is not so important in this method. As the authors wrote, they may compute $z_{2:t}$ by multiplying $A$ to $\bar{z}$. But this is just a way to construct a linear system. At the end, since they are using the Koopman operator, the latent state should satisfy the relation, $z_{t+n} = A^nz_t$. Then, they end with the same problem. If one of the eigenvalues is bigger than one, $z_{t+n} \rightarrow \infty$, or if less than zero, the model end up with a degenerate system.

We express our gratitude to Reviewer nGir for their favorable remarks regarding the simplicity of our model, the superior effectiveness of our approach, and the clarity of our writing. Furthermore, we appreciate their valuable contributions in posing significant questions and offering constructive suggestions for enhancing the quality of our paper. In what follows, we address the specific comments put forth by Reviewer nGir. Given the chance, we will be happy to integrate the suggested modifications outlined below into the final revision.

There is a lack of explanation about Koopman in the Background section. In the case of this paper, the main point is Koopman, and readers may also want to know more about Koopman. Therefore, an explanation of Koopman should be in the main paper.

Following the reviewer's suggestion, we moved the appendix section that gives background information on Koopman theory and practice into Section 3 (Background) in the main paper.

There are no results for the predictive loss term in the ablation study in Section 5.5.

Regrettably, there seems to be a typo in our ablation results in Table 4. Specifically, the submitted paper mentions $\beta$, suggesting that we removed the KL divergence term that minimizes the (pseudo-)distance between the prior and posterior distributions. However, in our ablation study, we tested the effect of the predictive loss term and the obtained results without utilizing this component. Thus, all appearances of $\beta$ in the ablation study should be replaced with $\alpha$.

For Section 5, Experiments seems to be a more appropriate word than Results.

Following the reviewer's suggestion, we changed the section heading of Section 5 to read Experiments.

It seems that 0 should be excluded from the MuJoCo results in Table 10.

Thank you for noticing this. We fixed it.

I am curious about the role of the predictive loss term. In this paper, a predictive loss term was added to the object function. Therefore, I am curious about how much the predictive loss term affects the performance that is superior to existing baseline models.

Indeed, ablating the effect of the predictive loss term is important. This ablation appears in Table 4 (after correcting the typo $\beta \rightarrow \alpha$). The results indicate that the benefit of the predictive loss term can be significant. Notable examples are stocks regular, energy $30\%$, and stocks $50\%$. More generally, removing this loss term results in inferior error estimates in most cases.

We would like to thank Reviewer SWgv for acknowledging the novelty of our approach, the positioning of the paper, and the compelling experiments. We also would like to thank them for their detailed comments, questions and suggestions that help to deepen our discussion and improve the paper. Below, we address the comments raised by Reviewer SWgv. Given the opportunity, we will be happy to incorporate the modifications listed below into a final revision.

... An ablation study with such a change of architecture would be highly informative.

To assess the contribution of our Koopman-based prior, we suggested the ablation study in Table 4. There (after fixing the typo $\beta \rightarrow \alpha$), we tested the behavior of the model without the predictive loss term, i.e., no linearity is imposed. Our results demonstrate that KVAE performs better in the majority of cases in comparison to the model with $\alpha=0$. Thus, while the GRU decoder may help in obtaining good results, it is not enough, and the Koopman-based prior improves these results further.

When designing KVAE, we opted for a GRU decoder in order to match existing generative models for a fair comparison. Modifying the decoder will have a strong impact on the performance. Naturally, using a stronger decoder may improve the results significantly, giving an unfair advantage to the approach. Similarly, employing a weaker decoder may damage the results. Thus, the decoder in all existing baselines should change to an MLP decoder for a fair and consistent comparison.

Why do you use two different letters, $y$ and $z$, for the prior part and the posterior part, respectively? ...

Regrettably, there seems to be a typo in our notation. Thank you for pointing this out. We fixed this issue in the revised version, where we always use $z$ for the latent variable.

Although the paper focuses on the generation capability of the models, some more experiments on the reconstruction / inference capablity could also be interesting.

Thank you for suggesting that. Following your suggestion, we added two new figures to the revised version showing several reconstruction and inference examples. Specifically, we plot in Fig. 15, the reconstructed signals in the regular setting for all datasets (dashed lines) vs. the ground-truth data (solid lines). In addition, we show in Fig. 16, the inferred signals (dashed lines) in the irregular 50% case. This experiment demonstrates the inference capabilities of our model since half of the samples were removed during training. We plot the complete signals (solid lines) for comparison. In addition, we want to emphasize that our climate experiments could also be considered as inference test cases. To evaluate our model abilities in the climate setting, we omit some of the signals, which we refer to as "points of interest", and generate them to compare with the ground truth.

... Do you have some observations when you did not use a nonlinear sequence-to-sequence model in the decoder?

In general, the predictive loss term improves the results in comparison to models without the predictive loss term (i.e., $\alpha=0$). This is shown in our ablation study in Table 4. In addition, our preliminary results with an MLP decoder show a similar trend. For further details regarding using an MLP decoder, please see the response above.

In practice, how linear the sequence of $y$ is? ...

Indeed, we preferred to promote linearity using a soft constraint instead of a hard constraint. We believe that it provides the learning algorithm with the flexibility to enforce linearity if possible. In addition, if the data is too complex or the encoder is not expressive enough, the predictive loss term will be high, serving as a tool for identifying potential modeling problems. In practice, the datasets we consider in this work admit a high-level of learnable linearity as we show in Table 1 below. Specifically, we report the relative predictive loss error, i.e., $|z_{2:T} - A\bar{z}_{1:T-1}|^2 / |\bar{z}_{1:T-1}|^2$, where $z_{2:T}, \bar{z}_{1:T-1}$ are the latent variables and $A$ is the approximated Koopman operator. Importantly, all errors are below $6e-2$, where some errors are one- and two-orders of magnitude below that.

We extend our appreciation to Reviewer kti2 for recognizing the robust performance of our approach, its simplicity, and its capability to incorporate physical constraints. Moreover, we are grateful for their insightful comments and suggestions aimed at enhancing the paper and delving into a more profound discussion. Below, we provide responses to the comments raised by Reviewer kti2. If given the opportunity, we would gladly integrate the suggested modifications outlined below into the final revision.

A large literature of sequential VAEs for time series data generation is omitted e.g. [1,2,3,4], despite a large number of baselines being used in the experiments section. Considering there is heavy development in this area, it would be useful to compare KVAE to these methods.

Thank you for bringing these works to our attention. We agree with the reviewer that sequential VAEs are actively developed in recent years, on a broad range of applications. In our work, we focused on comparing with baseline generative approaches that target the same benchmarks and evaluation environment, for a fair and consistent evaluation of approaches. That being said, we are also happy to consider papers [1-4]. Specifically, paper [1] can be viewed as a close variant of our approach. In particular, omitting the predictive loss term in KVAE forms method [1]. The results of this method appear in our ablation study in Table 4 in the submitted paper. Please notice that there was a typo there, and all appearances of $\beta$ should be replaced with $\alpha$ (we have corrected this in the revised manuscript). Further, we are currently running method [2], and we will report the results as soon as we have them. As for method [3], it can be viewed as a VAE only in the time-continuous limit. Nevertheless, we are currently working on migrating the available code to fit into our training and evaluation framework. Finally, we could not find an open-source implementation of method [4], unfortunately. We started to implement this approach from scratch, however, the time constraints of the rebuttal may be too limiting to obtain preliminary results.

More discussion in the experiments section is required on the topic of analysing "the behaviour of the system...using dynamical systems theory tools" in order to claim this as an additional feature of KVAE

Following the reviewer's suggestion, we extended the discussion in Section 5.2 on the physics-constrained generation example. The revised text details the particular loss term $\mathcal{L}_\text{eig}$ that we use. Moreover, we analyze the nonlinear pendulum system from a stability viewpoint, where Koopman eigenvalues represent growth and decay modes. The discussion is complemented by Figure 3, which shows the spectra of Koopman operators for unconstrained KVAE vs. constrained KVAE. Our results indicate that constraining the eigenvalue modulus improves generation results and positively affects the operator's spectrum. We also compute the spectra associated with the computed operators in the regular setting and further discuss the results in Appendix F.