Skip the Equations: Learning Behavior of Personalized Dynamical Systems Directly From Data

Dear Reviewer htLB,

Thank you so much for such a comprehensive review. We deeply appreciate your time and attention spent on our paper. We are glad that you found our methodology interesting, comparisons comprehensive, and claims justified. We answer the six weaknesses in your summary, then the two questions. Finally, we address your comment about the incremental nature of our work.

W1. Datasets with more than one dimension

We have now added experiments on the viral dynamics model of HIV [1] ($M=3$) in two noise settings, where the features include not only the initial conditions but also four additional parameters. We have also tested our method on the SIR model with more noise. Overall, we have added three additional multidimensional datasets (all results can be seen here.

The main challenge of extending Semantic ODE to multidimensional systems lies in the fact that these trajectories no longer can depend on a single initial condition $x_0\in\mathbb{R}$ but on a multidimensional $\mathbf{x}_0 \in \mathbb{R}^M$ (where $M>1$). Thus, the main contribution is the extension to multidimensional inputs, which we demonstrate in all our experiments.

We hope that this explanation and the additional experiments provide sufficient evidence for our claim.

Actions taken: Added additional experiments on multidimensional datasets in Appendix A.

W2. Details of $F_{\text{traj}}$

Actions taken: Added description of the trajectory predictor in Appendix E as well as details on compositions, motifs, and properties to make the paper self-contained.

W3. Similar examples in the introduction

Actions taken: Replaced some of the examples and references in the introduction to limit overlap with the paper on Semantic ODEs.

W4. Predicted trajectories

Actions taken: Included plots of sample trajectories predicted by EPISODE in Appendix A. These can be seen here.

W5. Metrics used

Please see our response to Reviewer g6tH (Evaluation metric).

Actions taken: Described the used metric in Section 5.

W6. Computational cost

We have now added a table to Appendix D showing the overall computational cost of all approaches (including five runs and hyperparameter tuning). We also show the individual times to train the composition map and the property maps. All experiments were performed on an 18-core Intel Core i9-10980XE with 60GB of RAM.

As mentioned in the limitations, training of the composition map is the most time-consuming process as it requires fitting every admissible composition to each sample (this can be parallelized). Note that we perform this preprocessing once for all five seeds and then subsample the results based on the split. The actual time to fit the decision tree is negligible. A more detailed breakdown of the time needed to train the composition map is provided in the following table. We discuss it further in our response to Reviewer evCm (Q1).

Actions taken: Added computation times in Appendix D.

Q1 Time horizon

Each dataset has a specific time horizon that is subsequently scaled to $(0,1)$. The details are in Appendix D.1. However, for unbounded motifs, our model can predict for any time $t\geq0$. We evaluate each model on a on a held-out dataset of samples, whose trajectories are observed at the same time points as the ones in the training dataset.

Q2 Robustness

Actions taken: Added additional five experiments on datasets with higher noise ($\sigma=0.1$) in Appendix A. EPISODE shows robustness to noise in these settings. The results can be seen here

Minor

Actions taken: Fixed the typo on page 4 (it should have been $\mathbf{v} \in \mathcal{V}$).

Incremental contributions

Semantic ODE only works for one-dimensional trajectories predicted from one-dimensional inputs. We have extended it to more realistic settings. In particular, we

1. Extended the composition map from interval partition to a decision tree and designed a novel optimization algorithm for fitting it.
2. Extended the property functions from univariate functions to GAMs and designed a novel optimization algorithm for fitting GAMs such that the predicted properties are consistent with the composition.
3. Improved composition map training by using Dynamic Time Warping distance.
4. Accommodated bounded compositions ($t_{\text{end}}\in\mathbb{R}$).
5. Accommodated categorical variables as features.
6. Implemented a visualization tool to visualize the fitted GAMs.

References

1. Hill, A. L., Rosenbloom, D. I., Nowak, M. A., & Siliciano, R. F. (2018). Insight into treatment of HIV infection from viral dynamics models. Immunological reviews, 285(1), 9-25.

---

We hope we have addressed all your concerns. Please let us know if you have any additional questions, we are eager to address them.

Kind regards,

Authors

Dear Reviewer g6tH,

Thank you very much for your review. We appreciate your time and effort spent reviewing our paper. We are glad you found our conceptual framework solid, the case study well-motivated, and the comparisons comprehensive. We address your comments below.

Reliance on inductive biases

The method seems reasonable but highly relies on the incorporation of semantic inductive biases.

Thank you for this comment, as it allows us to clarify the experimental setup in our paper. Although our method can incorporate semantic inductive biases, it does not require this information. The results in Table 2 (EPISODE) are for a generous set of compositions without much prior knowledge. We consider all compositions up to length 4 (or length 8 for the Bike sharing dataset) except for compositions where two consecutive transition points are inflection points. This ends up being, respectively, 26 and 34 compositions. Thus, EPISODE achieves competitive results for a relatively large choice of compositions and minimum prior knowledge. Often, the results are not significantly different for EPISODE*, where we incorporate such knowledge (see Tacrolimus or Bike sharing dataset). Prior knowledge is, however, useful in reducing the training time, satisfying requirements, and improving extrapolation.

Actions taken: We have now clarified the amount of prior knowledge used in experiments in Section 6.

End-to-end

Additionally, the realization is not end-to-end, requiring manual verification and editing ...

We would like to clarify that, although manual verification and editing are big advantages of our approach, they are not necessary. Although the training process consists of two steps (training the composition map and then the property maps), the whole pipeline is end-to-end. There is no manual intervention in experiments performed in Section 6.

Actions taken: We have now clarified the absence of manual intervention in experiments in Section 6.

Evaluation metric

Thank you for pointing this out! We have now described the used metric in Sections 5 and 6. The error is given by

$\mathcal{L} = \frac{1}{D} \sum_{d=1}^D \sqrt{\frac{1}{N_d} \sum_{n=1}^{N_d} \left(\frac{1}{M} ||\mathbf{F}(\mathbf{v}^{(d)})(t_n^{(d)})-\mathbf{y}_n^{(d)}||_2^2\right)}$

where $\mathbf{F}$ is the predictive model (EPISODE), $D$ is the number of samples, $M$ is the dimensionality of the system, $N_d$ is the number of measurements of the $d^{\text{th}}$ sample $(\mathbf{v}^{(d)}, (t\_n^{(d)}, \mathbf{y}\_n^{(d)})\_{n=1}^{N\_d})$. We choose this metric because for $M=1$ it reduces to the standard mean RMSE, i.e., $\frac{1}{D}\sum_{d=1}^D \sqrt{\frac{1}{N_d} \sum_{n=1}^{N_d} (F(\mathbf{v}^{(d)})(t_n^{(d)})-y_n^{(d)})^2}$ (used in Semantic ODE paper) and we normalize by $M$ so that it is easier to compare results between systems of different dimensionality.

The error is calculated on a held-out dataset of samples, whose trajectories are observed at the same time points as the ones in the training dataset.

Actions taken: Described the used metric in Sections 5 and 6.

Computational cost

We have now added a table to Appendix D showing the overall computational cost of all approaches (including five runs and hyperparameter tuning). We also show the individual times to train the composition map and the property maps. All experiments were performed on an 18-core Intel Core i9-10980XE with 60GB of RAM.

As mentioned in the limitations, training of the composition map is the most time-consuming process as it requires fitting every admissible composition to each sample (this can be parallelized). Note that we perform this preprocessing once for all five seeds and then subsample the results based on the split. A more detailed breakdown of the time needed to train the composition map is provided in the following table. We discuss it further in our response to Reviewer evCm (Q1).

Actions taken: Added computation times in Appendix D.

Performance

We have run additional experiments, including a new dataset with the viral dynamics model of HIV and settings with higher noise. All of the results can be seen here.

EPISODE* achieves nearly perfect performance on the SIR dataset, demonstrating superior noise robustness to SINDy whose constrained search space should have given it an advantage on this simple problem. Both variants of EPISODE show superior performance on the PK and HIV datasets (in both low and high noise settings) compared to both ODE discovery and black-box models. Both variants also achieve superior performance to ODE discovery methods on the two real datasets (Tacrolimus and Bike sharing).

---

We hope we have addressed all your concerns. Please let us know if you have any additional questions, we are more than happy to address them.

Kind regards,

Authors

Dear Reviewer evCm,

Thank you very much for your positive review! We are glad you found our method novel and our claims well-evidenced. We reply to your questions first, and then we discuss other weaknesses and suggestions.

Q1. Runtime complexity of the composition map training

As mentioned in the limitations, training the composition map is the most time-consuming process as it requires a preprocessing step of fitting every admissible composition to each sample (this can be parallelized). The actual time to fit the decision tree is negligible in our experiments. This preprocessing step has time complexity $O(D M |\mathcal{C}'|)$ where $D$ is the number of samples, $M$ is the dimensionality of the trajectory, and $|\mathcal{C}'|$ is the number of admissible compositions. Crucially, this time does not depend on $K$ (the dimensionality of static features $\mathcal{V}$).

We have now included a table showing the time needed to train the composition map for each dataset (see here). We currently use a computationally intensive procedure of minimizing a Dynamic Time Warping (DTW) distance and then the standard MSE error. There is a trade-off between the accuracy of fit and the computation time. Practitioners may opt for more approximate fits (e.g., by not using DTW).

The advantage of training the composition map separately from the property map is that the errors table $e^{(d)}[c]$ (Eq. 1) can be computed once, saved, and then reused for different training settings of the composition decision tree and the property maps. This makes introducing changes to the model much quicker. The training time of the property maps (for five seeds and with tuning) can be found here.

Actions taken: Added a discussion on the computational complexity in a newly created Appendix F (Additional Discussion) and tables with computation times in Appendix D.

Q2. Sensitivity to the initial choice of compositions

The results in Table 2 (EPISODE) are for a generous set of compositions. We consider all compositions up to length 4 (or length 8 for the Bike sharing dataset) except for compositions where two consecutive transition points are inflection points. This ends up being, respectively, 26 and 34 compositions. Thus, EPISODE achieves competitive results for a relatively large choice of compositions (and minimum prior knowledge). EPISODE* uses prior knowledge about the shape of the compositions, constraining the set to between 1 and 4 compositions depending on the dataset (details in Appendix D.3). We can see that it significantly improves the performance in some settings (Tumor dataset) but often has no significant impact (Bike sharing dataset or Tacrolimus). In these settings, EPISODE is able to autonomously determine which compositions should be used in the composition map. This finding generalizes to the additional experiments that we have run (see here).

Actions taken: Added a discussion on the sensitivity of the method to the choice of the set of admissible compositions in a newly created Appendix F (Additional Discussion).

Q3. Automatic strategy for composition selection

As described above, the algorithm can usually autonomously determine which compositions should be used in the composition map. Thus, a viable approach is to start with a large number of compositions and let the algorithm narrow it down. In the future, we would like to design a fast algorithm that provides an approximate fit between a composition and a trajectory that narrows down the search space before the actual fitting.

Q4. Periodic or oscillatory behavior

EPISODE could, in the future, be extended to periodic or oscillating trajectories by extending the definition of semantic representation to accommodate periodic compositions. Let us assume that we know that the trajectories are described by an infinite composition consisting of repeating $(s_{+-c},s_{--c},s_{-+c},s_{++c})$. Then we can describe this composition segment using properties like "amplitude" or "frequency". For each trajectory in our training dataset we can extract additional trajectories describing how these properties change over time. We can now model these auxiliary trajectories using a framework similar to EPISODE. However, we leave a full exposition of this idea for a future paper.

Actions taken: Added a discussion on extending EPISODE to infinite compositions in a newly created Appendix F (Additional Discussion).

W1. Finite compositions

Please see our response to Q4.

W2. Complexity of composition map training

Please see our response to Q1.

Minor

Actions taken: We have now added explicit guidelines on interpreting GAM plots in Appendix E.

---

We hope we have addressed all your concerns. Please let us know if you have any additional questions, we are more than happy to address them.

Kind regards,

Authors