Meta-Learning Nonlinear Dynamical Systems with Deep Kernels

The paper mainly compares with [1], which is a single task approach, and is slow for using ODE/PDE solvers. The main argument made here is that the proposed approach uses inference to get rid of the numerical solvers, and uses task encoding to broaden to multi-task setting, and so when applied to many tasks at the same time (about 200 as reported in Figure 6), the proposed method will achieve both a quick inference and an accurate one. However, training the proposed model requires an expensive generation of training datasets, which is not required in [1]. I understand that the approach is trying to show task representation is useful in generalizing to fast multi-task setting, but some information on the training cost of the proposed approach compared to that of [1] would be appreciated.

Besides, the empirical results are mostly focused on very low-dimensional tasks, and are not showcasing the power of LFMs. It would be great to see some high-dimensional experiments.

[1] Jacob D Moss, Felix L Opolka, Bianca Dumitrascu, and Pietro Lio. Approximate latent force model inference. arXiv preprint arXiv:2109.11851, 2021.

The model formulation needs clarification.

• Missing observation (output) $\mathbf{y}$ likelihood.
• Latent force $\mathbf{f}_n$ and $f$ (likelihood) are the same or not?
• Better use symbols consistently. The same symbols in GP preliminaries and model formulation with different meanings are confusing.

The figures captions lack description.

It is unclear how this is and should be in framework of meta-learning. What meta knowledge is learned and used, the task representation? The encoder for task representation is undermotivated.

However, despite the strengths, the paper has a few major weaknesses. I will focus especially on the Experiments and Related Works sections.

Major weaknesses:

1. The authors place their work within the Meta-Learning field, but they do not compare the proposed model with any other Meta-Learning method. Moreover, the lack of even mentioning the Meta-Learning approaches within the Related Works section. Unfortunately, it is an important issue, because the Meta-Learning field is known from considering many GP-based approaches. To mention just a few: [1], [2], [3], [4], and [5]. I strongly suggest to compare the proposed method with other Meta-Learning approaches. It could also be a good source of new ideas for the future work.
2. If I understand correctly, the authors for each experiments used the same kernel - RBF. However, in the Deep Kernel setting, we can utilize any kernel function we would like - even the scalar product in an embedding space is able to imitate many other kernels. From my perspective, the lack of comparison across various kernels (e.g., family of Matern kernels, spectral or cosine kernel) is another important issue. Even if the RBF kernel would be the best for all of the presented experiments, I would like to see any justification or ablation study proving that.
3. The DKLFM method is compared with only two other methods: DeepLFM and Alfi, from which one is providing the exact solutions. However, taking into consideration the Results in Table 2, we can see that even the DeepLFM is significantly better than DKLFM. I would like to see incorporated DeepLFM into similar comparison as presented in Figure 6.

Minor weaknesses:

1. I am not particularly sure if the MSE is the best metric to compare between the solutions. The interesting will be to see how it looks like in different measures, like NLL/ELBO, since it should be available in the GP setting.
2. I am not certain if I understand correctly the comparison between Alfi, DeepLFM and DKLFM. What is the size of datasets on which each result in those tables are computed? If I understand it looks like this: the results for Alfi and DeepLFM are from the set of 20 examples and the results for DKLFM is from the set of 256 examples? Moreover, regarding the computation times if they are averaged across all examples? Please if you could elaborate more on this? If the results for 2 methods are from the smaller set of examples than for the last method, it will not be a fair comparison.

References:

[1] Snell, J., & Zemel, R. (2020, October). Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes. In International Conference on Learning Representations, 2020.

[2] Patacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. J. (2020). Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 16108-16118.

[3] Wang, Z., Miao, Z., Zhen, X., & Qiu, Q. (2021). Learning to learn dense gaussian processes for few-shot learning. Advances in Neural Information Processing Systems, 34, 13230-13241.

[4] Sendera, M., Tabor, J., Nowak, A., Bedychaj, A., Patacchiola, M., Trzcinski, T., ... & Zieba, M. (2021). Non-gaussian gaussian processes for few-shot regression. Advances in Neural Information Processing Systems, 34, 10285-10298.

[5] Chen, W., Tripp, A., & Hernández-Lobato, J. M. (2022, September). Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. In The Eleventh International Conference on Learning Representations.

• The use of terms such as “multi-task modeling” and “meta-learning” may require some careful considerations. The task in the manuscript is defined as each data instance. For example, in the dynamics modeling context, a single task here corresponds to learning a dynamics function (or inferring a parameter as in system identification) from a set of measurement, which is collected from simulating a set of ODEs with a specific parameter. This way of defining “task” may be considered as a non-standard way in “multi-task learning” and, thus, a proper definition would be needed. Relatedly, the use of the terminology “meta-learning” also seems to require proper justification. Without these, the current method can be seen as a single framework, which is a mixture of parametric and non-parametric ML algorithms, where the parametric part of the framework is being trained on multiple input instances (which are denoted as tasks in the paper) via likelihood maximization with a regular gradient descent algorithm (without specifically learning meta-learned model parameters).

• The assumption on the availability of the latent force seems to be less practical. As shown in one of the benchmarks (the ODE benchmark), it does seem that latent force needs to be inferred from another software package (Alfi). The paper does not provide much information on how reliable this process is or if there are other alternatives for collecting latent forces (if they are not readily available). Based on this limited information, it is less convincing if the proposed method is practical. A side note is that the software, Alfi, seems to be based on the algorithm, appeared in a preprint, which has not been published; this makes it ever harder to assess the practicality of the proposed method.

• The description of the experimental section is not very clear. There seem to be two separate ODEs, mRNA and LV. But the descriptions on those two ODEs are mixed together and it’s unclear how the tasks are defined and how the actual experiments have been performed. Moreover, it is unclear if the authors provide all sufficient information on their experiments.

• An empirical side of the paper is weak. The proposed method has been tested on relatively simple sets of benchmark problems, which does not seem to provide much insight on the effectiveness of the method or the scalability of the method. It does not seem that there is no place where the scalability of the method is discussed. Finally, the output MSE of the method on the second benchmark problem seems to be very poor.

[Update after the rebuttal] I acknowledge that I have read the authors' responses and I left my comments on the authors' responses and the revised manuscript. As in my comment, I will keep my score.