Contrastive Meta Learning for Dynamical Systems

Dear reviewer:

We thank you for the insightful questions, let us reply them in the following:

$\bullet$ Multistable dynamics:

we thank you for the observation, Figure 1 is just an intuitive explanation of the proposed method, and does not indicate that the system has to converge to a single attractor or behave periodically. In the contrastive learning part, we are learning a mapping from a truncation of trajectory (usually 5-10 steps) to the embedding which could represent the system coefficients. Taking an extreme but basic system identification problem with linear system ($dx/dt=Ax+b$), if we want to use a linear model to approximate it, we only need some time steps to characterize A and b, and we don’t need to assume the trajectories are converging or close to each other. Especially when we use the local linear feature extractor, the time evolution of the state matters much more than the absolute coordinates.

$\bullet$ Dimensional collapse for nonlinear correlation between different dimensions:

The covariance loss is inspired by Ref [1] below. Unfortunately, we cannot prevent this from happening by linear covariance regularizer as it cannot capture the nonlinear correlation. However, such a scenario is rare. Because the different dimensions are all from the last fully connected neural network, linear correlations are much more likely to happen. In practice, we never meet nonlinear correlations (e.g. the 2D embedding distribution shrinked into a curve/circle). The only scenario when the dimensional collapse happens shows a linear relationship in the embedding space.

Ref 1: VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, ICLR 2022

$\bullet$ Equation 5:

Yes, we agree with you that the noise might affect the derivative approximation for local feature extractor. In our method, this feature extract functions as a quick and rough pre-processing of the data. Since we use multiple steps to approximate each local $A$ and a trajectory of $A$ is flattened as LSTM input, we do not require each approximation to be accurate.

$\bullet$ More complex systems:

We provide high-dimensional system experiments in the grid-based system study (Section 5.2 where the system has an image-like input). Since this research serves primarily as a proof of concept in this new field, we do not include experiments with chaotic or bifurcation systems. Nevertheless, we anticipate that meta-learning's performance would likely degrade when dealing with chaotic systems, particularly when test coefficients produce chaotic behaviors different from those in the training data. This is because meta-learning aims to extract and generalize common patterns across different environments. In chaotic systems, even slight deviations from the training distribution can lead to dramatically different behaviors, which may be beyond the meta-learning algorithm's ability to capture. Again, our experiments are designed to demonstrate the feasibility of our proposed algorithm. Given that system identification of chaotic systems remains challenging even for traditional methods, we choose to maintain a focused scope rather than expand into more complex chaotic scenarios.

$\bullet$ qualitatively different systems:

Thank you for the suggestion for bringing up a new direction. If different types of equations are introduced, then the “environment” space needs to be redefined as a new functional space. The proposed method in this could be a starting point/baseline for this new problem. Suppose we have a few different equation classes, I would suggest using probabilistic focused contrastive learning losses (e.g. Info-NCE) to train it, as it fits better for classification problems than regression on continuous space. However, it still would be challenging for the meta learning method to perform well especially on an unseen type of equations.

$\bullet$ How many parameter sets are required for effective training?

In appendix A, we provide all the parameter set/experiment details for standard experiment in the main paper. We also provide extra experiments for the performance gain with the proposed method over the standard training method under different numbers of coefficient sets in table 5 in Appendix C. The proposed method achieves better accuracy than standard experiment across all the setups. While “effective training” is a relatively ambiguous word, we believe as long as the systems exhibit shared characteristics and distinct variations, the meta learning method can hold an advantage because it utilizes additional information from the datasets.

We thank you for the thorough reading of the paper and overall positive comment.

$\bullet$ Baselines:

As meta learning for dynamical systems is a new area, there is limited prior work. For the grid based PDE problem, we do compare to and slightly outperform the existing baselines such as DyAN ref[2] in our table 2 and 3, despite their encoder being learned in the supervised way instead of our unsupervised learning. This echoes with your comments “If the proposed method is still better than the baselines even though baselines require additional information, that would increase the contribution of the paper.”

For the vector based systems, there are existing works such as ref[3][4]. Similar to the paper you referred to for the more general meta learning setup (Ref [1]), they all require additional data for the fine-tune process when a new testing environment is introduced. In our problem setup, we do not have additional data for the new environments, therefore all the existing methods are not applicable and so we can only compare with standard training. To fairly compare with these baselines, we need to dramatically modify the training pipeline and the data generation process, and this opens a new discussion of how much data is needed for few-shot learning (if limited data is added, then our proposed method will perform better; otherwise the fine-tune methods will do certainly do better). Due to limited rebuttal time, we are not able to perform this level of code change and incorporate other people’s work into our repo. We hope you can understand this, as we are the first to propose unsupervised pre-training + zero-shot learning for meta-learning dynamical systems.

[1] Learning to Generalize: Meta-Learning for Domain Generalization. AAAI 2017

[2] Meta-learning dynamics forecasting using task inference, NeurIPS 2022

[3] Learning dynamical systems that generalize across environments. Advances in Neural Information Processing Systems, NeurIPS 2021

[4] Generalizing to new physical systems via context-informed dynamics model, PMLR 2022

$\bullet$ Histogram loss:

Thank you for your suggestion, we managed to test this loss function as baseline/ablation studies. In general, the performance of histogram loss is similar to triplet loss and worse than our proposed method. Here are the results for Histogram loss compared with standard training and our proposed method and we will update to our table 4 and cite the paper in the next version.

$\bullet$ Code release:

Yes we will surely release the code once the paper is accepted.

$\bullet$ lambda hyper-parameter:

We do provide the lambda tuning experiment in table 4 with lambda = 0,0.2,0.5. The experiments reveal a wide tolerance window for this tuning since 0.2/0.5 does not change the result dramatically and both of them can output an embedding mapping similar to Figure 3. The purpose of this regularizer is to prevent collapse of embedding dimensions and 0.5 is strong enough for this purpose. Further increasing the number will not improve it any more. Therefore we choose 0.5.

Dear reviewer:

Thank you for bringing out the question and we are glad to clarify.

$\bullet$ Equation 9:

“*” is just the multiplication sign, we use it to emphasize that this is element-wise multiplication.

$\bullet$ Explanation on how contrastive learning works:

In general, with the knowledge of trajectory categorization (e.g. which trajectories below to same system/different systems), we use contrastive learning to learn the encoder which maps a truncation of trajectory (multiple concatenated system states) to the system embedding. This idea is identical with the classic contrastive learning application in the computer vision field. The learned embedding contains the system coefficient information. In the forecasting stage, given a few past state observations (e.g $\{x_{t-3},x_{t-2},x_{t-1}\}$), we first use the encoder to infer the embedding, and the embedding is later used in the dynamics model to infer the system dynamics. Taking the true system function $dx/dt=f_\phi(x)$ as an example, where $x$ is the state and $\phi$ is the system coefficient. We know the output $dx/dt$ should be dependent on both $\phi$ and $x$, therefore a lump model only taking $x$ cannot identify the output accurately. In our method, the encoder is used to infer the embedding $z$ contains the information of $\phi$, and another dynamical model taking both $x$ and $z$ is used to infer $dx/dt$.

$\bullet$ Regarding your question on the organized embedding like figure 3:

To clarify, our unsupervised learning does not ‘’identify’’ the actual coefficient, it aims to learn a representation of the coefficients. This representation could be a linear transformation or more complex transformation. In these two problem setups, it turns out that the rotated linear transformation is the easiest way for the model to describe the embedding, therefore it looks organized and ordered. We also tried to decrease the “number of coefficient sets” and see the results, for these two problems, once we have more than 4 coefficient variants per dimension, the output is always organized. Consider the 1D case with only 3 coefficient sets (e.g. 1,2,3), the model cannot differentiate the magnitude because it only knows 1,2,3 are from different groups, therefore the learned embedding could be ordered as (1,3,2) instead of organized (1,2,3) or (3,2,1).

However, despite the mapping being non-linear (e.g. cubic function) or even non-monotonic (e.g. square functions), as long as the encoder contains useful information regarding the system coefficients and the testing coefficients are not too far away from training distribution, it will always benefit the forecasting stage.

$\bullet$ Regarding the question on what dynamical system our proposed method can “handle”:

We are sorry that we have an ambiguous understanding of “handle” here. If you mean “generate the organized embedding like Figure 3”, when we have two varying coefficients and they vary independently, and the number of coefficients set per dimension is not too small, we can capture that. If you mean “benefit the forecasting”, as we discussed above, if the encoder contains useful information, then it will improve forecasting performance.

Dear reviewer:

Thank you for the insightful review, while we agree with most of them, we find there could be some misunderstandings and we hope to clarify below.

$\bullet$ element-wise loss function:

The major contribution of our element-wise design of the contrastive loss is to prevent certain dimensions to be “constant”; this cannot be achieved by other loss functions (SRL,triplet,Info-NCE) as they directly compare two vector embeddings. Let us consider a simple case with three examples [[1,0],[2,0],[3,0]], where the second dimension faces dimensional collapse. For SRL, Triplet or Info-NCE, the loss function will be identical to [[1],[2],[3]] with fewer embedding dimensions. However, with our element-wise design, it will evaluate the 2nd dimension individually, and penalize the loss function by moving the denominator close to 0. In this extreme case the value will be $\frac{0}{0}$, while it will be a large value before reaching this numerical error.

$\bullet$ covariance-based regularization:

Yes, we agree with you that we directly use it from the cited paper for our purpose.

$\bullet$ local linear square feature extraction:

We might also have some misunderstanding here. It is not a linear approximation of the input space (e.g. outputs Wx with parameterized W and input x). Instead, it tries to estimate the system state function (e.g. $min_A ||Ax-\dot{x}||$) and use A to infer the system coefficient. This treatment is inspired by classical system identification, where the linear model is the to-go basis function to use and its learned variable directly contains the physical properties (system coefficients). Actually, even for linear problems, the mapping from the trajectory of x to A is quite complex. Therefore, using a directly deep learning method (RNN or MLP) cannot easily capture the features and thus empirically performs worse.

$\bullet$ unsupervision of system coefficients:

Yes we agree with your comment that many previous work assume the system coefficients are unknown. Although there are indeed multiple coefficients/systems, the users treat it as a single lump model and do not explicitly differentiate the systems. Differently, our approach explicitly learns the inter-system difference and uses it in forecasting. There are three folds of benefits: 1. Better interpretability as we explicitly utilize the inter-system information 2. Better forecasting accuracy because of the explicit system coefficients are indeed useful for forecasting (consider system state function f(e,x), if we know environment e, it will improve regression accuracy by a lot) 3. We also empirically showed the performance gain over vanilla neuralODE. As a distant analogy, our method vs lump method is like contrastive pretrain + supervised fine-tune vs pure supervised learning in computer vision. We incorporated the “class”/”category” information in training so we have better performance.

To clarify our claim in line 441-444 “Previous research …, assuming the system coefficients are already known”, by ”previous research” we specifically mean the previous works in meta dynamical system learning that explicitly differentiate inter-system differences. While in the majority of system identification/dynamical learning papers, most of the papers do not differentiate that. This is consistent with your impression. We will update the wording into “Previous research in meta dynamical system learning” in the next version to make it clear.

$\bullet$ missing experiment details:

We delayed the experiment details and coefficient distribution of table 1,2,3 to appendix A: A.1 for dual spring mass system, A.2 for LV system, A.3 for fluid system, A.4 for GS system. Throughout all experiments, the testing environment is unseen to the training process, therefore, we claim “generalize from a set of training systems ϕtrain to new, unseen test systems with properties ϕtest”.