Continual Slow-and-Fast Adaptation of Latent Neural Dynamics (CoSFan): Meta-Learning What-How & When to Adapt

Presentation of results and figures

We agree that some of the figures’ presentation could be improved. We plan to continue to improve the quality of the figures.

Real world datasets

We acknowledge the need of future works to extend CosFan to real-world datasets, although the benchmark data we considered are quite representative of what are being used in the current literature of latent dynamic modeling. One challenge of identifying appropriate real-world datasets is that we are looking for time-series of high-dimensional data (e.g., image sequences) where the dynamics being learned should be in the latent space (not directly in low-dimensional data space as many available real-world time-series datasets).

Comparison on CML performance trade-offs

We did show improvements in both accuracy and speed by CosFan over MAML-based approaches, and provided further training performance comparisons to BGD-based baselines in Appendix B.2. A particularly notable performance trade-off between gradient-based methods and the feed-forward methods is that, when bi-level meta-optimization is enabled via the Task-Aware Reservoir Sampler, the gradient-methods face significant slowdowns in the face of multiple task IDs as they must sequentially process the per-task losses before aggregation to the meta-loss. We show that feed-forward methods are agnostic to the number of present dynamics within a batch and are easy to parallelize. We would like to hear from the reviewer any additional suggestions to further improve these results and discussion.

Computational cost of GMM cluster/Task-Relational replay

We appreciate the reviewer’s feedback regarding the lack of documentation on computational costs and agree that this is an important aspect to address. In response, we have added Appendix C.2, which provides analysis of memory and processing requirements across a range of reservoir sizes and increasing numbers of unique tasks. Additionally, we include a Time-to-Train comparison between the Task-Aware and Task-Relational mechanisms. Our findings show that the Gaussian Mixture Model (GMM) exhibits favorable scalability in both memory and processing time. The memory requirements for fitting the GMM on the meta-embeddings are minimal, largely due to the benefit of embedding meta-knowledge into a lower-dimensional space.

Gradual task shift experiments

We agree that exploring the stated limitation of detectable task shifts is an important aspect to include within the work. Please refer to our overall response 1 where we describe an ablation study and discussion over blurred task boundaries that was added to the work.

Hello,

Thank you for addressing all the points and the additions to appendix C.

Please make a real effort in addressing point 1 on the presentation, figures 15-19 of said appendix would be a good start.

The added results on the task boundaries are very interesting, however the std of the actual data points in F.23 & 24 is very large. I am not sure they are small enough to make clear conclusions. If your pipeline is setup, running a few more seeds would be quite beneficial.

Almost missed this: you have a figure mis-reference at the start of paragraph 3 of C.8. should be 23&24 if I'm not mistaken.

Comparison to additional meta-learners

We appreciate the reviewer’s suggestion that incorporating a comparison with sequence learners, such as Transformers, could enhance the robustness of the meta-learner baselines. Based on relevant work [1, 2], we have identified two promising directions to include sequence learners in our evaluation and have added an additional Related Works section dedicated to additional algorithmic priors to consider, including sequence learners. Chen et al. [1] propose a transformer-based meta-learner that combines a shared initial parameter set with available data tokens to adapt the weights, which would be conceptually similar to gradient-based meta-learner adaptation. Vladymyrov et al. [2] present a recent CML approach using a Transformer as a hyper-network to generate target network weights based on the context set. In their work, the generated weights of the previous task are used as parameter tokens for the current task’s weights, alongside active task samples. They omit the use of a replay buffer, using only the weights updated over time on active samples. While their methodology primarily focuses on image classification, adapting their approach to our latent dynamics setting is an interesting direction for future exploration. We have added this discussion in Appendix A with a reference to it in the conclusion of the main text.

We refer to the overall response 2 regarding the feed-forward meta-learner and additional reasons for choosing the hyper-network specifically. Additionally, we would like to note a key advantage of our current hyper-network architecture: its parameter efficiency, as it generates the target network’s weights from a low-dimensional context embedding. Given the well-documented scaling limitations of Transformers, applying them to our setting of high-dimensional time-series may pose significant computational challenges, particularly when generating full target network parameter sets. While we have begun implementing these baselines, adapting these architectures to the forecasting setting is non-trivial. Due to the complexity of this adaptation and the rebuttal timeline, we may not be able to provide results within this review period.

References: Chen, Yinbo, and Xiaolong Wang. "Transformers as meta-learners for implicit neural representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. Vladymyrov, Max, Andrey Zhmoginov, and Mark Sandler. "Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learning." Transactions on Machine Learning Research.

Explanation of s in T^s_j

$s$ refers to the context set for meta-learning. To support clarity in the text, we have added an explanation to the main text.

Line 216, variable z

$z$ refers to the latent space which the dynamics function forecasts in, and the encoder/decoders embed to and from. Section 3 Problem Formulation details what $z$ represents as well as what the equation $\mathbf{z}_t = f_\theta(\mathbf{z}_{<t}; \mathbf{c}(\mathcal{T}_j))$ represents.

Equation 3, variable l

Variable $l$ refers to the length of the observation subsequence $\mathbf{x}^q_{j,0:l}$ available to the initial state encoder derived from the full ground truth sequence $\mathbf{x}^q_{j,0:T}$. We recognize that the definition of $l$ is missing from the text and have added an explanation. Thank you for the catch.

Exact Replay metrics

Thank you for the astute observation. Based on this remark, we re-evaluated the exact-replay implementation for all baselines and identified an issue in metric reporting, which led to incorrect results. We have rerun all baselines on exact-replay and have updated figures and tables showing that, as expected, exact-replay performs the best overall in general.

Table 1 updates

We have added a description of what MAML-1 and MAML-5 represents in the table caption, as well as adding the unit of each metric to the table itself. We define metrics TTA-1 and TTA-12 within the relevant metric section, which due to space constraints of elaborating that metric, we hope is sufficiently clear.

Component motivation and trainable embedding networks

Please refer to our overall response 2 for clarification on the feed-forward methodology and experiments on comparing alternative conditioning mechanisms, such as trainable embedding networks.

Task boundary detection novelty

Please refer to our overall response 3 for clarification on the novelty of the task boundary detection mechanism and the primary contribution of the work.

Clarification of notations

Thank you for bringing up confusions in the notation. Indeed the superscript s refers to the context set for meta-learning. We have done a passthrough notation and have cleaned up notation throughout the paper, including making the internal sequences of T_j dependent on j.

Clarification of metrics and intended results in Figures 2 and 4

We used both MSE and DST across all experiments as MSE measures averaged pixel-level accuracy while DST the average Euclidean distance between the predicted object’s location and its ground truth. Due to space restrictions we are not able to report complete results from all metrics in all experiments in the main text – we thus decided to use DST as an example in Fig 2 whereas MSE in Fig 4; complete results on both metrics are in Appendix.

Results in Figure 2 and Figure 4 are intended to demonstrate different points. Results in Figure 2 (Section 5.2) are to show that both continual and meta methods are required simultaneously to achieve good performance. The non-meta models use CL strategies (except in naive learning) without context data to demonstrate the benefit of meta-learning (i.e., knowing how to adapt) in this setting. Because latent dynamic forecasting has not been studied in this setting of non-stationary and heterogeneous dynamics distributions, we felt that it is important to first demonstrate that both continual and meta components are essential in this setting. Once this is established in Section 5.2 / Figure 2, Figure 4 (and Section 5.3) then shows how the proposed continual and meta strategies improve over existing CML works (the baselines).

Task shift experiments

We agree with the reviewer’s observation that the method performs slightly worse than MAML-based meta-learners in the task-agnostic setting. Please refer to our overall response 1 for an additional ablation study we performed on gradual task shift experiments and additional justification as to the advantages of Task-Aware replay over Task-Agnostic replay.

Differences between Task-Agnostic and Task-Aware methods

Indeed each task is trained with the same number of samples, and both Task-Agnostic and Task-Aware Replays maintain an approximately equivalent sample distribution across tasks during training, based on similar usage of the Reservoir Sampling algorithm.

Please refer to our overall response 3 for our clarification on the key difference between the Task-Aware methodology over the Task-Agnostic approach.

Hypernetwork scaling

There is no required scaling with respect to the number of tasks in either the parameter or input size of the feed-forward hyper-network approach. It is more a matter of representational capacity of the network with respect to the complexity of the underlying task diversity and distribution. We argue that the algorithmic prior of this learned function transformation provides better scaling than gradient-based techniques, which have to share manifold capacity from an initial static point across the tasks. Computationally, the feed-forward adaptation approach benefits from being inherently agnostic to the number of heterogeneous tasks and efficiently parallelizing the adaptive forward pass across context sets. In contrast, gradient-based meta-learners require costly per-task test-time fine-tuning. This efficiency advantage is demonstrated in the adaptation efficiency comparison presented in Table 1.

Clarification on context-query pairing

For the task which is actively streaming in T_j, the previous k sequences are treated as the context set for the active samples. However, when we sample from the reservoir to get past-task query samples for experience replay, we additionally sample a relevant context set from the reservoir based on their assigned pseudo-labels (obtained by our two task identification strategies)

3. Clarifying on Contribution Related to Task Identification and Bi-Level Optimization (Reviewer GEod).

We would like to clarify the benefit of the Task-Aware methodology over the Task-Agnostic approach as it underpins a key contribution of the paper: our main innovation is not on the boundary detection mechanism, but the task identification strategies used to enable per-task context-query pairing to enable bi-level meta-optimization that is not achieved in existing CML approaches. The Task-Agnostic Replay mechanism does not leverage task information, relying instead on the approximate equivalence between meta-learning and continual learning objectives in aligning the current task’s gradient with the average gradient of previous tasks. This involves adapting only on the current task’s context set and using the resulting parameter set on the past tasks’ samples to get the meta-loss. While sufficient for image classification and low-dimensional regression problems in prior work, we found this approach inadequate for latent dynamics forecasting. In contrast, the Task-Aware Replay mechanism incorporates the estimated task ID via the boundary detection mechanism, enabling traditional per-task context-query pairing for reservoir samples. This allows each task to adapt to its relevant context set sampled from the reservoir, with per-task losses aggregated to form the meta-loss. The enables meta-learning to be done in a continual fashion without compromising full bi-level meta-optimization, which we believe was the cause of the significant performance gain in the proposed task-aware over task-agnostic settings across both the gradient-based and our considered feed-forward meta-learners.

4. Documentation of Computational Costs (Reviewer JYok).

To address suggestions regarding the computation cost of the Task-Relational Experience Replay, Appendix C.2 has been added to document the computational costs of Gaussian Mixture Model (GMM) clustering and task-relational replay. We analyze memory and processing requirements across varying reservoir sizes and task numbers and include a Time-to-Train comparison between Task-Aware and Task-Relational mechanisms. Our findings show that the GMM scales favorably in both memory and processing time, with minimal memory requirements due to the low-dimensional embedding of meta-knowledge.