Continuous Temporal Domain Generalization

Thanks for reviewing and acknowledging our work! We see your main concerns are about assumption and evaluations. Please read our answers along with the Rebuttal PDF.

W1. discuss more the limitations; Assumption 1 that states the conditional probability distributions follow an ODE, need to discuss the validity. abrupt changes throughout time that might not be well captured by a continuous ODE formulation; this assumption is likely to be violated in real-world settings.

1. Assumption 1 does not state that the conditional probability distributions P(Y(t)∣X(t)) follow an ODE. Rather, it assumes that if continuous temporal domains exhibit gradual concept drift, then the conditional probability distributions P(Y(t)∣X(t)) change continuously. In this context, 'change continuously' means that the distribution changes are smooth and incremental. This assumption of continuity is reasonable because, in the field of concept drift research, gradual (or incremental) concept drift is considered to be characterized by small, incremental changes[R1]. While in real-world gradual concept drift is more complex and may imperfectly fit the continuity assumptions, here we do not lose much generality, as the underlying factors leading to gradual concept drift are primarily continuous physical, biological, social, or economic processes.
2. Given the continuity of gradual concept drift, it is appropriate to use differential equations as a tool for modeling. Differential equations provide a mathematical framework to capture the essence of continuous processes. They are particularly useful when the system's evolution can be represented by smooth and continuous dynamics. We chose to model the continuity of model parameters using ODEs as an initial attempt because ODEs offer a simple yet clear and flexible toolset for understanding how systems evolve over time. While it is possible to replace ODEs with more advanced differential equations (e.g., Partial Differential Equations (PDEs), Stochastic Differential Equations (SDEs)) to incorporate factors such as noise and uncertainty, this may detract from the key focus on modeling the fundamental continuous processes. Indeed, our experiments on both synthetic and real-world data have shown that even the simplest ODEs are enough to achieve a pretty significant success.

Discussion of Limitation:

1. The limitation regarding capturing abrupt drift arises from the assumptions inherent in the research area of Temporal Domain Generalization (TDG) framework. TDG assumes that domain pattern drifts follow certain predictable,smooth patterns, allowing for modeling future changes as a sequence. CTDG extends TDG to continuous time, further generalizing the problem to handle domains collected at arbitrary times. Thus, their assumptions are aligned in focusing on smooth, predictable concept drift.
2. To address abrupt drift, alternative frameworks are required. These include domain-invariant learning in static models, or ensemble learning/importance weight sampling with a detect-and-restart strategy. These methods operate under different assumptions and frameworks compared to TDG and CTDG. It can be challenging to determine which model is superior without considering the specific application scenario, as different models come with different implicit biases.
3. However, different frameworks can be combined to address the full spectrum of domain generalization challenges. Our approach does not conflict with methods designed to handle abrupt drift, such as ensemble learning or detect-and-restart strategies. This integration could create a more robust and adaptable system. Exploring such a comprehensive approach may represent a promising direction for future work.

W2. the paper does not seem to follow the checklist; there is no justification for each points and if I am not mistaken, some "Yes" answer are not correct e.g. a Broader Impact section is missing.

We apologize for our all answers are yes. We review and justify each point in the checklist to ensure compliance.

1. Broader Impact: There is no potential negative societal impact is found in this work. our research is foundational research and not tied to particular applications.
2. Q11, Yes should be NA.
3. Q13, Yes should be NA.
4. Q14, Yes should be NA.
5. Q15, Yes should be NA.

Q1. How does it compare to domain invariant approaches (e.g. IRM, V-REx etc.)? A domain-invariant approaches might be better at handling abrupt changes.

1. We implemented IRM and V-REx, we also implemented an advanced adversarial learning-based baseline CIDA [42] for comparison. CIDA is a powerful strategy built upon adversarial learning. It enhances domain-invariant representations by considering the distance between domain indexes.
2. Results are shown in Table 4 (in rebuttal PDF), demonstrating that our proposed Koodos performs competitively against these methods. The effect of Koodos is still by a huge margin.
3. However, as we discussed before, different frameworks can be combined to address the full spectrum of domain generalization challenges. We think exploring a comprehensive system is a promising future direction.

[R1] Learning under Concept Drift: A Review, TKDE 2018

Thanks for reviewing.

W1. needs more empirical studies.

1. The Koopman operator is proven theoretically that can linearize any nonlinear dynamical system in [4].
2. Our empirical evidence supports its applicability across various domains. Datasets are summarized in Tab. 5
   • Experiments span a wide range of real-world domains (7 in total): social media, culture, climate, economics, business, physics, and energy.
   • Experiments includes various modalities (tabular, image, text) and tasks (classification, regression).
   • Experiments use high-dimensional and complex datasets. : Yearbook (32x32=1024 pixels), Cyclone (64x64x2=8192 pixels).
   • Datasets from complex, noisy real-world sources: Influenza (millions of tweets for flu prediction), Cyclone (satellite images for weather systems).
3. Koodos demonstrates robustness and significant success across all 12 datasets, which evidence supports the effectiveness of it and highlights its applicability across various domains.

W2. loss function

We conducted three rounds of convergence tests on the 2-moons dataset. Fig. 11 (rebuttal) shows that:

1. The loss function and its terms converge effectively, showing Koodos' robustness and ease of training.
2. Each term in the loss functions is easy to train and converge using off-the-shelf optimizers such as Adam without requiring special optimization tricks.
3. Ablation tests (Section A 1.5) show performance drops when any term is removed, justifying their inclusion.

W3. method complexity. A hyperparameter sensitivity.

The loss weights are designed to balance each term's magnitude, ensuring no term dominates training. for example, 2-moons is set 1,100, and 10 as each loss around 1,0.01,0.1. Koodos is designed to be simple, comprising only a Koopman Network and a differential equation solver. We conducted sensitivity analysis, shown in Fig.10

1. Koodos exhibits stable behavior and robust convergence across wide varied hyperparameters, evidencing its ease of training, robust convergence, and insensitivity to hyperparameters.
2. Setting the loss weights in accordance with the magnitude of each loss term is sufficient for achieving good performance.
3. A few hundred Koopman operator dimensions are sufficiently good;

W4. Theorem 2

We analyze the risks associated along the time line based on the cumulative error on the training set, and it will propagate to the test set. So the cumulative error on the test set will also be smaller.

W5. comparison limited. AdaRNN Diversify not discussed. Including more TDG

1. AdaRNN and Diversify address different problems compared to CTDG/TDG:
   • Different Objective: The core problem AdaRNN and Diversify solved is creating domain-splitting algorithms to solve the problem of lacking predefined domain divisions in time series classification datasets. In contrast, TDG/CTDG have predefined domains, with the primary challenge is capturing the evolutionary pattern among temporally sequential domains and generalize the pattern to future domains.
2. We recognize that AdaRNN and Diversify used adversarial learning strategy DANN. Therefore, we implemented CIDA, an advanced adversarial learning baseline compared to DANN. We also implemented IRM and V-REx to together show domain-invariant learning effectiveness. Results in Tab. 4 show none of these methods having fair results.
3. We implemented the state-of-the-art discrete TDG method TKNet. Results in Tab. 4 indicate it does not handle CTDG. Notably, currently no more continuous TDG methods are available for comparison.

Q1. How does Koopman Theory align temporally varying distributions?

1. TDG differs from traditional DG. TDG assumes there is widespread smooth, predictable distribution changes over time, so the field aims to predict future distributions and models by building dynamic models with time-varying parameters, aligning the model function with predicted future domain distributions. CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test domain not limited to immediate one-step future domains. Defining like this challenging because it breaks the discrete-time models (e.g., state-space models, LSTM) that traditional TDG relies on. Therefore, we theoretical proof that the parameter of time-varing model for continuous temporal domains is also continuous, leading to a dynamic predictive model described by differential equations.
2. Solving differential equations analytically is impossible. So we use Koopman operator to simplifie learning these complex differential equations, enabling effective numerical solutions.

Q2. How deal with non-periodic/irregular data variations/significantly change? Can we integrate other mechanisms?

1. TDG and CTDG assume future predictability. If domain changes are irregular or unpredictable, this violates the model's biases. Since different models have different implicit biases, we need to find a suitable frameworks whose prior assumptions match the setting, such as domain-invariant learning, ensemble learning, or importance weight sampling with restart.
2. Our approach complements other DG methods. For example, a static feature extractor from domain-invariant learning can be used in Koodos, or equipted Koodos to onlining system with detection-and-restart strategies. Exploring comprehensive systems is a promising future direction.

Q3. Is datasets are with covariate shifts?

The datasets we used represent changes in P(Y∣X) and are widely recognized in the TDG field: Rotated 2-Moons: Labels change with rotation, e.g., point (1,0) changes from label 0 to 1 after 180 degrees. Rotated MNIST: Rotating '6' to look like '9' should still be labeled '6'. Yearbook: Changing fashion trends cause the same features (e.g., clothing styles) to have different gender labels over time. Cyclone: Changes in atmospheric pressure or temperature cause similar satellite images to represent different storm intensities.

Thanks for reviewing and acknowledging our work! Please read along with the Rebuttal PDF.

W1. The explanation of how the article addresses domain changes with arbitrary temporal sampling is not very clear.

1. TDG assumes there is widespread smooth, predictable distribution changes over time, so the field aims to predict future distributions and models by building dynamic models with time-varying parameters, aligning the model function with predicted future domain distributions. CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test not limited to immediate one-step future domains.
2. Inferring future model parameters under CTDG is challenging. In our work, we prove that if the domain distribution changes continuously, then the corresponding predictive model parameters should also change continuously. This insight allows us to formulate a differential equation for the predictive model parameters. Through the differential equation, we can predict future model parameters at any continuous time.
3. However, analytically solving this difference equation is almost impossible. Note that the Koopman operator allows us to linearize the nonlinear dynamics of the difference equation by mapping the original nonlinear predictive model parameter space into a higher-dimensional linear space. By leveraging this operator, we can effectively solve the differential equations numerically, ensuring that the model parameters evolve precisely over time and generalize well to the far future.
4. Finally, the learned differential equation will directly compute the predictive model parameters in any specific future tume.

W2. There is some overuse of formula characters, and the use of L_pred2 seems rather arbitrary.

1. Before submitting for review, we carefully checked the use of symbols and tried our best to make them concise and clear. We apologize for any confusion. We will carefully review the manuscript again and try to streamline the notation to ensure greater clarity and readability.
2. The loss function L_pred2 is used to quantify the prediction error of dynamically integral parameters performed on domain data. L_pred2 works together with L_pred to constitute the evaluating term of the integral and intrinsic model parameters on the domain task. We acknowledge that the notation might be unclear. To improve understanding, we will rename L_pred and L_pred2 to L_intrinsic and L_integral, respectively. This change will more accurately reflect their roles.

W3 & Q3. Additionally, the numerous hyperparameters in the loss function increase the cost of tuning for different datasets.

The loss weights are just designed to balance the magnitude of each loss term, ensuring that no single term dominates the model's training process.

1. We conducted a sensitivity analysis to understand the hyperparameters in Koodos: loss weights $(\alpha, \beta, \gamma)$ and the dimensions $n$ of the Koopman operator $\mathcal{K}$. Fig. 10 (in rebuttal PDF) shows the results.
2. The loss weights are set based on the magnitude of each loss term. For instance, in the 2-Moons dataset, the cross-entropy loss $L_{pred}$ and $L_{pred2}$ are approximately 1 after convergence, the $L_{recon}$ and $L_{consis}$ in the Model Space are around 0.01, and $L_{dyna}$ in the Koopman Space is around 0.1. Accordingly, we set the initial values of $\alpha, \beta, \gamma$ to 1, 100, and 10, respectively. We then adjust each weight term independently within its respective range: $\alpha$ and $\gamma$ are varied within 1 to 100, and $\beta$ within 10 to 1000. The dimensions $n$ of the Koopman operator vary within the range of 16 to 2048.
3. It can be seen that:

• Koodos exhibits stable behavior and robust convergence across wide varied hyperparameters, evidencing its ease of training, robust convergence, and insensitivity to hyperparameter.
• Setting the loss weights in accordance with the magnitude of each loss term is sufficient for achieving good performance.
• A few hundred Koopman operator dimensions are sufficient good; too many can lead to overfitting.

Q1. The article repeatedly emphasizes the importance of "C" in CTDG and the arbitrary selection of time t. However, does the proposed method specifically address this aspect?

1. As we discussed in W1., CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test not limited to immediate one-step future domains. Defining tasks in continuous time is challenging because it breaks the discrete-time models (e.g., state-space models, LSTM) that traditional TDG relies on.
2. Therefore, we theoretical proof that the parameter of time-varing model for continuous temporal domains is also continuous, leading to a dynamic predictive model described by differential equations.
3. Solving differential equations analytically is almost impossible. The Koopman operator simplifies learning these complex differential equations, enabling effective numerical solutions.
4. By the learned differential equations, the parameters of the predictive model can be calculated at any specific time using ODEssolvers.

Q2. In the methodology section, the article mentions that domain conditional probability distributions will not experience abrupt changes. However, this situation holds true only when the sampling time intervals are equal. The method addresses arbitrary sampling; does this imply it also deals with abrupt changes?

The assumption that domain conditional probability is continues changes is based on the nature of the underlying processes being continuous. The observation intervals, whether equal or arbitrary, do not influence the continuity of the underlying processes themselves. For example, the temperature change throughout a day is a continuous process, regardless of whether measurements are taken every hour or at irregular intervals. The continuity of the temperature change remains inherent to the process itself.

Thanks for reviewing and acknowledging our work! We see your main concerns are about related works and evaluations. Please read our answers along with the Rebuttal PDF.

W1. Limited evaluation on small datasets/models; lacks high-dimensional datasets.

1. We provide a summary of the datasets and predictive models used in our work in Tab. 5. From it, it is evident that we have utilized high-dimensional datasets and modest-sized datasets and models in our evaluation: the Yearbook dataset comprises around 20k images with 32*32=1024 pixels, and the model used for this dataset has approximately 163k parameters. Similarly, the Cyclone dataset, with image dimensions of 64x64x2=8192, is tested with a model having 135k parameters.
2. As a comparison, the existing discrete TDG datasets benchmark only uses data below 100 dimensions, and the minimum size of the model parameter is 12k.
3. Combined with the intrinsic challenge in TDG field, further increasing model size to a very large scale (e.g., LLMs) is a promising open area that is considered as a great future direction.

W2. CTDG's continuous nature resembles CIDA and AdaGraph settings, needing further discussion and comparisons.

1. Our work differs from their setup in the following points:
   • Different Fields: CIDA and AdaGraph focus on domain adaptation with access to target domain data during training. CTDG extends TDG and focuses on domain generalization without accessing target domain.
   • Different Semantics: the term 'continuous' is used differently in each work. CIDA: Domain index as a continuous variable instead of a categorical variable to aid the discriminator with a distance-based loss. AdaGraph: Continuous arrival of test data and online adaptation. CTDG: the temporally sequential domains are collected on continuous-time instead of fixed interval discrete-time.
   • Different Objectives: CIDA and AdaGraph follow traditional DA/DG settings without modeling temporal evolution dependence like CTDG. Although modeling such temporal patterns has been theoretically shown to have a lower generalization bound [50].
   • CTDG's novelty lies in its unique requirenment to capturing temporal dependencies and generalizing in continuous-time temporal domains, a challenge not addressed by CIDA, AdaGraph, or existing TDG methods.
2. Modifications to CIDA's then can apply to our dataset. We implemented it. Tab. 4 shows that CIDA cannot deal well with the CTDG problem.

W3. TKNets need further discussion and comparisons.

1. Our work and TKNets have different starting points, frameworks, and contributions.
   • Focus and Motivation: Our work addresses domain evolution over continuous-time, which existing TDG methods, including TKNets, fail to capture effectively as they rely on discrete-time models (e.g., state-space model, LSTM).
   • Central Contribution: Our primary contributions are theoretical proof that the model for continuous temporal domains is also continuous, based on which we construct a dynamic system described by differential equations. The Koopman operator is used to simplify learning complex differential equations. In contrast, TKNets' core contribution is proposing to use the Koopman operator to model the domain state transition matrix, serving a different role from ours.
2. We implemented TKNets as a baseline for the CTDG task. Results can be checked in Tab. 4 and it shows TKNets cannot deal well with the CTDG problem.

W4. Including the training and inference costs would be better.

We have added related experiments; please check Tab 6. Our model strikes a good balance between training time and effectiveness.

Q1. Does CTDG still adhere to the assumption of smooth distribution shifts?

1. Yes, CTDG extends TDG and retains the smooth assumption.
2. Moreover, CTDG mitigates two key assumptions of traditional TDG:
   • train domains are collected at fixed time intervals without missing, e.g., t=1,t=2,t=3
   • test domains appear only at a time interval in the immediate future, e.g., t=4
3. Unlike these assumptions, CTDG allows training domains to be collected at arbitrary times, e.g., t=1.2, t=2.432, t=5.4693…, and adapts to test domains that appear at any point in the future, e.g., t=6.324, t=8.3, t=9.45, not limited to immediate subsequent steps, nor limited to the future one domain.

Q2. Is the quantitative evaluation in the paper still performed on the last domain?

1. No. As we answered in Q1, one important task of CTDG is to generalize the model to any future point. Therefore, we perform evaluations in multiple test domains arbitrarily and irregularly distributed after the training period.
2. For the number of test domains for each dataset, one can refer to the Tab. 5, while the time distribution of these multiple test domains can be found in Figure 6, marked with gray shading.

Q3. Is the Yearbook dataset discrete but dense as continuous? How about others?

1. No. We do not use dense discrete domains to approximate continuous data. All datasets are designed to be consistent with real-world CTDG case situations: (1) Discrete-Time Domains but with Missing Domains; (2) Event-Driven Domains; (3) Streaming Data with Arbitrary Data Collecting Times.
2. All three of these scenarios require CTDG. Therefore, the datasets are designed as follows:
   • The YearBook dataset represents (1). The raw data is collected by years; we randomly sampled 40 years from 83 years to represent the incomplete temporal domain collection process.
   • The Cyclone dataset represents (2). When each cyclone occurs, the satellite collects a series of images for its entire life cycle, with the date of its occurrence representing a temporal domain time.
   • Situation 3 is simulated by setting multiple randomly started data collection windows on the tweet stream and the price stream, to create Influenza dataset and House dataset.