Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains

Q1. Modeling the drift factors may be not necessary.

R1. We argue that modeling $Z^d$ is indispensable. Although both $Z^d$ and $Z^{dy}$ are influenced by $L$ and contribute to $Y$, $Z^{dy}$ operates in the feature space while $Z^d$ pertains to the label space which is designed to capture mechanism drifts. To be clear, $Z^d$ essentially characterizes the evolution of the classifier itself, which is crucial in EDG, as also emphasized in [1-2]. Furthermore, we conduct an ablation study on $Z^d$ and show result below, which clearly demonstrates the necessity of modeling drift factors.

[1] LSSAE, ICML’22.

[2] DDA, AAAI’23.

Q2. Regarding the posterior distribution model for $z_t^{dy}$.

R2. We clarify that the posterior $q_{\theta}(z_t^{dy}|z_{<t}^{dy},x_t)$ in our model is valid.

In VAE frameworks, the variational posterior $q_\theta(z_t^{dy}|z_{<t}^{dy},x_t)$ is naturally learned through the inference network. The conditional dependency on $z_{<t}^{dy}$ reflects a data-driven modeling of temporal dynamics, independent of prior knowledge about $p(z_{<t}^{dy})$. Crucially, while both modeling $z_{<t}^{dy}$ and $x_{<t}^{dy}$ are reasonable, $z_{<t}^{dy}$ represents a ​low-dimensional encoding of $x_{<t}^{dy}$, compressed via LSTM hidden states to bypass high-dimensional data handling. This approach aligns with established sequence modeling practices [3], where latent variables effectively capture temporal dependencies.

[3] S3VAE, CVPR’20.

Q3. About conflicting optimization goals.

R3. We believe that the reviewer may have some misunderstandings. First, the misunderstanding fundamentally arises from a misinterpretation of the collider structure $Z^{st} \rightarrow Y \leftarrow Z^{dy}$, where $Z^{st} \perp Z^{dy}$, $Z^{st} \not \perp Z^{dy}|Y$ hold. This is a well-established principle in causal science, commonly referred to as the collider effect. Therefore, these two objectives are reasonable. In addition, the second objective does not directly take the static and dynamic factors as input. Instead, it operates on the representations processed by the causalizer.

Q4. Misunderstanding about information leakage during inference.

R4. We respectfully disagree with the reviewer's view. During inference, static and dynamic causal factors are obtained by corresponding VAE encoder and causalizer, and the drift factors are inferred solely from the learned prior $p(z_t^d|z_{<t}^d)$, without access to ground-truth labels. Hence, the concern about possible information leakage may be misplaced. Please refer to Appendix. D for full details about training and inference procedures of our approach.

Q5. Regarding for showing results in domains.

R5. We stress that EDG is inherently designed for future-domain generalization. Accordingly, following the common practice adopted by most existing EDG methods such as [1, 4], we evaluate on the last third of the domains to assess this capability. However, to address potential concerns, we also report the average test performance across all domains on Circle and RMNIST, demonstrating the superiority of SYNC.

[4] SDE-EDG, ICLR’24.

Q6. Lack of sufficient comparison with other baselines in Figures 5 and 6.

R6. For Fig. 6(a-b), LSSAE [1] is chosen as the baseline since other methods don't explicitly model both static and dynamic factors. Although MMD-LSAE, an extension of LSSAE, considers these factors, it models them deterministically, making mutual information (MI) computation infeasible under our MWS-based estimation scheme. For the remaining figures, we have added the optimal baselines and redrawn them, as shown in Fig. 3 and Fig. 4 of https://anonymous.4open.science/r/2150-1CBD. The results demonstrate that SYNC still outperforms other methods, as claimed.

[5] MMD-LSAE, TPAMI'23.

Q7. Clarification of this work's contribution to resolving spurious correlations.

R7. Our method disentangles static and dynamic causal factors to learn time-aware causal representations and address spurious correlations. It works as follows: First, MI $I(z_t^{st},z_t^{dy})$ is minimized to disentangle the factors. Then, Proposition 3.3, Lemma 3.4, and Lemma 3.5 guide the learning process. Specifically, static causal factors can be identified by maximizing conditional MI between static components of consecutive domains given the class (Eq. (4)). Dynamic causal factors can be obtained by anchoring on static causal factors and maximizing conditional MI within the same domain given both the class and static causal factors (Eq. (8)). Based on these theoretical results, we use causalizer modules to extract finer-grained static and dynamic causal factors by optimizing the above objectives.

Q8. Regarding for some symbol errors.

R8. Thanks. We will fix these errors and thoroughly revise the manuscript.

Q1. Further analysis of spurious correlation caused by time factors and more concrete examples.

R1. Thanks. Herein, we provide a formal characterization of the spurious correlation problem. In the proposed time-aware SCM, the time factor constitutes a latent confounder whose components $G$ and $L$ establish backdoor paths between static and dynamic spurious factors and the label, namely $Z_s^{st} \leftarrow G \rightarrow Y$ and $Z_s^{dy} \leftarrow L \rightarrow Z^d \rightarrow Y$, thereby introducing spurious correlations. Under this configuration, naively maximizing mutual information between features and labels (that is, minimize the cross-entropy loss) will cause the model to learn spurious features. Specifically, the model may erroneously utilize brightness features with strong temporal correlations as discriminative cues for vehicle detection in Figure. 1, rather than learning essential shape semantics. To make the example more concrete, we provide more visualization results and analysis. Please refer to R2.

Q2. Improvement on visualization to tell the trained model learned the spurious correlations.

R2. Thanks. To more clearly demonstrate the effectiveness of our method, we have enhanced the visualization and presented it in Fig. 2 of https://anonymous.4open.science/r/2150-1CBD. It can be found that in nighttime images, LSSAE [1] identifies the image as “No Car” based on lighting, while our method correctly focuses on the semantic information of the car, identifying it as “Car”. In daytime images, although LSSAE successfully classifies the image as car, it relies on environmental factors, whereas our method correctly identifies it based on the car's semantic information. Therefore, the trained model may learn the spurious correlations.

[1] LSSAE, ICML 2022.

Q3. Comparison with methods using time factors to avoid spurious correlations.

R3. Thanks. Time-series causal modeling constitutes the most pertinent research direction to this work. Most existing time series causal methods [2-4] construct temporal SCMs that incorporate causal factors and aim to learn causal representations based on the properties of these factors. However, due to the complexity of dynamic scenes, the behavior and nature of causal factors also become intricate, often requiring strong assumptions, such as the reversibility of the generation function and an additive noise model.

In contrast to them, meticulous deliberation has been conducted regarding the majority of factors in our method, wherein causal variables are explicitly decomposed into static and dynamic components for joint modeling. This modeling approach allows our method to learn complete causal representations by first learning easily obtainable static causal factors by and using them as anchors to learn dynamic causal factors, without requiring stringent assumptions, thus endowing it with extensibility to more complex scenarios. Furthermore, the integration of causal mechanism drift enables better adaptation to the underlying data distribution.

[2] Causal-HMM, CVPR’21.

[3] TDRL, ICLR’22.

[4] CtrlNS, NeurIPS’24.

Q4. Visualization improvements and the idea for mitigating significant deviations from the trend.

R4. Thanks. In the displayed visualization results, the similarity in the hot map positions may be attributed to the fact that the vehicle's position is directly in front of the camera. We have improved the visualization, as shown in Fig. 2 of https://anonymous.4open.science/r/2150-1CBD.

When faced with unseen distributions significantly deviating from the training trends, the performance of EDG methods inevitably deteriorates, as they are designed under the assumption of slow and regular distribution changes. However, our approach incorporates the learning of static causal factors, enabling the model to maintain relatively stable generalization capabilities. To illustrate this, we randomly reorder the test domains of RMNIST and evaluate different methods. Specifically, we keep the training and validation sets unchanged, while rearranging the domains in the original test set, which previously arrived sequentially from 130°, 140°, ..., 180°, into 170°, 140°, 180°, 160°, 130°, 150°. The results in the table show that our method still outperforms the baselines. In addition, techniques such as out-of-distribution detection can be used to identify moments of significant distribution shifts and adjust the model to mitigate the impact of abrupt and violent shifts.

Finally, we also conduct an experiment on the more challenging FMoW dataset as shown in R2 of Reviewer GuDA. it can be found that our approach remains effective in generalizing in more challenging scenarios, exhibits the potential for deployment in complex real-world applications.

Q1. The significance of Proposition 3.3 and its connection to SCM.

R1. Thanks. Here we provide a detailed explanation of Proposition 3.3.

The Significance for learning causal representation: The two points of Proposition 3.3 respectively guide the learning of static and dynamic causal representations.

From Proposition 3.3 (i), it can be found that for various static factors in two consecutive temporal domains, the static causal factors $Z_c^{st}$ are those that maximize the conditional mutual information (CMI) under a given category. Since we extract static causal factors using network $\Phi_c^{st}$, static causal representations can be learned by maximizing the CMI $I(\Phi_c^{st}(X_t);\Phi_c^{st}(X_{t-1})|Y)$. Proposition 3.3 (ii) shows that among various dynamic factors within the same temporal domain, the dynamic causal factors $Z_c^{dy}$ are those that maximize the CMI with the static causal factors under a given category. Therefore, static causal factors $Z_c^{st}$ can serve as anchors to facilitate the learning of dynamic causal factors $Z_c^{dy}$ by maximizing $I(\Phi_c^{dy}(X_t);Z_{c,t}^{st}|Y)$.

Connection towards the SCM: Although Proposition 3.3 can be derived straightforwardly using information theory, its inspiration actually comes from SCM and can be intuitively explained within the SCM framework. Since the two points of Proposition 3.3 are quite similar, for the sake of simplicity, we will explain using the second point. As shown in the time-aware SCM in Figure. 2, there is a clear collider structure and a fork structure, namely $Z_c^{st}\rightarrow Y \leftarrow Z_c^{dy}$ and $Z_s^{dy} \leftarrow L \rightarrow Z_c^{dy}$. According to d-separation, when conditioned on $Y$, both $Z_c^{dy}$ and $Z_s^{dy}$ are related to $Z_c^{st}$. Consider a boundary case where $Z_c^{dy}$ and $Z_s^{dy}$ are independent, this implies that the backdoor path between $Z_c^{dy}$ and $Z_s^{dy}$ is blocked, leading to independence between $Z_c^{st}$ and $Z_s^{dy}$, while $Z_c^{st}$ remains related to $Z_c^{dy}$. Proposition 3.3 generalizes this observation, under certain entropy inequalities, static causal factors are more strongly related to dynamic causal factors than dynamic spurious factors.

Q2. More details about ablation study.

R2. Thanks. Here we explain the ablation study in detail and illustrate the contribution of the modeled causal factors to performance. As the table shown below, Variant A serves as the base method trained solely with evolving pattern loss $\mathcal{L}\_{\text{evolve}}$. Variant B builds upon the base method by additionally training with MI loss $\mathcal{L}\_{\text{MI}}$, serving as an ablation for $\mathcal{L}\_{\text{causal}}$. Variants C and D build upon Variant B by incorporating additional training with static causal loss $\mathcal{L}\_{\text{stc}}$ and dynamic causal loss $\mathcal{L}\_{\text{dyc}}$, respectively. We conducted additional ablation experiments on the Portraits dataset, excluding RMNIST, and recorded their worst-case performance (W) and average performance (A). The results are presented below.

First, both Variant C and Variant D show performance improvements over Variant B, validating the effectiveness of modeling static and dynamic causal factors.

Then, it is clear that Variant C achieves a greater improvement in worst-case performance compared to Variant D, indicating that static causal factors can ensure stable generalization under continuous distribution shifts. However, focusing solely on them ignores evolving pattern in EDG, limiting further generalization gains. Learning dynamic causal factors captures features related to task evolving over time, enabling to generalize better to the current distribution. Variant D outperforms Variant C in average performance and provides evidence for this claim.

Finally, SYNC learns static and dynamic causal representations jointly, achieving the best performance, demonstrating their significant contribution to overall performance.

Q3. Typographical errors.

R3. Thanks. As your suggestion, we will thoroughly revise the manuscript to improve clarity and readability.

Q1. The contribution of causal factors to performance.

R1. Thanks. We perform a further analysis of causal factors. Please refer to R2 of Reviewer RWjv for details.

Q2. Model assumptions and their impact on performance.

R2. Thanks. Given the challenging of the EDG problem, most existing methods make appropriate assumptions to derive their objectives. To name a few, LSSAE [1] assumes the latent variable's Markov property, while DDA [2] assumes consistent evolution across domains. Our approach adopts a causal perspective, relying only on the fundamental SCM assumptions like Markov condition, to build a reasonable SCM for modeling time causal factors. These widely used assumptions have been validated in real-world scenarios in literature [3-5]. Besides, our method generally outperforms other methods on real-world datasets in our paper, supporting the assumptions.

Since causal sufficiency and faithfulness are relatively important in our assumptions (violating causal sufficiency disrupts static-dynamic independence, while violating faithfulness affects conditional dependencies), we conduct experiments to evaluate their impact. Results below show little performance change. Moreover, incorporating these relaxed assumptions improves performance.

Finally, we conducted experiments on the more challenging FMoW. The results below show that SYNC outperforms others, demonstrating its applicability in more complex scenarios.

[1] LSSAE, ICML’22.

[2] DDA, AAAI’23.

[3] MatchDG, ICML’21.

[4] CIRL, CVPR’22.

[5] iDAG, ICCV’23.

Q3. Comparison with transformer-based DG methods.

R3. Thanks. We discuss transformer-based DG methods here. [5] generates data from unseen domains using CycleGAN, while [6] investigates the deployment of vision transformers (ViT) in DA and DG. However, neither addresses the DG using transformers. Here we compare SYNC with two highly cited transformer-based DG methods, i.e., DoPrompt [5] and GMoE [6]. While ViT helps capture robust semantic features, they fail to consider the continuous structure in EDG. The results below show SYNC's superior temporal generalization.

[5] DoTra, IJCNN’22.

[6] Vision...Generalization, Neural Comput Appl’24.

[7] DoPrompt, arXiv.

[8] GMoE, ICLR’23.

Q4. Evaluation on more real-world benchmarks.

R4. Thanks. We evaluate our method and baselines on FMoW and the results are detailed in R2. It is known that SYNC remains effective in generalizing in more challenging scenarios, exhibits the potential for deployment in complex real-world applications.

Q5. Computational complexity and scalability analysis.

R5. Thanks. Our network largely follows LSSAE [1] and MMD-LSAE [9], with the only addition being two MLP-based maskers. Similar to them, the time complexity and memory complexity of sequential VAE are $\mathcal{O}(T\cdot B\cdot [\sum\_{l=1}^LH^{l}W^{l}C^{l-1}C^l(K^l)^2+D^2])$ and $\mathcal{O}(T\cdot B\cdot [\sum\_{l=1}^LH^lW^lC^l+D])$ respectively, where $T$ is the number of time domains, $B$ represents batch size, $D$ is the dimension of latent features. For the $l$-th layer of the decoder, $H^l$ and $W^l$ denote the output feature map size, $C^l$ represents the output channel, and $K^l$ denotes the kernel size. For the MI loss function, the time complexity and memory complexity are $\mathcal{O}(T\cdot B^2\cdot (D+1))$ and $\mathcal{O}(T\cdot B\cdot D)$, respectively. In the implementation, $D$ and $K$ are set to a relatively small value ($D=32, K=5$) and the complexity is acceptable.

Besides, we conduct an experiment on FMoW using DenseNet-121. The results show the effectiveness of our approach in challenging real-world scenarios. Additionally, we record memory usage and runtime per iteration. The results below show that our method requires almost the same cost to achieve better performance.

[9] MMD-LSAE, TPAMI'23.

Q6. Comparison with Koodos [10].

R6. Thanks. Koodos considers the continuous temporal DG and models the evolving pattern via Koopman theory. Our method is tailored to the EDG and enhance temporal generalization by tackling spurious correlations through time-aware causal representations. We evaluate Koodos on two datasets and the results below verify the effectiveness of SYNC.

[10] Koodos, NeurIPS’24.

Q7. Practical robustness checks.

R7. Thanks. We conduct a sensitivity analysis experiment on $\alpha\_1$ and $\alpha\_2$, the results are shown in Fig. 1 of https://anonymous.4open.science/r/2150-1CBD, indicating insensitivity within a certain range. Additionally, our evaluation on FMoW shows the method's adaptability to more complex scenarios.