Time-sensitive Weight Averaging for Practical Temporal Domain Generalization

It will be great to know how much this assumption is agreed and violated on individual tasks on experiments.

For synthetic datasets like Rotated MNIST, the assumption of smooth distribution shifts holds because distribution shifts on such datasets are controllable. For real-world datasets, the performance variation of the ERM baseline across different domains can serve as an intuitive indicator. If the changes in ERM performance are smooth, it suggests that distribution shifts are also smooth.

It would be nice if the theoretical guarantees appeared in the paper instead of in the appendices.

Due to page limitations, we had to move the theoretical section to the appendix. We will consider incorporating some key parts into the main text in the revised version.

I think one improvement the authors can consider is to add some discussions about the computational complexity and running times.

Yes, we discussed efficiency in Table 11 of the paper. We further add more efficiency and comparisons in the following table.

Having explanations about methods used for comparison could significantly increase the readability.

Thank you for the suggestion; we will attempt to incorporate more detailed descriptions of the baselines in the revised version.

The authors' assumption about the distribution is the usual assumption made in supervised classification under concept drift (References below). I think the authors should mention that this is a common assumption (gradual drift assumption) in supervised classification under concept drift as well as describe how the methods that make this assumption adapt to changes in the distribution.

Thank you for the suggestion. We will consider including more discussions regarding concept drift.

How does the method perform if you use a dataset with tasks in which the distribution does not change? For example MNIST dataset. Or what happens if the distribution changes too fast? Could the method work well but the theoretical guarantees would not be true?

We will add the corresponding ablation study in the revised version. In theory, even if the distribution does not change, TWA can still improve performance. This is because weight averaging itself is a commonly used technique to enhance the generalization ability.

Can the method be used on text datasets? The authors use the image datasets from the wild time library. Could the method be extended to the other three datasets?

Yes, it can be used on text datasets. In the Section C.2 of appendix, we simply evaluated TWA on 2 text datasets, arXiv and HuffPost. The results show that TWA can also boost generalization ability on text datasets.

Can the method be used for any loss function?

In theory, TWA can be combined with various loss functions. However, it's unclear how different loss functions might impact the performance of TWA.

The motivation is a little weak in my opinion. Specifically, the authors criticize that existing methods rely too much on time-sensitive mechanisms, which may cause troubles for large generic models. However, given the task is temporal DG, it is quite intuitive to rely on time-sensitive information, and authors seem to agree with the strength of using the time-sensitive information. For most of the generic models like ChatGPT, the problem of DG may not be a big deal. Applying weight averaging to these foundation models is also a resource-consuming task.

We need to clarify that we are not criticizing using time-sensitive mechanisms. In fact, we also believe that time-sensitive mechanisms are essential for TDG tasks. Our difference in motivation from previous works lies in that, previous works primarily concentrated on improving generalization performance through better utilization of temporal information, and we aim at incorporating more practical considerations into the design and evaluation of TDG. And we believe these 2 motivations are equally important.

The major issue we are addressing is that the time-sensitive mechanisms within many existing methods cannot generalize well to larger and more realistic application scenarios. As demonstrated in Table 1 of our global response, they either result in a severe performance drop or lead to a significant increase in training costs. These results also underscore the importance of considering TDG issues from a practical view.

While large models trained on large datasets may directly achieve strong generalization, there are many other application scenarios that models and dataset are not “large” enough to make DG/TDG less necessary. However, these scenarios could still be larger in scale than those studied in previous TDG approaches. Therefore, incorporating more practical considerations into TDG is a necessary step in bridging TDG with real-world applications.

The technical design is a little trivial and odd to me. The motivation says existing methods rely too much on temporal information, we should use weight averaging instead. However, the proposed method still seems to leverage a selector network to model the trajectory of the distribution shift over time. In addition, the model is built upon SWAD (Cha et al., 2021) with some additional contributions to learning the averaging coefficient.

Rather than opposing using temporal information, we are actually suggesting to improve the way to use temporal information from a practical perspective, making TDG methods more generic, easier to be applied to various application scenarios. And the weight averaging method exhibited the desired properties, so we chose to combine the weight averaging method with the temporal mechanism.

In the global response, we reiterated our novelty and contribution. When specifically comparing with SWAD, we drew inspiration from SWAD in the aspects of theoretical guarantees and overfit-aware sampling scheduling. And the other parts, including model training (e.g. normalization), model averaging approach (dense or not), and sampling strategy for TDG (late sampling) are totally different from SWAD.

Finally, the presentation of the manuscript still has a large room for improvement. For example, all the notations in Figure 1 are not introduced, which makes the visual presentation not self-contained. In addition, some sentences are hard to understand, e.g., "They assumed that understanding say how a mobile phone’s appearance has changed in the past may help predict future changes."

Thank you for pointing out these areas for improvement. We will refine these details in the revised version to ensure clearer expressions.

TWA also relies on the “smooth distribution shift” assumption, which could limit its potential applications.

Yes, TWA also follows the “smooth distribution shift” assumption of TDG tasks.

Evaluations are limited to relatively “simple” tasks, e.g. most TDG methods are evaluated with classification and regression tasks only. It’s unclear whether these TDG methods could still work when generalized to more complex tasks, such as image segmentation or common object detection.

Moving from simple datasets and network structures to realistic datasets with larger networks is the first step in making TDG practical. We will explore the potential applications of TDG in various more complex tasks in future work.

When you are applying TWA with different tasks, datasets or models, are you using a constant “number of snapshots”? If not, is there any strategy to select the best number of snapshots for different application settings? It would be helpful if there could be results of “number of snapshots vs. task complexity”.

For different tasks, we may use different numbers of snapshots. We will add the specific numbers of snapshots to the experimental details.

Based on our experience, it is better to use more snapshots when dealing with complex tasks. Conversely, selecting too many snapshots on simple tasks may harm performance.

As for the “number of snapshots vs. task complexity”, we refer to Fig.3 (b) to show how performance changes with the number of snapshots on a relatively complex task. We further vary the number of snapshots on the Elec2 dataset, which is a relatively simple task, and get the following results.

The proposed method does not strictly adhere to the problem formulation outlined in Equation (1). Notably, while the original problem is cast as a bi-level optimization, the proposed algorithm utilizes a two-phase training approach to optimize the two terms sequentially. The authors should clarify the rationale behind their model design.

We provide Eq.1 as a way to introduce the TDG formulationally generally. However, Eq. 3 represents the specific form of Eq. 1 that we used in our paper.

The two-stage training approach employed by the proposed method raises concerns about both time (for training the selector network) and space (for storing model snapshots) complexity.

According to Table 11, we can observe that TWA does not significantly increase training time, only resulting in an increase of 2-20% of the total cost. As for the storage of snapshots, on one hand, the memory consumption of model weights is significantly smaller than that of feature maps. On the other hand, in practical applications, we can categorize a period of time into a domain, where all data within that period share the same domain index. This allows us to directly combine weights to obtain a single averaged model to save memory. For instance, store all snapshots and selectors on servers and use the single averaged model on memory-constrained devices.

The explanation of how the theoretical results influence the development of the proposed method is somewhat lacking in clarity. From my point of view, the current theoretical findings appear to be more apt for providing a theoretical assurance of TWA rather than serving as the primary guiding influence on the algorithm's design.

Yes, in the Section 3.3, we are stating ”we have derived some theoretical guarantees on the ability of TWA to generalize from past to future data”

Could the authors offer more detailed insights to distinguish their work from SWAD? From my point of view, the technical contribution seems somewhat limited. It appears that the primary contribution lies in the utilization of Time2Vec to adapt the weight-averaging method to the temporal-shift scenario.

In the global response, we reiterated our novelty and contribution. When specifically comparing with SWAD, we drew inspiration from SWAD in the aspects of theoretical guarantees and overfit-aware sampling scheduling. And the other parts, including model training (e.g. normalization), model averaging approach (dense or not), and sampling strategy for TDG (late sampling) are totally different from SWAD. We also want to argue that we are specifically targeting the TDG task, and excluding the TDG-specific algorithm designs and theoretical analysis from technical contributions is unreasonable and not objective.

The authors assert that they introduce a comprehensive benchmark, encompassing several realistic datasets and baselines for temporal domain generalization. However, the comparison with existing temporal domain generalization methods is conducted solely on the Portrait dataset. For other real datasets such as CLEAR, FMoW, arXiv, and HuffPost, the authors only provide comparisons with the weight-averaging method (SWAD [1]). It is recommended that the authors expand their analysis to include results from other methods, particularly those tailored for temporal domain generalization (e.g., GI [2], LSSAE [3], DDA [4], DRAIN [5], DPNET [6]), on these datasets to bolster the strength of their contribution. (NOTE: This is a significant concern. I am open to revising my evaluation if the authors can address and resolve this issue.)

We appreciate your suggestions. In the following, we are glad to provide more results of TDG baselines, the reason why some results are not available and our motivation to split the comparison into 2 parts.

   Accuracy and efficiency evaluation on CLEAR-10 and CLEAR-100

First, we evaluate various TDG methods on CLEAR-10 and CLEAR-100. The last domain is used as the target domain. These methods include CIDA, AdaGraph, GI, DDA, and LSSAE. The results are presented in the following Table. It can be observed that TWA exhibits a more pronounced advantage when compared to these TDG methods on CLEAR-10 and CLEAR-100.

   Missing Comparison with GI and DRAIN

GI is hard to be adapted to large datasets due to the gradient interpolation loss. Training a network with TReLU is feasible in terms of efficiency and accuracy. However, simply using GI loss leads to a significant performance drop. While finding adaptation with extensive experiments might alleviate this issue, the high cost of GI loss prevents us from this. On CLEAR-10, a single finetuning epoch with GI loss requires 23 GPU hours, and on CLEAR-100, each epoch even demands 153 GPU hours.

DRAIN is missing as we cannot facilitate its proper training on new benchmarks. Methodologically, DRAIN is generic and can easily integrate with various model architectures. However, its implementation and empirical evaluation are primarily designed for small datasets, such as MNIST, and small models like MLP or 2-layer CNN. When we attempt to apply DRAIN to larger datasets, such as CLEAR, and larger CNN networks, such as ResNet, we find it hard to make the network learn properly, resulting in random guessing networks.

   2 parts of comparisons

When many TDG methods struggle to generalize to the new settings, comparing with them only using new benchmarks and TDG methods may raise concerns about unfair comparisons. Perhaps better method adaptations could enhance the accuracy. But the significant cost increases are basically unavoidable, which also makes it hard to explore better adaptations. Therefore, we additionally selected some datasets commonly used by these TDG methods to enhance the fairness and comprehensiveness of the evaluation.

   Other relevant clarifications.

CIDA and AdaGraph are used as TDG baselines, as they are also selected as baselines by most other TDG methods. And TDG can be regarded as a special case of their original Continuous Domain Adaptation task.

ArXiv and HuffPost are relatively less crucial, as they are simply used to test whether TWA can generalize to NLP tasks, instead of serving as the major benchmarks. And they also require using pretrained weights, making many TDG methods cannot be evaluated.

DPNET seems to be not published yet.