Temporal Source Recovery for Time-Series Source-Free Unsupervised Domain Adaptation

Methodologies Questions:

Q1. Regarding Recovery-initialization, which parts are initialized in Figure.1 framework? Why initializing with the target distribution satisfies the first requirement. Ablation study by initializing randomly? Why GAN does not meet the two requirement?

AWS: The masked data in Fig. 1 is initialized and masked from the target data. If the initial distribution significantly diverges from the source, aligning the two distributions becomes far more challenging. Initializing with a highly random or distant distribution forces the recovery model to search through a much larger solution space to approximate the source-like distribution. This expanded search space not only increases the complexity of finding an optimal solution but also heightens the risk of converging on a sub-optimal solution. A closer initial alignment with the source distribution narrows the search space, allowing the model to more efficiently and accurately capture essential source characteristics, leading to a more effective source-like domain.

We have also included an ablation study with random initialization to evaluate its impact. The results show that this random initialization for the source-like domain significantly reduces performance. Furthermore, the standard deviation increases, highlighting the greater difficulty in identifying an optimal solution.

Regarding GANs, it fail to meet the two outlined requirements for two reasons: 1. GANs typically use a random noise vector sampled from a standard distribution (e.g., Gaussian or uniform) as the initial input to the generator. This random initialization normally diverges significantly from the source distribution, expanding the solution space and making it challenging to converge to an optimal source-like distribution. 2. GANs are not inherently designed to handle sequential data or temporal dependencies, as they treat each generated sample independently without enforcing continuity between data points, so it may generate the random time points and fail to capture the temporal coherence.

Q2. More details on the masking strategy? Similar with MAE?

AWS: To clarify the masking strategy, suppose we use a masking ratio of 3/8. In this case, we divide each TS data into 8 segments, and 3 of these segments are randomly selected for masking by setting their values to 0. For multivariate TS data, where multiple channels are present, we apply this masking process independently to each channel.

Regarding MAE, this technique aims to mask and reconstruct the original samples, while our approach focuses on masking and recovering the masked samples toward a source-like distribution. As the source-like samples are not available for direct reconstruction, we cannot use reconstruction loss as in MAE. Instead, we introduce a regularization loss L_Seg to guide the recovery process, encouraging the masked samples to approximate the source-like distribution.

Weak 1 (W1). The notation in Figure 1 is inconsistent with the notation in the text. Specifically, is the L_sse in Figure 1 the same as L_seg mentioned in line 259?

AWS: Sorry for the mistakes. L_sse is L_Seg, and we have revised this in the manuscript.

W2. The non-source-free baseline models used are outdated. W3. Insufficient source-target pairs.

AWS: 1. We have included comparisons with CLUDA and RainCoat for the HAR dataset. Due to time constraints, experiments on additional datasets are still in progress and will be reported in the camera-ready version. From the results, we observe that these recent methods, particularly RainCoat, demonstrate strong performance. However, it is important to note that RainCoat is a traditional UDA method that requires access to source data during adaptation. In contrast, TemSR operates in a source-free setting and still achieves comparable performance, underscoring its effectiveness despite the stricter constraints.

2. To address the concern regarding the number of cross-domain cases, we have conducted experiments on five additional cross-domain pairs for the HAR dataset, following the setting in [ref TKDD 2023]. Due to time limitations, these results are currently limited to MAPU as the baseline. Further experiments on other datasets and with more comprehensive baselines will be included in the camera-ready version. From these additional evaluations, TemSR continues to demonstrate its effectiveness, further validating the robustness and adaptability of our approach across diverse cross-domain scenarios.

Q5. (2) why using the same CNN backbone?

AWS: While task-specific backbones can improve performance, our goal is to evaluate TemSR’s effectiveness rather than the impact of the backbone. Therefore, we use a consistent backbone, as done in similar TS studies [ref ICML 2023, ICLR 2023, TPAMI 2024], to ensure fair comparisons and attribute performance differences to the algorithm rather than the backbone.

Q5. (3) Model details, particularly on the recovery model. Why not considering Transformer structure?

AWS: Due to space limitations, model details are included in the appendix, as is common in other works [ref ICML 2023, ICLR 2023, TPAMI 2024]. We use a two-layer LSTM for the recovery model due to its strong imputation ability for TS data [ref KDD 2023]. We also evaluated a transformer-based recovery model. However, its performance was slightly lower, with average results of 89.21%±1.45%, 64.38%±0.39%, and 92.45%±3.07% on the HAR, SSC, and MFD datasets, respectively, compared to the two-layer LSTM. This underperformance is attributed to the transformer's reliance on attention mechanisms to learn temporal dependencies. While effective, these mechanisms typically require larger datasets to fully capture patterns. Unlike video or image tasks, where abundant data points is often available, time-series data is typically limited, making transformers less effective in this context. More introduction regarding the recovery model will be provided in the camera ready version.

Q5. (4) Not convincing results, as TemSR is the winner in 2 of 5, 1 of 5, and 3 of 5 tasks in the three datasets.

AWS: The main advantage of TemSR is its ability to transfer temporal dependencies without requiring domain-specific designs during source pretraining, and it achieves competitive performance in such restricted cases. For example, in the SSC task, where TemSR wins only 1 of 5 tasks, MAPU, which relies on source-specific designs, achieves the best performance in most cases. However, MAPU’s dependency on such pretraining makes it less practical in real-world scenarios, while TemSR achieves comparable performance without requiring any source specific designs.

Q5. (5) Not convincing sensitivity analysis by only using 3 datasets. The hyper-parameter generalization ability. What is “EEG” in figures 4,5,6?

AWS: We apologize for the lack of clarity. The sensitivity analysis presented in Figures 4 and 5 reflects the average results across all cross-domain cases in each task, encompassing a total of 15 cross-domain scenarios. This broader evaluation provides a more comprehensive perspective, rather than being limited to only three datasets. Furthermore, additional experiments conducted on extended cross-domain scenarios on HAR reaffirm TemSR's consistent and strong performance compared to the second-best method, MAPU. Regarding "EEG" in Figures 4, 5, and 6, we apologize for the oversight. "EEG" refers to the SSC task, as the SSC dataset is based on EEG data collected for experiments. We have corrected this mislabeling in the revised version.

[ICLR 2017] VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION

[ICLR 2023] CONTRASTIVE LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION OF TIME SERIES

[TPAMI 2023] CALDA: Improving Multi-Source Time Series Domain Adaptation With Contrastive Adversarial Learning

[TPAMI 2024] SEA++: Multi-Graph-Based Higher-Order Sensor Alignment for Multivariate Time-Series Unsupervised Domain Adaptation

[ICML 2023] Domain Adaptation for Time Series Under Feature and Label Shifts

[CVPR 2023] Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation

[KDD 2023] Source-Free Domain Adaptation with Temporal Imputation for Time Series Data

Weak 1 (W1). Regarding technical novelty, mask and then recovery has been shown the adaptation and transfer capability in generation tasks, e.g., masked autoencoder and videoMAE. The application of such an idea to source-free UDA setting makes the technical significance limited although it may not be explicitly done before.

AWS: We would like to highlight the novelty of our work in three key aspects, distinguishing it from MAE and its variants:

1. Different focus: MAE emphasizes masking and reconstructing the original sample, using the masking process as a core mechanism. In contrast, our work uses masking as a tool, with the primary focus on recovering masked samples as source-like samples rather than reconstructing the original sample. This shift in objective fundamentally changes the methodology and application.

2. Different loss functions: Due to differing aims, MAE relies on reconstruction losses, such as MSE, to align the reconstructed and original samples. However, our approach seeks to recover source-like samples, which do not explicitly exist. Therefore, we use entropy minimization as a guiding principle. By leveraging the property that models with minimized entropy produce deterministic outputs, we employ this constraint inversely to guide the recovery.

3. Specialized design for time-series data: Our work is specifically tailored for TS-SFUDA tasks, leveraging masking as a tool to capture and incorporate the intrinsic temporal dependencies of TS data for enhanced TS-SFUDA performance. To the best of our knowledge, this is the first approach to employ such a concept in the TS-SFUDA setting.

W2. Missing video-based UDA related works, and video is a popular time-series data. W3. Overlook the spatial divergence and only focus on the temporal one.

AWS: Video, time-series, and language data share similarities as sequential data, but they also exhibit distinct characteristics. For instance, video data is an extension of image data, with frames containing rich spatial and temporal information. Consequently, video-based UDA often emphasizes both spatial and temporal modeling. In contrast, TS data consists of discrete limited data points at each time step, so most TS-related works focus primarily on temporal dependencies [ref ICLR 2017, 2023, TPAMI 2023]. Furthermore, most TS-focused research [ref ICLR 2017, 2023, TPAMI 2023, 2024] does not explore the relevance of video data, as the emphasis typically lies on the unique properties of TS data. We will incorporate a discussion of relevant video-based UDA works in the related work section to provide a more comprehensive perspective.

Q1. Figure 2 inadequate to represent complex temporal and spatial divergence across TS data. What does “target distribution” mean here?

AWS: The "target distribution" refers to the data distribution in the target domain, which we use as the initialization for generating the source-like domain. We acknowledge that due to the complex dependencies in time-series data, accurately depicting the divergence between domains with a simple diagram is challenging. However, the figure is intended to convey a high-level conceptual understanding, aligning with the simplified visualizations commonly used in prior works (e.g., [ref ICML 2023, TPAMI 2024, CVPR 2023]) to illustrate key ideas. Specifically, our diagram illustrates that while the target domain differs from the source domain, the two domains are still related, and the target domain is not drastically divergent from the source domain. This relationship motivates to use the target distribution for initialization.

Q2. Regarding the recovery component, is a reconstruction loss included? Or using a pretrained reconstruction model as backbone?

AWS: In the recovery process, we design a regularization loss L_Seg, rather than a reconstruction loss, to guide the masked samples toward becoming source-like samples. As our goal is not to reconstruct the original samples, we argue that using a recovery model pretrained with target data may not provide significant benefits. To evaluate this, we conducted an experiment comparing the performance of a variant using a pretrained recovery model with our vanilla model. The results (90.76 w/ pretrained vs. 90.93 w/o pretrained, regarding the average performance) indicate that the pretrained recovery model does not provide a notable advantage in this context.

Q3. Ablation studies on low entropy difference between segments to show its effectiveness. What does {C,E,L,R} mean?

AWS: In Eq. (2), {C, E, L, R} represent the complete recovered sample (C), the early segment (E), the late segment (L), and the segment containing all recovered parts (R).

The equation minimizes the entropy differences between these segment pairs to encourage consistent temporal dependencies. The rationale for the smooth dependencies among segments is as follows:

1. Entropy minimization for aligning with the source distribution:

Models with minimized entropy can produce deterministic outputs, and this ideal output constraint can be inversely employed to guide adaptation. Thus, with the minimized entropy on source data, the source model can produce deterministic outputs for distributions with source characteristics. By minimizing the entropy computed by the fixed source model for recovered samples, this constraint can inversely regularize the samples, forcing them to align with the source distribution. Here, the recovery model is forced to capture source temporal dependencies, as only by understanding these dependencies can the model effectively reconstruct masked parts, minimize entropy, and ensure recovered samples align with the source distribution.

2. Minimizing Entropy Differences to Ensure Smooth Dependencies:

By minimizing entropy across segments, as per Eq. (1), and reducing the entropy differences between segment pairs (Eq. (2)), the model ensures smooth temporal dependencies among segments. This consistency is crucial for recovering the global and local structure of TS data.

We also conducted ablation studies on L_SegSim, showing that removing this loss results in performance drops across all three datasets. While not substantial, the results can also highlight the effectiveness of this loss in enhancing recovery and alignment.

Q4. The test flow?

AWS: After the adaptation stage, the optimization target encoder extracts features from the testing sample, which are then classified by the pretrained source classifier for final predictions.

Q5. (1) Video benchmark datasets

AWS: As discussed earlier, video data differs significantly from TS data, and TemSR is specifically designed for TS tasks, focusing on temporal dependencies unique to TS data, so it is hard to directly adapt TemSR to video tasks. Extending TemSR to video tasks will be considered in future work.

Thank you for your thoughtful feedback. We appreciate the opportunity to address your concerns regarding W2 and W3. While we acknowledge that some works consider both temporal and spatial divergences in their approaches, we would like to highlight two key points relevant to this work:

1. We focus on demonstrating the effectiveness of our idea

The central goal of this work is to validate the effectiveness of our approach: generating a source-like domain by recovering inherent dependencies within time-series (TS) data for TS-SFUDA. Temporal dependencies are inherent in TS data, making them a universal feature across tasks. While introducing additional spatial information can enhance performance in specific tasks where extensive sensors are used, our focus here is on temporal dependencies to establish a solid foundation for our method. In future work, we plan to incorporate spatial information into TS data to further improve performance.

2. Generality through temporal dependency modeling

Unlike video data, many TS tasks lack strong spatial characteristics, making temporal dependency modeling a more universally applicable starting point. For example, in scenarios where only one sensor or a single channel is used (e.g., due to cost constraints), no spatial information exists. Even when TS data is N×d-dimensional, the channels may represent diverse signals without meaningful spatial relationships. For instance:

• In certain human activity recognition datasets [1], three channels are collected, representing the x, y, and z axes from a single sensor, which do not inherently encode spatial structure.

• Similarly, in the UCI-HAR dataset used in this work, the nine channels correspond to signals from three different types of sensors (e.g., body acceleration and angular velocity), each comprising x, y, and z axes. These signals lack the clear spatial semantics observed in images or videos.

By focusing on temporal dependencies, our method addresses a broader range of TS tasks and lays a strong foundation for future advancements.

We hope this clarification addresses your concerns. Please let us know if you have any further feedback.

[1] Activity recognition using cell phone accelerometers

Weak 1 (W1). More ablation study, such as the consistent segment entropy loss and the ARDM loss.

AWS: We have expanded the ablation study to analyze the components of the segment entropy loss and the contributions of various elements in the ARDM loss. Specifically:

• For the segment entropy loss, w/o Early - w/o Complete refer to the removal of the corresponding segments in Eq. (1), while w/o L_SegSim indicates the exclusion of the segment similarity loss.

• For the ARDM loss, w/o Anchor denotes the removal of anchors, which also eliminates the additional objective since it is designed to enhance anchor guidance. Meanwhile, w/o Additional Obj refers to the exclusion of only the additional objective while retaining the anchors.

From the ablation study on the segment entropy loss, we observe that each component plays a critical role, as removing any of them results in performance degradation. Notably, the "complete" is the most significant, as it captures global dependencies. When this component is removed, only local dependencies are captured, which adversely affects the model's performance. Among the components targeting local dependencies, the early and late segments are relatively more impactful than the recovered segment. This is likely because the early and late segments sometimes intersect with the recovered parts. However, this does not diminish the importance of the recovered segment, as it focuses on the masked parts, encouraging them to align with source temporal dependencies and further enhancing performance. Additionally, removing L_SegSim results in a noticeable drop in performance, highlighting the effectiveness of maintaining the consistent temporal dependencies across these segments.

Regarding the ARDM loss, removing the anchor results in a significant performance decline due to insufficient diversity among the recovered samples, especially when using small masking ratios. This lack of diversity negatively impacts the effective recovery of source-like samples. Introducing the anchor (w/o Additional Obj) improves performance by providing guidance; however, excluding the additional objective prevents optimal results. Without this objective, directly pushing all recovered samples toward the anchor risks convergence to a collapsed solution, reducing the overall effectiveness of the recovery process.

The detailed ablation study will be provided in the camera-ready version.