Diffusion Auto-regressive Transformer for Effective Self-supervised Time Series Forecasting

Q5: This paper uses an autoregressive prediction paradigm, however autoregressive approaches generally suffer from error accumulation and high inference time overheads. To the best of our knowledge, TimeDART has not designed a special training-inference strategy or model architecture to mitigate these problems. Further discussion of TimeDART's limitations in the autoregressive prediction paradigm is urgent and necessary.

A5: We would like to clarify that our use of the autoregressive mechanism is limited to the self-supervised pretraining phase. During this phase, we leverage the Diffusion Model to capture complex patterns and generate robust representations. However, when transitioning to the downstream forecasting tasks, we do not reuse the denoising network for prediction. Instead, we adopt a traditional one-step prediction approach, which avoids the issues typically associated with auto-regressive error accumulation and the uncertainty introduced by the reverse sampling process in diffusion models.

Q6: As the authors say 'we adopted the channel-independence setting'. However, for datasets with a very large number of channels, such as ECL and Traffic, how can a channel-independent design establish connections between multiple channels? Meanwhile, the experimental results of Crossformer[10], SAMformer[11] and iTransformer[12], show that cross-channel modelling is crucial for multivariate time series, and we believe that a discussion on inter-channel modelling for multivariate time series is necessary. However, to the best of our knowledge, TimeDART does not consider these factors in its modelling.

A6: We appreciate the reviewer’s insightful comment regarding the importance of cross-channel modeling for multivariate time series.

We acknowledge that both channel-independent (CI) and channel-dependent (CD) strategies have their respective strengths. Channel-dependent models, as demonstrated by works such as Crossformer, SAMformer, and iTransformer, can effectively capture inter-channel dependencies, thereby leveraging richer information for improved performance in certain tasks. However, CI models offer distinct advantages in terms of robustness and generalization, particularly in cross-domain scenarios.

Recent research, such as [1] highlights the trade-offs between these two strategies. The study shows that while CD approaches may exhibit higher capacity by modeling inter-channel dependencies, the CI strategy achieves superior robustness, especially in handling non-stationary and diverse time series data. Specifically, the CI approach minimizes the risk of overfitting to domain-specific inter-channel relationships, making it better suited for real-world applications where cross-domain generalization is critical.

We acknowledge that inter-channel modeling is important, as you have rightly pointed out. Cross-domain performance is indeed critical for evaluating the generalization and robustness of time series models. Given this, we adopted the channel-independent (CI) strategy, which is currently one of the simplest and most effective paradigms for conducting cross-domain experiments. CI allows us to avoid overfitting to specific inter-channel relationships, thereby enhancing the model’s robustness and adaptability across diverse datasets. This design choice aligns with our objective of building a model capable of generalizing well across domains.

[1] Lu Han et al. "The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting" 2023

Q7: See Weakness 2,3 for our concerns about the experimental results, and we believe that the introduction of competitive and up-to-date baselines is considered necessary. In addition, the authors are expected to explain the phenomena in the ablation experiments, details of which can be referred to Weakness 4.

A7: See A2, A3 and A4.

Q1: As a Diffusion-based model, we would like to introduce more valuable metrics to fully examine the performance of the proposed TimeDART. Specifically, in order to evaluate the prediction accuracy and generalization capability of TimeDART from both forecasting and generation perspectives, existing studies often include the following four metrics (including Context-FID, Correlational Score, Discriminative Score, and Predictive Score) for comprehensively evaluating the performance of Diffusion methods. In addition, the proposed TimeDART is not compared with advanced Diffusion-based approaches, such as Diffusion-TS[1], mr-Diff[2], and MG-TSD[3]. We believe that the introduction of more competitive and up-to-date approaches can demonstrate the effectiveness of the proposed method more objectively.

A1: We have noted your concerns regarding the evaluation metrics and the comparison with other Diffusion-based approaches. However, it seems there is a significant misunderstanding about our framework, which we would like to address and clarify.

Unlike traditional Diffusion-based models that utilize the denoising process for both training and inference, TimeDART employs the Diffusion Model exclusively for self-supervised pretraining. In our framework, the denoising process is designed to optimize the representation network, which is then transferred to downstream tasks. Importantly, the denoising network itself is not used during forecasting. Instead, we adopt a simple yet effective one-step prediction scheme for downstream tasks. This design allows us to avoid issues such as error accumulation in auto-regressive settings and the computational overhead of the reverse diffusion process.

Given this structure, metrics such as Context-FID, Correlational Score, Discriminative Score, and Predictive Score—which are typically employed to evaluate the generative capabilities of Diffusion-based models—are irrelevant for assessing TimeDART's downstream performance. Our focus is on leveraging the pretraining benefits of the Diffusion Model, not on direct generative forecasting, and thus these metrics do not align with the objectives of our method.

Regarding your suggestion to compare our model with advanced Diffusion-based approaches like Diffusion-TS, mr-Diff, and MG-TSD, it is crucial to note that our method operates in a fundamentally different paradigm. These methods rely heavily on the denoising process for direct probabilistic forecasting, leading to substantial computational and inference costs. Moreover, their performance on long-term, multi-variable forecasting tasks is often subpar. To provide a fair and concrete comparison, we specifically selected an open-source baseline, Diffusion-TS, for evaluation. The results under comparable settings, summarized below, highlight its inefficiency and inferior accuracy, further substantiating the advantages of our approach:

It is also important to note that most diffusion-based models focus on probabilistic forecasting and are evaluated on metrics like Context-FID, as you mentioned. Consequently, their performance on point-level evaluations, such as those in our study, tends to be significantly worse, resulting in a considerable gap compared to our method.

We hope this explanation resolves any misunderstandings and provides a clearer perspective on the contributions of our work.

Thanks for your rebuttal. However, I still have some concerns.

1. The authors mention that TimeDART has a unique prediction paradigm (not relying on noise reduction networks), so additional metrics do not apply to this model. However, we believe it is necessary to discuss TimeDART's performance in unsupervised generation of time series tasks, such as DiffusionTS, which supports both unsupervised generation tasks and supervised prediction tasks. We therefore believe that validating TimeDART's performance in generative tasks using four additional metrics will help to paint a comprehensive picture of TimeDART's generative capabilities.

2. Regarding the performance of ‘Random Init’ being slightly worse than supervised PatchTST, we believe that the difference in message encoding and tokeniser operation alone does not lead to such a significant performance gap. The authors should further explore the underlying reasons, such as ensuring sufficient training time.

3. The performance improvement in cross-domain training is not significant, in Table 1 of the authors‘ response, TimeDART only improves 0.002, 0.004, 0.003 on ETTh2, ETTm2, and Exchange compared to UniTime, which is not in line with the authors’ description of the performance ‘TimeDART significantly outperforms models including UniTime’. Does this mean that TimeDART lacks sufficient cross-domain generalisation capabilities?

Q1: In the original paper [1], the performance of patchTST seems to be better, and it is suggested that the author should evaluate it more fairly.

A1: We sincerely thank the reviewer for their insightful comment regarding the performance comparison with PatchTST. To address this, we would like to emphasize that all our experiments were conducted using the official implementations from the baseline repositories, and the results for PatchTST were obtained through our own runs. To ensure a fair comparison, we maintained consistency in key hyperparameters such as the learning rate schedule and dropout rate across all models. It is important to note that the official PatchTST implementation performs additional fine-tuning of these parameters, which we believe is not a fair comparison, given the significant computational overhead required for such optimizations. As a result, the performance we observed for PatchTST was slightly lower than the results reported in the original paper.

Q2: Why didn't the author consider evaluating the performance of classification tasks? Currently, aside from forecasting tasks, most self-supervised learning models [2] also focus on the performance of classification tasks. This is because the main purpose of self-supervised learning is to enhance the quality of representations generated by the model and to uncover key semantic features, which is especially important for classification tasks.

A2: We sincerely thank the reviewer for raising this insightful question regarding the evaluation of self-supervised learning models on downstream classification tasks. Our initial motivation was to address forecasting tasks by capturing the causal characteristics of time series data while minimizing the gap between pretraining and downstream tasks. To achieve this, we combined two generative paradigms—auto-regressive mechanisms and the Diffusion Model—within a self-supervised framework. This design choice was driven by the nature of forecasting tasks, which benefit from leveraging both causal dependencies and robust data representations. That said, we fully agree with your observation that self-supervised learning should aim to produce high-quality, versatile representations applicable to a wide range of downstream tasks, including classification. In response, we worked tirelessly to conduct preliminary classification experiments during the rebuttal phase. Specifically, we evaluated our model on the HAR and EEG datasets. The results are summarized below:

These results show that our self-supervised framework achieves competitive performance on classification tasks. Notably, TimeDART outperforms its random initialization counterpart, demonstrating the effectiveness of our pretraining in improving representation quality. On the HAR and EEG dataset, TimeDART achieves the highest accuracy, underscoring its robustness in handling diverse classification tasks. Due to time constraints, we have only been able to complete experiments on these datasets so far. However, if we are fortunate enough to be accepted, we will extend our evaluation to a broader range of datasets and tasks to further validate the generalizability and robustness of our approach.

Dear Reviewer:

Let us now address the concerns you raised based on the above rebuttal:

Q1: Due to differences in parameter size and structure among various models, fine-tuning some key hyperparameters is crucial. For example, the input length in paper [1] is fixed at 96, which results in a 50% performance drop for PatchTST on the Traffic dataset. I don't think this is fair.

A1: Thank you for raising concerns regarding the fairness of baseline performance comparisons. We acknowledge that our initial rebuttal may not have fully addressed your question.

In our first-round response, under A1, we mentioned, "To ensure a fair comparison, we maintained consistency in key hyperparameters such as the learning rate schedule and dropout rate across all models." There was a wording mistake here; parameters like dropout are not key hyperparameters but rather non-core parameters. We sincerely apologize for any misunderstanding this may have caused. Below, we will systematically outline the efforts we made to ensure fairness in our evaluations:

We would like to clarify that our experimental setup was conducted with fairness as a priority. Specifically, we used a consistent look-back window length of 336 and evaluated prediction horizons of [96, 192, 336, 720] across all self-supervised baselines, ensuring absolute fairness in the comparison. Moreover, we adhered to the model-specific parameters (e.g., d_model, patch_size, etc.) provided in the official open-source implementations of other methods, without any modification. To ensure a fair evaluation, we standardized certain non-core parameters, such as the dropout rate, across all models. Fine-tuning these parameters would risk overfitting, which we aimed to avoid.

To sum up, we are committed to conducting rigorous and unbiased evaluations of the different self-supervised approaches through careful standardization and consistent experimental setups, ensuring the fairness of our experiments.

We appreciate your detailed feedback and hope this clarifies our rationale.

Q2: Although the ETT and Exchange datasets are widely used, they exhibit severe biases and distribution shifts, leading to simple models like DLinear [2] outperforming classical Transformers by a significant margin. If datasets like PEMS04 and PEMS08, which have less severe distribution shifts, are used, DLinear performs far worse than Informer [3]. Moreover, recent studies [1] have also started evaluating models on PEMS04 and PEMS08. I don't believe comparing performance on such datasets is unfair.

A2: Thank you for raising the issue of datasets. Due to time constraints, we have now conducted experiments on the PEMS04 and PEMS08 datasets. For the comparison, we selected the best-performing self-supervised methods, including PatchTST and SimMTM. We followed the commonly used settings for the PEMS datasets as outlined in iTransformer, specifically using a look-back window of 96 and prediction horizons of [12, 24, 36, 48]. The experimental results are as follows:

As shown in the table, TimeDART outperforms our baseline methods on these datasets, demonstrating its effectiveness. Our implement scripts can be found in https://anonymous.4open.science/r/PEMS-scripts-2024

Q1: The reviewer is concerned about the computational overhead associated with this approach. The reviewer's core concern stems from the introduction of diffusion mechanisms, which can lead to a substantial increase in training and inference overhead for the model as a whole. If the model is expensive, the computational conditions required to solve the real task will be severe. Therefore, the authors need to report the actual time of the training and inference phase of other baseline models in the future and report the GPU and server model.

A1: We sincerely thank the reviewer for highlighting the concern regarding computational overhead. This is indeed a critical factor for the practical application of our method. To address this, we would like to clarify that all our experiments, including those involving the denoising diffusion mechanism, were conducted on a single NVIDIA RTX 4090 24GB GPU. This demonstrates that TimeDART can be efficiently trained and evaluated on high-end GPUs without excessive computational requirements. To further illustrate the computational feasibility, we have summarized the training and inference times, as well as memory usage(look-back window=336, predicted window=96), for TimeDART and the baseline models in the table below. These results show that TimeDART maintains competitive efficiency and scalability even for larger datasets like Traffic.

Traffic

ETTh1

Q2: The motivation for choosing to use a causal mechanism in Transformer requires further explanation. After all, time series data is encoded with more complex patterns. In particular, there are random changes caused by extreme weather events in the meteorological data, and this causal relationship is strong. But many things cause and effect is unclear, so whether such a component is appropriate needs to be used for the specific task.

A2: We sincerely thank the reviewer for raising this insightful question regarding the use of a causal mechanism in the Transformer. This is indeed a critical component of our approach, and we appreciate the opportunity to clarify its role and motivation. Regarding the use of a causal mechanism in the Transformer, I would like to clarify that the Causal Mask is only employed during the pre-training phase of the model’s representation network. In the downstream tasks, we align with other baselines by removing the causal mask and using full attention instead. The motivation for introducing the causal mechanism during pre-training has two main aspects:

1. Mechanistic Consideration: The causal mechanism ensures that the auto-regressive property is properly implemented during pre-training. Without it, there would be a risk of information leakage, which would compromise the training process.
2. Motivational Consideration: Forecasting tasks inherently involve causal relationships, and introducing the causal mechanism during pre-training allows the representation network to naturally learn such causal dependencies. From our ablation experiments, we found that the performance improvement with the causal mask is significant compared to not using it. We appreciate your point about the ambiguity of causal relationships in some time series data. Indeed, some sequences, such as those involving random changes (e.g., weather events), may not exhibit clear causal patterns. To address this, we removed the causal mask in the downstream tasks to maintain consistency with the baselines, which in turn encourages the model to explore non-causal dependencies in the data. This approach has proven effective in improving performance across a broader range of forecasting tasks. In future work, we will further explore the role of non-causal dependencies in time series forecasting and evaluate how different types of dependencies impact the model's performance.

We appreciate your point about the ambiguity of causal relationships in some time series data. Indeed, some sequences, such as those involving random changes (e.g., weather events), may not exhibit clear causal patterns. To address this, we removed the causal mask in the downstream tasks to maintain consistency with the baselines, which in turn encourages the model to explore non-causal dependencies in the data. This approach has proven effective in improving performance across a broader range of forecasting tasks. In future work, we will further explore the role of non-causal dependencies in time series forecasting and evaluate how different types of dependencies impact the model's performance.

Q3: The cross-domain evaluation shows strong results on energy datasets but weaker performance on datasets like Exchange. Is TimeDART overly sensitive to the type of data it is pre-trained on? For instance, does it struggle with financial or highly volatile datasets because of the lack of shared characteristics between domains (e.g., energy vs. finance)? Could this method benefit from domain adaptation techniques to make the cross-domain transfer more robust?

A3: We sincerely thank the reviewer for raising this important and insightful question. Our response is structured as follows:

• Clarification on Cross-Domain Experiments We respectfully clarify that our original submission did not include cross-domain evaluations on the Exchange dataset. At the time of submission, these experiments had not yet been conducted. Therefore, the observed results on Exchange reflect in-domain performance rather than cross-domain evaluation. We appreciate the opportunity to clarify this point and acknowledge that a more comprehensive cross-domain analysis is valuable.
• New Cross-Domain Experiments and Model Robustness To address your concerns regarding model sensitivity and the generalizability of cross-domain experiments, we worked tirelessly to design and conduct additional evaluations. Specifically, we performed a general pretraining on all eight datasets and fine-tuned TimeDART on four datasets: ETTh2, ETTm2, Exchange, and Electricity. We selected UniTime, GPT4TS, and PatchTST as baselines, ensuring alignment with UniTime’s experimental setup (look-back window of 96, predicted windows of {96, 192, 336, 720}). These settings allowed for a fair and consistent comparison across models. Due to time constraints, we have so far focused on these four datasets but plan to extend our evaluations to additional datasets to thoroughly assess the model’s cross-domain performance. The new results, summarized in the table1 below, highlight TimeDART's robustness and potential for further improvement through domain adaptation techniques.

Q4: The model uses a denoising diffusion loss, which may not be the most suitable for every type of forecasting task. How does TimeDART perform with other loss functions (e.g., Quantile Loss, Huber Loss) that are often used in time series forecasting tasks where the goal is to forecast confidence intervals or robustly handle outliers?

A4: We sincerely thank the reviewer for raising this insightful question. Our response is structured as follows:

• Clarification of Diffusion Loss Usage We would like to clarify that the diffusion loss is used exclusively during the self-supervised pretraining phase, not in downstream tasks. Its purpose is to model the underlying data distribution via a denoising process, which helps TimeDART capture richer representations. While the diffusion loss has a form similar to MSE, it is fundamentally different from traditional forecasting losses like Quantile Loss and Huber Loss. Unlike these losses, which extend MSE or MAE for specific objectives, the diffusion loss is designed to improve the model’s ability to generate diverse, high-quality predictions for complex forecasting tasks.
• MSE in Downstream Tasks In downstream forecasting tasks, we follow the standard practice of using MSE as the optimization objective, which aligns with the majority of self-supervised methods. To ensure a fair comparison, all baselines in our experiments also use MSE as their loss function. This allows us to isolate and evaluate the benefits of the pretrained representations. We acknowledge the reviewer’s suggestion to explore alternative loss functions, such as Quantile Loss and Huber Loss, for downstream tasks. This is indeed a valuable direction, and we plan to extend our experiments in future work to assess the impact of these losses on forecasting performance.

Q1: The combination of Transformer-based attention mechanisms with the denoising diffusion process introduces substantial computational overhead. The paper mentions running experiments on a single NVIDIA RTX 4090 GPU, but it doesn't provide detailed insights into the time and memory consumption required for training. How scalable is TimeDART for larger datasets or real-time applications?

A1: We sincerely appreciate the reviewer’s insightful comments regarding the computational overhead. To clarify, the denoising diffusion process in our framework is only used during pretraining, while the downstream tasks adopt a lightweight linear layer for single-step predictions, significantly reducing inference complexity. To address scalability concerns, we conducted experiments on runtime and memory usage (look-back window=336, predicted window=96). Results, including the largest Traffic dataset, demonstrate that our model requires less time and memory compared to other methods. Detailed results are presented in the table below.

Traffic

ETTh1

Q2: The experiments conducted on noise scheduling, the number of diffusion steps, and the number of layers in the denoising network highlight a significant degree of sensitivity to hyperparameter selection. How the model will perform well in practical settings with minimal tuning?

A2: We appreciate the reviewer’s valuable comment on hyperparameter sensitivity. Indeed, hyperparameter tuning can be time-consuming, and we have devoted significant effort to exploring this aspect. Through extensive experiments, we identified a fundamental guideline: ensure a smooth noise addition process and maintain a balance between the denoising network and the representation network’s parameter scales. Specifically:

• Total Noise Steps: We found that performance differences across values (e.g., 750, 1000, 1250) are relatively minor. The commonly used 1000-step setting strikes a good balance between performance and training efficiency, requiring no significant adjustment.
• Noise Scheduler: Our experiments confirm that the cosine scheduler consistently outperforms the linear scheduler. As such, we recommend the cosine scheduler as a robust default.
• Denoising Network Layers: Using overly deep networks can adversely affect representation learning. Through experimentation, we determined that a one-layer denoising network is optimal, providing an effective trade-off between model capacity and computational efficiency.

Dear Reviewer:

Thank you for your valuable feedback. We would like to clarify that our experimental setup was conducted with fairness as a priority. Specifically, following the practices of other works such as PatchTST and SimMTM, we used a consistent look-back window length of 336 and evaluated prediction horizons of [96, 192, 336, 720] across all self-supervised baselines, ensuring absolute fairness in the comparison. Moreover, we adhered to the model-specific parameters (e.g., d_model, patch_size, etc.) provided in the official open-source implementations of other methods, without any modification. To ensure a fair evaluation, we standardized certain non-core parameters, such as the dropout rate, across all models. Fine-tuning these parameters would risk overfitting, which we aimed to avoid.

To sum up, we are committed to conducting rigorous and unbiased evaluations of the different self-supervised approaches through careful standardization and consistent experimental setups, ensuring the fairness of our experiments.