TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation

All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/

[A1 for Q1]--->para4,para5, [A3 for W1] —> para3

A1 for Q1

Please refer to A1 for Q1 in our rebuttal to the second Reviewer TNxq.

A2 for Q2

Of course, we are willing to elaborate on our understanding of these two generative paradigms in the context of time series representation learning, and we welcome your corrections if there are any inaccuracies:

From the perspective of the characteristics of the data itself, we believe that time series data possesses traits of both language data and image data. First, language data generally has high information density and exhibits strong contextual dependencies. Autoregressive modeling methods share a natural alignment with human language [1], and similarly, time series data emphasizes dependencies from past to current states. Secondly, image data typically has lower information density [2] and places greater emphasis on jointly modeling global spatial relationships and locality. Time series data also shares similar characteristics.

Recently, there have been excellent works applying diffusion to autoregressive language models, such as LlaDA [3], as well as works applying autoregressive approaches to image generation, such as VAR [4]. These are inspiring contributions. As contemporaneous work, we carefully examined the distinct roles that diffusion modeling and autoregressive modeling play in time series representation learning.

On one hand, autoregressive modeling can capture relationships from left to right. However, we recognize that time series data also shares characteristics with image data, including locality features. Due to its relatively low information density, it is challenging to effectively fit time series data using autoregressive methods without employing discretization techniques like VQVAE.

On the other hand, diffusion models can effectively model locality features in time series, such as abrupt weather changes within a single day or week. However, modeling time series requires a significant focus on trend modeling [5],[6]. otherwise, the model risks overfitting to drift.

This is the source of our motivation: by combining these two approaches, we can capture both long-term dynamic evolution and subtle local patterns in a unified manner .

[1] GPT-1: Improving Language Understanding by Generative Pre-Training

[2] MAE: Masked Autoencoders Are Scalable Vision Learners

[3] LlaDA: Large Language Diffusion Models

[4] VAR: Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

[5] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

[6] FDF: Flexible Decoupled Framework for Time Series Forecasting with Conditional Denoising and Polynomial Modeling

A3 for W1

For non-diffusion self-supervised methods, comparisons are provided in Tables 2, 3, and 4 of the paper, including methods based on mask modeling and contrastive learning. Specific baseline details can be reviewed in Section 4.1 (Baselines) of the article. The experiments demonstrate that, compared to the baseline methods we selected, TimeDART achieves superior downstream fine-tuning performance after pre-training.

For LLM-based methods, please refer to A2 for Q2 in our rebuttal to the first Reviewer baxN where we compare TimeDART with LLM-based methods like UniTime.

A4 for W2

Thank you for your sincere suggestions. We will continue to refine our paper based on the feedback from all reviewers, especially the analysis regarding the differences between autoregressive and diffusion modeling that you mentioned in question 2.

Thank you for your encouraging score. We sincerely look forward to your reply.

All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/

[Table1]--->para7, [Table2]--->para8

A1 for Q1

We conduct detailed few-shot experiments on the performance of the model with 5% or 10% fine-tuning data, including forecasting and classification tasks. The results can be found in the [Table1] below.

[Table1]

A2 for Q2

The following is our detailed derivation from formula [7] to formula [8], which we will integrate into the paper in the future to highlight the comprehensiveness and rigor of the paper.

The original theoretical loss should be in the form of cross entropy:

L_{ideal}=H(p_\\theta(x^0\_{1:j}),q(x^0_{1:j}))=\\mathbb{E}_{q(x^0\_{1:j})}(-\\log p\_\\theta(x^0\_{1:j}))

Use the Evidence Lower Bound (ELBO) method and ignore the patch index subscript:

$ L\_{ideal} &\\leq \\mathbb{E}\\_{q(x^0)} \\left(\mathbb{E}\_{q(x^{1:T}|x^0)}\\left[\\log \\frac{q(x^{1:T}|x^0)}{p\_\\theta(x^{0:T})}\\right]\\right) \\\\ &=\\mathbb{E}\_{q(x^{0:T})} \\left[\\log \\frac{q(x^{1:T}|x^0)}{p\_\\theta(x^{0:T})}\\right] \\\\ &:= L\_{diff} $

Expand and use the Bayes formula:

\ $ L &= \\mathbb{E}\_{q(x^{0:T})} \\left[ \\log \\frac{q(x^{1:T}|x^0)}{p\_\\theta(x^{0:T})} \\right] \\\\ &= \\mathbb{E}\_{q(x^{0:T})} \\left[ \\log \\frac{\\prod\_{s=1}^T q(x^s|x^{s-1})}{p(x^T) \\prod\_{s=1}^T p\_\\theta(x^{s-1}|x^s)} \\right] \\\\ &= \\mathbb{E}\_{q(x^{0:T})} \\left[ \\log \\frac{ q(x^T|x^0)}{p(x^T)} + \\sum\_{s=2}^T \\log \\frac{q(x^{s-1}|x^s, x^0)}{p\_\\theta(x^{s-1}|x^s)} - \\log p\_\\theta(x^0|x^1) \\right] \\\\ \ $

The first term is a constant, the third term is the reconstruction loss, and only the second term is discussed:

\ $ \\mathbb{E}\_{q(x^{0:T})} \\sum\_{s=2}^T \\log \\frac{q(x^{s-1}|x^s, x^0)}{p\_\\theta(x^{s-1}|x^s)} &= \\int\\text{d}x^{0:T}~ q(x^{0:T})\\cdot\\sum\_{s=2}^T \\log \\frac{q(x^{s-1}|x^s, x^0)}{p\_\\theta(x^{s-1}|x^s)} \\\\ &=\\sum\_{s=2}^T\\mathbb{E}\_{q(x^0,x^s)}\\left[D_{KL}(q(x^{s-1}|x^s, x^0)\\|p\_\\theta(x^{s-1}|x^s))\\right] \ $

Applying Bayes' theorem again:

$$q(x^{s-1} | x^s, x^0) = \\mathcal{N}(x^{s-1},\\tilde{\\mu}_t(x^s,x^0),\\tilde{\\beta}_s I)$$

It can be assumed that the predicted distribution can also be expressed as:

p_\\theta(x^{s-1}|x^s) = \\mathcal{N}(x^{s-1};\\mu_{\\theta}(x^s,s),\\sigma_s^2 I)

According to the KL divergence of two Gaussian distributions with the same variance, we can get:

$$L\_{diff}=\\mathbb{E}\_{q(x^0,x^s)}\\left[\\frac{1}{2\\sigma\_s^2} ||\\tilde{\\mu\_s} (x^s, x^0)-\\mu\_\\theta(x^s,s) ||^2\\right]$$

Since $\\tilde{\\mu}_s$ and $x^0$ are linearly related, we turn to predict the original space, so:

$$L_{diff}\\sim \\mathbb{E}_{\\epsilon, q(x^0)} \\left[ || x^0 - x^{out} ||^2 \\right].$$

Adding the cumulative sum of subscripts and the notation from the original paper, we get:

$$L_{ours} = \sum_{j=1}^{N} \\mathbb{E}_{\\epsilon, q(x_j^0)} \\left[ || x_j^0 - g(\hat{z_j}^{in} - f(z\_{1:j-1}^{in}) ||^2 \\right]$$

A3 for Q3

Below is a table of the key parameters for the complete model, pre-training, and fine-tuning process, where p1/p2/… means we searched for these parameters (such as d_model, learning_rate), or we dynamically modified these parameters based on the size of the dataset (such as batch_size):

[Table2]

Thank you for your encouraging score. We sincerely look forward to your reply.

All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/

[Table1]--->para4,para5, [Table2]--->para6, [A3 for Q3] --->para1, [A5 for Q5]--->para2

A1 for Q1

We abandon autoregressive modeling and diffusion in the downstream task based on three considerations: First, we not only performed forecasting tasks but also discriminative tasks such as classification, which are difficult to directly adapt to this paradigm. Second, almost all evaluations of self-supervised methods for time series follow the approach of transferring only the encoder and attaching different heads for various downstream adaptations (e.g., TimeSiam, SimMTM). Therefore, to enable comparisons and ensure fairness, we adhered to the same transfer method. Lastly, autoregressive inference and diffusion sampling would be extremely slow.

Early of this work, we have designed a downstream denoising autoregressive paradigm for forecasting tasks and conducted experiments. However, based on the considerations above, we decided not to use this approach further. We appreciate the reviewer's question, as it allows us to revisit our initial design. The results can be found in the [Table1] below.

[Table1] The following table shows the average of predicted window on [96,192,336,720]. Detailed designs can be found at para5 in link above. The results show that retaining the decoder for prediction can achieve similar results.

A2 for Q2

Even though the denoising decoder is removed during fine-tuning, it remains necessary, as the ablation experiments show: removing the diffusion model during pre-training leads to a decline in performance for both downstream forecasting and classification tasks.

We can use a simpler MLP as a decoder. For simplicity, we directly use the concat strategy to concatenate the output of the causal encoder and the embedding of the noise added patches in the dim axis. In order to keep the parameters consistent with the original 1-layer Transformer (about $12d_{model}^2$), we use a Squencial(Linear($2d_{model}$, $4d_{model}$), ReLU, Linear($4d_{model}$, $d_{model}$) ) MLP as the denoising decoder. The results can be found in the [Table2] below.

[Table2] The following table shows the average of predicted window on [96,192,336,720]. The results show that simpler's MLP is slightly weaker than TRM, but almost the same.

A3 for Q3

Please refer to A1 for Q1, paragraph 1 in our rebuttal to the first Reviewer baxN, where the table of gpu memory and calculation is given.

A4 for Q4

You have raised an important point, however, we must clarify the entire data processing and experimental evaluation section:

You mentioned that in /data_provider/data_factory.py#L41, setting drop_last to True might lead to discarding some samples, potentially biasing the results. However, at #L44 and #L45, when the downstream task is forecasting, the batch_size is set to 1. With this, no forecasting test samples will be dropped even drop_last is set to True. Furthermore, at #L42 and #L43, when the downstream task is classification, although batch_size is set to args.batch_size, at #L69 and #L70 drop_last is explicitly reset to False. Thus, the classification task is also unaffected. For the sake of rigor: we double-checked the code and printed the key parameters. The results confirm that the critical parameters are as described above.

A5 for Q5

Please refer to A1 for Q1, paragraph 3 in our rebuttal to the first Reviewer baxN, where the table of different look-back windows is given.

A6 for W1

As mentioned in a3 for w1 in our rebuttal to first Reviewer baxN, we did not mechanically combine these two techniques in pre-training. Instead, we approached the uniqueness of time series data with careful consideration. Simply using autoregressive optimization during pre-training does not yield satisfactory results when transferred to downstream tasks. However, through our novel approach of explicitly introducing noise, we found that the pre-training performance indeed improved, which inspired the development of our current complete work.

A6 for W2

Please refer to A1 for Q1.

A7 for W3

Please refer to A3 for Q3 above or A1 for Q1, paragraph 1 in our rebuttal to the first Reviewer baxN.

Your score is very important to us. We hope we can solve your questions. We sincerely look forward to your reply.

All full results are at https://anonymous.4open.science/r/TimeDART-results-ECBD/

[Table1] —>para1,[Table2] —>para2,[Table3] —> para3

A1 for Q1

1. For efficiency, we used a lightweight denoising decoder during pre-training. After training, similar to masked autoencoders, only the embedding layer and encoder network are transferred for downstream tasks. As a result, TimeDART introduces small additional computational overhead in terms of GPU memory usage and computation speed. Specific comparative results can be found in the [Table1] below.
2. In forecasting tasks, we have evaluated performance on large datasets especially traffic and pems. Dataset details, including variables and lengths, are in Section 4.1. We also pre-trained on all six datasets combined, reaching approximately 43M training time points (channel independence). This data volume is significantly larger than the baseline in the article.
3. To evaluate the scalability of TimeDART for longer sequences, we conducted additional experiments with different look-back window sizes, especially extending it to 576 and 720 in ETTh2. We observed a further reduction in prediction MSE, indicating that TimeDART scales well for longer sequences. The detailed results can be found in the [Table2] below.

[Table1] The data in the following table is recorded in the Traffic dataset with a look-back window=336 and a predicted window=336.

[Table2] The following table shows the average of predicted window on [96,192,336,720].

A2 for Q2

1. Due to limitations in GPUs and the time constraints of the rebuttal process, and given the significant disparity in the scale of training data, it is challenging for us to directly compare with large foundation models in a short period. To address your concerns regarding representation quality and transferability, we have instead conducted some comparisons with SOTA methods based on LLMs (e.g. UniTime, GPT4TS), which we hope will help alleviate your doubts and concerns. Specific comparative results can be found in the [table3] below.
2. Regarding the cross-domain issue, we mixed all in-domain datasets for general pre-training in forecasting tasks and fine-tuned across domains. As noted in a1 for q1, this yielded approximately 43M training time points. Experiments (Section 4.2) demonstrate that TimeDART’s cross-domain representation learning, under large-scale pre-training, shows improvement compared to no pre-training. As for going beyond the current benchmark datasets, we are considering conducting general pre-training on larger public datasets (e.g., UTSD) to further validate TimeDART’s representation ability.

[Table3] The look-back window is reset to 96 same to UniTime to ensure fair comparison. Model is pretrained on ETT(4subsets), Exchange, Electricity, Traffic datasets. The following table shows the average of predicted window on [96,192,336,720]. Experiments show that TimeDART can still learn better representations compared to LLM-based methods.

A3 for W1 Due to time constraints, we just provide a brief explanation. Our assumption is that traditional autoregressive pre-training excels in capturing long-term dependencies but struggles with inherent time series noise, which is hard to avoid. By introducing noise into the pre-training process, we encourage the encoder to learn both long-term dependencies and the distribution of gaussian noise $f(x_j^0|x_{1-j-1}^0, \epsilon_1^{s_1}, ..., \epsilon_{j-1}^{s_{j-1}})$. This improves the model's adaptability to inherent noise in downstream tasks. We are now working on a formal theoretical proof of combining autoregressive modeling with diffusion, which will be included in future work.

A4 for W2 Please refer to a1 for q1.

Thank you for your encouraging score. We sincerely look forward to your reply.