Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting

Thank you very much for your careful review and precious advice. We have made great effort to address your questions and concerns.

W1: Concerns on the motivation and behaviors of the proposed complementary contrastive learning.

Indeed, contrastive learning have been shown to discover effective vision or language representations by learning to discriminate between positive and negative sample pairs. But this might not be a valid manner as well in time series forecasting, since time series do not have the explicit and human-recognizable temporal structures, like the definite semantics of image and text data. In this work, we aim to explicitly maximize the mutual predictive information between limited past observations and future forecasts, which is beneficial to improve the accuracy and reliability of the generated prediction intervals. To achieve this goal, we specifically design the time series contrastive learning adapted from the classic InfoNCE loss to implement the intention of mutual prediction-related information maximization. Meanwhile, in order to improve the training efficiency of standard denoising diffusion, we particularly endow a step-wise noise regression form into the InfoNCE loss which can be seamlessly consistent with the naive denoising training manner. The additional contrastive training can help time series diffusion models to discern the low-probability areas (constituted by diffusion paths of negative samples) in the target predictive distribution and keep away from them. Besides, CCDM can be learned end-to-end on specific datasets without laborious self-supervised pretraining on large-scale corpus and fine-tuning on downstream tasks. We additionally provide more detailed theoretical interpretations on the benefits of the devised information-theoretic contrastive diffusion training in $\underline{\text{Appendix A.1.2 and Theoretical Insights Section of the revised paper}}$.

W2: Insufficient statement on the novelty of the proposed channel-wise denoising network.

As the reviewer said, channel-centric time series structure design has been broadly investigated in the field of multivariate deterministic forecasting, such as the excellent work of iTransformer and Crossformer. But this issue has not been well explored in the specific multivariate diffusion-based probabilistic forecasting task. In this area, a key architectural challenge is how to design an effective conditional time series denoising network, which is able to identify the faithful multi-channel temporal properties with varying degrees of diffused noise disturbing each variate sequences. Improper treatment for the diffused noise can cause training instability and hurt the capacity to tackle long-term dependencies and complex inter-channel correlations. In this regard, we propose a composite channel-aware conditional denoiser with both channel-independent CiDM and channel-mixing DiT. It can tackle the side effect of noise corruption and recover the plausible heterogeneous temporal correlations.

W3.1&Q1.1: More comparisons with up-to-date competitive forecasting models.

To further verify the forecasting capability of CCDM, we implement extra four classes of baseline models for a more comprehensive comparison, including iTransformer [5], SimMTM [6], mr-Diff [7] and Moirai [8]. We compare their deterministic forecasting capability on three real-world datasets and present the MSE and MAE results as follows:

We can observe that CCDM and SimMTM can achieve the state-of-art and second-best ranks respectively. It reveals that designing complementary learning strategies like contrastive refinement in CCDM or masked pretraining in SimMTM versus standard predictive training is able to enhance the forecasting capacity on specific time series. This justifies the design principles in our work.

We have incorporated these additional comparison outcomes in $\underline{\text{Appendix A.8.2 and Table 9 of the revised paper}}$.

W3.2&W4.1: Improvement on CCDM forecasting results presented in Table 1, 2.

Following the training implementation in CSDI, we add a MultiStepLR scheduler in Pytorch to adaptively adjust the learning rate during contrastive diffusion training. Besides, we modulate the weight decay of Adam optimizer to $1e-6$, which can regularize the network parameters learned by CCDM using L2-norm to avoid overfitting. These two operations can further improve the contrastive training stability and efficiency, as the step-wise noise injection in diffusion and the erratic variability of real-world time series could render time series diffusion hard to attain the optimal learning effectiveness. Furthermore, we slightly increase the batch size and negative sample number for Electricity and Traffic datasets. We present both the updated and original average results in Table 1 and performance decrease by w/o contrastive learning in Table 2 as follows:

We can see that CCDM outcomes exhibit a moderate growth after we refine the training settings. The detailed overall comparison and component ablation results have been updated in $\underline{\text{Table 1, 2, 10 of the revised paper}}$.

[1] Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng and Bin Yang. TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. PVLDB, 17(9): 2363 - 2377, 2024.

[2] Luo D, Cheng W, Wang Y, et al. Time series contrastive learning with information-aware augmentations. Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4534-4542.

[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.

[4] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations, 2021.

[5] Liu Y, Hu T, Zhang H, et al. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.

[6] Dong J, Wu H, Zhang H, et al. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 2024, 36.

[7] Shen L, Chen W, Kwok J. Multi-Resolution Diffusion Models for Time Series Forecasting. The Twelfth International Conference on Learning Representations. 2024.

[8] Woo G, Liu C, Kumar A, et al. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592, 2024.

We sincerely hope our responses address some of your concerns. If you have any further questions, please feel free to ask.

Q2.2: Comparisons for diverse types of negative time series augmentation methods.

To enable contrastive learning for time series diffusion models, we consider the following four types of augmentation methods to produce negative sequences, and the latter two operations are borrowed from [2]:

• Intra-series variation shuffling. It alters the ground-truth temporal variations along each channel by patch shuffling, since recovering the correct dynamic evolution is a vital challenge for time series diffusion models.
• Magnitude scaling. It scales up or scales down the magnitudes of individual time points, as an ideal prediction interval should well cover every point forecasts without any of them falling outside. Thus, we uniformly sample a scaling factor between $\left [ 0, 0.5 \right ] \cup \left [ 1.5, 2.0 \right ]$ and multiply each univariate time series by this factor.
• Jittering. It samples a random Gaussian noise from $\mathcal{N}(0, 0.3)$ and adds it to the ground-truth time series.
• Cutout. It randomly masks out the true values on $10\%$ timestamps from real sequences by zeros.

We attest the effect of these four negative construction methods on ETTh1 dataset with $L=96, H=168$ and report prediction results below:

We can find that utilizing scaling and variation augmentation methods incurs modestly better quality of prediction intervals than normal Gaussian jittering and zero cutout. Thus in the devised CCDM, we combine the scaling and variation methods to produce more informative negative instances at each diffusion step. We also integrate this clarification on negative time series augmentations in $\underline{\text{Section A.9.1 and Table 11 of the revised paper}}$.

Q3: Time overhead analysis on contrastive training.

As the reviewer commented, introducing additional contrastive learning to time series diffusion forecasters can give rise to larger computational time consumption. We just leverage this auxiliary scheme to trade training time efficiency for prediction accuracy. It is not straightforward to conduct theoretical time complexity analysis for contrastive training, since it could be affected by a variety of potential factors, including the volume of dataset, batch size in each iteration, negative sample number, model architecture, hardware capacity and so on. In $\underline{\text{Section A.7 and Table 7 of the revised paper}}$, we present both training and inference time of CCDM versus other baseline models like other contrastive time series forecasting work have implemented [4]. And corresponding model dimension, batch size and negative sample size are provided in $\underline{\text{Table 5, 6 of the revised paper}}$. We can see that although CCDM costs more training time, its sequential inference procedure is notably faster than other models. Besides, for the large-scale Electricity and Traffic dataset, we only impose contrastive training to the last 20 epochs to reduce the time overhead.

W4.2&Q1.2: Explanation on the results of channel-aware DiT ablation study for ECL dataset.

There could be two possible reasons to explain this phenomenon: 1) As the channel dimension in ECL is considerably larger than that in ETTh1 and Weather, it would be much harder for channel-wise attention to discover the plausible cross-channel correlations by diffusion training on limited ECL data. 2) According to the claims in Section 5.3.2 of a nice benchmark paper [1], the inter-variate correlations in ECL are relatively weaker than those in another two datasets. Thus, adding channel-wise blocks to capture the cross-channel temporal properties on time series with weak variate correlations may not bring a benefit as prominent as that with strong variate correlations.

Q2.1: Influence of different forms of density ratio functions on temporal contrastive learning.

To seamlessly align with the standard denoising diffusion training paradigm, we specifically dictate the density ration function $f(\cdot)$ using the step-wise noise regression form, which adopts the same MSE loss to optimize the conditional denoising network. In fact, we can also prescribe $f(\cdot)$ as another similarity-based form, which is widely employed in vision representation learning [3]. We present this similarity-based design for density ratio function as:

where $\mathrm{sim}(\cdot)$ indicates the cosine similarity between the ground-truth and predicted noise. The essence of above two equations is that the predicted noise of positive time series should be more similar to the imposed noise label $\epsilon{}'$, while that for negative instances is repelled from the true $\epsilon{}'$. We validate the efficacy of these two disparate density ratio forms for time series contrastive diffusion learning, and report the forecasting outcomes on two real-world datasets with $L=48, H=96$ as follows:

We can clearly observe that the MSE noise regression form is slightly better than the cosine similarity type, which suggests that aligning the additional contrastive training with naive denoising manner is more effective to enhance time series diffusion models.

We have contained this part of discussions in $\underline{\text{Appendix A.9.2 and Table 12 of the revised paper}}$.

Q4: Elaboration on the specific challenges of applying contrastive learning to time series modeling.

There are two specific challenges on integrating contrastive learning to time series generative modeling:

• Constructing effective negative time series instances to inform temporal contrastive learning. As claimed in [3], [4], how to select appropriate time series augmentation methods is a key factor for ultimate contrastive learning effect, since time series do not have diverse and human-recognizable temporal structures like images and languages. In this work, we consider four kinds of augmentation methods including intra-series variation shuffling, scaling, jittering and cutout to produce negative samples for contrastive diffusion training. More detailed discussions can be found in $\underline{\text{Section A.9.1 and Table 11 of the revised paper}}$.
• Adapting an efficient task-specific contrastive learning objective for the target time series analysis task. In this specific probabilistic forecasting scenario, we wish to maximally utilize the beneficial predictive information in historical conditioning time series. For this purpose, we specifically leverage an information-theoretical contrastive learning objective to explicitly maximize the prediction-related mutual information between past observations and future forecasts. Besides, we specifically prescribe this contrastive loss as a noise regression form to seamlessly align with the standard denoising diffusion training. More detailed explanations can be seen in $\underline{\text{Appendix A.9.2 and Table 12 of the revised paper}}$.

[1] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from a multi-view perspective. In International Conference on Learning Representations, 2020.

[2] Yunshu Wu, Yingtao Luo, Xianghao Kong, Evangelos E Papalexakis, and Greg Ver Steeg. Your diffusion model is secretly a noise classifier and benefits from contrastive training. arXiv preprint arXiv:2407.08946, 2024.

[3] Luo D, Cheng W, Wang Y, et al. Time series contrastive learning with information-aware augmentations. Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4534-4542.

[4] Woo G, Liu C, Sahoo D, et al. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575, 2022.

[5] Cao D, Ye W, Zhang Y, et al. Timedit: General-purpose diffusion transformers for time series foundation model. arXiv preprint arXiv:2409.02322, 2024.

[6] Feng S, Miao C, Zhang Z, et al. Latent diffusion transformer for probabilistic time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(11): 11979-11987.

Thank you again for your thorough review, and we hope our responses can address your questions.

Q1: Clarifying the difference between "neural mutual information perspective" and the combination of denoising and contrastive loss?

According to the motivation clarified in W2 response, we aim to explicitly maximize the prediction-related mutual information between past observations $\mathbf{x}$ and future forecasts $\mathbf{y} _{0}$. As claimed in [1], standard generative diffusion learned by log-likelihood maximization (i.e. $\max p _{\theta}(\mathbf{y} _{0}|\mathbf{x})$) and derived denoising loss is a forward predictive way to maximize the mutual information between $\mathbf{x}$ and $\mathbf{y} _{0}$. [1] both theoretically and empirically proved that combining predictive learning and contrastive learning can enhance the effectiveness of neural mutual information maximization on target tasks. Inspired by this, we propose to complement the contrastive learning to naive denoising diffusion learning. Besides, we specifically design a novel noise regression form for the canonical InfoNCE contrastive loss to make the additional contrastive training seamlessly align with original denoising training.

Q2: Distinctions and comparisons with DiT-based TimeDiT [5] and LDT [6].

In addition to the novel contrastive learning framework, the proposed composite channel-aware DiT distinctly leverage the unified channel-centric strategy to model the multivariate predictive distribution, and can differ from TimeDiT and LDT in two ways below:

• Multi-head attention usage for temporal correlations modeling. We switch the naive point-wise attention over the time dimension to a channel-wise attention along the variate axis. It allows to represent complex cross-channel correlations in given conditioning time series $\mathbf{x}$ and corrupted targets $\mathbf{y} _{k}$, apart from the temporal dependencies captured by channel-independent dense encoders.
• Incorporation scheme of conditioning time series. We directly concatenate the conditioning $\mathbf{x}$ with corrupted $\mathbf{y} _{k}$ and capture their temporal features by the subsequent channel-wise attention. This operation can fully exploit the useful predictive information in limited historic observations. Whereas TimeDiT and LDT simply pass the given $\mathbf{x}$ to linear adaLN layers, which may cause the predictive information loss for DiT blocks.

Furthermore, we compare CCDM with TimeDiT and LDT on Exchange and Electricity datasets with lookback window of 168 and prediction horizon of 24. The comparison outcomes are presented below:

We can see that channel-aware CCDM can obviously outperform TimeDiT and LDT. We have updated above discussion on DiT adaptation differences and comparisons in $\underline{\text{Section 3.1 and Appendix A.8.1 of the revised paper}}$.

Q3: Explaining Figure. 5 more clearly to demonstrate the positive effect of contrastive learning.

Following the reviewer's suggestion, we have replotted Figure. 5 to exhibit the positive effect of contrastive learning more clearly. As shown in $\underline{\text{Fig. 5 of the revised paper}}$, where we display MSE, MAE, CRPS and CRPS_sum incurred by assigning various values to the contrastive weight $\lambda$. Fig. 5 aims to illustrate two points: 1) how different weights (i.e. $\lambda=0.01, 0.005, 0.001, 0.0005, 0.0001$) of the proposed contrastive loss in Eq. 6 can influence the diffusion forecasting capacity. 2) Adding contrastive training to naive diffusion predictive learning can actually enhance the forecasting capability, as the four metric outcomes of $\lambda=0$ (i.e. without contrastive refinement) is significant larger than those of $\lambda>0$ (i.e. imposing different degrees of contrastive learning).

Many thanks for your meticulous review and insightful questions. In the following, we make great effort to address each of your concerns and questions.

W1: The theoretical insights do not seem cohesive.

Both [1] and [2] have demonstrated that denoising diffusion models can benefit from additional contrastive training, which can lead to a higher-quality approximation for the target distribution, but they work from two distinct perspectives to interpret this merit. Specifically, the former reference starts from the view of neural mutual information maximization to state two complementary training objectives, see Q1 response below for details. While the latter reference stands from the view of out-of-distribution (OOD) evaluation, claiming that learning on extra negative samples can help diffusion models to recognize the low-probability regions in the target distribution. Since the less powerful training objectives and distribution shifts are actually two key issues in the scope of time series diffusion forecasting, the Theoretical Insight part just claims that introducing contrastive learning can tackle these two problems simultaneously.

W2: Insufficient statement on major challenges and contributions.

Here, we want to highlight the main contributions, as well as clarifying the related challenges tackled by this paper more clearly as follows:

• First composite channel-aware architecture for temporal conditional denoising. In the time series forecasting community, it remains an open issue on the design of effective conditional denoising network, which is able to identify the faithful channel-specific and cross-channel temporal properties with varying degrees of diffused noise disturbing each variate sequences. To this end, we propose a composite channel-aware conditional denoising network consisting of both channel-independent dense encoders and channel-mixing diffusion transformers, which can tackle the side effect of noise corruption and recover the plausible heterogeneous temporal correlations.
• Novel integration of time series contrastive learning with diffusion forecasting for more efficient exploitation of useful predictive information. Given limited historical time series, it is an under-explored issue to find effective schemes for enhancing the utilization efficiency of the implicit temporal predictive information. The auxiliary training methods leveraged by existing diffusion forecasters need to expose prior knowledge on task-specific temporal properties, and can be inconsistent with standard step-wise denoising training. To this end, we devise a complementary denoising-based temporal contrastive refinement to boost diffusion training efficiency and explicitly amplify the prediction-related mutual information between generated forecasts and past observations.
• Comprehensive evaluation on multivariate probabilistic forecasting. Previous generative forecasting methods only verify their performance on short-term settings and do not release their capability on more challenging scenarios with long-term dependencies. To this end, we construct a more comprehensive benchmark with varying prediction horizons and channel numbers to completely compare different diffusion forecasters' capacity. The proposed CCDM can achieve the state-of-art performance on diverse setups.

We have restated the key challenges, motivations and contributions in $\underline{\text{Introduction section of the revised paper}}$.

W3: More probabilistic showcases like Figure 4.

In $\underline{\text{Figure 14 of the revised paper}}$, we visually compare the quality of prediction intervals and point forecasts produced by four different models on each channel of ETTh1. We can clearly see that the prediction intervals generated by contrastive diffusion CCDM hold better accuracy, sharpness and reliability to encompass the real observations versus other models. We can also observe that the faithfulness of the approximated predictive distribution can be enhanced after introducing auxiliary contrastive training to time series diffusion models.

We are grateful for your positive support and beneficial comments. Below we address each question and concern.

W1: Concerns on extra hyper-parameter searching for effective contrastive refinement.

As the review suggested, empirically determining the optimal contrastive training configuration can vary case by case. However, the remarkable forecasting outcomes achieved by CCDM as shown in $\underline{\text{Table 1, 9 of the revised paper}}$ can reveal that: simply initializing CCDM using the uniform setting provided in Table 5 without any extra hyper-parameter search is sufficient to acquire more excellent forecasting capability versus other baseline models on a wide variety of real-world datasets and prediction scenarios. We have added this part of discussions to $\underline{\text{Appendix A.9.3 of the revised paper}}$.

Q1: p3: $\mathbf{y} _{0}$ and $\mathbf{y} _{K}$ is introduced one section too late.

$\mathbf{y} _{0}$ denotes the clean ground-truth time series at diffusion step 0, while $\mathbf{y} _{K}$ indicates the prior Gaussian noise at diffusion step K. We want to calrify that we have introduced them at Section 2.1 and 2.2.

Q2: Unclear statement in p4: accounts for … and can be any types of … ().

As proved in the foundational contrastive representation learning work [1], the density ratio function $f(\mathbf{y} _{0},\mathbf{x})=\frac{q(\mathbf{y} _{0}|\mathbf{x})}{q(\mathbf{y} _{0})}$ can be any positive real-valued forms to support the contrastive training governed by the derived InfoNCE loss. The noise regression form $f(\mathbf{y} _{0},\mathbf{x})=\exp (-||\epsilon{}'-\epsilon _{\theta}(\cdot)||^{2} _{2}/\tau)$ proposed by this work is inspired by this fact. In $\underline{\text{Section 3.2 and Appendix A.9.2 of the revised paper}}$, we provide more detailed discussions on this density ratio function design.

Q3: Unclear statement in p5: replacing point-wise attention by channel-wise attention.

Actually, previous DiT-based diffusion forecasters such as LDT and TimeDiT place attention mechanism along the time dimension to capture the temporal dependencies among each time points, which we call it point-wise attention. While we put the attention module along the channel axis to represent the cross-channel temporal correlations, which we call it channel-wise attention. More clear discussions on these two different DiT adaptations can be found in $\underline{\text{Section 3.1, paragraph 3 of the revised paper}}$.

Q4: p6 \l 275: both a positive sample.

We have addressed the typo in the revised paper.

Q5: p7 \l 327: efficiency.

We have corrected this word mistake in the revised paper.

Q6: Rendering Figure. 5 more clear and meaningful to show the positive effect of contrastive refinement.

We have depicted a new version of Figure. 5 to exhibit the positive effect of contrastive learning more clearly. As shown in $\underline{\text{Fig. 5 of the revised paper}}$, where we display MSE, MAE, CRPS and CRPS_sum incurred by assigning various values to the contrastive weight $\lambda$. Fig. 5 aims to illustrate two points: 1) how different weights (i.e. $\lambda=0.01, 0.005, 0.001, 0.0005, 0.0001$) of the proposed contrastive loss in Eq. 6 can influence the diffusion forecasting capacity. 2) Adding contrastive training to naive diffusion predictive learning can actually enhance the forecasting capability, as the four metric outcomes of $\lambda=0$ (i.e. without contrastive refinement) is significant larger than those of $\lambda>0$ (i.e. imposing different degrees of contrastive learning).

[1] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

We sincerely appreciate your positive feedback again and hope our response can address your concerns. If you have any further questions, please feel free to ask.

[1] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from a multi-view perspective. In International Conference on Learning Representations, 2020.

[2] Yunshu Wu, Yingtao Luo, Xianghao Kong, Evangelos E Papalexakis, and Greg Ver Steeg. Your diffusion model is secretly a noise classifier and benefits from contrastive training. arXiv preprint arXiv:2407.08946, 2024.

[3] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Research, 2023a.

[4] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2023.

[5] Kim T, Kim J, Tae Y, et al. Reversible instance normalization for accurate time-series forecasting against distribution shift. International Conference on Learning Representations. 2021.

[6] Liu H, Kamarthi H, Kong L, et al. Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learning. Forty-first International Conference on Machine Learning. 2024.

[7] Lu W, Wang J, Sun X, et al. Out-of-distribution Representation Learning for Time Series Classification. The Eleventh International Conference on Learning Representations. 2023.

[8] Chen M, Shen L, Fu H, et al. Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift[C]. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024: 341-352.

[9] Liu Y, Wu H, Wang J, et al. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 2022, 35: 9881-9893.

[10] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pp. 8857–8868. PMLR, 2021.

[11] Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, and Jiang Bian. Mg-tsd: Multi-granularity time series diffusion models with guided learning process. arXiv preprint arXiv:2403.05751, 2024.

We sincerely hope our responses address your concerns. If you have any further questions, please feel free to ask. Thank you.

W3.3&Q4.3&Q5.2: No empirical evidence to support the OOD performance improvement.

The OOD generalization issue, a.k.a temporal distribution shift is an imperative and fundamental issue in time series forecasting. Since the real-world time series are always non-stationary and stochastic, the intrinsic temporal dependencies and inter-variate correlations can alter over time, which will render the training distribution different from the unknown test distribution. Previous works [5], [6], [7], [8], [9] have statistically demonstrated the OOD phenomenon is ubiquitous in real-world time series datasets (e.g. six datasets utilized in our work), and incorporating auxiliary learning strategies to combat with test distribution shift can enhance the forecasting performance, like the invariant learning in [6] and adversarial training in [7].

In this work, we find that the proposed complementary contrastive learning can also improve the forecasting capability of naive time series diffusion models on OOD test time series data. In Fig. 3 of [5], we can observe that the distribution of unknown test time series over each step is hard to overlap with that of collected training sets. We verify this by comparing the proposed contrastive diffusion CCDM with three plain denoising diffusion forecasters purely trained by step-wise noise regression, including CCDM-w/o contrastive, CSDI, TimeGrad. We choose six real-world datasets from the field of Energy, Weather, Finance and Traffic, which have been shown to have the OOD issue in [5]. We present the comparison results below:

We can see that contrastive diffusion model CCDM can achieve the state-of-art results across various datasets, which reflects that adding the proposed contrastive training can improve the forecasting capacity on OOD test time series. More detailed outcomes over diverse prediction horizons and other baseline models can be found in $\underline{\text{Table 1, 2, 8, 9, 10 of the revised paper}}$.

W3.4&Q4.4&Q5.3: The absence of mean and variance statistics to quantify uncertainties inherent in diffusion models.

As the reviewer suggested, time series diffusion models will randomly generate a group of possible forecasting trajectories according to its approximated predictive distribution, which constitute a prediction interval. In probabilistic forecasting area, both the deterministic prediction accuracy and probabilistic prediction uncertainties should be well quantified. Adhering to many other canonical time series generative forecasting models [10], [11], we also utilize MSE, MAE, CRPS and CRPS_sum metrics to assess the quality of diffusion generated prediction intervals. In detail, the MSE (Mean Squared Error) and MAE (Mean Absolute Error) are employed to quantify the mean difference between the obtained median forecast and true target, which can indicate the mean statistics as you mentioned. The CRPS (Continuous Ranked Probability Score) and CRPS_sum are employed to characterize the divergence between the generated prediction uncertainties and the real observed time series distribution, which can indicate the variance statistics as you mentioned. And Table 8 is easy to interpret, since auxiliary contrastive training strategy and channel-aware DiT denoising network are totally two disparate designs to improve time series diffusion forecasters, they will exert different influences on the forecasting outcomes. We have updated the descriptions of evaluation metrics in $\underline{\text{Appendix A.5 of the revised paper}}$, which aims to quantify the accuracy and uncertainties of diffusion generated forecasts.

W2.2&Q3.3: Lack of ablation study on the necessity of CiDM components in DiT blocks.

We leverage the channel-independent dense modules (CiDMs) in TiDE [3] to capture the univariate temporal dependencies along each channel. TiDE has verified that MLP-based CiDM with residual connections is an excellent structure to represent dynamic temporal variations. To verify its necessity in the proposed channel-centric denoising network, we replace it by a normal channel-independent linear encoder as shown in iTransformer [4], and report the respective results on a short-term forecasting setup in CSDI below:

We can see that if CiDM is replaced by by ordinary MLP entices, a moderate and consistent decline is observed for prediction accuracy. It reflects that adding residual shortcuts to channel-independent MLP encoders can indeed boost the expressivity for dynamic temporal variations, and verifies the virtue of residual CiDM modules in TiDE [3] again. We have incorporated this CiDM ablation study in $\underline{\text{Appendix A.8.1 and Table 8 of the revised paper}}$.

W3.1&Q4.1&Q5.1: Lack of short-term forecasting results as shown in CSDI.

We verify the capability of CCDM on short-term probabilistic forecasting at which previous diffusion forecasters are displayed to be adept. As you suggested, we follow the same short-term setting in CSDI with lookback window of 168 and prediction horizon of 24. We report the short-term prediction results below:

We can clearly observe that CCDM consistently outperforms other baselines on this short-term forecasting setup, which further validate the superior forecasting capacity of CCDM. We have incorporated this short-term comparisons in $\underline{\text{Appendix A.8.1 and Table 8 of the revised paper}}$.

W3.2&Q4.2: The two-stage training process is not clearly explained.

We apologize for unclear description for such two-stage training method. As introducing extra negative samples to every training batch and iteration will induce additional time overhead. When the data volume is extremely large, the training time consumption will be prohibitive. Hence, we propose a two-stage training method to reduce the training costs on large-scale datasets like Weather, ECL and Traffic. Concretely, we first train a naive low-cost diffusion model by the standard denoising loss in 100 epochs, and then proceed to train such intermediate model by the auxiliary contrastive loss in extra 20 to 30 epochs. $\underline{\text{Table 1, 8, 9, 10 of the revised paper}}$ illustrated that such two-stage strategy with less training costs can attain satisfactory forecasting outcomes as well.

Thank you very much for your careful review and constructive suggestions! Hope our response can address your questions and concerns.

W1.1&Q1: Concerns on novelty and contribution of combing contrastive learning with denoising diffusion.

Here, we would like to reexplain and clarify the core motivations and contributions of the proposed contrastive denoising diffusion forecaster CCDM. It is an under-explored issue on how to enhance the utilization efficiency of the implicit temporal predictive information hidden in limited historical time series. The auxiliary training methods leveraged by existing diffusion forecasters need to expose prior knowledge on task-specific temporal properties and can not be consistent with standard step-wise denoising training. To this end, we propose to devise a complementary temporal contrastive learning scheme to improve diffusion training efficiency and explicitly maximize the prediction-related mutual information between generated forecasts and past observations.

Our inspiration to integrate contrastive learning with denoising diffusion models come from two perspectives: 1) [1] has proved that combining both reconstruction-based generative learning and information-theoretic contrastive learning can enhance the usefulness of discovered task-specific representations. 2) [2] has shown that diffusion models can benefit from conducting more OOD evaluations on low-probability samples. Besides, we specifically design a noise regression form for the contrastive loss to seamlessly align with standard denoising diffusion training, which thus is efficient to implement. We are one of the first to integrate time series contrastive learning with diffusion forecasting for more efficient exploitation of useful predictive information.

W1.2&Q2: Concerns on Theoretical insights and Proposition 1 in Section 3.3.

Here, we want to interpret the role of the theoretical insights subsection more clearly. In this part, we aim to demonstrate how naive time series diffusion models can benefit from auxiliary contrastive learning to approximate the target predictive distribution more faithfully. [1] and [2] demonstrate two disparate perspectives to motivate combining auxiliary contrastive learning with naive denoising diffusion. We further adapt these claims for the specific time series diffusion forecasting task as clarified in W1.1&Q1 and W3.3&Q4.3&Q5.2 response. We have presented additional theoretical interpretations on the benefits of information-theoretic contrastive learning in $\underline{\text{Appendix A.1.2 and Theoretic Insights Section of the revise paper}}$.

As the reviewer suggested, the upper bound of probabilistic diffusion forecasting can not be calculated in exact closed-form. However, we want to clarify that utilizing the KL-divergence can represent such upper bound. Moreover, it reflects that the forecasting capacity of conditional diffusion models is closely associated with the step-wise noise regression accuracy of trained denoising network on OOD test time series. We believe this is still important for guiding CCDM training. In this regard, resorting to temporal contrastive learning or other auxiliary training regimes is significant to boost conditional denoiser behaviors and the ultimate prediction outcomes.

W2.1&Q3.1&Q3.2: Improvement for Figure 1,2,3.

We apologize for the ambiguities in Figure 1,2,3. For Figure 1, we have deleted the bidirectional arrows and let two branches represent two complementary lower bounds for predictive mutual information. For Figure 2, the detailed negative time series augmentation methods can be found in $\underline{\text{Appendix A.9.1 of the revised paper}}$, where we discuss about four kinds of negative construction methods and validate their effectiveness for contrastive learning. For Figure 3, the channel-independence and channel-dependence are articulated in $\underline{\text{Section 3.1 of the revised paper}}$, where we also talk about the distinctions from other DiT-based time series diffusion models. The necessity of CiDM modules has been verified in W2.2&Q3.3 response below.