Non-stationary Diffusion For Probabilistic Time Series Forecasting

Q1: Why does NsDiff not adopt a fully end-to-end joint optimization approach? Is it necessary to pre-train the two networks separately?

A1: yes, the networks can be trained jointly without large performance loss. Below is an example to train on ETTh1 with and without pretraining, where we report CRPS metric.

As can be seen, although joint train experiences a slight performance degradation (1.86%), it still outperforms the previous state-of-the-art TMDM (0.452). However, compared to pretraining, co-training is slightly harder to converge. We will clarify this in the updated version.

Q2: Figure 2 should be modified, e.g. remove curved lines.

A2: Thanks for this suggestion, we will modify Figure 2 according to your advice in the lastest version.

Q3: Report additional metrics such as MSE/MAE.

A3: Thanks for your insight. We give the results on real and synthetic datasets at the following tables with an addtional baseline CSBI (as required by Reviewer gSNF), the settings are consistent with the main context. The repo is updated accordingly to include CSBI code.

We present the results of MSE/MAE of synthetic datasets at follows:

As seen in this Table, NsDiff still achieves SOTA in uncertainty variation conditions.

We present the results of MSE/MAE of real datasets at follows:

Note: TimeDiff is a model speciffically designed for long-term point forecasting.

As in this Table, in datasets with high non-stationarity, NsDiff still achieves SOTA, attributed to the dynamic mean and variance endpoint and the uncertainty-aware noise schedule.

Here we give the results of an addtional baseline CSBI, where NsDiff still remains SOTA.

Q1: The differences from TMDM.

A1: We believe there are some misunderstandings regarding the relationship between our method and TMDM, particularly in the contributions and derivation aspects. We believe the differences from TMDM are clearly presented throughout the paper. To aid clarity, we list some key differences between NsDiff and TMDM along with parts of where they are discussed in the paper. We hope the following table helps the reviewer quickly identify and understand these distinctions with TMDM.

We believe that in the derivation presented in Appendix A, all the relevant parts listed in the table above are different from those in TMDM. For example, NsDiff infer the reverse distribution variance by solving Eq. 38, while TMDM does not introduce this step.

Q2: the method is based on fine-tuning, which requires path sampling during training. Is this computationally expensive for DDPM in time series?

A2: The fact is no path sampling is required during training. The training and inference procedures are the same as those of a standard DDPM, except for using a different endpoint and noise schedule. We only introduce some basic operations during training and inference, so the overall computational complexity remains. See Q5 for experimental results.

Q3: What if the variance predicted via MLE is too large (when some spiky data points appear)? Using sliding windows might not address this issue perfectly.

A3: We do introduce a design to address this issue by using an uncertainty-aware noise schedule, which incorporates the true variance into the diffusion training process. This design reduces reliance on the variance predicted by the estimator (e.g., via MLE), which can be overly large in the presence of spiky or noisy data. See Table 5 for the experiments results, where it can be seen that by explicitly learning from the true variance, NsDiff becomes more robust to such cases, rather than relying solely on sliding window heuristics or estimator outputs.

Q4: The numerically stability of NsDiff.

A4: In NsDiff, the only potential numerical issue is the solvability of the equation in Eq. 15. We have already provided the conditions for solvability in Eq. 17. As stated in the paper (lines 269-272), this equation always has a solution. Therefore, theoretically, NsDiff does not introduce additional numerical instability issues. In practice, no numerical instability has been observed in our experiments. We kindly invite reviewers to check and run our code to verify this.

Q5: The efficiency and memory costs in training and inference phases vs. performance improvements.

A5: As shown in the following table, compared to TMDM, NsDiff achieves SOTA and has smaller memory costs and higher efficiency. This is because NsDiff does not introduce additional hidden variables and only adds a small number of basic operations.

The results are tested on ETTh1, with 100 diffusion steps.

Q6: Some metrics are missing, e.g., MAE.

A6: NsDiff still achieves SOTA on these metrics, see Reviewer ZzLi Q3.

Q1: should Eq. 1. be given as the ODE/SDE integral form?

A1: Thanks for the insight. However, we believe the reviewer may be referring to Eq. 7 instead of Eq. 1: Eq. 1 describes the LSNM and is not a stochastic process, so it cannot be written as SDE; Eq. 7 defines the data perturbation process and can be written as SDE. We give a theoretical discussion as follows. First, the Euler-Maruyama discrete form of NsDiff is

$$\mathbf{Y}(t+\triangle t) = Y(t) -\frac{1}{2}\beta(t)(\mathbf{Y}(t) - f_\phi(\mathbf{X}))\triangle t + \sqrt{\sigma_{Y_0} - \beta(t)(\sigma_{Y_0} -g_\psi(\mathbf{X}))\triangle t} \sqrt{\beta(t)\triangle t}\mathbf{z}(t)$$

The diffusion coefficient is related to the $\triangle t$, which is non-Itô-integrable, making it intractable to define a clean continuous-time reverse SDE for the process. To give more theoretical insight, we resort to the perfect estimator assumption (i.e., $\sigma_{Y_0}=g_\psi(\mathbf{X})$) and give the forward/reverse SDE as follows:

$$d\mathbf{Y} = -\frac{1}{2}\beta(t)(\mathbf{Y} - f_\phi(\mathbf{X}))dt + \sqrt{g_\psi(\mathbf{X})\beta(t)}d\mathbf{w}$$ $$d\mathbf{Y} = [-\frac{1}{2}\beta(t)(\mathbf{Y} - f_\phi(\mathbf{X}))-g_\psi(\mathbf{X})\beta(t)\nabla_\mathbf{Y}\log p_t(\mathbf{Y})]dt + \sqrt{g_\psi(\mathbf{X})\beta(t)}d\mathbf{\bar w}$$

We will include more detail in the latest version.

Q2: improve Figure 2 and the description of Eq. 21.

A2: Thanks for this suggestion, we will revise the paper according to your advice.

Q3: SB problem and include SB-based baselines.

A3: Indeed, the problem can be viewed as a SB problem. Taking CSBI as an example, we can introduce an uncertainty-aware prior for it by replacing its $p_\text{prior}$ with the prior of NsDiff $\mathcal{N}(f_\phi(X),g_\psi(X))$, to combine the advantages of NsDiff. Compared to CSBI, NsDiff can estimate the true variance $\sigma_{Y_0}$ by $\sigma_\theta$ (see App. A.4) to learn uncertainty more accurately (as reflected in the results of Sec. 5.4). We include a SB-based baseline CSBI in Table 3 and 4, see Revewer ZzLi Q3.

Q4: MSE/MAE, computational time.

Q4: NsDiff still achieves SOTA on MSE/MAE. Furthermore, compared to previous SOTA, NsDiff has smaller memory costs and higher efficiency. This is because NsDiff does not introduce additional hidden variables and only adds a small number of basic operations.

Q5: Figure 5, the TMDM model seems to perform better, why?

A5: In Figure 5, TMDM actually performs worse than NsDiff. Of course, the difference is less significant compared to Traffic as ETT datasets have relatively small uncertainty variation as in Table 2 (e.g. 1.2 ETTm1, 1.3 ETTm2). Specifically, Figure 5 shows TMDM produces a less accurate mean prediction, leading to larger MAE and MSE than Nsdiff; and, TMDM's predictions do not sufficiently cover the true values, resulting in poorer CRPS and QICE scores.

Q6: Is it possible to apply the diffusion model solvers during inference stage?

A6: Yes, following A1, NsDiff has the following ODE form:

$$d\mathbf{Y} = [-\frac{1}{2}\beta(t)(\mathbf{Y} - f_\phi(\mathbf{X}))- \frac{1}{2}g_\psi(\mathbf{X})\beta(t)\nabla_\mathbf{Y}\log p_t(\mathbf{Y})]dt$$

where the ODE follows a semi-linear structure, remaining compatible with DPM-Solver. However, since time series tasks typically require only a few steps (less than 100) for effective performance, the inference efficiency is already sufficient. Therefore, we do not recommend applying DPM-Solver at the cost of prediction accuracy, given an additional assumption in A1.

Q7: Bridge-based Models [1-3] and noise editing [4] should be discussed.

A7: Thanks for these references. we have discussed SB-based methods in Q3. The noising editing method is interesting and relevant. While NsDiff gives a distribution-wise improvement on the initial noise, the given paper consider a sample-wise perspective to optimize the initial noise to a more reasonable space. We will provide more discussion in the latest version.

Q8: What would happen if the pre-trained model does not predict well (stiffness). Include various random seeds to demonstrate the robustness.

A8: NsDiff incorporates the true variance $\sigma_{Y_0}$ into the learning process to alleviate this stiffness problem. Although the pre-trained model $g_\psi(X)$ may not predict well, NsDiff use Eq. 18 to estimate the true variance. As shown in Table 5 of the ablation experiments (we report results mean and std on various seeds), incorporating this variance estimation improves both the performance and the robustness of our method.

Q9: The convergence of NsDiff has not been discussed.

A9: We acknowledge that the convergence analysis is a critical problem, which is missing in basically all previous works, e.g. TimeGrad, TMDM etc. We will explore this more in future work.

Q1: additional key references [1-2].

A1: Thanks for your recognition of our work. We agree that the references [1–2] provide meaningful context and will help strengthen our discussion. In particular, we find several aspects of these works especially relevant to our setting:

The work by Chen et al. [1] proposes a novel probabilistic forecasting framework based on Föllmer processes and stochastic interpolants. Their approach to learning noise schedules through tailored loss functions resonates with our motivation to adapt diffusion endpoints for better uncertainty modeling, especially under non-stationary conditions.

Jiang et al. [2] address the challenge of long-term forecasting in chaotic systems by preserving invariant measures. Their use of contrastive learning to stabilize dynamics over time without requiring domain-specific priors is an inspiring direction that aligns with our interest in modeling non-stationary behavior robustly.

We will cite and briefly discuss these works in the revised version to better contextualize our contributions within the broader landscape of diffusion-based and non-stationary forecasting techniques.

[1] Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, N. M., & Vanden-Eijnden, E. (2024). Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes. arXiv preprint arXiv:2403.13724.

[2] Jiang, R., Lu, P. Y., Orlova, E., & Willett, R. (2023). Training Neural Operators to Preserve Invariant Measures of Chaotic Attractors. Advances in Neural Information Processing Systems, 36, 27645–27669.