TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

Table 1: Different transformer backbone for TimeDiT, compared on the zero-shot forecasting task.

W3 TimeDiT's Transformer condition on Diffusion Model

Thank you for raising these important points about conditional diffusion models. We respectfully disagree with the characterization of TimeDiT's conditioning mechanism as overly simplistic. While ControlNet and other approaches offer valuable insights, TimeDiT's design reflects a deliberate balance between architectural complexity and practical effectiveness, as evidenced by our strong empirical results across diverse temporal tasks.

Our method's sophistication lies in its integrated approach: the conditional information works synergistically with adaptive layer normalization (AdaLN) and the Time Series Mask Unit to maintain temporal correlations. The diffusion process naturally handles multi-resolution features through its progressive denoising, while our varied masking strategies (random, block, and stride) enable learning at different temporal scales. This is demonstrated by our superior performance in tasks requiring multi-scale temporal understanding (Tables 1-4), often outperforming more architecturally complex approaches. Rather than adding architectural complexity through multi-level extractors, TimeDiT achieves effective temporal modeling through its unified framework, suggesting that sophisticated temporal patterns can be captured without necessarily increasing model complexity.

W4 Calrify on Physics-Informed Forecasting VS. PDE Solving

The experiments in Table 2 actually demonstrate TimeDiT's ability to predict future values while respecting physical constraints, not just solving classical PDEs. The physics-informed component serves a dual purpose: it guides the denoising process to ensure physically plausible predictions while improving forecasting accuracy. Theorem 4.1 provides the theoretical foundation for this integration, showing how the energy-based prior $K(y;F)$ derived from PDEs combines with the learned distribution p(y|x) during inference. This framework allows TimeDiT to incorporate domain knowledge about the underlying physical system without requiring model retraining.

For example, when predicting future values in systems governed by the Burgers equation or Navier-Stokes equations, TimeDiT uses PDE-derived gradients to ensure predictions respect conservation laws and physical constraints while maintaining temporal coherence. This is particularly valuable in real-world applications where physical consistency is crucial for reliable forecasting. The empirical results show that this physics-informed approach improves both prediction accuracy and physical plausibility compared to baselines that lack such physical constraints. We will clarify these points more explicitly in the revised paper to better communicate how TimeDiT's physics-informed component directly contributes to time series forecasting performance.

W5&Q4 Adidtional Experiment

Table 2: Forecasting results on practical scenarios

Table 7 Predictive Score of TimeDiT compared to baselines on 100% training data

Q6 Multimodal Contribution

Thank you for asking about our multimodal alignment design. As shown in Figure 1, TimeDiT was intentionally architected with multimodal capabilities through the conditional information interface 'c'. Our implementation leverages GPT-2's pre-trained representations to process textual information (both frequency descriptions and category labels), which are then integrated as conditional information alongside temporal features during the diffusion process. This approach allows the model to naturally learn the alignment between textual and temporal modalities during denoising, without requiring complex explicit alignment mechanisms. Figure 2 empirically validates the effectiveness of this design, where TimeDiT+Fre+Cat achieves the best results, while the introduction of other textual information may harm the model. These improvements demonstrate that our model successfully learns to selectively utilize relevant textual information during the denoising process, effectively aligning the two modalities internally through the diffusion framework.

Q7 Details on the Data

Thank you for raising this important point about our unified pre-training methodology. As detailed in Appendix A, our approach addresses the challenge of integrating diverse time series datasets through a standardized pipeline that handles varying channel counts, sampling rates, and data sizes. Specifically, we pre-define maximum dimensions and employ intelligent padding and segmentation: inputs with fewer channels are padded to Kmax, while those exceeding it are segmented into ⌈k/Kmax⌉ blocks. Front padding achieves uniform temporal dimensions, with padding masks tracking valid positions. Our key innovation lies in the mask unit, which generates random, stride, and block masks during pre-training, enabling the model to learn from multi-resolution data without explicit resampling naturally. We randomly sampled from all datasets during pre-training, allowing the transformer architecture to learn relevant patterns across domains through masked self-supervised learning rather than enforcing strict dataset uniformity. This approach maintains domain-specific characteristics while enabling effective unified pre-training, as demonstrated by our strong experimental results across various tasks and datasets. We plan to provide additional details about our dataset unification process in future versions of the paper.

Q8 Ablation Study

Thank you for raising this important point about ablation studies. We agree that comprehensive ablation experiments are crucial for understanding component contributions and have conducted extensive analyses as shown in the provided table.

Table 7: Ablation Study on the Model Design Space.

Thank you for your detailed review of our TimeDiT paper. While completing the suggested experiments took longer than anticipated, we can now address each of your comments comprehensively. Our response comes in two waves: First: A point-by-point address of your concerns in this document; Second: An updated manuscript with clearly marked revisions reflecting these changes

Now we begin with your specific comments:

W1 Mask for forecasting

Thank you for your thoughtful analysis. First of all, we would like to respectfully mention that masking technologies offer robust solutions for forecasting under complex conditions while maintaining theoretical rigor in many works [1,2,3,4]. In addition, we want to clarify that while TimeDiT uses mask-reconstruction as an important mechanism, it's not merely self-supervised pre-training followed by direct sampling. Our framework incorporates task-specific adaptations through the Time Series Mask Unit's different masking strategies (random, block, and stride) that are specifically designed to capture different temporal dependencies relevant to each task. The block masking strategy specifically addresses the forecasting scenario by masking continuous future segments, while the stride masking helps capture periodic patterns and seasonal trends by systematically masking at regular intervals. The combination of different mask types during training enables TimeDiT to learn both local and global dependencies simultaneously.

The empirical results demonstrate that TimeDiT effectively transfers its learned representations across tasks, as shown by our strong performance in forecasting benchmarks (Tables 1, 3), indicating that the model successfully captures the required temporal patterns despite using a mask-based approach. Additionally, the diffusion process adds another layer of flexibility by allowing the incorporation of task-specific priors during sampling, helping bridge any potential gaps between masking and specific task requirements. Our experimental results across diverse tasks validate that this unified approach not only works in theory but delivers state-of-the-art performance in practice.

[1] Che, Zhengping, et al. "Recurrent neural networks for multivariate time series with missing values." Scientific reports 8.1 (2018): 6085.

[2] Liu, Gang, et al. "Self-Supervised Spatiotemporal Masking Strategy-Based Models for Traffic Flow Forecasting." Symmetry 15.11 (2023): 2002.

[3] Liu, Jingwei, et al. "Retrieval-Augmented Diffusion Models for Time Series Forecasting." The Thirty-eighth Annual Conference on Neural Information Processing Systems.

[4] Kollovieh, Marcel, et al. "Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting." Advances in Neural Information Processing Systems 36 (2024).

W2 TimeDiT's Transformer backbone

Thank you for raising these points about TimeDiT's architecture. While we acknowledge the use of a standard transformer backbone, the innovation of TimeDiT lies not in reinventing the transformer architecture itself, but in its novel integration with diffusion models and the Time Series Mask Unit to address temporal challenges. Our diffusion process inherently handles multi-scale temporal patterns by progressively denoising from coarse to fine temporal resolutions, while the combination of different masking strategies (random, block, and stride) explicitly enables learning of multi-period patterns at various scales. This is evidenced by our superior performance across multiple benchmarks, including those specifically testing multi-scale and periodic modeling capabilities (as shown in Tables 1-4).

The strength of our approach lies in maintaining the proven benefits of transformer architectures while introducing novel mechanisms for temporal modeling through diffusion and masked learning. Rather than viewing the transformer backbone as a limitation, we see it as a deliberate design choice that provides a stable foundation for our innovative diffusion-based temporal modeling approach. The empirical results validate this design philosophy, demonstrating that TimeDiT effectively captures complex temporal dependencies despite, or perhaps because of, its architectural choices. In addition, we add a more comprehensive comparison with different transformer designs, which including temporal-wise transformer (current model), channel-wise transformer and dual-attention transformer with both temporal and channel attention in Table 1. This suggests that architectural complexity in the transformer backbone itself may not be necessary when combined with appropriate temporal modeling mechanisms.

Thank you for your detailed and thoughtful response. I have carefully reviewed the authors' rebuttal along with the comments from other reviewers. While some of my concerns have been addressed, a few remain unresolved:

Q9 We have described the gap between masking and prediction in detail in the weaknesses. Therefore, it is difficult for us to endorse the authors' claim of "bridging the potential gap between specific tasks with masking strategies". The authors should address our concerns and specifically point out what novel designs exist in TimeDiT to overcome the following challenges:

1. The mask task instructs the model to capture contextual information from multiple random masks and discontinuous subsequences to reconstruct the mask part.
2. Prediction tasks require models to capture global dependencies and local fluctuation patterns, such as seasonal trends, multi-scale, and multi-period information, from observed sequences that are often corrupted by random masking operations.
3. The reconstruction paradigm allows the use of local up and down information to generate masked parts, while the prediction paradigm only allows the use of observed sequences to predict future sequences.

Q10 From a methodological point of view, there is no innovation in TimeDiT, as we state in Weakness 2. TimeDiT's Transformer backbone still adopts the traditional Encoder-only architecture consisting of Input Embedding Layer, Multi-layer Encoder, Output Projection Layer, which is in line with most of the existing articles (PatchTST, iTransformer, UniTime, Timer, Moment, Moirai) and thus lacks innovation. And the author's response acknowledges this, "we acknowledge the use of a standard transformer backbone, the innovation of TimeDiT lies not in reinventing the transformer architecture itself".

In fact, in the response, the authors still refer to masking strategy and Adaptive Normalization (AdaLN) as TimeDiT's innovations, but these mechanisms have long been included in the self-supervised pre-training paradigm and are not original to TimeDiT.

Q11 Supplementary experiments on PDE Solving. The author still begs the question, does TimeDiT have a clear advantage over the algorithms PDE-Diffusion(ICLR2024) and DiffusionPDE(ICML2024), which are specifically designed for PDE solving?

Q12 Regarding the supplementary experiments in Table 2. The authors do not in widely circulated ETT, Electricity, Traffic, Exchange, a standard data sets and advanced TimeMixer, TimeLLM comparison, but some have never been published in the official papers as benchamrks data sets. Thus, the fairness of the comparative experiments is questionable. Extensive experiments with explicit observation and prediction lengths on the above standard datasets are considered necessary.

Q13 Discussion on introducing multimodal information to enhance TimeDiT. To the best of our knowledge, the authors simply feed textual information into the pre-trained GPT2 model, which is subsequently fed into the noise reduction network as a condition variable. However, in our opinion this is no different from TimeLLM's LLM-based Tokenizer operation, which leads us to insist that TimeDiT is an incremental work rather than innovative.

Q14 About ablation experiments. We feel it is necessary for the authors to respond to the reasons for the complete absence of ablation experiments in the manuscript.

In fact, the author did not address any of my confusion, so I decided to keep my score the same.

Q13 Multimodal TimeDiT

First, while we do use GPT2 for processing textual information, its role is strictly limited to initial text encoding, serving merely as a feature extractor. This is fundamentally different from TimeLLM's approach where the LLM serves as a core tokenizer for the entire model architecture. Our use of pre-trained language models is purposefully minimal and focused solely on text understanding.

Second, and more importantly, our approach to multimodal alignment occurs during the diffusion process itself, representing a novel integration methodology. Unlike TimeLLM's transformer-based alignment, TimeDiT's diffusion-based framework allows for probabilistic alignment between textual and temporal information through the progressive denoising steps. This enables more flexible and robust integration of multimodal information while maintaining uncertainty quantification capabilities - a key differentiator from deterministic transformer-based approaches.

Q14 Ablation Study

We acknowledge the importance of ablation studies and have conducted comprehensive ablation experiments that we added to the manuscript. In the next sections of our rebuttal, we will introduce the modification of the new version.

New Version of TimeDiT

Thank you for further engaging in this conversation. Based on your feedback, we have updated our draft and emphasized the motivation and real-world challenges underlying this work in the general modifications (highlighted in red).

We hope that by re-emphasizing how the choices of masking, transformers, and diffusion models were specifically tailored for real-world scenarios—and reflecting years of thoughtful design inspired by advancements in artificial general intelligence—you will reconsider the value of our work. These choices, coupled with our efforts and results, address a significant and practical challenge, underscoring the importance of our contributions.

The general response includes:

General Modifications (marked in red)

• Abstract Enhancement (lines 15–17):
  Added explicit discussion of how realistic data characteristics challenge the development of general-purpose time series models.

• Current Limitations (lines 64–69):
  Expanded the analysis of why existing approaches struggle to achieve robust foundation model capabilities. This analysis also underscores TimeDiT's methodological contributions in successfully handling multiple tasks and diverse data characteristics, including irregular sampling patterns and multi-resolution inputs.

• Data Understanding (lines 77–78):
  Emphasized the critical importance of understanding underlying data characteristics.

• Design Motivation (lines 84–87):
  Clarified how TimeDiT's architecture is fundamentally guided by real-world application requirements, demonstrating that our component choices directly address specific practical challenges rather than pursuing novelty for its own sake.

These revisions provide a clearer motivation for our work and better contextualize TimeDiT's contributions within the broader landscape of time series modeling.

Specific Responses to Your Comments (marked in yellow)

• For more comparisons:
  We added TimeMixer, TimeLLM, and MG-TSG to Table 1 for the forecasting task. Additionally, we included the results of Timer, TimeMixer, and iTransformer for the imputation task, and also added TimeMixer and iTransformer to the anomaly detection task. Furthermore, we compared the zero-shot performance of the foundation model with TimeMixer, TimeLLM, and Timer in Table 15. These results collectively highlight the strong performance of TimeDiT in handling real-world data. A detailed analysis can be found in Sections D.1.2, D.1.5, D.2.1, and D.4.

• Clarification on physics-informed TimeDiT:
  We updated line 420 to explicitly state that we are using the sampling strategies introduced in the referenced baselines by adding “introduced in.”

• Ablation study:
  We added the ablation study in Section E.1 to outline the design choice space of our TimeDiT clearly.

We hope that the updates we have made to clarify the motivation behind our work have resolved most of your concerns and provided a clearer understanding of our objectives. Additionally, we trust that the enhancements to our experimental analysis have highlighted the significance of our contributions and the practical value of the proposed approach.

Dear Reviewer aczi,

Thank you again for your insightful summary of the fundamental challenges in developing a general-purpose time series foundation model as our strengths. We appreciate your recognition of the key issues we address: handling variable channel sizes, missing values, and multi-resolution characteristics in real-world data, while maintaining compatibility across multiple downstream tasks.

We are encouraged that most reviewers have responded positively and are satisfied with our revisions. As we are working on providing a robust and practical solution for real-world time series challenges, we welcome any specific questions or concerns you may have about TimeDiT to improve your evaluation of our paper further.

Best regards,

Authors

W3 Comparison with Established Benchmarks

You raise a valid point about benchmark comparisons. We explicitly compare TimeDiT with GPT2(3), which represents a prominent "one-fits-all" benchmark in time-series modeling. This comparison is particularly relevant as GPT2(3) similarly aims to provide a general-purpose solution for time-series tasks, even with multiple single models. We wil enhance the presentation of these comparative results in our manuscript to better highlight this important baseline.

W4 Adaptability to Different Channel Numbers and Input Resolutions

TimeDiT employs a flexible architecture designed to handle varying input dimensions through an adaptive processing mechanism detailed in Appendix A (lines 921-930). For channel adaptation, we utilize a predefined maximum channel number Kmax, where inputs with k < Kmax channels are padded accordingly. For cases exceeding Kmax, the input is automatically segmented into ⌈k/Kmax⌉ blocks, each containing Kmax channels that are processed independently. In this paper, we utilize a predefined maximum channel number Kmax = 40, which was chosen based on comprehensive analysis of real-world multivariate time series datasets across different domains. This value represents an efficient balance for many common multivariate time series applications, while maintaining computational efficiency. We further provide the influence of channel number as follows. The results show that while performance is notably worse with only 10 channels, it does not significantly affect the model's results for larger channel numbers.

Table 2: Difference channel number’s influence on the zero-shot performance.

Regarding input resolution, TimeDiT implements front-padding to achieve uniform dimensions up to a maximum length Lmax. This approach preserves positional integrity while efficiently managing high-dimensional data. The maximum resolution limits weren't arbitrarily chosen but were determined based on practical computational constraints and common dataset characteristics across multiple domains. Importantly, our masking mechanism is designed to be resolution-agnostic, allowing the model to adapt to various input resolutions without assuming a fixed resolution structure. This flexible architecture enables TimeDiT to handle diverse time series tasks while maintaining computational efficiency and providing reliable uncertainty estimates through confidence intervals.

We sincerely appreciate the time and effort you invested in reviewing our TimeDiT paper, along with the insightful and constructive suggestions you provided. We are now in the process of responding to your concerns. We will first tackle your feedback points in this email, ensuring that each concern is addressed comprehensively. After that, following your suggestions and our responses, we will upload a revised draft of the paper. This version will include (1) Incorporation of your suggestions. (2)Clearly marked changes for your ease of review.

W1 Methodological Clarity and Missing Technical Details

• Token Definition: As mentioned in #62-65, current tokenization approaches often lack flexibility, either sacrificing information or requiring predefined window lengths based on prior knowledge. Instead of constraining the model with human-discovered patterns, TimeDiT adopts a "What You See Is What You Get" (WYSIWYG) design where tokens are directly represented as contiguous arrays of inputs. Specifically, for a time series with $M$ channels and $T$ time steps, each token encodes $M$-dimensional features at a specific timestamp, resulting in $T$ tokens per sequence. This straightforward approach treats each timestamp's complete feature vector as an atomic unit, allowing the model to discover temporal patterns naturally rather than through pre-imposed structures. We will this explicit definition to Section 4 for clarity.

• Embedding Architecture: The embedder in TimeDiT follows a deliberately simple design - it is implemented as a single linear projection layer that maps each time point $x_i$ from its original M-dimensional feature space to the model's hidden dimension $d$. We intentionally chose this straightforward approach over more complex architectures (like convolutional layers) to maintain model simplicity and computational efficiency while allowing the transformer's self-attention mechanisms to learn temporal relationships. This design choice aligns with our philosophy of letting the model discover patterns through computation rather than encoding them through architectural complexity.

• Noise Embedding Justification: TimeDiT's noise embedding approach plays multiple key roles in the diffusion modeling framework. The diffusion process operates directly in a continuous embedding space, allowing for smoother transitions between noise levels and better preserving the inherent time dependence, thus enabling the model to learn a more robust representation of the underlying time series structure. This approach has several technical advantages[1, 2, 3]: the embedding space provides a continuous representation in which the diffusion process can operate more efficiently. The direct embedding of noisy samples helps prevent the embedding space from collapsing during training. From a practical point of view, this approach allows for parallel processing of multiple time steps, handles varying degrees of noise through a unified framework, and makes the diffusion process more stable compared to traditional generation methods. In addition, the embedded noise representation allows for the seamless incorporation of physical constraints and maintains temporal continuity while progressively denoising, thus contributing to a better quantification of the uncertainty in the generated samples. Direct prediction of the INPUT is also an option available, and we add new experiments as shown in the table below. This also demonstrates the advantages of reconstructive noise.

Table 1: Results on predicting the input of TimeDiT, compared on the zero-shot forecasting task.

W2 Irregular Time-Series Data Claims and Evaluation

Irregular Time-Series Data Claims and Evaluation: Thank you for highlighting this point and providing helpful references. We address irregular time-series scenarios directly in Section 5.1, where our masking mechanism naturally handles non-uniform temporal data. Specifically, the PhysioNet variants (a, b, c) in Table 1 represent real-world irregular time-series data, as detailed in Appendix Table 8. These datasets contain naturally occurring missing values and irregular sampling intervals, demonstrating TimeDiT's practical capability in handling irregular time-series. We will enhance the clarity of this section to better emphasize these aspects and their connection to real-world applications.

Table 4 Anomaly Detection Performance of TimeDit compared to additional baselines evaluated on F1 score

In zero-shot forecasting, TimeDiT significantly outperforms recent models, including TimeMixer, TimeLLM, and Timer, across all datasets, which was originally designed for deterministic time series forecasting. The improvements are substantial: on Solar (0.457 vs 0.999/0.997/1.101), Electricity (0.026 vs 0.302/0.303/0.301), and Traffic (0.185 vs 0.403/0.368/0.384). For imputation tasks, TimeDiT demonstrates superior performance on most datasets, achieving significant improvements over Timer, TimeMixer, and iTransformer, particularly on ETT datasets where we see reductions in MSE by up to 60%. While TimeMixer shows slightly better performance on Weather and Electricity datasets, TimeDiT maintains strong overall performance while offering greater versatility. In anomaly detection, TimeDiT consistently outperforms both TimeMixer and iTransformer across all datasets, with particularly notable improvements on SMAP (95.91 vs 67.63/66.76) and SWAT (97.57 vs 88.84/92.63). These comprehensive comparisons against the latest models demonstrate TimeDiT's effectiveness as a unified framework for time series analysis, often achieving state-of-the-art performance while maintaining broader applicability across diverse tasks.

C2: We appreciate your concern about baseline comparisons on generation tasks and are actively working on incorporating additional recent models into our evaluation framework. Given the time-sensitive nature of the review process, we will try our best to update the new results before the end of the rebuttal stage. Otherwise, we are committed to expanding our comparisons and will incorporate additional baselines in the final version. We would welcome specific suggestions of models you believe would strengthen our comparative analysis. This would help us further strengthen our evaluation framework in the most relevant directions.

Moreover, we would like to clarify several points about our current comparisons. First, Diffusion-TS (2024) represents the current state-of-the-art in time series generation, making it a crucial and contemporary baseline. TimeGAN and TimeVAE were included as representative benchmarks for GAN-based and VAE-based approaches respectively, as they remain influential reference points in the field. Furthermore, Table 12 represents only a subset of our experimental evaluation - our work encompasses a broader range of tasks and comparisons beyond generation, including forecasting, imputation, and anomaly detection, each with comprehensive baseline comparisons.

C3: We are currently updating the manuscript with these revisions but have not yet uploaded the new version. We will submit the updated draft, including the clarifications about irregular time-series handling in Section 5.1, by November 27. This will ensure sufficient time for your further review. We will notify you immediately upon uploading the revised version.

Best,

Authors

Dear Reviewer d3F8,

Thank you for your continued engagement with our work and for highlighting these important points. Your remaining concerns are addressable through our new experimental results, which we are currently finalizing. These additional experiments and clarifications will significantly enhance both the presentation and quality of our manuscript. Now, we have a chance to clarify the concerns right before we upload the new draft:

C1: Thank you for raising concerns about additional baseline comparisons. While we are working on updating the new results in the draft, we would like to paste the new update here to address your concerns.

Table 1: Forecasting results on practical scenarios

For the practical time series forecasting task TimeDiT demonstrates strong performance against the latest models across different paradigms, showing consistent improvements over TimeMixer and TimeLLM across all datasets, with particularly significant gains in MIMIC-III (0.517 vs 0.769/0.787 MAE) and NASDAQ (0.516 vs 3.267/3.125 MAE), along with substantial improvements in uncertainty quantification as shown by better CRPS scores. When compared with the diffusion-based MG-TSD, TimeDiT achieves comparable or better performance on applicable datasets (Air Quality and NASDAQ), while importantly, TimeDiT can handle irregular sampling with heterogeneous input (MIMIC-III and PhysioNet datasets) which MG-TSD cannot process. These results demonstrate that TimeDiT not only competes with the latest specialized models but often outperforms them, while offering greater flexibility and broader applicability, validating our unified approach to time series modeling.

In addition, we have actually conducted more comprehensive comparisons with the latest state-of-the-art models, and the results strongly validate TimeDiT's effectiveness:

Table 2: Forecasting results on zero-shot setting

Table 3: Imputation Performance of TimeDiT compared to additional baselines evaluated on MSE and MAE

Thank you for your detailed and thoughtful response. I have carefully reviewed the authors' rebuttal along with the comments from other reviewers. While some of my concerns have been addressed, a few remain unresolved:

• Comparison with Established Benchmarks: The authors have not provided additional comparisons. For instance, it remains unclear how their model performs on forecasting tasks using well-established benchmarks.

• Time-Series Generation: The comparisons presented focus primarily on older methods (see Table 12 and Figures 7–10), limiting insight into the model's performance against more recent advancements.

• Certain revisions promised in the response have not been incorporated into the main paper. Specifically, the clarification regarding the handling of irregular time-series in Section 5.1 is still missing.

Dear Reviewer d2F8,

Thank you for acknowledging our feedback and for your positive evaluation of our work. We have continuously strived to enhance the manuscript based on your insightful suggestions. These modifications strengthen both the technical presentation and the comprehensive discussion of our work's capabilities and limitations. Before addressing your specific comments (marked in purple), let us outline the general improvements made in the new version of TimeDiT (in red):

General Modifications (marked in red)

• Abstract Enhancement (lines 15–17):
  Added explicit discussion of how realistic data characteristics challenge the development of general-purpose time series models.

• Current Limitations (lines 64–69):
  Expanded the analysis of why existing approaches struggle to achieve robust foundation model capabilities. This analysis also underscores TimeDiT's methodological contributions in successfully handling multiple tasks and diverse data characteristics, including irregular sampling patterns and multi-resolution inputs.

• Data Understanding (lines 77–78):
  Emphasized the critical importance of understanding underlying data characteristics.

• Design Motivation (lines 84–87):
  Clarified how TimeDiT's architecture is fundamentally guided by real-world application requirements, demonstrating that our component choices directly address specific practical challenges rather than pursuing novelty for its own sake.

These revisions provide a clearer motivation for our work and better contextualize TimeDiT's contributions within the broader landscape of time series modeling.

Specific Responses to Your Comments (marked in purple)

We have addressed your feedback as follows:

• For W1: Regarding the Token Definition and Embedding Architecture, we found your feedback valuable. We have directly incorporated explanations into the main paper by modifying lines 224–226. For the noise embedding justification, we added Section E.4 to clarify the design choice.
• For W2: We have addressed this point directly in Section 5.1 of the main paper, providing explanations tailored to each dataset’s unique characteristics.
• For W3: We greatly appreciate your additional comments (C1), which helped us better understand your needs. In response, we have made the following updates:
  -- Added TimeMixer, TimeLLM, and MG-TSG to Table 1 for the forecasting task.
  -- Included results for Timer, TimeMixer, and iTransformer in the imputation task.
  -- Added results for TimeMixer and iTransformer in the anomaly detection task.
  -- Additionally, we compared the zero-shot performance of the foundation model with TimeMixer, TimeLLM, and Timer in Table 15. Collectively, these results demonstrate the strong performance of TimeDiT in handling real-world data. Detailed analyses can be found in Sections D.1.2, D.1.5, D.2.1, and D.4.
• For W4: To address your concerns regarding adaptability to different channel numbers, we have added Section E.5, which provides a clear and focused study on this topic.

We sincerely hope that our previous response offers a clearer perspective on our work and further addresses your concerns thoroughly. Please feel free to let us know if there is anything you want to discuss.

Best,

Authors

Dear Reviewer d2F8,

Thank you for your positive evaluation of TimeDiT again! We have expanded our experimental validation with advanced baselines to address your remaining concerns about comparative analysis. Our efforts in those experiments are not only to address your suggestions but to provide insights and practical value to address real-world challenges, establishing TimeDiT as a robust, general-purpose time series model. Your insights have been invaluable in strengthening our work, and we welcome any further suggestions for enhancement.

Best,

Authors

Q12 Incorporation of Physical Constraints

The diffusion schedule plays a crucial role in how physical constraints are incorporated into TimeDiT's generation process. The forward process gradually adds noise according to variance schedule $βt$, while the reverse process must balance physical constraints with denoising at each timestep. When $βt$ is too large, the noise overwhelms the physical structure making it harder to recover during denoising. Conversely, when $βt$ is too small, the model may struggle to explore the full solution space constrained by physical laws.

We find empirically that a cosine schedule works well for most physical systems as it provides a smooth transition between preservation of physical constraints and exploration of the solution space. This is evidenced by our results in Table 2, where TimeDiT achieves lower MSE and RMSE compared to baselines across different PDEs. For example, in the Burgers equation case, TimeDiT achieves MSE of 0.011 compared to DDPM's 0.016, suggesting better preservation of physical dynamics. The schedule also affects how the physics-informed energy function $K(x^{tar}; F)$ guides the sampling process. During early diffusion steps when noise is high, the physical constraints provide broader guidance. As noise decreases in later steps, the constraints become more precise in shaping the final output. This progressive refinement helps explain TimeDiT's superior performance in maintaining physical consistency while generating diverse samples. These observations point to the importance of carefully tuning the diffusion schedule when applying TimeDiT to new physical systems. Future work could explore adaptive schedules that automatically adjust based on the complexity of the physical constraints.

Q13 Method Exploration

Several key stabilization techniques were integrated into TimeDiT's training process to manage the complex interaction between transformer and diffusion components. The Time Series Mask Unit provides structured guidance through random, block, and stride masks, creating a curriculum of increasingly challenging reconstruction tasks, while adaptive layer normalization (AdaLN, Equation 5) stabilizes training by controlling feature representation scaling using conditional information. The training proceeds in two phases: task-agnostic pre-training with varied masking strategies followed by task-specific fine-tuning. Standard transformer training techniques like learning rate warmup and gradient clipping are employed alongside the weighted MSE loss (Equation 4) for the diffusion process, which helps balance the contribution of different noise levels. The effectiveness of these stabilization methods is demonstrated by TimeDiT's consistent performance across multiple tasks and datasets.

Q14 Trade-off between Sample Quality and Steps

Based on Table 14 and the analysis in Appendix E.1, we carefully examined the trade-off between sample quality and computational efficiency across different sampling steps (50-500). The results show dataset-dependent optimal sampling steps - for example, the Solar dataset achieves best CRPS at 450 steps and best CRPS_sum at 350 steps, while the Traffic dataset shows different behavior with optimal CRPS at 500 steps but best CRPS_sum at just 50 steps. Importantly, the performance variations across different sampling steps are relatively small (e.g., Traffic CRPS ranges from 0.425 to 0.327), suggesting that TimeDiT maintains robust performance even with fewer sampling steps. Given these findings, we typically recommend using 200-300 sampling steps as a practical default that balances computational efficiency with sample quality. This provides nearly optimal performance while significantly reducing computation compared to using 500 steps, especially since the marginal improvements beyond 300 steps are minimal for most applications. The stability of performance across different sampling steps also indicates that TimeDiT is robust to this hyperparameter choice within a reasonable range.

Q9 Core Component

Our performance improvements stem from three key architectural innovations working in concert. The Time Series Mask Unit serves as a foundational component by providing flexible masking strategies (random, block, and stride masks) that enable the model to learn robust temporal dependencies by reconstructing strategically masked segments. This masking mechanism forces the model to understand both local and long-range temporal relationships during training. The transformer backbone then leverages these masked inputs through its self-attention mechanism, which is particularly effective at capturing complex temporal dependencies across different timescales while preserving the sequential nature of the data. Finally, the diffusion process acts as a powerful regularizer that gradually denoises the entire sequence as a whole, maintaining temporal coherence throughout the generation process. The combination of these three elements - mask unit guiding the learning process, transformer capturing temporal dependencies, and diffusion ensuring coherent generation - creates a synergistic effect that enhances the learned representations.

Q10 Underperforming Case Analysis

Thanks for your comments. We have a deeper understanding of the limitations, and will include it in our revised version. Specifically, TimeDiT may face challenges with high-dimensional, rapidly changing system metrics where temporal patterns are less structured and more chaotic. As shown by SMD dataset for anomaly detection (Table 5) where it achieves 83.28% F1 score versus GPT2's 86.89%. This dataset represents cloud server machine metrics with high-frequency sampling and complex feature interdependencies. Additionally, when dealing with extremely short-term patterns or highly localized anomalies, specialized architectures like GPT2 that focus intensively on recent temporal context may outperform TimeDiT's more holistic approach, as our diffusion-based generation process may occasionally smooth over abrupt local changes. These limitations point to potential areas for future improvement, particularly in handling high-dimensional, rapidly changing systems and very localized temporal patterns.

Q11 Intuition on Theorem 4.1

Please also refer to the previous answer in #Q3. Theorem 4.1 provides an elegant mathematical framework for balancing physical constraints with learned distributions in TimeDiT through an optimal Boltzmann distribution based on an energy function that combines two key components: K(x^tar; F), which measures adherence to physical laws through PDE residuals, and α log p(x^tar | x^con), which represents the model's learned patterns from training data. This theoretical foundation has important practical implications: it enables TimeDiT to generate samples that simultaneously respect both physical laws and learned patterns without retraining, allows control of the trade-off between physical consistency and learned behaviors through the α parameter, and can be implemented through an iterative sampling procedure that gradually refines predictions to minimize the energy function. The effectiveness of this approach is demonstrated in our experimental results (Table 2), where TimeDiT successfully generates physically plausible sequences while maintaining the benefits of data-driven learning, significantly outperforming baselines that lack this physics-informed sampling capability.

Q6 Handling missing values

Our experiments leverage naturally occurring missing values from the datasets themselves, primarily arising from irregular sampling rates and multi-resolution data collection processes. Rather than artificially generating missing patterns, we work with real-world missing data scenarios inherent in irregular time series and multi-resolution sampling. This approach ensures our model's robustness is validated against authentic missing data patterns. To connect your concerns with our paper, the model employs various mask types including random masks (MCAR) generated using uniform distributions, block and stride masks (MAR) for capturing structured missing patterns and dependencies between non-contiguous observations, and reconstruction masks with physics-informed sampling (MNAR) for cases where missing patterns relate to unobserved variables. These masks are applied simultaneously through self-supervised learning, allowing the model to learn robust representations without requiring explicit knowledge of the missing mechanisms. For fully address your problem, our ablation study demonstrates the importance of each masking strategy. The results show that removing any mask type degrades performance, with the future mask having the most significant impact. This validates our comprehensive masking strategy for handling diverse missing data scenarios.

Table 3: Mask mechanisms for TimeDiT, compared on the zero-shot forecasting task.

Q7 Anomaly Detection Threshold

Yes, we have experimented with other thresholds. In general, the 99th and 99.5th thresholds achieve better performance and we see a performance degradation as the threshold lowers. However, we did not find it fair to choose the threshold value based on the testing performance and retained the same experiment setting and threshold with previous methods.

Table 4: Threshold Sensitivity Analysis on Anomaly Detection Performance evaluated on F1 score

Q8 Temporally Coherent

We would emphasize that we ensure temporal coherence of generated samples through multiple complementary mechanisms. Our transformer-based architecture explicitly models temporal dependencies through self-attention mechanisms that capture both local and long-range temporal relationships, while the diffusion process maintains temporal consistency by gradually denoising the entire sequence as a whole, rather than generating each timestep independently. During training, we use a comprehensive masking strategy that forces the model to learn temporal dependencies by reconstructing masked segments while considering surrounding context, and the model is trained on both short and long sequences to capture temporal patterns at different scales. We validate temporal coherence both quantitatively and qualitatively - the predictive score metric specifically evaluates temporal consistency by measuring how well future timesteps can be predicted from generated sequences, our PCA visualizations (Figures 7-10 in the paper) demonstrate that TimeDiT-generated samples maintain similar temporal trajectories to the real data, and our superior performance on the predictive score metric (Table 6) indicates that TimeDiT generates more temporally coherent samples compared to baselines like TimeGAN and TimeVAE. Additionally, for domains governed by physical laws, our physics-informed sampling procedure (Section 4.3) provides an additional mechanism to ensure generated samples follow valid temporal dynamics by incorporating domain knowledge during the generation process. These combined approaches help ensure that TimeDiT generates samples that are not just statistically similar to real data, but also maintain appropriate temporal relationships and dynamics, as demonstrated by our empirical results across multiple datasets.

Q2 Handling Different Channel Dimensions

TimeDiT handles varying channel dimensions through an efficient adaptive processing mechanism. For inputs with fewer channels than Kmax (40), direct padding to Kmax ensures uniform processing while invalid channels are masked during sampling to prevent false information propagation. For inputs exceeding Kmax, automatic segmentation into ⌈k/Kmax⌉ blocks occurs, where each block contains Kmax channels, enabling independent processing while maintaining data integrity. For example, with an input of 75 channels and Kmax = 40, TimeDiT processes it in ⌈75/40⌉ = 2 blocks: Block 1 processes channels 1-40 directly, while Block 2 handles channels 41-75 with padding to 40 channels (35 actual + 5 padded). During sampling, the 5 padded channels in Block 2 are masked to prevent false information, and results from both blocks are integrated to reconstruct the full 75-channel output. This segmentation strategy ensures efficient processing while maintaining the integrity of the original data structure.

Q3 Physics distribution VS. Learned distribution

The tension between physical constraints and learned distributions in TimeDiT is managed through a sophisticated energy-based optimization framework that combines two key components: the physics knowledge represented by function $K(x^{tar}; F)$, which measures PDE residuals for physical law conformity, and the learned probabilistic distribution $p(x^{tar} | x^{con})$ from the diffusion model. This balance is achieved through an energy function $E(x^{tar}; x^{con}) = K(x^{tar}; F) + α log p(x^{tar} | x^{con})$, where the parameter $α$ controls the trade-off between physical consistency and distribution fidelity. Rather than directly modifying model parameters, TimeDiT implements this balance through an iterative sampling procedure that starts with samples from the learned distribution and gradually refines them using physical gradients while maintaining probabilistic characteristics. This approach allows the model to generate samples that respect both the learned patterns in the data and the underlying physical laws without significantly compromising either aspect, ultimately resolving the tension through a theoretically-grounded Boltzmann distribution as the optimal solution.

Q4 Condition scheme for TimeDiT

AdaLN's superior performance stems from its ability to dynamically adjust feature distributions across different layers while maintaining computational efficiency. This approach aligns well with the inherent nature of time series data, where temporal dependencies typically exhibit gradual rather than dramatic changes in both seen and unseen time steps. We conducted comparative experiments to evaluate different conditioning mechanisms in TimeDiT: (1) Additive Conditioning, which adds conditional information directly to the diffusion input; (2) Cross-attention, which uses conditional time series as keys/values and noisy time series as queries to fuse conditional information; and (3) Token Concatenation, which concatenates conditional time series with noisy time series at the input level before TimeDiT processing. The experimental results across Solar, Electricity, and Traffic datasets consistently show that AdaLN achieves superior performance compared to the next best alternative. This significant performance gap validates our choice of AdaLN as TimeDiT's primary conditioning mechanism.

Table 2: Condition scheme for TimeDiT, compared on the zero-shot forecasting task.

Q5 Architectire Design

Our architectural choices reflect a deliberate balance between model effectiveness and efficiency, as demonstrated in Table 15, the performance with different model size. While larger models achieve marginally better performance, the increased computational cost is substantial. Our medium configuration (127M parameters) strikes an optimal trade-off, though users can opt for smaller or larger variants depending on their specific accuracy and computational requirements.

We are profoundly grateful for the meticulous attention and insightful feedback you've provided on our TimeDiT paper. Your constructive suggestions have been invaluable, and although the completion of the recommended experiments took us longer than initially planned, this extended period has allowed us to engage deeply with your critiques.

Here is our plan moving forward:

Detailed Response: We will now address each of your comments in detail, ensuring that your feedback is met with thoughtful consideration.

Updated Manuscript: Subsequent to our responses, we will submit a revised manuscript that integrates your suggestions and highlights changes for an easy review.

We eagerly anticipate your feedback on these updates and are dedicated to improving our research through this collaborative dialogue.

W1 Computational Costs

We address computational costs and training efficiency in Appendix A (lines 959-971). TimeDiT achieves computational efficiency through scalable model variants (33M to 680M parameters) and optimized memory usage via adaptive input processing (Kmax=40). Training is completed on NVIDIA A100 (40GB) GPUs, and our zero-shot capability eliminates task-specific retraining needs. The parallel processing and efficient inference enable strong performance across multiple tasks using a single model, substantially reducing computational overhead compared to traditional multi-model approaches. In order to fully address your comments, for additional clarity, we have included inference time comparisons for single-sample generation:

Table 1: Comparison of inference times for single-sample generation.

W2 Model Scalability

TimeDiT's adaptive input processing naturally addresses sequence length scalability through front-padding and segmentation strategies, as detailed in Appendix A. For very long sequences, our model's memory complexity scales linearly via temporal dimension segmentation, automatically breaking inputs into manageable segments while preserving temporal dependencies. Moreover, our pretrained model has already captured fundamental time series patterns, making extremely long historical contexts often unnecessary. This foundation enables effective modeling with moderate sequence lengths while maintaining strong performance across various tasks.

W3&W4 Discussion on Limitations

We have discussed the current limitations in lines #533-539, including scalability for very long sequences, handling highly variable multivariate time series, understanding domain information contributions, and potential extensions to multi-modal time series analysis. Thank you for suggesting the inclusion of failure cases and physics knowledge integration. We provide a more comprehensive analysis of specific failure scenarios and have identified three key scenarios where TimeDiT's performance degrades:

1. Highly irregular sampling rates that deviate significantly from the training distribution
2. Complex, non-stationary patterns underrepresented in the pretraining data
3. Domain-specific patterns requiring expert knowledge beyond general time series characteristics

These limitations stem primarily from the model's dependence on learned foundational patterns. They become particularly relevant in specialized industrial applications and unique financial scenarios. Understanding these boundaries is crucial for informed model deployment decisions and highlights promising directions for future research.

Q1 Mask Ratios

The model's robustness to mask ratios stems from our dynamic masking strategy during pretraining. Instead of using fixed mask ratios, we randomly vary the mask ratio for each training instance. This approach, while increasing training complexity, better reflects real-world scenarios where missing or predicted segments can occur at varying scales. Our experiments show that this adaptive strategy enables TimeDiT to handle diverse prediction tasks without sensitivity to specific mask ratios, enhancing its generalization capabilities as a foundation model.

Dear Reviewer p3cf,

We sincerely appreciate your recognition of our work and your encouraging observation that we have demonstrated strong technical merit and a comprehensive understanding of the strengths and limitations of our study. With the updated version of TimeDiT has now been uploaded; we are excited to highlight further the improvements we’ve made based on your valuable suggestions. To ensure our response is clear and well-structured, we will first outline the general modifications (which relate to Q9) and then address your specific comments (marked in green).

General Modifications (marked in red)

• Abstract Enhancement (lines 15–17):
  Added explicit discussion of how realistic data characteristics challenge the development of general-purpose time series models.

• Current Limitations (lines 64–69):
  Expanded the analysis of why existing approaches struggle to achieve robust foundation model capabilities. This analysis also underscores TimeDiT's methodological contributions in successfully handling multiple tasks and diverse data characteristics, including irregular sampling patterns and multi-resolution inputs.

• Data Understanding (lines 77–78):
  Emphasized the critical importance of understanding underlying data characteristics.

• Design Motivation (lines 84–87):
  Clarified how TimeDiT's architecture is fundamentally guided by real-world application requirements, demonstrating that our component choices directly address specific practical challenges rather than pursuing novelty for its own sake.

These revisions provide a clearer motivation for our work and better contextualize TimeDiT's contributions within the broader landscape of time series modeling.

Specific Responses to Your Comments (marked in green)

We have carefully addressed your feedback as follows:

• W1: Added Table 7 comparing inference times for single-sample generation and analysis (lines 1060–1062).
• W2, Q5, Q13: These points were already addressed in Sections A and E.8; no changes were needed.
• Q14: Previously addressed in Section E.6; no changes were required.
• W3, W4, Q10: Added Section E.7, which discusses the limitations of our approach.
• Q1: Provided an explanation in lines 1019–1020.
• Q2: Addressed in lines 1000–1005 with specific examples showcasing different channel dimensions.
• Q4: Added a new Section E.3 along with Table 22, detailing the condition scheme for TimeDiT.
• Q6: Included Section E.2, which discusses how missing values are handled.
• Q7: Added a discussion on anomaly detection in Section D.4, accompanied by Table 19.
• Q8: Clarified in our previous correspondence.
• Q3, Q11, Q12: Addressed in Section C.2 (lines 1399–1432).

We greatly appreciate your thorough review and continued feedback. Please don’t hesitate to reach out if you have any additional suggestions or need further clarification. We truly hope that the updated version of our paper provides a clearer evaluation, allowing you to reconsider and potentially restore your original score (back to 8), which is incredibly important to us.

Best,

Authors

We are delighted to respond to your thoughtful review of our TimeDiT paper! We'll proceed by first addressing your concerns here, followed by submitting a revised manuscript with tracked changes highlighting our responses to your feedback:

W1&Q1 Details and Model Capabilities

We appreciate your thoughtful comments and we are glad to find that we should share the same vision for advancing the foundation model research. First of all, we appreciate your attention to the details of our work. Comprehensive descriptions of our datasets and training procedures can be found in Appendices A and B, as well as in Tables 7 and 8. However, given that TimeDiT addresses a wide range of generalized time series problems, including tasks like forecasting with missing values and time series generation, the accompanying textual explanations can become quite complex. We greatly value your suggestion and have reorganized our tables to follow the format of Table 2 in the referenced survey [1] for enhanced clarity. Additionally, we will clearly indicate in Tables 7 and 8 when data is utilized for pre-training, testing, and fine-tuning. Regarding our data usage strategy, due to limited resources, we streamline our pre-training process by overlapping datasets to maximize reuse without compromising task-specific integrity. Specifically, to maintain comparability with current mainstream foundation models, we employ extensive pre-training data that may include datasets pertinent to imputation tasks. For instance, while our prediction tasks utilize more practically relevant datasets excluding ETT, the ETT dataset itself is reserved exclusively for our final prediction models. In imputation tasks, we ensure that the pre-training datasets do not encompass ETT data. Furthermore, fine-tuning is performed on specific datasets without introducing additional external data. In essence, rather than selecting subsets from the pre-training datasets, we incrementally incorporate training data in a sequential manner to ensure fair and unbiased comparisons across tasks.

Table 1: A comparable analysis of representative general purposes time series models.

General Model Capabilities: TimeDiT qualifies as a general model through several key aspects that demonstrate its versatility and generalizability. First, it handles variable channel sizes and sequence lengths natively through its unified mask mechanism, allowing it to process diverse types of time series data without requiring task-specific architectures. Second, the model supports multiple downstream tasks including forecasting, imputation, anomaly detection, and synthetic data generation within a single framework. Third, TimeDiT incorporates physics-informed sampling through an energy-based approach, allowing it to integrate domain knowledge during inference without requiring model retraining. This combination of flexible architecture, task-agnostic design, and ability to incorporate external knowledge positions TimeDiT as a powerful foundation model capable of addressing diverse time series challenges across various fields. We acknowledge your concerns about the definition of foundation models, will make it more accurate in this paper as a general-purpose approach.

Dear Reviewer v6Mm,

Thank you for your detailed feedback on our TimeDiT paper. We're grateful for your thorough review of our responses regarding model capabilities, architecture design rationale, and physics-informed approach.

We are incorporating these clarifications into our revised draft and would greatly appreciate your feedback on whether we have adequately addressed your concerns. We hope these improvements will further strengthen your evaluation of our work.

Best,

Authors

Dear Reviewer v6Mm,

Thank you for your positive response and your willingness to enhance your score! We believe that addressing these questions will significantly aid readers in understanding our article more comprehensively. We deeply appreciate your thoughtful and insightful comments. Now we have a chance to address your concerns further:

PS: While we acknowledge the length of our explanation, we believe it is necessary to fully convey both the design principles behind TimeDiT and the comprehensive effort required to develop a truly functional time series general model, outperforming multiple downstream tasks.

C1: The decision to exclude specific datasets was made based on careful consideration of practical constraints. Let’s take anomaly detection as an example, this task requires full reconstruction of time series to identify anomalies via reconstruction error, which conflicts with our foundation model's masking strategies that are designed to learn normal temporal patterns. Including anomalous data in pre-training could compromise the model's ability to learn genuine temporal relationships. Furthermore, while the foundation model aims to minimize reconstruction error for normal patterns, including anomaly detection data in pre-training would force the model to reconstruct outliers, creating a fundamental conflict with the task's goal of identifying anomalies through reconstruction differences.

Another crucial consideration is the variability in label definition for anomaly detection tasks. Unlike forecasting tasks where the objective is consistently focused on temporal progression, anomaly labels are highly context-dependent and vary across different scenarios. This variability necessitates fine-tuning to adapt to specific definitions of anomalies in different applications. Our design maintains flexibility for downstream applications by allowing specific anomaly detection scenarios, preventing potential degradation of anomaly detection performance from pre-trained reconstruction capabilities. This pinpoints the useful design of the foundation model for down stream tasks with complex and variable time series.

Based on this discussion, we would like to extend it to Timedit's position as a general-purpose model. When we looked at the literature and asked the search engine or LLM what they considered to be a foundational model for time series, the key words we could find always matched the capabilities TimeDIT demonstrated and the design philosophy behind it. Thus, we would like to add the following clarification in our draft: “While acknowledging that there is always room for advancement, TimeDiT demonstrates the essential attributes expected of a foundation model: a generalized architecture capable of handling diverse input formats, strong multi-task capabilities across different time series challenges, inherent domain adaptability, and consistent performance across varied applications.”

C2: We apologize for any confusion regarding our data incorporation process. To clarify, we do not train on datasets sequentially in any specific order. Rather, our mention of "incremental" incorporation reflects the practical constraints of our research process - as we progressed with our work, we gradually gained access to different datasets and computational resources. While we initially validated our model's performance on certain tasks (like imputation) before moving to others (like forecasting), this was part of our development process rather than a training requirement.

In terms of the actual training implementation, once we specify the datasets for pre-training, our dataloader randomly samples batches from the entire pool of available data. This means that during any training iteration, the model can encounter samples from any of the included datasets, ensuring truly randomized training. The final pre-trained model therefore learns from all datasets simultaneously, not sequentially. We agree with the reviewer that there's no theoretical reason to incorporate datasets in any particular order, and our implementation reflects this through random sampling across all available data.

Would you find it helpful if we clarify this point in the manuscript to prevent any misunderstanding about our training procedure?

Dear Reviewer v6Mm,

Thank you for your thorough efforts and willingness to increase the score! Let us address our revisions systematically: the general modifications to the abstract and introduction are highlighted in red, while your specific suggestions are marked in blue. These comprehensive revisions strengthen our manuscript across several key areas and provide a more robust foundation for presenting TimeDiT's contributions:

General Modifications (marked in red)

• Abstract Enhancement (lines 15–17):
  Added explicit discussion of how realistic data characteristics challenge the development of general-purpose time series models.

• Current Limitations (lines 64–69):
  Expanded the analysis of why existing approaches struggle to achieve robust foundation model capabilities. This analysis also underscores TimeDiT's methodological contributions in successfully handling multiple tasks and diverse data characteristics, including irregular sampling patterns and multi-resolution inputs.

• Data Understanding (lines 77–78):
  Emphasized the critical importance of understanding underlying data characteristics.

• Design Motivation (lines 84–87):
  Clarified how TimeDiT's architecture is fundamentally guided by real-world application requirements, demonstrating that our component choices directly address specific practical challenges rather than pursuing novelty for its own sake.

These revisions provide a clearer motivation for our work and better contextualize TimeDiT's contributions within the broader landscape of time series modeling.

Specific Responses to Your Comments (marked in blue)

We have systematically addressed your feedback through comprehensive revisions, as outlined below:

General Revisions

To address your comments on W1, W2, and C3, we clarified the position of TimeDiT in lines 976–987 and lines 1008–1012. These updates are designed to be read in conjunction with the general modifications in the introduction, providing a cohesive understanding of the highly motivated real-world challenges and the value of our proposed TimeDiT.

Specific Revisions

• W1 & Q1:

  • Added detailed training specifications (lines 1025–1027).
  • Included a comparative analysis in Table 8.
  • Added data usage clarification in Table 9.
  • Introduced a new data usage strategy section (lines 1068–1079) to address C1 and C2.

• W3 & Q2:

  • Added a discussion on limitations (lines 537–539, Section 6).
  • Introduced a new section on physics-informed modeling (Section C.2, lines 1395–1426).
  • Included a comparative analysis in Table 11.

These revisions provide a more comprehensive presentation of TimeDiT's capabilities and methodological choices. We welcome any additional questions or concerns you may have and look forward to your feedback.

Dear v6Mm,

For the overall comments, thank you for your insightful review and for highlighting TimeDiT's key strengths, which include the general-purpose time series analysis, comprehensive experimental validation, and novel incorporation of physics-based knowledge into the diffusion framework!

For the follow-up, thank you for recognizing our response and increasing your score! Your comments have greatly helped us clarify and strengthen our paper's presentation and contribution. Please let us know if you have any further questions, and we will respond promptly!

Best,

Authors

The program chairs reviewed this case. The ICML is a workshop paper and does not count for dual submission. The URL was redacted.

• Transformer attention mechanisms for temporal relationship modeling
• Impact of different mask unit configurations on general performance
• Effectiveness of various conditional generation strategies
• Necessity of patch tokenization for handling complex, inconsistent input structures

These comprehensive experiments consistently demonstrate TimeDiT's superior performance across diverse scenarios, validating our unified approach to time series modeling. The results not only showcase TimeDiT's effectiveness but also provide valuable insights into the design principles behind its success.

[1] Wang, Shiyu, et al. "TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting." The Twelfth International Conference on Learning Representations, 2024.

[2] Jin, Ming, et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." The Twelfth International Conference on Learning Representations, 2024.

[3] Fan, Xinyao, et al. "MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process." The Twelfth International Conference on Learning Representations, 2024.

[4] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023.

[5] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations, 2023.

[6] Kidger, Patrick, et al. "Neural controlled differential equations for irregular time series." Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

[7] Liu, Yong, et al. "Timer: Transformers for time series analysis at scale." Forty-first International Conference on Machine Learning (2024).

[8] Liu, Yong, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting." The Twelfth International Conference on Learning Representations, 2024.

3.2 Summary of the Changes Made to TimeDiT Paper

The modifications to the TimeDiT paper can be organized into two major categories: general structural improvements (in red) and specific technical additions (in different colors corresponding to the reviewer). On the structural side(#all[red]), the paper was enhanced to communicate its real-world motivation and practical significance better. The abstract and introduction were revised to address how realistic data characteristics challenge existing models explicitly, why current approaches struggle with foundation model capabilities, and how TimeDiT's architecture was specifically designed to address practical requirements rather than pursuing novelty for its own sake. These changes create a stronger narrative around TimeDiT's practical value in handling real-world time series challenges.

On the technical side, the paper was substantially expanded with new experimental results and detailed analyses. Key additions include comprehensive comparisons with state-of-the-art models (TimeMixer, TimeLLM, Timer, iTransformer) across multiple tasks(#aczi[yellow], #d2F8[purple]), detailed ablation studies demonstrating the contribution of each component(#p3cf[green], #aczi[yellow] ), expanded sections on physics-informed modeling(#v6Mm[blue]), and new analyses of the model's adaptability to different channel numbers(#p3cf[green]). The paper now includes clearer explanations of implementation details(#v6Mm[blue]), from token definition(#d2F8[purple]) and embedding architecture(#d2F8[purple]) to data usage strategies(#v6Mm[blue]) and handling of missing values(#p3cf[green]), as well as failure scenarios analysis(#p3cf[green]). These technical additions strengthen the paper's empirical validation while making its methodological choices more reproducible and transparent. All changes were carefully tracked and cross-referenced, ensuring consistency throughout the document while maintaining a clear connection between the theoretical foundations and practical implementations.

Once again, we extend our sincere gratitude to all reviewers for their thorough and constructive feedback. The diverse perspectives offered have helped us better articulate TimeDiT's contributions while addressing important methodological questions. Through this collaborative review process, the paper now presents a more robust and well-validated contribution to the field, for which we are deeply appreciative.