SDformer: Similarity-driven Discrete Transformer For Time Series Generation

Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.

Q1. I wonder about the motivation behind this work.

A1. Our motivations include the following three points:

• Inference efficiency. The inference time of previous SOTA models based on DDPMs is lengthy. To reduce this, we use vector quantization to compress the length of time series in latent space, significantly speeding up the second-stage Transformer. Additionally, non-autoregressive masked token modeling further decreases inference time.
• Generation quality. There is room to improve the quality of generated time series. To address this, we use similarity-driven vector quantization to capture correlations within time series, resulting in higher-quality discrete token representations. Additionally, we apply NLP techniques like autoregressive and masked token modeling to better learn time series distributions, effectively enhancing the quality of the generated time series.
• Compatibility with LLMs. In recent years, LLM-related technologies have advanced rapidly. By aligning the format of time series data with natural language data (i.e., learning discrete time series tokens), we can directly leverage these advancements to tackle time series tasks. Our work confirms the feasibility of this approach and sets the stage for future research on integrating time series with LLM technology.

Q2. Can the authors suggest an idea to use their method for other conditional time series generation, such as trend or global minimum?

A2. You are correct that, compared to diffusion frameworks, VQ-based methods face greater challenges in handling conditional information, making this a valuable research direction. One approach could be to replace the [BOS] token in autoregressive token modeling with conditional input information from a conditional encoder. For instance, during training, we could input the trend or global minimum into the encoder and integrate it into the Transformer for joint training.

Q3. In terms of mask selection for masked token modeling, time series models often use special masking, such as geometric masking. Randomly selected masks cannot fully learn the properties of the time series domain.

A3. Thank you for your insight. Geometric masking is indeed more appropriate than random masking for capturing the properties of the time series domain. In time-domain datasets, a single missing point can often be accurately predicted using surrounding values, such as through interpolation, without requiring complex neural network modeling. Thus, such masking instances do not contribute to learning the time series data distribution. However, for SDformer-m, the processed data is a token sequence, where each token represents $r \times d$ data points from the original data space, with $r$ being the temporal downsampling factor and $d$ the number of time variables. In this case, accurate prediction using adjacent tokens via interpolation or other methods is not feasible, necessitating neural network modeling. Additionally, we compare the impact of the two masking strategies on the Context-FID Score, as shown in the table below. The results indicate that random masking is more effective for token sequence modeling.

Q4. In the proposed vector-quantized samples, is there any relationship between the distance of the embedding and the generated data samples, i.e., do small distances in vectors really exhibit similar time series data after decoding?

A4. Yes, typically, the closer the latent vectors are, the more similar their decoded time series data will be. To validate this, we construct positive and negative sample pairs. Positive samples represent data pairs that belong to the same token after quantization (i.e., the latent vectors are similar), while negative samples represent data pairs that do not belong to the same token after quantization (i.e., the latent vectors are dissimilar). Subsequently, we compute the average time series distances of the positive and negative sample pairs after decoding, as shown in the table below. It is evident that the average distance of the positive sample pairs is significantly smaller than that of the negative sample pairs, indicating that the closer the latent vectors are, the more similar their decoded time series data will be.

Q5. Can the authors suggest more ablation study about model size and code embedding size?

A5. Thank you for your suggestion. We will incorporate these ablation studies into the main text. The following two tables show the Context-FID Score of SDformer on the Energy dataset with varying model sizes and code embedding sizes, respectively. It can be observed that as the model size increases, the performance improves significantly. Moreover, a larger code embedding size also contributes to performance enhancement, albeit to a limited extent.

Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.

Q1. How did the authors choose the size of the codebook K.

A1. K is a hyperparameter that requires experimentation to determine the optimal value. Specifically, we choose from {128, 256, 512, 1024}. The table below presents the Discriminative Score of SDformer under different K. Our final choice can be found in Appendix B (Experimental Settings).

Q2. Code collapse is mentioned in the intro and not discussed further.

A2. Code collapse occurs when only a small part of the codebook gets updated during training, which degrades the performance of the VQ-VAE. The table below shows how code collapse happens on the Energy dataset due to the lack of EMA and Code Reset. I'll include it in the introduction.

Q3. Adding more precise descriptions for implementing the random replacement would be helpful for reproduction.

A3. I'll revise the original text as follows: "In this approach, each token is processed individually. A random number is compared to a probability threshold, π. If it meets the threshold, the token is replaced randomly; otherwise, it remains unchanged." Pseudocode is provided below for clarification.

if Uniform(0,I) <= π:
    y_i <- randint(0,K)
else:
    y_i <- y_i

In this pseudocode, Uniform(0,I) denotes a number randomly generated between 0 and 1, randint(0,K) signifies a token index randomly chosen from the codebook, and y_i represents the i-th token in the original sequence.

Q4.1. When zero-shot forecasting, it can be helpful to compare with some baselines.

A4.1. In our task setting, we focus on training for unconditional generation and performing zero-shot forecasting. As a result, it would be unjust to directly compare our approach with specialized forecasting models, which are typically trained exclusively on conditional generation tasks. However, if you have any suggested baselines, we would be more than happy to conduct a comparison with them.

Q4.2. Is the diffusion-TS trained specifically in the conditional case? or is it compared to the same setting where you train for unconditional generation and perform zero-shot forecasting?

A4.2. For the prediction task, our settings are the same, both training for unconditional generation and performing zero-shot forecasting.

Q4.3. The qualitative examples are nice but do not give us a thorough understanding of the model performance in that case.

A4.3. The qualitative examples in Fig. 3 and 9-12 are randomly selected instances from the prediction task, with their corresponding quantitative metrics shown in Fig. 4.

Q5. Why the unconditional results in Fig. 4 (bottom) have value > 0.5?

A5. Because the evaluation metrics for the prediction task and the unconditional generation task differ. To display them on the same bar chart, we have scaled the results accordingly, which is indicated in the caption of Fig. 4.

Q6. For the long-range benchmark, comparing with LS4 would be beneficial.

A6. Regarding LS4, we have attempted to reproduce the results but faced unforeseen environmental issues and time constraints. We are continuing to address these challenges and will update you promptly with any progress.

Q7.1. The authors do not compare the number of parameters for their original results.

A7.1. The parameter counts for our models are detailed in the following table, which we'll update in the main text.

Q7.2. Can you add the results of the reduced size model on Energy and MuJoCo？

A7.2. The table below presents the results of the reduced-size model on the Energy and MuJoCo datasets. It can be observed that the smaller SDformer model still maintains a performance advantage.

Q8. Some time-series generation-related work and comparisons are missing, such as KoVAE and GT-GAN.

A8. The table below shows the results of my reproduction of KoVAE and GT-GAN in the same environment. Some results differ from those in the original paper. We are in contact with the corresponding authors and will provide updates as they come.

Thank you for your response and the additional results. Some of the results you reported for KoVAE and GT-GAN differ significantly from those in the original ICLR and NeurIPS papers. Additionally, the model parameter counts you provided (around 40M and sometimes even 100M) are much larger compared to competitive methods that have approximately 40K parameters.

and performing zero-shot forecasting. As a result, it would be unjust to directly compare our approach with specialized forecasting models, which are typically trained exclusively on conditional generation tasks.

If it does not perform well, what advantages does your model offer for forecasting?

Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.

Q1. The description of the proposed method is not very detailed.

A1. Thank you for your feedback. SDformer is a two-stage method.

• First Stage: In the first stage, we develop a variant of VQ-VAE [1], introducing an alignment method based on maximizing the similarity between discrete and continuous representations, abandoning the distance-minimization alignment approach used in VQ-VAE, as shown in Eq. 5. To further enhance the precision and stability of the training process, we apply Exponential Moving Average (EMA) for codebook updates and reset inactive codes (Code Reset) during training, which is included in line 48-50 of introduction. Also, the specific structures of the encoder and decoder in the first stage are presented in Tables 8 and 9 in Appendix D, and the detailed training process for the first stage is described in Algorithm 1.
• Second Stage: In the second stage, we use the trained encoder and tokenizer in the first stage to obtain the discrete representation of the target time series, as shown on the right side of Figure 1. In this stage, we employ two different strategies: autoregressive (SDformer-ar) and non-autoregressive (SDformer-m) using a Transformer structure to accomplish the task of generating discrete indexes. The Transformer structures for the two different strategies in the second stage are presented in Tables 10 and 11 in Appendix D. The detailed processes for training and inference are described in Algorithms 2-5.

To further address your concern, we would appreciate it if you could specify the aspects of the method description that you found unclear or insufficient. This will allow us to better understand your concerns and enhance the clarity and detail of our method description accordingly.

Q2. The limitations of the method have not been well explained. It should be compared with other methods, rather than just stating that achieving better generation results will lead to an increase in the number of parameters.

A2. Thank you for your suggestion. The table below shows the Context-FID Score of SDformer and Diffusion-TS at varying parameter quantities. While SDformer demonstrates competitive performance with smaller parameters, it’s evident that there is significant potential for improvement as the parameters increase. This suggests that achieving higher performance may require managing increased memory demands. Conversely, using smaller parameters could result in a notable drop in performance.

On the other hand, Diffusion-TS shows a gradual performance improvement initially, but with a large enough parameter quantity, its performance may become distorted. This indicates that one can opt for smaller parameters in Diffusion-TS without a substantial loss in performance. We will incorporate these findings into our paper to better address our limitations. Your insights are highly appreciated and play a crucial role in improving our manuscript.

Q3. What's the exact limitation of the proposed model?

A3. The SDformer has a slightly larger parameters to ensure high accuracy, which results in relatively higher memory requirements. In future work, we will explore how to maintain the model's accuracy while reducing the parameters.

[1]. Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).

Thanks for your constructive reviews and suggestions. In the following, we will answer your question.

Q1. One concern of the current submission is the lack of sample code provided to reviewers for verifying the reported experimental results.

A1. Thank you for your suggestion. We have provided the anonymous link containing the sample code to the AC. This will allow you to directly verify our experimental results.