Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

Thank you so much for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.

---

Additional baselines

Based on your and Reviewer BBt2’s suggestion, we evaluated TimeMOE and TTM-R2 models. WaveToken clearly outperforms these models on both benchmarks. See our reply to Reviewer BBt2 for specific details.

Task-specific baselines: We already reported results for TFT and PatchTST, which are known state-of-the-art models. Recent independent work found PatchTST performs comparably to iTransformer on many datasets. Additional task-specific results are also available in Ansari et al. (2024). We thank you for suggesting iTransformer, but due to the limited rebuttal time, we couldn’t finalize experiments on all 42 datasets. We’ll include these results in the final manuscript.

---

For in-domain settings, why WaveToken only performs better than Chronos on VRSE?

Comparisons with Chronos have to be made by looking at model pairs of the same size. Doing so, WaveToken achieves lower (i.e., better) scores than Chronos 75% of the times across all metrics, and achieves the best average rank across all metrics for both Benchmarks I and II (Figures 9 and 10).

WaveToken’s superiority in VRSE specifically arises from its wavelet decomposition, explicitly capturing complex time-frequency structures. VRSE measures forecast-truth similarity in terms of frequency magnitude, aligning naturally with wavelet-based tokenization.

The core motivation of our work is to develop a general purpose tokenizer that seamlessly captures global and local patterns. Qualitative results (Figure 1 and Section 3.3) demonstrate the impressive performance of WaveToken on complex edge cases that are pervasive in practical applications, where existing foundation models fail almost completely.

---

For zero-shot settings, PatchTST performs on par with WaveToken on MASE and TFT performs on par with WaveToken on WQL, but they lag far behind on other metrics. What causes this difference?

This is due to the original design principles of these models. PatchTST was originally a point forecaster. We use the implementation from GluonTS, which adapts PatchTST for probabilistic forecasting. However, it is possible that some design elements of PatchTST are better suited for point predictions, hence its superior performance on the MASE. Similarly, TFT was originally designed to be a probabilistic forecaster trained with quantile regression, which is similar to the WQL up to scaling.

Neither PatchTST nor TFT directly aim at producing forecasts whose frequency content closely matches that of the true time series, which is the target property of the VRSE metric.

---

[...] in zero-shot settings with VRSE as metric, a small size model WaveToken(Small) is much better than a large size model WaveToken(Base), can you provide some discussion on it?

We indeed noticed this pattern as well while conducting experiments. Chronos (Ansari et al. (2024)) exhibits a similar phenomenon. Both Chronos and WaveToken share the same architecture (T5), so the answer might lie in this model. However, we do not currently have a definitive explanation.

---

Can you provide some detailed comparison results about inference time?

The wavelet tokenizer does not influence inference speed and adds only a minimal overhead, since it is a linear-time convolution-and-downsampling operation. Inference times (average of 10 runs, batch of 32 series, context=512, horizon=64) for both Chronos Base and WaveToken Base — which only differ in the tokenizer — are reported below. The difference is negligible, and could be further reduced by applying the DWT on a GPU.

• Chronos: 6.56s +/- 1.02ms
• WaveToken: 6.86s +/- 1.14ms

The main bottleneck remains the autoregressive T5 structure. Future work could explore integrating wavelets with PatchTST or TimesFM architectures so that they can process wavelet coefficients and retain the expressivity of the time-frequency domain, while achieving significant speed improvements.

---

Please provide the visualization results on some benchmark datasets to make the evaluation results in Section 3 more persuasive.

Figure 1 demonstrates WaveToken’s superiority over existing models on controlled practical scenarios (e.g., non-stationarity, sparse spikes). Due to the tight rebuttal deadline, we prioritized additional baseline evaluations requested by Reviewers BBt2 and Yi8n. We'll include further visualizations on a diverse set of real-world datasets in the final manuscript, so as to strengthen the qualitative evaluation of Section 3 as you suggested.

---

Thank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.

Thank you for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.

---

The code is not publicly available, hindering replication.

We plan to release a user-friendly research package complete with details on how to train and evaluate our tokenizer and models. Unfortunately, we could not share our code for review at this stage due to pending legal approvals.

---

The autoregressive nature of the T5 architecture paired with wavelet tokenization may lead to slower inference times compared to non-autoregressive models (e.g., patch-based approaches like TimesFM). This could hinder real-time applications requiring rapid predictions.

We agree that competitive inference times are paramount to achieve an effective utilization of these models in real-time application. We note that in practice the wavelet tokenizer does not influence inference speed and adds only a minimal overhead, since it is implemented as a convolution-and-downsampling operation which runs in linear time. Below we report running times (averaged over 10 repetitions) for both Chronos (Base) and WaveToken (Base) — which use the same underlying architecture and only differ in the tokenization pipeline — when forecasting a batch of 32 time series, with context_length=512 and prediction_length=64.

• Chronos: 6.56s +/- 1.02ms
• WaveToken: 6.86s +/- 1.14ms

The difference in inference times is negligible, and could be further reduced by applying the wavelet decomposition in parallel on the GPU.

Similar to Chronos, what limits inference speed for WaveToken is the autoregressive nature of forecast generation. Crucially, this is the reason why patch-based methods such as TimesFM are faster than WaveToken. It would be interesting to apply ideas from our work to patch-based architectures as part of future work. In this way, one would retain the expressivity of the time-frequency domain while achieving significant speed improvements.

---

The choice of wavelet family (e.g., Biorthogonal-2.2) and thresholding method (e.g., no-thresholding) is task-dependent and may not universally optimize performance across all datasets. Hyperparameter sensitivity could limit reproducibility in diverse real-world scenarios.

As detailed in Section 3.4, we choose hyperparameters by analyzing their probabilistic and point forecasting performance across all the tasks in the in-domain and zero-shot benchmarks, not just on a specific task/dataset. These benchmarks are already quite large (42 datasets in total), hence they ensure that the chosen hyperparameters indeed provide optimal performance for a variety of real-world domains, seasonalities and frequencies. These choices lead to superior generalization performance, as it can be seen from the results on the Zero-Shot benchmark of Figure 3 (panel B), where WaveToken even outperforms (or is competitive with) task-specific models that were trained separately on each single dataset.

---

While WaveToken performs well for H×2 horizons, its WQL performance slightly lags behind TimesFM for H×3 forecasts. This suggests potential limitations in extrapolating patterns over very long horizons, which may require architectural adjustments (e.g., hierarchical attention).

We note that TimesFM seems to outperform WaveToken for $H \times 3$ horizons only with respect to weighted quantile loss, which measures the accuracy of probabilistic forecasts. On the other two metrics, namely mean absolute scaled error and visual relative squared error (VRSE), WaveToken outperforms all other models.

---

Thank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.

Thank you for these useful comments and questions that helped improve our paper. Below we provide point-by-point answers to each section in your review.

---

I would like to know if the proposed method can handle variable-length inputs and outputs with a single model.

The current implementation can readily handle variable-length inputs and outputs. More specifically:

• At training time, any input of size up to 512 and output of size up to 64 can be processed directly. If the inputs or outputs are shorter, we simply pad them with NaNs to ensure consistency of the shapes. In practice, we observed that these maximum lengths cover the vast majority (if not all) of practical applications. Note that other popular foundation models for time series forecasting adopt the same strategy, e.g. Chronos by Ansari et al. (2024) and TimesFM by Das et al. (2024).
• At test time, the model can provide forecasts of unlimited length due to its autoregressive nature.

---

Regarding thresholding, are the filtered coefficients set to zero or completely removed? In other words, does the number of input coefficients remain C? Additionally, how are excessively large coefficients handled?

The filtered coefficients are set to zero. Removing them would lead to variable-length coefficient vectors and a potential mis-representation of different groups of coefficients. In other words, since the model processes concatenated groups of wavelet coefficients, it is important to keep the length of each coefficient group fixed (i.e. no removals) so that the model can learn to forecast (and attend to) approximation coefficients differently from details coefficients.

---

Is there a possibility to allow the model to dynamically select the thresholding method based on the data characteristics?

All the thresholding methods we tested do indeed try to adapt to the data characteristics. CDF-thresholding, for example, directly looks at the empirical distribution of the detail coefficients (in absolute value) to determine the cutoffs. VisuShrink, on the other hand, applies a threshold based on an estimate of the variance of the input time series. In general, one could in principle choose entirely different thresholding methods for different datasets or domains. Although we have not implemented this feature yet, it represents an interesting area for future work. Thank you for raising this point!

---

From both theoretical and experimental perspectives, why is quantization necessary?

and

Ablation on quantization ia required, i.e. directly usinig the raw coefficients without as quantization input and perform continuous autoregressive with MSE loss.

The quantization step is not strictly necessary to apply the ideas presented in our paper. However, the primary focus of this work was to improve tokenization in the context of time series foundation models with discrete vocabularies. Therefore, we need the quantization step to construct a finite vocabulary of tokens that can be directly processed by the T5 architecture without substantial modifications. We focus on models with discrete vocabularies because they present an exciting alternative method to address forecasting problems compared to traditional regression-based approaches and have demonstrated superior performance in prior works.

In principle, one could also sidestep the quantization process by feeding the raw coefficients as embeddings to the transformer, or, as you suggested, by learning to forecast the raw coefficients directly via a continuous-input loss such as MSE/MAE. Both these approaches are perfectly valid ideas. However, they entail completely different models with different analyses and are better suited for independent future works rather than ablations of WaveToken. It would also be interesting to explore ideas based on Wavelets in the context of models that operate on patches (e.g., TimesFM and Moirai). We are hopeful that the community will build upon our work to investigate these ideas.

---

Do outputs share the same quantization bins with inputs?

Yes, the vocabulary is shared between inputs and outputs. Both are mapped to and from the same bins using the procedure described in Section 2.2 - Quantization.

---

Thank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.

Thank you for your review and comments that have helped us improve our manuscript. Below we provide point-by-point answers to each section in your review.

---

why not train Chronos with WaveToken?

Chronos itself is based on the T5 architecture (Ansari et al. (2024)) and we use the same model hyper-parameters (Table 8, Appendix H in our paper). Thus, Chronos effectively serves as an ablation of WaveToken with respect to a different tokenization method (wavelet tokenizer vs simple discretization). Hence, the performance gain of WaveToken can be attributed entirely to the wavelet tokenizer.

---

The experiments do not include some of the best performing models, such as TabPFN-TS, TTM, and MOMENT.

Thank you for these suggestions. We ran preliminary experiments for the pre-trained TTM-R2 and TimeMOE models (also suggested by Reviewer Yi8n). TTM-R2 and TimeMOE are point forecasters, so the WQL metric is not meaningful for them.

Both TTM-R2 and TimeMOE exhibit significantly inferior performance compared to WaveToken. These findings are consistent with the limited zero-shot capabilities of these models noted on benchmarks like GIFT-Eval.

• MOMENT: We faced some challenges setting up experiments correctly with its linear probing setup. We will finalize the experiments for the final manuscript. We note that while MOMENT has performed well in contexts beyond forecasting (e.g., classification and anomaly detection), there isn’t enough evidence in existing benchmarks showing that MOMENT is a state-of-the-art zero-shot forecasting model.

• TabPFN-TS: The paper and codebase was released publicly on Jan 5, 2025. This falls well within the four months time frame specified by the concurrent work guidelines for ICML 2025. Nevertheless, we thank the reviewer for this suggestion and will include the results for TabPFN-TS in the final version of our paper.

---

I find that a few notations are used without clearly defined.

We agree that some of the notation is not clear. Thank you for pointing this out. We will make sure to update the final version of the manuscript with the clarifications below:

• In $[a_k]_J$, the symbol $k=1,\dots,K$ indexes approximation coefficients from the DWT. For an input of size (say) 512, we get about $K=256$ approximation coefficients (depending on the wavelet basis).
• In $F^{−1}_{|d_j|} (b^{J−j+1})$, $F^{-1}$ is the (empirical) inverse-CDF of the absolute values of the detail coefficients $|d_j|, \forall j=1,\dots,J$. The thresholding percentile $b^{J−j+1}$ grows exponentially in the decomposition level so that finer detail coefficients, which tend to capture more noise, are thresholded more aggressively. We set $b=2$
• $z_{1:C}$ denotes the concatenated wavelet coefficients after decomposing the time series context.
• $z_{C+1:C+H}$ denotes the coefficients obtained from the time series horizon at training time, which are used to compute the loss.

---

Can you be more clear about compactness in what sense?

The “compactness” of wavelets manifests itself in two ways:

1. Wavelets concentrate most of the signal energy (input variance) on a few coefficients of high magnitude. Thus, after thresholding the effective number of non-zero coefficients can be much lower than the size of the input
2. Empirically, WaveToken has excellent performance while using a much smaller vocabulary (1024 tokens) relative to Chronos (4096 tokens), which instead directly quantizes the time series.

---

I guess w∈[[ak]J,[dk]j] should be w∈[[ak]J,[dk]j=1J]. Right? Anyway, w is a vector, e.g., [ak]J Then how do you quantize a vector?

Thanks for catching the typo! Yes, $j=1,\dots,J$ is correct. Also, note $w$ is a single coefficient, without distinguishing between approximation and detail. In other words, we quantize all the coefficients element-wise by assigning each to bins based on their empirical distribution.

---

Also, in the quantization, how did you decide the bin size that can be universally applied to diverse data set?

We construct the vocabulary by scanning the training set and choosing the optimal bin size according to Freedman and Diaconis (1981), which selects the bin width minimizing the reconstruction error. Clearly, the universality of this binning scheme relies on having a representative training corpus, the reason why we trained WaveToken on 28 different real-world datasets from different domains.

---

Thanks for pointing out the reference to Yeo and Melnyk (JCP 2019) regarding prior quantization approaches. We’ll add this reference in the final version.

---

Thank you again for your time and engagement in the review process. We hope our responses above have satisfactorily addressed your concerns. If so, we request you to consider raising your score to reflect that. If you have further questions, we would be happy to respond to them.