Investigating Hallucinations of Time Series Foundation Models through Signal Subspace Analysis

Dear Reviewer A1xq,

Thank you for acknowledging the novelty of our problem definition, methodological clarity, and potential impact. We want to address the posed questions to improve our paper's quality and clarity. Please find the below replies to your comments.

W1 The knowledge rules we propose are basic rules that capture fundamental characteristics of general time series (trend, frequency, and patterns). These rules are scale invariant, robust to noise, and generalizable across different domains. Similar TS pattern matching approaches with sliding windows have been adopted in [1]. Experiments on real-world datasets across five different domains show that our rules effectively differentiate TSFM forecast performance (Table 2 and Appendix Table 4).

W2 The thresholds of knowledge rules allow for a reasonable degree of generalization of context signals in TSFM forecasts. As mentioned in $\S5.1$, we tune the thresholds based on validation performance such that TSFM performance differentiation is maximized.

W3 While our approach is intervention-based, the center-project-scale (CPS) operation is not used in existing works to our best knowledge. Our approach adaptively determines the scaling factor. Moreover, few existing works investigate the use of activation variance as the hallucination detector and forecast performance predictor.

W4 As discussed in $\S2$, the work in [38] does not address specific challenges, such as hallucination detection and mitigation and performance prediction in our work.

W5 Thanks for the suggestions. We add hallucination mitigation baselines. We are working on the experiments.

W6 The neuron activity score (Equation 1) stems from our observations that hallucinated forecasts associate with low activation variance and high cosine similarity of hidden states across TS steps (Figure 1), while non-hallucinated forecasts with clean context signal associate with high activation variance and low cosine similarity on average (Figure 2). The intuition and motivation are elaborated in $\S4.1$. Essentially, activation variance reflects the sensitivity of a neuron to the context TS. As such, we propose to leverage the standard deviation of activations across TS steps as a measure of neuron activity. To distinguish neurons that are sensitive to context signal from those sensitive to context noise, we compute the contrastive neuron activity score (Equation 2) between data inputs without causing hallucinations and Gaussian noises. We select the activity score of the signal neuron with top contrastive score from the last hidden layer as the hallucination detector and forecast performance predictor, i.e., SSAS.

W7 While our observations in $\S4.1$ are drawn from synthetic data, experiments on real-world datasets show that cosine similarity and mean activation variance alone have hallucination detection and performance prediction power to some extent (Table 3 and Appendix Table 2), validating the generality of observations. Our approach only utilizes synthetic datasets without requiring any domain-specific real-world data.

W8 Our experiments have been conducted on real-world datasets from five different domains. We add case studies in the revision.

Q1 The rules are invariant to scale as they consider the relative differences. Centering the context TS may have an impact on the results, but we do not normalize the context TS in the experiments. As mentioned in Appendix A, the rolling window size is always the same as the forecast horizon, which is necessary for computing the relative differences between the forecast and context windows.

Q2 We add hallucination mitigation baselines to our evaluations. We are working on the experiments.

Q3 We think SSAS could generalize to classification and anomaly detection tasks. SSAS detects hallucinations by characterizing the abnormal behaviors in hidden states when hallucinated forecasts are generated. Similarly, it could be useful for detecting whether anomalies exist in the context TS.

[1] Shokoohi-Yekta, Mohammad, et al. "Discovery of meaningful rules in time series." Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015.

Dear Reviewer A1xq,

Thank you for raising the score. We sincerely appreciate your insightful review and feedback. We provide further clarification to address your concern. Our work focuses on the hallucinations of TSFMs in zero-shot time series forecasting. Analogous to LLMs processing the query and auxiliary information (such as few-shot examples) from the input context, TSFMs extract signal information from the context and generate extrapolations. A forecast is considered to be hallucinated if its dynamics drastically deviate from those of the context, as this suggests a failure of accurately processing the context information. To check if such discrepancy between the context and forecast occurs, we design knowledge rules to extract and compare basic dynamics of time series covering several perspectives. Additional ablation results on knowledge rules are provided at Note 1 and Note 2. Our work draws parallels with the works on the relationships between hallucinations and context information processing in the LLM domain [2, 3]. We would be happy to answer if you have any further concern.

[2] Chen, Shiqi, et al. "In-context sharpness as alerts: an inner representation perspective for hallucination mitigation." Proceedings of the 41st International Conference on Machine Learning. 2024.

[3] Sun, ZhongXiang, et al. "ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability." The Thirteenth International Conference on Learning Representations. 2025.

Dear Reviewer A1xq,

Thank you for your comment.

The dynamics that our knowledge rules extract from the context time series can be regarded as some form of facts. From this perspective, our rules catch hallucinated forecasts that heavily violate the facts provided by the context.

When using an LLM to solve a mathematical problem, people rely on external tools to verify the correctness of the LLM's output that looks right. The same goes for zero-shot TS forecasting. Since it is hard to judge whether the forecast exhibits different dynamics from the context with human eyes, we apply knowledge rules that involve trend, frequency, pattern, and ARMA analyses to check this. Lowering the rule tolerance thresholds $\epsilon$ would lead to a greater proportion of forecasts that look right being regarded as hallucinations but would sacrifice the degree of generalization in the forecasts.

We believe the issue with context information processing relates to not only model capacity but also model pre-training, so it can be addressed with intervention.

Dear Reviewer Q2NU,

Thank you for acknowledging the novelty of our work and the clarity of our paper. We want to address the posed questions to improve our paper's quality and clarity. Please find the below replies to your comments.

W The knowledge rules we propose are basic rules that capture fundamental characteristics of general time series (trend, frequency, and patterns). These rules are scale invariant, robust to noise, and generalizable across different domains. Experiments on real-world datasets across five different domains show that our rules effectively differentiate TSFM forecast performance (Table 2 and Appendix Table 4). It is possible to further include domain knowledge as rules, but this may affect the generalizability of the knowledge set to other domains.

Q1 Experiments on real-world datasets show that cosine similarity and mean activation variance alone have hallucination detection and performance prediction power to some extent (Table 3 and Appendix Table 2). As discussed in $\S5.2$ RQ3 and Appendix E.1, the phenomena of cosine similarity are applicable to the Chronos family and Chronos-Bolt. The phenomena of mean activation variance are applicable to all three types of models but are less consistent across domains.

Q2 As explained in $\S3$, the ARMA rule is purposed to complement the pattern rule when the TS exhibits strong ARMA dynamics. Since only a small proportion of TS exhibit strong ARMA dynamics, we have not reported the experimental results of the standalone ARMA rule. We choose to use the first-order ARMA model to capture the fundamental characteristics of general time series and avoid overfitting. We are working on separate results for the pattern and ARMA rules.

Q3 Thank you for pointing this out. We fix the typo in the revision.

Dear Reviewer Q2NU,

Thanks for your patience. Below, we report separate results for the pattern rule and ARMA rule to address Q2 in rebuttal. From Table 1, both the pattern and ARMA rules contribute to the differentiation of TSFM forecast quality, with positive differentiation in all cases. The pattern+ARMA rule narrows down the scope of hallucinations by considering forecasts that violate both the pattern and ARMA rules as hallucinations. It captures a smaller set of hallucinations with poor quality and provides superior performance differentiation overall. Please refer to this comment for further ablation results on knowledge rules. Table 2 reports the distributions of rule violations without and with SSIM intervention. Please note that per our definition, a forecast is also labeled as violating the ARMA rule when the context time series does not exhibit strong ARMA dynamics. SSIM primarily reduces the violations of the pattern rule.

---

Table 1

Table 2

Dear Reviewer 43oT,

Thank you for acknowledging the clarity of motivation and the completeness of experimental results. We appreciate your time and efforts.

Dear Reviewer pd2E,

Thank you for acknowledging the importance of our research topic, novelty of our methodology, and potential impact. We want to address the posed questions to improve our paper's quality and clarity. Please find the below replies to your comments.

W1 We have discussed on [38] in the paper. We add discussions on [1,2,3]. If there are additional works, we would appreciate pointing them out.

W2 The knowledge rules we propose are basic rules that capture fundamental characteristics of general time series (trend, frequency, and patterns). These rules are scale invariant, robust to noise, and generalizable across different domains. Similar TS pattern matching approaches with sliding windows have been adopted in [4]. Experiments on real-world datasets across five different domains show that our rules effectively differentiate TSFM forecast performance (Table 2 and Appendix Table 4). The thresholds of rules allow for a reasonable degree of generalization of context signals in TSFM forecasts. As mentioned in $\S5.1$, the thresholds are tuned based on validation performance such that TSFM performance differentiation is maximized.

W3 The neuron activity score (Equation 1) stems from our observations that hallucinated forecasts associate with low activation variance and high cosine similarity of hidden states across TS steps (Figure 1), while non-hallucinated forecasts with clean context signal associate with high activation variance and low cosine similarity on average (Figure 2). The intuition and motivation are elaborated in $\S4.1$. Essentially, activation variance reflects the sensitivity of a neuron to the context TS. As such, we propose to leverage the standard deviation of activations across TS steps as a measure of neuron activity. To distinguish neurons that are sensitive to context signal from those sensitive to context noise, we compute the contrastive neuron activity score (Equation 2) between data inputs without causing hallucinations and Gaussian noises. We select the activity score of the signal neuron with top contrastive score from the last hidden layer as the hallucination detector and forecast performance predictor, i.e., SSAS. Experiments on real-world datasets show that while cosine similarity and mean activation variance alone have hallucination detection and performance prediction power to some extent, SSAS yields superior performance by considering the activity of identified signal neurons (Table 3 and Appendix Table 2).

In SSIM, we scale the activations of candidate signal neurons via our proposed Center-Project-Scale (CPS) operation described in $\S4.3$. The properties of CPS operation are mathematically proved in Appendix B. The algorithm of SSIM is outlined in Appendix C.

W4 a) We add hallucination mitigation baselines. We add an ablation study on how each rule contributes to hallucination detection. We are working on these experiments.

W4 b) SSIM could fail when the number of pattern occurrences in the context is low. It is easier for TSFMs to capture frequent patterns than rare patterns from the context TS.

The comparison of SSIM performance for models of different complexity is presented in Appendix E.1. The computational overhead of SSIM at each layer is $O(nk)$, where $n$ is the number of TS steps and $k$ is the number of signal neurons. The computational cost of perturbation averaging is the number of perturbations times the original computational cost of TSFM inference.

W4 c) Denoising with sliding windows does not involve randomization. We set the window size to 5 as it works the best overall. In Perturbation+Averaging, we perturb the input TS with Gaussian noise for 10 runs with different random seeds and average the outputs.

W5 b) Figure 1 (c) and (d) compare the distributions of last layer hidden states corresponding to the hallucinated forecast in Figure 1 (a) and non-hallucinated forecast in Figure 1 (b). We observe the hidden states in (d) are more evenly distributed within each cluster than those in (c). Figure 1 has been referenced in $\S1$ and $\S4.1$ main text. We note that the subfigures should instead be listed as (a) (b) ... (c) (d) ... in Figure 1 caption.

W5 c) We compute the mean pairwise cosine similarity of hidden states across TS steps, which has also been used as a measure in [5].

W5 d) We assume the activation statistics in Figure 1 (c) (d) are being referred to in the comment. These show the difference of the mean activation standard deviation of all neurons. By identifying the signal neurons, SSAS yields greater gaps between hallucinated and non-hallucinated forecasts with substantially higher hallucination detection and performance prediction accuracies than mean activation standard deviation (Table 3 and Appendix Table 2)

W5 e) Our observations suggest that hallucinated forecasts associate with low activation variance and high cosine similarity of hidden states across TS steps (Figure 1), while non-hallucinated forecasts with clean context signal associate with high activation variance and low cosine similarity on average (Figure 2). TSFM hallucinations are partly caused by the inactivity of signal neurons, leading to more similar hidden states. Experiments on real-world datasets show that cosine similarity and mean activation variance of hidden states have hallucination detection and performance prediction power to some extent (Table 3 and Appendix Table 2). We mathematically show in Appendix Theorem 3 that our CPS intervention operation effectively reduces the cosine similarity of hidden states that have different projection directions in the signal subspace.

W5 f) Through UMAP projection, we aim to show in Figure 1 (c) (d) the difference in the distributional characteristics of last layer hidden states corresponding to the hallucinated forecast in Figure 1 (a) and non-hallucinated forecast in Figure 1 (b) rather than alignment to certain directions. We observe the hidden states in (d) are more evenly distributed within each cluster than those in (c).

[1] Santamaria-Valenzuela, Inmaculada, et al. "Decoding Latent Spaces: Assessing the Interpretability of Time Series Foundation Models for Visual Analytics." arXiv preprint arXiv:2504.20099 (2025).

[2] Li, Yuening, et al. "Towards learning disentangled representations for time series." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

[3] Oublal, Khalid, et al. "Disentangling time series representations via contrastive independence-of-support on l-variational inference." The Twelfth International Conference on Learning Representations. 2024.

[4] Shokoohi-Yekta, Mohammad, et al. "Discovery of meaningful rules in time series." Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015.

[5] Nguyen, Tam, Tan Nguyen, and Richard Baraniuk. "Mitigating over-smoothing in transformers via regularized nonlocal functionals." Advances in Neural Information Processing Systems 36 (2023): 80233-80256.

Dear Reviewer pd2E,

Thanks for your patience. Below, we report the ablation results on the effect of each knowledge rule that constitutes our definition of TS forecast hallucinations to address W4 a) in rebuttal. The distributions of rule violations have been provided in Figure 4. From the table below, we note that all rules contribute to the differentiation of forecast quality, with the aggregated Pearson correlation of non-hallucinations substantially higher than that of hallucinations in all cases. The pattern+ARMA rule effectively differentiates both $R^2$ and correlations in all cases. The frequency rule differentiates correlations but not $R^2$, due to the misalignment between forecasts and ground truths on some outlier test instances. The synergy of the three rules gives the best performance differentiation overall.

Dear Reviewer pd2E,

Thank you for your comment. We provide further clarification to address your concern.

It is unclear where such a large gap in HAL performance comes from.

The additional rule ablation results are consistent with Table 2 in the paper. We find that the huge negative $R^2$ scores come from a few outlier test instances. On domains with large peaks, such as WebOps, Chronos sometimes generates a forecast with the peak misaligned with the ground truth, resulting in a huge negative $R^2$ score. Since Chronos-Bolt and TimesFM adopt patching on the input, they generate smoother forecasts than Chronos, so their results are less affected by extreme $R^2$ scores. You may also refer to the correlation results for clearer performance comparison as Pearson correlation coefficients range within $[-1,1]$ without yielding extreme scores.

The performance difference between HAL and non-HAL variants remains marginal and not clearly justified.

We assume the performance differences in Table 2 in the paper and the Rule Ablation table are being referred to in the comment. In these tables, $R^2$ ranges within $(-\infty,1]$, and Pearson correlation coefficient ranges within $[-1,1]$. The Diff column shows that there is a significant gap of $R^2$ and Pearson correlation between HAL and non-HAL forecasts, as validated by the $\chi^2$ test. The inferior performance of hallucinated forecasts is because the TSFM fails to accurately process the context information and generates hallucinations in the forecast that do not ever exist in the context, such as a peak, slope, or pattern. These hallucinations generally deviate from the ground truths, resulting in low evaluation scores.

The authors’ claim that “it is easier for TSFMs to capture frequent patterns than rare patterns” remains unconvincing.

In this claim, we refer to the number of pattern occurrences within a fixed length of input context. For instance, $sin(2x)$ has a higher frequency than $sin(x)$, and it is easier for TSFMs to capture the patterns of $sin(2x)$. On real-world datasets, we observe that TSFMs generally have better performance when the patterns are frequent and regular, such as the ones from the Energy domain in our experiments.

The metrics section needs significant restructuring.

We apologize for the confusion. We revise the metrics section as follows:

We evaluate forecast quality with $R^2$ and Pearson correlation coefficient, which are scale invariant. $R^2$ measures the goodness of fit to the ground truth; Pearson correlation coefficient measures the strength and direction of the linear relationship with the ground truth (invalid values are imputed with 0). Whether a forecast is hallucinated is determined according to the knowledge rules defined in $\S3$. We evaluate the effect of hallucination mitigation with hallucination rate reduction and forecast quality improvement. We evaluate the accuracy of hallucination detection with AUROC and performance prediction with Spearman rank correlation coefficient.

Would this help?

Dear Reviewer,

Could you be specific which issues we promised to resolve have not been addressed? Thanks.

I am not enumerating all missing revisions because I have not gotten a chance to do it until today (Reviewer pd2E's comment reminded me to do so). Here are some missing revisions involved in the rebuttal to my review:

1. In the response to W8. The authors reply that "We add case studies in the revision". However, I did not find any new case study in the camera-ready version. The only case study that I saw is the Figure 1, which seems not to be a real-world data case and already included in the original version
2. In the response to my W5: "Thanks for the suggestions. We add hallucination mitigation baselines. We are working on the experiments." I did not see the new hallucination mitigation baselines in the camera-ready revision, neither following my suggestion to use some diffusion based model to post-process the output or following some recent hallucination-related works in NLP area. The "full-parameter finetune (FT) TSFMs" in the rebuttal is not included in the paper, neither.
3. I did not see the Note 1 and 2 ablation study on different rule combinations (they put the hyper links to the rebuttal to reviewer pd2E) in the camera-ready version.

Dear Reviewer,

Could you be specific which issues we promised to resolve have not been addressed? Thanks.