TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval

Thanks for your constructive feedback. We have addressed your concerns as follows.

Weakness 1: The availability of aligned time series and textual data may limit its deployment in real-world scenarios.

While perfectly aligned multimodal datasets may require additional effort, such data are increasingly accessible in practice recently. For instance, the TimeMMD [2] already provides multi-domain, multimodal time series data across Health, Energy, and Environment sectors. Furthermore, many time series datasets (e.g., weather, finance, traffic) can be enriched with textual context through simple and scalable methods, including web scraping, report parsing, or LLM-based description generation (as we did in our curated NOAA dataset). Therefore, multimodal alignment is practical with modest preprocessing (We refer to Appendix B for our detailed preprocessing steps on weather dataset).

In addition, we note the growing trend in both academia and industry towards integrating language and time series for improved performance in forecasting and reasoning tasks. As the benefits of multimodal modeling become more evident, we expect more such datasets to emerge. TRACE offers a general and extensible framework that not only works with existing aligned datasets but also encourages scalable dataset construction and integration of textual context into time series modeling.

Weakness 2 and Question 1.2: The paper omits comparison with multimodal methods like TimeCMA, which proposes channel-wise retrieval for cross-modal alignment and improved forecasting.

The core objectives, architectures, and applications of TRACE and TimeCMA are fundamentally different. Specifically, TimeCMA's reliance on LLM-generated synthetic prompts for multivariate time series forecasting presents a key limitation: the lack of genuine alignment between different modalities. TimeCMA generates its own textual modality by wrapping numerical data in a prompt. This means the "textual" information isn't independently sourced or inherently meaningful in the same way an actual weather narrative would be. TimeCMA is inherently built for forecasting and classification tasks. Achieving cross-modal retrieval with TimeCMA would require substantial model modifications, making direct comparisons to dedicated retrieval models challenging. We will add a more detailed discussion on TimeCMA as relevant work in our revision.

Weakness 3: The design of channel-biased attention and RoPE for modeling channel dependencies is not easy to read and not clearly motivated in the context of multimodal modeling.

Due to space constraints, we used compact in-line formulations for both the channel-biased attention mask and RoPE integration. We acknowledge this and will provide detailed formulation and motivational context in our revision.

Here, we elaborate on the motivation behind both components. Our goal is to build interpretable and modular representations by encouraging each Channel Identity Token (CIT) to first abstract information from its own channel (via masked attention to intra-channel patch tokens). This ensures that each CIT captures localized semantics independently, which is crucial in multimodal alignment where channel-level descriptions may map to different textual spans. The global CLS token, which is allowed to attend across all tokens, then aggregates these per-channel abstractions to enable inter-channel reasoning. This staged structure balances local disentanglement (for channel grounding) and global integration (for sample-level alignment and retrieval).

RoPE injects relative positional information while preserving sequence length generalization, and we apply it independently within each channel. This ensures that each CIT benefits from phase-aware encoding of local temporal dynamics, which is especially important when channel-level trends vary (e.g., temperature vs. humidity). Unlike absolute or learned position embeddings, RoPE is scale-invariant and better suited for cross-modal alignment across variable-length sequences.

We will better communicate these motivations to aid readability in our revision.

Question 1.1: The assumption regarding highly aligned time series - text pairs, is a bit too strong in real-world time series contexts.

While full alignment is ideal, TRACE is designed to be robust in semi- or weakly-aligned settings, and we already implement several countermeasures:

• Synthetic descriptions from raw TS: In cases where aligned text is unavailable, channel-level descriptions can be automatically generated using LLMs with structured prompts based on raw statistics (e.g., min/max/mean/volatility). This is how we constructed our NOAA dataset (see Appendix B.4).
• Optional text conditioning: TRACE can operate in TS-only mode during inference, or use retrieved text from similar samples, even if not perfectly aligned. This makes it applicable to domains like finance or industry where explicit text pairs are sparse.
• Flexible supervision: Our contrastive training allows text–TS alignment even with partial or noisy matches, encouraging shared semantics without requiring perfect ground-truth pairing.

Question 2: Did the authors consider other moderated time series encoders in Table 1?

Yes, we did explore alternative time series encoders like PatchTST and iTransformer during the pre-training stage, and our architectural design in TRACE is, in fact, informed by their respective strengths. Specifically, given a multivariate time series input of shape $[N, T]$, where $N$ is the number of channels and $T$ is the temporal length, PatchTST applies token-wise attention across time (i.e., $T \times T$), treating each channel independently. In contrast, iTransformer focuses on channel-wise attention ($N \times N$), but ignores temporal locality. Both approaches are partially expressive. Instead, TRACE simultaneously attends over channel identity tokens (CITs) and temporal tokens, which is a practice increasingly standard in time-series foundation models.

This architectural choice is empirically validated. Our early experiments demonstrate that token-only variants (akin to PatchTST) or channel-only variants (akin to iTransformer) struggle to learn discriminative representations for downstream tasks (also shown in Tables 2 and 3). Moreover, dual-level cross-modal alignment consistently fails under these alternatives, as the resulting encoders cannot generate sufficiently distinct channel-level and sample-level embeddings, which are critical for our contrastive alignment framework. We will add these discussions and experimental results to the appendix in our final version.

We thank the reviewer for constructive comments. We hope the above responses clarify your questions. We are more than happy to provide more explanations if needed.

Thanks for your responses, which address some of my questions and concerns, including the motivation of channel-based design, and the choices of time series encoders. I remain positive about this paper and appreciate the provided clarifications.

Thanks for your positive feedback on the idea and experimental results. We have addressed your concerns as follows.

Weakness 1: The proposed method requires curated multimodal datasets, which are not easy to achieve in the time series domain.

While perfectly aligned multimodal datasets may require additional effort, such data are increasingly accessible in practice recently. For instance, the TimeMMD [2] already provides multi-domain, multimodal time series data across Health, Energy, and Environment sectors. Furthermore, many time series datasets (e.g., weather, finance, traffic) can be enriched with textual context through simple and scalable methods, including web scraping, report parsing, or LLM-based description generation (as we did in our curated NOAA dataset). Therefore, multimodal alignment is practical with modest preprocessing (We refer to Appendix B for our detailed preprocessing steps on weather dataset).

In addition, we note the growing trend in both academia and industry towards integrating language and time series for improved performance in forecasting and reasoning tasks. As the benefits of multimodal modeling become more evident, we expect more such datasets to emerge. TRACE offers a general and extensible framework that not only works with existing aligned datasets but also encourages scalable dataset construction and integration of textual context into time series modeling.

Weakness 2: The paper presents a curated Weather dataset but lacks intuitive or quantitative metrics to demonstrate its quality or reliability.

We provide multiple forms of evidence to support the quality of the curated Weather dataset, detailed in Appendix B. For your reference:

• Coverage: The dataset includes diverse weather events (e.g., flash floods, heatwaves, snowstorms) across different regions and time periods.
• Examples: Realistic channel-level textual descriptions are paired with corresponding time series, demonstrating semantic alignment and meteorological plausibility.
• Statistics: With over 74,000 samples across 7 variables, it is among the largest publicly available multimodal weather datasets.

Weakness 3: More baselines should be included in Table 4, especially multimodal time series models.

• For TimeCAP: We have already reached out to the authors to request the official code and are awaiting a response to ensure a fair and reproducible evaluation.
• For Multimodal PatchTST and iTransformer: We have implemented the MM-TSlib of these models and included initial results on the Weather dataset (see below). It is worth noting that models in Table 4 do not take textual context as the input for time series forecasting. Therefore, we also implemented TRACE w. Text by concatenating the textual embedding with the time series for downstream forecasting. Our experiments show that TRACE still outperforms them across multiple forecasting horizons (H=7 and H=24) in multimodal scenarios.

We commit to including comprehensive comparisons with all three models, including TimeCAP, once the official implementation becomes available. These results will be added to the camera-ready version of the paper.

Question 1: Have you evaluated the overall quality of the descriptions? How do you ensure the quality of the generated descriptions?

To ensure both factual accuracy and semantic utility, we applied a controlled generation pipeline with multiple layers of validation:

• Structured Prompting with Constraints: As detailed in Appendix B.4, we designed a strict template that limits generation to factual, numeric summaries per channel (e.g., min, max, volatility, trends), and enforces keyword annotations from a closed set (e.g., {Clear, Rainy, Foggy}). The prompt explicitly avoids speculative or narrative content.

• Grounded Generation Using Raw Data: The full time series values are provided as input to ChatGPT, which ensures that every generated description is directly derived from actual statistics, thus enabling traceability and reproducibility.

• Qualitative Examples: Appendix B.4 provides representative samples. For instance:
  “The temperature ranged from a low of -21.57°C to a high of 26.44°C, with noticeable trends showing a gradual warming as the months progressed from winter to summer, indicating a clear seasonal transition. ” These examples illustrate coherent, granular, and faithful text grounded in underlying trends.

• Empirical Validation: The alignment between text and time series is quantitatively validated in Section 4.2. TRACE achieves high cross-modal retrieval accuracy (e.g., Text-to-TS P@1 of 89.63%), and the UMAP plots in Appendix D.3 show tight clustering between paired text and time series embeddings, hence indirectly confirming consistency.

• Scale and Balance: As shown in Appendix B.5, over 74,000 multivariate samples were processed. Channel-level text was generated for all event-labeled samples and half of the non-event samples to ensure diversity and reduce leakage.

We agree with the reviewer that showcasing a side-by-side comparison with real descriptions would further strengthen the case, and we will include such qualitative comparisons in the camera-ready version.

Question 2: How to obtain soft tokens from time series and how these tokens affect TRACE’s performance?

The process of obtaining the soft tokens from a set of retrieved time series snippets (with their associated textual descriptions) is detailed in Appendix D.5 in the supplementary. After retrieving the top relevant time series–text pair, TRACE encodes both modalities into embeddings: $h_{\text{ts}}^{(i)}$ for the time series (from [CLS] token) and $z_{\text{text}}^{(i)}$ for the text using a frozen SentenceTransformer. These are fused via a trainable MLP over $[h_{\text{ts}}^{(1)};z_{\text{text}}^{(1)};......;h_{\text{ts}}^{(R)}; z_{\text{text}}^{(R)}]$ to form the final soft tokens.

Importantly, this soft token is not part of TRACE itself, but serves as a lightweight augmentation signal for any downstream time series foundation model. Therefore, the performance of TRACE itself is not affected by the learned soft token. The purpose of soft tokens is to demonstrate that TRACE, as a retriever, can produce semantically rich, task-relevant representations to condition external models in retrieval-augmented settings. As shown in Table 2, stronger soft tokens provide richer semantic cues, leading to better predictions, particularly in models like Time-MoE and Timer-XL. This highlights TRACE’s utility as a plug-and-play retriever that delivers highly informative prompts for downstream time series models.

Question 3: How does TRACE prevent information leakage in textual description?

Yes. TRACE implemented specific safeguards during dataset construction and modeling to prevent any information leakage within the textual description as follows.

• Time Series Anchoring: As described in Appendix B.1–B.2, each input time series is truncated to end exactly at the event start time (or the reference time for non-events). Forecasting targets lie strictly in the future, beyond the input window.

• No Use of Post-Event Text in Training: While NOAA event reports may contain post-hoc narrative content, TRACE does not use these raw reports directly in retrieval or training. Instead, the only text used during alignment and RAG stages are:

  • (1) Channel-level descriptions synthesized from the observed input window only, and
  • (2) Sample-level context summaries, similarly generated or aligned to pre-event data.

• Synthetic Description Control: As detailed in Appendix B.4, LLM-generated descriptions are constrained to summarize only the observed (pre-event) portion of the time series. No future values are included in the generation prompt.

This setup ensures strict temporal alignment and rules out any leakage of ground truth labels or future signals into the input space. TRACE’s design adheres to the forecasting causality constraint throughout training and evaluation.

We thank the reviewer for constructive comments. We hope the above responses clarify your questions. We are more than happy to provide more explanations if needed.

Dear reviewer,

Thank you for your thoughtful review! Please take a moment to acknowledge the authors' rebuttal and specify if it addresses your concerns.

Best, AC

Thanks for your constructive feedback. We have addressed your concerns as follows.

W1: The channel-level descriptions may be overly simplistic, potentially limiting realism and their alignment quality.

To ensure the accuracy and usefulness of LLM-generated channel-level descriptions, we applied a carefully controlled and validated generation process. As detailed in Appendix B.4:

• Factual Grounding: Each prompt includes raw time series data, and the model is instructed to summarize statistical properties (e.g., min/max, volatility) only from the observed input. This ensures the descriptions are grounded in actual trends, not speculative narratives.
• Prompt Structure and Constraints: We use a strict prompt format that enforces concise, factual summaries, reducing the chance of hallucination.
• Qualitative and Empirical Validation: TRACE achieves strong retrieval performance (Text-to-TS P@1 = 89.63%) and high cross-modal alignment in embedding space (Appendix D.3), demonstrating that the synthetic texts are semantically informative and align well with the time series.
• Scalability and Diversity: As described in Appendix B.5, we generated over 74,000 such descriptions with balanced event/non-event coverage, confirming both scale and robustness.

W2: The experimental comparisons rely on adapted unimodal models rather than purpose-built cross-modal baselines like TimeRAF.

While TimeRAF is a strong retrieval-augmented forecasting method, it is not a cross-modal model, and the method does not involve text encoders or cross-modal contrastive alignment. In contrast, TRACE is explicitly designed for bidirectional and fine-grained alignment between time series and text, with a dual-level (sample + channel) contrastive objective that supports both text-to-TS and TS-to-text retrieval. These tasks are outside the scope of TimeRAF and thus not directly comparable.

To further strengthen our evaluation, we implemented Multimodal PatchTST and Multimodal iTransformer from MM-TSlib and compared them against TRACE w Text on the Weather dataset:

We appreciate the suggestion and will explore adapting retrieval-augmented forecasters like TimeRAF into TRACE’s multimodal retrieval framework in future work.

W3: The hierarchical hard-negative mining strategy has quadratic complexity for batch size ($\mathcal{O}(B^2)$), raising concerns about its scalability and computational cost.

While the full similarity computation for hard-negative mining does scale as $\mathcal{O}(B^2d)$ (sample-level) and $\mathcal{O}(B^2C^2d)$ (channel-level), we mitigate the practical overhead in several ways:

• Efficient Hard-Negative Filtering: As described in Appendix D.9.1, we use a hierarchical mining strategy that combines lightweight prototype filtering with partial sorting. This reduces the actual mining cost to $\mathcal{O}(B\log R + BC \log R)$, which is negligible compared to full pairwise similarity computation and does not scale quadratically in practice.

• Empirical Efficiency: Table 7 (Appendix D.9.2) shows that TRACE trains faster than larger baselines like FSCA and Moment (6.05s vs. 11.7s for Moment), while activating only 0.12M parameters during fine-tuning. This confirms that our dual-level contrastive training, including hard-negative mining, is not a bottleneck at realistic batch sizes.

• Future Scalability Plans: We plan to integrate scalable approximations for even larger settings, including:

  • Memory-bank-based mining (e.g., MoCo-style gradient caching),
  • Approximate nearest neighbor (ANN) search using tools like FAISS,
  • Stochastic sub-batch or masked-channel mining, reducing sampling complexity to $\mathcal{O}(B\log B)$ or better.

These extensions can be integrated modularly into TRACE's framework without architectural changes.

W4 and Q2: RoPE assumes regular sampling intervals. This may limit its applicability to irregularly sampled or event-driven time series.

It is correct that RoPE, in its standard form, assumes regularly sampled sequences. However, we would like to emphasize:

• Common Assumption in the Literature: This is a shared limitation among most Transformer-based time series models (e.g., TSFormer, FSNet, Timer-XL), which use absolute or relative positional encodings tailored to uniformly sampled data. Our focus on regular time series aligns with the standard evaluation setting adopted in prior works, including those in TimeMMD and weather forecasting.

• Modular Design of TRACE: TRACE is designed with modularity in mind. RoPE is not a fundamental dependency—it was selected due to its superior performance in structured, densely sampled domains (e.g., weather, energy). The architecture can be extended to support irregular or sparse data by swapping in alternative temporal encoding mechanisms.

• Future Extensions: TRACE can readily incorporate time-aware mechanisms such as:

  • Delta-timestamp attention bias (e.g., Informer, RAIM),
  • Learned timestamp embeddings or time-to-vector encoders (e.g., Neural ODE-style Time2Vec),
  • Adapted RoPE with relative phase shifts or interpolation strategies to handle sparse or non-uniform time steps.

We plan to explore these directions in future work to broaden TRACE’s applicability to irregularly sampled or event-driven settings, and appreciate the reviewer’s suggestion to highlight this in the final version.

Q1: How does training/inference memory behave if C >> 1000?

We thank the reviewer for the insightful question on scalability. While our CITs scale linearly with the number of channels, handling extreme-channel scenarios (C >> 1000) is a broader challenge for the entire time series field. Such inputs create ultra-long token sequences that exceed the capacity of current Transformer-based time-series models. We agree this is a crucial area for future research. Fortunately, TRACE's modular design is well-suited for future extensions to address this, such as by compressing channels via clustering or sharding them into manageable groups. We consider this a promising direction for future work.

Q2: Please refer to our response to Weakness 4

Q3: Would incorporating frequency-domain or season–trend decomposition losses help the downstream alignment?

This is an insightful question. Indeed, frequency-domain or season–trend-aware objectives can sometimes promote better modeling of temporal structures. However, we note the following:

In many models (e.g., PatchTST, DLinear), seasonal–trend decomposition (e.g., STL, Fourier transforms) is applied as a preprocessing step rather than as part of the loss function. This modularity allows for flexible architecture choices and avoids hard-coding inductive biases into training.

• TRACE’s masked MSE loss operates directly on raw values and does not assume any specific temporal structure (e.g., periodicity or stationarity). This makes it more broadly applicable across diverse domains and datasets, especially where such assumptions may not hold (e.g., medical signals, financial trends).

• As shown in Table 1 (retrieval) and Table 3 (classification), as well as the UMAP visualization in Appendix D.3, TRACE achieves strong representation quality and alignment performance with the current loss setup. The simplicity and generality of the masked MSE loss proved to be both stable and effective in our experiments.

That said, exploring season–trend-aware objectives as auxiliary tasks or plug-in modules could be a promising extension, and we thank the reviewer for suggesting this direction.

Q4: Would soft gating allow better modeling of causal interdependencies?

We thank the reviewer for this observation. As correctly noted, our current design employs a hard attention mask where each Channel Identity Token (CIT) is restricted to attend only to patch tokens within its own channel. This enforces channel-specific abstraction, which is beneficial for alignment and interpretability.

However, the reverse is not restricted—patch tokens are allowed to attend to all CITs. This design enables implicit cross-channel interactions through shared CITs, allowing the model to propagate interdependencies while maintaining a disentangled structure. We agree that introducing soft gating mechanisms, such as learnable attention masks, cross-channel adapters, or attention dropout, could allow more flexible modeling of causal relationships across channels without fully discarding the benefits of CIT-driven modularity.

We consider this an avenue for future work and thank the reviewer for the constructive suggestion.

Q5: Have you considered using attention-based fusion to better weigh relevance among retrieved items?

We appreciate the reviewer’s suggestion. While more expressive fusion mechanisms (e.g., Set Transformers, gated attention, or cross-attention pooling) may give certain benefits, we chose mean pooling for its simplicity, training stability, and computational efficiency. This choice aligns with our paper’s core focus on retrieval and alignment quality, rather than optimizing the fusion architecture.

Importantly, TRACE is designed to be modular. The aggregated embedding (i.e., soft prompt) is decoupled from the encoder and can be easily replaced with more advanced fusion modules, such as attention-based pooling or set-wise encoders, without retraining the time-series or text encoders. We consider exploring such plug-and-play fusion enhancements a promising direction for future work.

Thanks for your constructive feedback. We have addressed your concerns as follows.

Question 1: What is the optimal ratio between $L$ and $T$ for different types of time series? How does this ratio affect performance, and are there any practitioner guidelines?

The ratio between the flattened token sequence length $L$ and the original temporal length $T$ is controlled by the patch length $P$ (i.e., $L = T/P$). As shown in Figure 4, we conduct a detailed ablation over $P$ and observe that mid-range patch lengths ($P = 6–8$) consistently yield the best performance across tasks. This corresponds to a flattened ratio $L/T ≈ 1/6$ to $1/8$, though the exact value depends on the number of channels and sequence length.

Key observations:

• Over-segmentation ($L$ ≫ $T$ / very small $P$): Leads to overly fine-grained tokenization, exposing the model to high-frequency noise and increasing computational cost. This often results in overfitting and unstable training.

• Under-segmentation ($L$ ≪ $T$ / very large $P$): Causes excessive compression and loss of temporal detail, harming both reconstruction quality and fine-grained alignment.

We recommend tuning $P$ based on the dominant temporal resolution of the dataset (e.g., shorter $P$ for hourly signals, longer $P$ for daily/monthly). Moderate patching offers a good trade-off between local detail and global context, and we found it to be robust across diverse domains (weather, energy, health).

Question 2: In Figure 4, why are the results for Relative Positional Encoding (PE) missing at embedding dimension d = 768? Were they excluded due to poor performance? Were there implementation difficulties?

Thank you for pointing this out. The omission of Relative PE results at d = 768 was due solely to space and figure clarity constraints, not performance concerns or technical limitations. They show slight improvement over Relative PE at $d = 384$ but still underperform Rotary PE, which remains our default due to its consistent superiority across tasks.

We appreciate the reviewer’s attention to detail and will make sure these results are explicitly presented in the final version.

Question 3: How is the [CLS] token initialized and trained to capture global information in TRACE?

We follow standard practice and initialize the [CLS] token using nn.init.normal_, with parameters learned during pretraining. The [CLS] token is fully trainable and optimized end-to-end as part of the masked reconstruction and contrastive objectives.

Importantly, during encoding, the [CLS] token is allowed to attend to the entire flattened sequence, including all channel identity tokens (CITs), which summarize per-channel information, and all patch tokens, which encode localized temporal patterns. This design enables the [CLS] token to act as a global aggregator, capturing context across time and variables. Its resulting embedding is used for sample-level alignment, retrieval, and forecast conditioning.