Forging Time Series with Language: A Large Language Model Approach to Synthetic Data Generation

Improvement of SDForger

We believe SDForger can be further improved. Its modular design is intentionally built to support flexible experimentation, making it easy to explore enhancements or tailor components to specific needs.

We see several promising directions:

• Embedding Strategies: While our current approach relies on linear methods like FastICA and FPC, future work could explore more expressive, nonlinear embeddings (e.g., AE) or multivariate-aware methods like Multivariate FPCA or Multivariate Singular Spectrum Analysis to better capture temporal and inter-channel dependencies.
• Parameter-Efficient Fine-Tuning: We currently use full fine-tuning for the LLM. Incorporating PEFT techniques such as LoRA or adapters could improve scalability, efficiency, and facilitate domain adaptation.
• Extended Utility Evaluation: While we focus on forecasting, SDForger could be evaluated and optimized for broader downstream tasks such as classification or anomaly detection.
• Context and Covariate Integration: By design, SDForger supports integration of external covariates (e.g., categorical or textual data). Expanding this functionality could enable richer conditional generation, and multimodal transfer learning.

In summary, SDForger offers a flexible foundation, and we see meaningful opportunities to improve it both architecturally and in terms of task generalization.

Non-stationary datasets

SDForger was evaluated on both stationary and non-stationary datasets. Weather, Bikesharing, and Australian Electricity mostly contain stationary series with clear seasonal patterns.

In contrast, the currency dataset is fully non-stationary, reflecting the volatility of financial data. Additionally, 25% of NN5 and 99% of Tourism series were identified as non-stationary using the Augmented Dickey-Fuller (ADF) test. Below are results for three example channels per dataset:

These experiments, coupled with the non-aggregated results in Table D.7 and D.11, confirm that SDForger is capable of handling both stationary and non-stationary time series effectively. However, we appreciate the suggestion and will consider adding a dedicated section in the paper to separate results on stationary and non-stationary datasets!

Temporal dependency

The embedding methods we use, FastICA and FPC, decompose each fixed-length time series segment into a linear combination of basis components. This strategy retains temporal dependencies, as the components span the entire window and reconstruction preserves their temporal structure. The choice of embedding dimension k controls the fidelity of this representation and how much information will be “lost”: lower-dimensional embeddings retain less variance, which can lead to some loss of fine-grained temporal or cross-channel details.

Below, we report the variance retained by FICA embeddings across datasets:

As detailed in Appendix A.2, we also implement a dynamic strategy to select the number of components based on a target variance threshold, offering practical guidance to adapt $k$ to each dataset’s complexity. In practice, when dealing with strongly correlated or complex sequences, using a sufficiently high embedding dimension mitigates information loss and helps maintain SDForger’s performance.

For inter-channel dependencies, we refer to Q3 of Reviewer V1ut, where we assess Frobenius distance between real and generated correlation matrices. This ablation confirms that GPT-2 can effectively learn cross-channel structure from compressed multivariate embeddings.

SDForger readiness

We kindly ask the Reviewer to clarify whether the question refers to multimodal readiness. If so, this is addressed in Section 6, where we demonstrate text-conditioned generation by combining structured embeddings with free-form natural language in the prompt. This example shows that SDForger can integrate textual information during both fine-tuning and inference without requiring any architectural modifications. The input prompts are presented in Section 6 (l. 323) and the generated prompts in this experiment are printed in the submitted conditional_generation.ipynb notebook (cell 11). Due to space constraints, we could not include them in the paper, but we would be happy to share excerpts in a follow-up. This experiment supports our claim of “readiness,” showcasing SDForger’s flexibility for multimodal data. That said, we agree with the Reviewer that more robust and efficient conditioning strategies will be important to fully unlock SDForger’s multimodal potential, and we plan to explore this direction in future work.

Alternative embedding methods

In our current study, we focus on classical linear embedding method, FastICA and FPC, due to their interpretability, computational efficiency, and well-understood properties in preserving variance, statistical independence, and temporal dependency. Alternative nonlinear embedding techniques like Autoencoders (AE) and t-SNE offer powerful representation capabilities and can capture more complex, nonlinear structures in the data. However, they also introduce challenges in terms of:

• Training stability and complexity, especially in low-data regimes,
• Deterministic reconstruction quality, which is critical for reliable time series synthesis
• Interpretability and direct control over embedding dimensions.

While t-SNE excels at visualization, it is not designed for reconstructive embeddings, limiting its direct applicability in our generation pipeline. That said, integrating or comparing these alternative embeddings with our LLM-based generation framework represents promising future work. We plan to explore nonlinear embeddings such as AEs in subsequent studies to assess their impact on generation fidelity and downstream task performance.

Dimensionality that can be handled

SDForger is not fundamentally limited by the raw length of the time series. Long sequences are split into fixed-length windows using periodicity-aware segmentation, and each window is embedded and treated as a separate training instance. Longer sequences thus yield more examples, boosting generalization without increasing prompt size.

The main limiting factor is the embedding dimensionality $K$, which determines the prompt length. As $K$ increases, prompts grow longer, inference slows down, and generation becomes less stable. This holds in multivariate settings: more variables imply higher-dimensional embeddings. The embedding size that can be handled depends on the LLM. In our tests, a small model like GPT-2 handles up to $K = 30$ while producing valid samples. Since ICA and FPCA embeddings span entire windows, window length is not a bottleneck. The challenge is representing temporal structure with a compact embedding to ensure efficient and stable generation.

Handling of external covariates

By design, SDForger supports the integration of external covariates, including categorical, numerical, and textual data, into the generation process. These covariates are directly incorporated into the textual prompt, allowing the LLM to condition its generation not only on the time series embeddings but also on rich contextual information.

A concrete example is a weather dataset where each time series channel (e.g., temperature, solar radiation) is associated with metadata about the station, such as geographic coordinates or station ID. These contextual features are encoded as natural language and injected into the prompt, enabling the model to generate sequences that align with both the temporal dynamics and the contextual attributes of the data.

We explore this conditioning strategy in Section 6, where we demonstrate SDForger’s ability to generate channel-specific sequences with high fidelity, identifying the origin of generated curves, indicating that the synthetic sequences effectively preserve contextual distinctions. Integrating richer forms of metadata (e.g. geospatial tags, or event descriptions) will unlock new applications in simulation, personalized forecasting, and beyond.

Balanced performances

We kindly refer to the last paragraph of Reviewer Ck4d rebuttal for a discussion on this.

In SDForger readiness, yes, I was suggesting that the author should include more prompt examples similar to that of Section 6, line 323. Due to the current space limit, I would suggest that the author consider creating a section in the appendix dedicated to illustrating example prompts.

I read through the authors’ rebuttal carefully, and it addresses my concerns satisfactorily. I will revise my evaluation in the author's favor.

Q1. Clarification on "data-scarce" performance

We thank the reviewer for raising this important point. In our paper, data-scarce refers to both the limited number of time points and the small number of training instances available for fine-tuning. Unlike most generative approaches requiring thousands of long multivariate sequences, SDForger is designed to operate effectively even when only a single short time series is available.

Our periodicity-aware segmentation (Section 3.1) enables the extraction of multiple training instances by segmenting the signal into windows aligned with its dominant period. Crucially, the number of training instances $I$ can be adjusted by varying:

• the window length $L$: shorter windows yield more segments, and
• the window stride: increasing overlap results in denser sampling.

To assess the sensitivity to small $I$, we conducted additional experiments on the “count” variable from the Bikesharing dataset, varying both the number of instances and total time points. We fixed the embedding strategy to ICA with $k = 3$, used the same LLM and fine-tuning configuration as in the main paper, and averaged results over 5 seeds. The table below reports:

• similarity metrics (MDD, ACD, SD, KD, ED, DTW, SHAP-RE),
• NaN%, indicating generations with missing values,
• discard rate during filtering due to the norm criterion, and
• average embedding norms (original, accepted, rejected).

Key takeaways:

• SDForger remains functional even with just 10 instances and as few as 500 time points.
• As $I$ decreases, quality degrades gracefully, with moderate increases in distance metrics (e.g., MDD, SHAP-RE).
• NaN% and discard rates increase, indicating greater generation instability, but the LLM still produces many valid, high-quality samples.
• The norms of accepted embeddings remain close to those of the real data, confirming that the LLM continues to model the embedding space reliably.

In summary, while extreme scarcity naturally impacts stability, SDForger remains robust and practical even in low-resource settings, with graceful degradation and built-in filtering mechanisms to preserve sample quality.

Q2. Choice of embedding method

The superior performance of the ICA-based variant likely comes from the nature of the components it produces. Unlike FPC, which orders components by explained variance and often concentrates most information in the first few components, ICA explicitly seeks statistically independent components. This tends to produce a more balanced and disentangled basis decomposition, where each component carries distinct information that have similar importance for data reconstruction. For our LLM-based generation pipeline, this disentanglement appears advantageous because the model can learn a joint distribution over a set of factors that all have the same “power” and are more interpretable. Thus, the results suggest that the LLM is indeed better equipped to model the joint distribution when presented with independent factors rather than a hierarchy of variance-ordered components. This aligns with the broader literature in representation learning, where disentangled representations often lead to improved generative performance and interpretability.

Q3. Generation rejection rates

Observed Rejection Rates Across Embedding Dimensions

The table below reports rejection statistics on the "count" variable from the Bikesharing dataset, averaged across 5 seeds. We vary the embedding dimension $K$ while keeping all other settings fixed:

We observe that:

• Discard% remains consistently low (<2%) across settings, indicating that most generations fall within a plausible norm range.
• NaN% increases with larger embedding dimensions, reflecting a higher likelihood of incomplete generations in longer textual outputs. However, these are detected and filtered early, preserving only valid samples.
• Importantly, norms of accepted samples closely match the originals, while rejected samples exhibit extreme values, justifying their removal.

Impact of Filtering on Generation Quality

We further evaluated the effect of filtering on similarity metrics for $K=3$, again using the "count" variable from the Bikesharing dataset and reporting average results over 5 random seeds.

Filtering leads to a notable improvement in higher-order statistics, especially Skewness (SD) and Kurtosis (KD), by removing rare but impactful outliers. These outliers, while small in number, disproportionately affect the quality and stability of the generated dataset. Overall, the observed rejection rates are low, and the LLM generates on-manifold embeddings in the majority of cases, even under tight data constraints. The filtering step acts as a safeguard, eliminating extreme outliers without undermining generation diversity or coverage.

More on the application of similar filtering procedures to alternative generative models (e.g., TimeVAE) is discussed in our response to Q4 from Reviewer Ck4d.

Q4. Comparison with specialized architectures for correlation preservation

COSCI-GAN and similar specialized architectures explicitly incorporate mechanisms, such as a central discriminator, to explicitly preserve inter-channel correlations in multivariate time series. These designs reflect a targeted approach to model dependencies within the data and have demonstrated strong performance on correlation preservation.

In contrast, SDForger implicitly learns the joint distribution of multivariate time series by encoding an entire instance, including all channels and temporal dynamics, into a single, unified textual prompt. This approach offers several unique advantages:

• Flexibility and Generality: Rather than engineering architectural components explicitly for correlation modeling, SDForger exploits the powerful contextual modeling capabilities of LLMs. LLM naturally captures complex, high-dimensional dependencies through its attention mechanisms and deep representations, without requiring hand-crafted modules.
• Multimodality: By representing multivariate embeddings as text, SDForger enables the use of off-the-shelf LLMs pre-trained on large, diverse corpora, unlocking transfer learning benefits and opening up opportunities for multimodal extensions, like text-guided generation.
• Modularity: SDForger’s design is inherently modular. While this work uses basis decomposition methods (e.g., FastICA) for general-purpose embedding, the framework allows for the integration of domain-specific or correlation-preserving embeddings. By coupling such embeddings with the LLM’s ability to model joint patterns in long textual sequences, future versions of SDForger could explicitly enhance inter-channel correlation preservation, combining explicit inductive biases with implicit sequence modeling.

While explicit architectures like COSCI-GAN excel in focused correlation preservation, SDForger’s implicit approach via LLMs provides a complementary and versatile framework that can generalize across tasks and modalities. We agree that a clearer discussion of these trade-offs would strengthen the paper.

Additionally, in our response to Q3 from Reviewer V1ut, we empirically assess SDForger’s ability to generate consistent multivariate sequences. Results show that our method can effectively preserve channel-wise structure even in low-data regimes.

Limitations on potential societal impact and privacy considerations

We appreciate the reviewer’s important observation regarding the dual-use potential of synthetic data generation in high-risk applications. By relying on basis decomposition and text-conditioned generative modeling, SDForger does not memorize or directly reproduce individual time series samples. We acknowledge that synthetic data models could potentially be misused in sensitive domains such as healthcare or finance. We will update the manuscript to reflect these societal and ethical considerations.

Q1: Choice of embedding dimensions

In our utility-based experiments, we evaluated multiple values of the embedding dimension k across datasets. The values presented in the main paper reflect the best-performing configuration for each dataset in terms of downstream forecasting performance. Indeed, we observe a clear relationship between dataset complexity and the optimal embedding dimension. As discussed in our response to reviewer V1ut, more complex datasets (traffic and etth1), benefit from higher-dimensional embeddings, that allow both better reconstruction and a better learning of inter-variables correlations.

We present the utility evaluation results for embedding 3, 5, 7 in the ablation below:

These results reinforce the embedding selection (realised on validation data) presented in our paper. Furthermore, we discuss in Appendix A.2 a variance-explained heuristic that could be used to dynamically select k for a given dataset based on the percentage of variance to retain.

Q2: Data leakage?

We confirm that only the training portion of each dataset is used for both fine-tuning the LLM and selecting the embedding dimension. The test set is strictly held out and is never used during LLM training, embedding selection, or any intermediate tuning step. As explained, this ensures that there is no data leakage nor biased evaluation between the synthetic data generation process and the evaluation phase. Embedding dimensions k=3,5,7 are evaluated using validation data (i.e., the 20% subset of the training set as explained in Section 5), and the best-performing configuration was selected.

Q3: Heuristic filtering procedure

As discussed in our responses to Q1 and Q3 from Reviewer 2HPq, SDForger includes an in-generation filtering step to discard invalid or outlier samples, specifically those with missing values (NaN%) or abnormally large total norms (Discard%), computed in the embedding space. This lightweight mechanism improves sample quality and stabilizes distributional properties with minimal overhead.

In the results reported in Q3, we show that:

• Across different embedding dimensions, Discard% remains consistently low (<2%), indicating that the LLM generates mostly on-manifold samples.
• Rejected samples have norms several orders of magnitude higher than accepted ones (e.g., 19.1 vs. 1.7), which strongly impact higher-order statistics such as skewness and kurtosis.
• The filtering step notably improves metrics like SD and KD without significantly affecting lower-order distances, showing its value in controlling rare but impactful outliers.

To extend this analysis, we implemented a post-generation filtering step for TimeVAE, where we discard generated samples based on the norm of the reconstructed time series (in the time domain). Specifically, we constructed SDForger-like intervals and removed curves whose norms fall outside the selected interquartile range $[Q1 - 3 \cdot \text{IQR}, Q3 + 3 \cdot \text{IQR}]$, with bounds computed from the original (real) samples.

The results below (Bikesharing, “count”, 5 seeds) show the effect of this post-hoc filtering:

We observe that:

• Only a small fraction of samples (~1.2%) are discarded, yet this leads to improvements in SD and KD, indicating that a few outliers can disproportionately affect distributional metrics.
• This supports the value of post-hoc validation, especially in models like TimeVAE that do not natively include such constraints.

However, we would like to highlight that while post-hoc filtering is a useful safeguard, embedding lightweight validation directly into the generation process, as done in SDForger, offers both practical and performance advantages. It enables real-time quality control, avoids unnecessary decoding, and ensures robustness without additional overhead.

Q4: Clarification on "balanced performances"

In our similarity-based evaluation (Table 1), different generative models exhibit strengths in distinct types of metrics: For example, TimeVAE performs very well on distribution-based metrics. Conversely, TimeVQVAE excels on distance-based metrics (euclidian distance, dtw ...). In contrast, SDForger achieves a more balanced performance across both metric categories, meaning it consistently performs well on both feature-based and distance-based metrics without heavily favouring one domain. To quantify this, we computed the normalised average score per metric category per dataset in Table D.2.

This balanced performance indicates that SDForger not only preserves statistical features but also captures distributional similarities effectively, contributing to more robust and versatile synthetic time series generation. We will emphasize this in the updated version of the paper.

We hypothesize that for downstream tasks such as forecasting, it is crucial to use synthetic data generated by models that do not overly prioritize a single metric type but instead achieve consistently strong performance across multiple metric categories and domains.

1. Question on natural cycles

The term "periodicity-aware" refers to our method of aligning segmentation windows with the dominant periodic patterns of each time series channel. This is done in a fully data-driven way, without requiring dataset-specific configurations.

Specifically, for each channel, we compute the Autocorrelation Function (ACF) and identify statistically significant peaks. We select the most prominent peak corresponding to a period $P < L/2$, where $L$ is the target window length. This peak defines the natural cycle of the signal.

We then segment the time series into windows of length $L$, using a step size that is a multiple of $P$. This ensures that segment boundaries align with intrinsic periodic patterns, improving both representativeness and diversity of the resulting training instances.

This segmentation strategy enables meaningful augmentation from a single sequence, adapts dynamically to each channel's structure, and avoids any manual tuning or domain-specific heuristics.

A detailed and mathematical description of this procedure is provided in Appendix A.1.

2. Method's efficiency

We address efficiency in Section 5.3, with detailed benchmarks in Appendix Table D.3, where we report generation times on the Bikesharing dataset using different window lengths and embedding sizes.

The efficiency of SDForger is primarily influenced by two factors: the embedding dimension $K$ and the underlying LLM architecture. As $K$ increases, the length of the generated output also increases. This affects inference time, as longer sequences take more time to produce, and it can impact generation quality, leading to a higher number of discarded samples and requiring more iterations to reach the desired number of valid outputs. However, even at higher $K$, SDForger remains highly efficient, consistently outperforming all the baselines by a wide margin (from 4 to 100 times faster).

The choice of LLM also plays a role. Models like Granite require more memory and computation time, typically a GPU for inference, but still remain significantly faster and more efficient than generative baselines such as diffusion models, especially in low-data regimes.

The data-to-text conversion step itself is negligible in terms of overhead. For example, converting an embedding table of size 10 × 30 instances takes less than 0.002 seconds on a Mac M1 Max. This step is efficiently handled by the data_to_text function in the SDForgorger.py file.

Overall, SDForger offers a favorable trade-off between expressiveness and efficiency, benefiting from compact embeddings and fast generation, while remaining scalable across LLM sizes and embedding dimension $K$. While we acknowledge that more complex or multivariate datasets could lead to longer contexts, our current experiments suggest that the method scales well within practical margins.

3. Efficient generation of multivariate data

To evaluate whether our structured tabular transformation effectively supports the generation of multivariate time series, we analyse the preservation of inter-variable dependencies in the generated embeddings. Specifically, we compute the Frobenius distance between the correlation matrices of the real and generated embeddings. This distance is normalised to account for differences in embedding scales (k=3,5,7,10) and serves as a proxy for how well the generated data retains the structural relationships observed in the training data.

Table below aggregates result over 5 seeds:

As the embedding dimension increases, the decreasing Frobenius distance indicates an enhanced ability of the LLM to capture and reproduce the structured multivariate relationships. This trend suggests that larger embedding spaces enable the model to more effectively learn intricate cross-channel dependencies, which are essential for faithful multivariate time series generation.

Clarification on utility-based evaluation

Our utility-based metric stems from a key limitation in standard synthetic data evaluation: metrics like distributional similarity or distances often favor models that closely replicate the training data, potentially leading to overfitting and reduced generalization. This contradicts the central goal of synthetic data generation, which, in our case, is to produce data that not only resembles the real distribution but also generalizes well to support downstream tasks. In this paper, we use forecasting as the primary utility benchmark, where synthetic data is employed to fine-tune a predictive model. That said, expanding utility-based evaluation to include other real-world task, such as anomaly detection or classification, presents a promising avenue for future research.

Clarification on LLM fine-tuning

We employ standard full fine-tuning of the LLM to adapt it to the structured time series generation task. At this stage, we do not use PEFT methods (e.g., LoRA or adapters), as our focus was to demonstrate that even lightweight models (e.g., GPT-2) can effectively learn compact tabular embeddings with minimal training data.

Notably, we achieve strong results without any specialized fine-tuning optimization, which highlights the accessibility and robustness of the proposed framework. This suggests that SDForger is effective even under default training setups, without requiring task-specific or resource-intensive configurations.

That said, we recognize the benefits of parameter-efficient fine-tuning, especially for scaling to larger models, supporting continual learning, or domain adaptation. We consider this a promising direction for future work.