❓ Can you provide more examples of refined text paired with time series?

💡 We provide an example of how the text description evolved with more iterations iteration.

Thank you for your really kind advice. We really appreciate your suggestion and already revise them in the new version of papers.

❓Are you going to release the code and dataset for this project?

💡 Thank you for your interest in our work. We plan to release the code and datasets used for the benchmark and all experiments in the paper, under an Apache-2.0 license upon acceptance. Currently, Anonymized code is available here: 

https://anonymous.4open.science/r/BRIDGE-CB58

❓How would removing parts of the agent architecture impact performance?

💡 In the experiment, the two teams showed completely different strategies: one team cared more about details in text description, while the other cared more about the overall trend and statistics. Specifically, we compare the single team with multi teams. Macro refers to a single team executing high-level information adjustments, while Micro focuses on details. Multiple teams indicate collaboration between two teams to complete the task. Overall, collaboration among multiple teams outperforms any single-team strategy, indicating that combining different strategies leads to more comprehensive and appropriate textual outputs. From a strategic perspective, the macro single-team approach performs better than the micro single-team approach, suggesting that overly detailed textual descriptions are still challenging to utilize effectively at this stage. Both teams chose to include statistical information, aligning with previous work that these factors most intuitively provide valuable insights, detail example can be find in Appendix A.5.

❓What about different text models?

💡 Thank you for the reminder. We have conducted an ablation experiment on two different text models. For more details please see our response to reviewer ZPw4

❓It is difficult to disentangle which parts of the agent architecture contribute to the improved text descriptions.

💡The proposed LLM-based multi-agent systems mainly has two components, one responsible for testing the performance of the text and the other responsible for refining the text, and they play equally important roles. The former is validated using a model that is widely used as a baseline and allows text as part of the input, where we hope that the results will be more reliable. The latter is responsible for modification, which includes an agent (manager) responsible for monitor the entire process and two teams responsible for analysis and modification respectively. We have rewritten this section in the section 3 and appendix A.5.

Comments, Suggestions And Typos: Thank you for your advice. We address all of them in the revised version. Since we have answered all your questions and have added all your requested experiments, please consider raising the score to give us an opportunity to present our work at proceedings.

We greatly appreciate your meaningful feedback and kind suggestions. In the new version of the paper submitted, we have focused on revising the issues you mentioned and added additional experiments. For your convenience, we have also attached the revised paper with highlighted modifications on GitHub for your review. Please consider raising your score!

❓ Could you provide more details on the agent design? Additionally, could you offer justifications for these design choices?

💡Our multi-agent framework have two main components , one responsible for testing the performance of the text and the other responsible for refine the text, and they play equally important roles.

Evaluation. The evaluation part is not a agent. Here, we using an famous work’s outputs as our backbone ([2402.16132] LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting). The input including Initial/Revised Text and Time Series and the output is predicted time series. Then the result had been measured be a series of metrics. We aiming to employed as many as metrics we can to provide the agent more dimension to refer.

Refine. There are five basic roles that make up the refine part of the agent. They are manager (monitoring the entire process, providing the input to each team and summarizing the final output), planner (as a small manager, monitor teams process), scientist (Analyze and provide feedback on which section probably should be add/delete/revise), engineer (following the scientist idea and conduct), and critic (judge the team work and point out any issues).

The pipeline is: (1) Given Input text, time series and metrics feedback, the manager will given the instruction to team A, where instruction shown in Figure 1. (2) Team A had be ask to focus on overview level revision. (3) When Team A finished the work, Planner will tell manager the result and Manager will let team B start to work (4) Team B focus on low-level revision. (5) After Team B finished the work, the two planner of each team and manager will going to discuss, and manager will writing the final template like “Background: <Description>; Sequence Length <Seq_len> …”. It is worth noting that the two teams have exactly architecture and members, differing only in the prompts for the tasks they execute. Additionally, in the initial experiments, we did not specify tasks, hoping the two teams would freely discuss and generate revision suggestions. However, the results showed that they would, intentionally or unintentionally, choose different directions.

We provide experiment on different strategies at the new version papers subsection 6.1. Overall, collaboration among multiple teams outperforms any single-team strategy, indicating that combining different strategies leads to more comprehensive and appropriate textual outputs. We also provide more further explanations in our response to reviewer x5xf, please also refer to that.

❓Could you provide more explicit details on the critical hyperparameters, model architecture, and training process for the diffusion model?

💡 Thank you for your point out, we added such information in the revised paper and more details algorithms in the appendix B.3. The reported result are all under following training settings. The number of prototypes are set to 16 for all the main evaluations. Models for each sequence length are trained for 50, 000 steps using a batch size of 128 and a the learning rate of $5 * 10^{-5}$ with 1, 000 warm-up steps. The model architecture can be found at the appendix B.3.

❓How do the pre-trained models, with their pretraining knowledge, affect the evaluation of the dataset? Does this introduce confounding factors that make it difficult to assess the dataset's impact?

💡 We explored two different text models for text encoder Llama and GPT2. The result has been shown below:

Results show that larger pre-trained model leads to slightly better generation performance, but the differences are marginal. This suggests that the impact from difference between language models does not surpass other model components.

Thank you so much for your response! I appreciate the added missing details as well as the suggested analyses, it effectively improved the clarity and the presentation of the paper. However, my concerns about the experiments seem not being addressed:

Section 6.1 (6.2 in the updated version), the usefulness of text is analyzed over the description generation stage (i.e., whether the description can be used to better reconstruct the time series) instead of the actual time series generation which is the real focus of interest of the proposed method.

I didn't see it being discussed in your response.

Section 6.2 (6.3 in the updated version), there is no comparison to diffusion model baselines which raises questions regarding fairness and completeness, especially given the observation that Bridge w/o Text seems to be essentially a vanilla unconditional diffusion model, but can already outperform most baselines.

It was not addressed. However, this is one major concern of the method.

Section 6.3 (6.4 in the updated version), the models are pre-trained, which introduces the confounding factors from the pretraining knowledge, and makes it hard to evaluate the effect of the dataset.

I don't think it can be addressed by evaluating larger models as the knowledge in even pre-training smaller models may already saturate for the test sets. The confounding from the pre-training still cannot be excluded.

Section 6.4 (6.5 in the updated version) lacks ablation on the prototypes themselves (e.g., comparing prototype vs. direct text embedding usage), making it unknown about the benefits of using prototypes.

I didn't see it being addressed.

As a result, the contribution of using text-condition, prototype, as well as the dataset, is still not justified effectively, while those are major contributions of the paper.

❓(Section 6.3, 6.4 in the updated version): How does the method address confounding factors introduced by pretraining, which complicate evaluating the dataset's effects?

💡 We sincerely apologize for misunderstanding your previous question. In this experiment (Table 4), our goal is to evaluate the quality of the synthesized data generated by the proposed method, particularly in comparison to real data and the Kernel Synth data augmentation method. In other words, we are asking, “Can the synthetic data generated by the proposed method be used to train models?”

We trained four different models and compared their performance across various data settings, rather than focusing on the differences between the models themselves. Regarding the selected models, we would like to clarify that none of the models directly use the ILI and M4 datasets in training. Specifically, LLM4TS [1] does not involve the ILI or M4 datasets, while Time-LLM [2] and GPT4TS [3] use these datasets only for evaluation. TEMPO [4] adopts a "Many-to-One" approach to evaluation, using multiple datasets for training and a different dataset for testing. This setup ensures that the influence of pretraining knowledge does not interfere with our evaluation process, which aims to compare the usefulness of different synthetic datasets.

For our experimental setup, we followed the evaluation setup from previous work on data generation [5]. We trained from scratch using the training frameworks of the four models mentioned, with three types of data: our model generated synthetic data, KernelSynth-generated data, and real data, to adapt the language models to time-series tasks. We then evaluated these models on the same real-time series test set. This setup ensures that pretraining knowledge is effectively leveraged without introducing confounding factors related to the time-series task itself.

In summary, we would like to state that in our experimental setting, pretraining knowledge in language models does not create any confounding factors for time-series problems, as the models used are language models that require fine-tuning to handle time-series tasks effectively. Moreover, the results show that our synthetic data leads to better performance than KernelSynth, underscoring the effectiveness of our time-series generation approach.

[1] Chang, C., Peng, W.C. and Chen, T.F., 2023. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469.

[2] Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J.Y., Shi, X., Chen, P.Y., Liang, Y., Li, Y.F., Pan, S. and Wen, Q., 2023. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728.

[3] Zhou, T., Niu, P., Sun, L. and Jin, R., 2023. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems, 36, pp.43322-43355.

[4] Cao, D., Jia, F., Arik, S.O., Pfister, T., Zheng, Y., Ye, W. and Liu, Y., 2023. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948.

[5] Jordon, J., Yoon, J. and Van Der Schaar, M., 2018, September. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International conference on learning representations.

Thank you once again for your thoughtful review and valuable feedback! We greatly appreciate the time and effort you put into evaluating our work. We have carefully addressed all your questions and resolved most of your concerns in our responses. We hope that our clarifications and updates demonstrate the strength and potential of our work. If you find our revisions satisfactory, we kindly ask you to consider raising your score, as it would be helpful for us to present in the conference. Thank you for your consideration!

❓(Section 6.4, 6.5 in the updated version): Why is there no ablation study comparing prototypes to direct text embeddings, leaving the benefits of using prototypes unclear?

💡 As requested, we will include an ablation study comparing (BRIDGE + text embeddings + prototypes) vs. (BRIDGE + prototypes) vs. (BRIDGE + text embeddings) in the revised version. Our previous results, shown in the table, demonstrate that BRIDGE consistently outperforms the other variants across most datasets.

• BRIDGE (w/ Text Embeddings + Prototypes) achieves the best performance, as it successfully models and generates distributions that closely match real data, with the lowest MDD and K-L divergence. For example, on the Electricity dataset, BRIDGE achieves the lowest MDD and K-L divergence, indicating its ability to effectively generate accurate time series.
• BRIDGE (w/o Text) shows a noticeable decline in performance across almost all datasets, with an increase in MDD and K-L divergence, underscoring the importance of text input in improving generation quality.
• BRIDGE (w/o Prototypes) exhibits the most significant performance drop. Removing the prototype mechanism results in a substantial increase in MDD and K-L divergence, as seen in the Taxi dataset, where MDD jumps from 0.591 to 0.633. This highlights the crucial role of the prototype mechanism in ensuring cross-domain generalization and maintaining high-quality generation.

We will include this ablation study in both the main paper and the appendix for a clearer comparison.

Dear Reviewer,

Thank you for your time and thoughtful feedback throughout the review process. As the reviewer response period concludes today (AoE), we respectfully request that you reconsider your score in light of the updates and clarifications we have provided.

We truly value your contributions and have worked diligently to address all nine of your questions, as well as to conduct extensive additional experiments in response to your comments. Given the significant revisions and improvements made to the paper, we believe we have thoroughly addressed the major concerns, including those related to the diffusion model baseline and the analysis of text.

We understand that the rebuttal phase can be a busy time, and we greatly appreciate your thoughtful input. While other reviewers have provided more positive feedback, with some adjusting their scores to 6 or 8, we respectfully ask if you might consider revisiting your score, as a 3 still reflects a relatively negative assessment, in light of the revisions and additional experiments we have provided.

We deeply value your evaluation, and given the importance of your feedback in this context, we fully respect your judgment, whether you choose to adjust your score or maintain it.

Thank you once again for your time and thoughtful consideration.

Sincerely,

The Authors

We sincerely appreciate your detailed and thoughtful suggestions. In the newly submitted version of the paper, we have addressed the mentioned issues. Specifically, we have clarified the differences in motivation between our work and Chronos, emphasizing that our focus is on cross-domain time series generation rather than foundational models.

❓ Does the BRIDGE paper fairly represent its comparison with Chronos, given that BRIDGE may actually be more resource-intensive?

💡 We apologize for any misunderstandings caused by the writing of the paper, and we have specifically addressed this in the revised version. As stated in lines 71-75 of the third paragraph in the original paper, our motivation is to use textual information to assist generation, rather than directly using an LLM backbone as the generative model. Therefore, a direct comparison with Chronos is inappropriate. Furthermore, we are investigating time series generation instead of time series forecasting. In our framework, the LLM mainly functions as an encoder to encode information and does not participate in the training process. Our diffusion model is relatively small (approximately 27 million parameters). As reported in the paper, when trained without text input, it only requires one hour and 20,000 steps on a single 32GB V100 GPU. As a proof, our trainable params is 26,972,641, where LLama is only used as an encoder for the text input and its weights are frozen during training.

❓ Did the authors consider using synthetic **datasets or different techniques like TSMixup, as introduced in the Chronos paper, to address data scarcity? 

💡In this work, we make two main contributions:

1. Validating the role of text and identifying what types of text are useful.
2. Proposing a diffusion model for generating time series in specific domains through text.

We didn’t consider using the mentioned methods, because our work aims at exploring whether and how text can assist generating time series for specific domains, even only using few-shot samples. This differs form the focus of synthetic datasets techniques like TSMixup, which aims to augment pre-training datasets for time series forecasting models without extracting specific pattern from specific domain. Therefore, these methods are not applicable in our setting. For your information, we added the description of the form of input and output in the Appendix B.2.

❓Looking at Definition 1... Does TG^2 and BRIDGE use implicit rather than explicit labeling mechanisms?

💡 Your understanding of our approach is correct, but what you mentioned represents only an intermediate step of our method. The proposed method implicitly captures domain information through its foundational representation system, using a weighted sum of orthogonal bases. Here, text is used to represent domain semantic information and provides domain-agnostic statistical information. In the generation phase, the diffusion model uses this as a condition to generate synthetic data that better aligns with the true data distribution.

❓How does the BRIDGE method ensure there’s no data leakage when the LLM agent retrieves external information for text descriptions?

💡We want to emphasize that the motivation of this work is to use text to enhance time series generation rather than time series forecasting. Therefore, there is no data leakage concerning possible future data when training our model. In addition, providing background information is not an uncommon practice. In Time-LLM ([2310.01728] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models) and TEST ([2308.08241] TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series), relevant dataset information is also provided as prompts to assist the model's forecasting. Since BRIDGE method retrieves external information for each domain that reflects the static property of the domain. The retrieved messages are generally applicable and not likely to vary along with time, so they are not concerned of future leakage.

Comments, Suggestions And Typos: We greatly appreciate your detailed feedback and constructive suggestions. We have sincerely followed your advice to revise the paper and supplement the experiments. To address the concerns, we have:

• Revising Section 3 and moving additional details to the appendix.
• Clarifying the definitions in Section 4 for greater clarity.
• Checking and correcting grammar and spelling errors throughout the paper.

Please consider improving our score and allowing us to present at conferences

❓ Did the authors consider using synthetic **datasets or different techniques like TSMixup, as introduced in the Chronos paper, to address data scarcity? 

💡In this work, we make two main contributions:

1. Validating the role of text and identifying what types of text are useful.
2. Proposing a diffusion model for generating time series in specific domains through text.

We didn’t consider using the mentioned methods, because our work aims at exploring whether and how text can assist generating time series for specific domains, even only using few-shot samples. This differs form the focus of synthetic datasets techniques like TSMixup, which aims to augment pre-training datasets for time series forecasting models without extracting specific pattern from specific domain. Therefore, these methods are not applicable in our setting. For your information, we added the description of the form of input and output in the Appendix B.2.

We also employed Chronos data augmentation methods for a comparison experiment. Since TSMixup is not open-sourced, we used another method mentioned in the same paper (link). We maintained the same dataset size for training and tested on the same real test set. Overall, both methods effectively provide valuable synthetic data (compared to completely random data). However, our proposed method is closer to real data, demonstrating its significance in cross-domain data generation. The results can be found below:

Thank you for your response! Some further questions:

1. Can you please clarify which of these is correct?

   • In your response, it says "As reported in the paper, when trained without text input, it only requires one hour and 20,000 steps on a single 32GB V100 GPU."
   • In the paper, it says "We implemented all the model and conduct all experiments on single NVIDIA Tesla H/A100 80GB GPUs" and "Models for each sequence length are trained for 50,000 steps"

2. I'm not convinced by your response to my concern about data leakage, I was hoping you could please add some more information? You say "In Time-LLM... relevant dataset information is also provided as prompts to assist the model's forecasting. Since BRIDGE method retrieves external information for each domain that reflects the static property of the domain. The retrieved messages are generally applicable and not likely to vary along with time, so they are not concerned of future leakage." I'm certainly not concerned with retrieving external information in general, but it seems to me that the TimeLLM method doesn't suffer from the same potential dataset leakage problem. Looking at Figure 4 in their original paper (link) it appears that the external information isn't scraped from the web-scale data in an open-ended fashion; in contrast, Figure 7 in the present manuscript makes it seem like the agents have access to internet-scale data. Is my understanding here correct? (Note: I agree that the results in Table 3 indicate that the time series generation using BRIDGE outperforms the other methods, but I'm not convinced that this was done without data contamination)

3. Thank you for running the additional KernelSynth experiments. On the downstream forecasting task, the ILI results indeed show that your method outperforms using synthetic data generated from Gaussian processes, on that dataset. However, Looking at Table 14 (M4 dataset), it seems like the average KernelSynth performance, and the average performance from your method, are very similar (and in the absence of error bars, they might even look like they yield equivalent performance) [and they both underperform the "Real" data, which isn't a problem]. This seems to indicate that on the M4 forecasting task, your method performs on par with just using KernelSynth synthetic data, is this correct? (And is this to be expected?)

Thanks for answering my questions!

We sincerely thank you for your reply and your constructive suggestions on our work! Since we have addressed most of your concerns, we kindly ask you to consider raising your score to help us present at the conference. Thank you for your consideration!

❓Can you please clarify which of these is correct? (1) w/o Text: 20,000 steps on a single 32GB V100 GPU (2) w/ Test: 50,000 steps on NVIDIA Tesla H/A100 80GB GPUs

💡 Both statements are correct, as they describe the resource consumption of our model from different perspectives. Our approach uses an LLM-based encoder to retrieve domain-related semantics from text and a diffusion-based generator to output synthetic time series samples. Therefore, we analyzed the resource consumption of these two components separately.

As reported in the paper, when trained without text input, it only requires one hour and 20,000 steps on a single 32GB V100 GPU.

This statement describes the minimum resource consumption of the diffusion-based generator module in our model.

We implemented all the models and conducted all experiments on a single NVIDIA Tesla H/A100 80GB GPU.

This statement refers to the actual computation platform used for each experiment involving the full version of our model. However, the resources were not fully utilized during our experiments. Since we do not fine-tune the weights of the LLM, our model uses relatively fewer computational resources than it might appear.

❓ Is the understanding correct that Time-LLM avoids data leakage by using controlled static information, whereas BRIDGE seems to access internet-scale data, potentially leading to data contamination?

💡 We apologize for any confusion caused and would like to clarify how our pipeline works. Our work is divided into two main stages: constructing “annotations” (text) for time series and generating time series using the text-time series pair dataset.

In the first stage, the text is finalized and transformed into static data, aligning with the input format of Time-LLM. During the second stage, in the time series generation process, there is no interaction with the web. Specifically, in our agent framework, all prompts are fixed. Web retrieval is only used during the pre-processing stage (as described in Appendix A.2) to search for relevant initial text descriptions.

Once the text template is finalized, it is applied to the entire dataset to generate textual "annotations" for the time series. Agents do not retrieve information about specific time series samples, but only help to format the prompt templates. These templates are then populated with time series statistics obtained from the training set. For example, a prompt generated by the multi-agent framework might look like this:

"The {dataset_name} dataset provides {frequency} totals of {data_description} from {start_date} to {end_date}. The prediction length is {prediction_length} time steps. Data statistics indicate that expected values for the time series range between a minimum of {min_value} and a maximum of {max_value}, with a mean of {mean_value} and a standard deviation of {std_value}, suggesting {variability_summary} compared to the beginning period. Anticipated peaks are likely to occur at steps {peak_steps}, while predicted dips are expected at steps {dip_steps}."

In conclusion, our method uses static input during the time series generation stage and does not interact with the web, thereby eliminating any potential for data leakage.

❓Does this indicate that, on the M4 dataset, your method performs on par with KernelSynth? If so, was this result expected?

💡 Thank you for your question. In the case of the short-term forecasting task on the M4 dataset, the performance difference was less noticeable due to the shorter prediction horizons. It's important to note that the M4 dataset does not have clearly defined domains; instead, it groups various time series based on sampling frequency. This is different from datasets like ILI and traffic, which consist of domain-specific time series (e.g., hospital admissions or traffic volumes). As a result, our domain-centric approach is less effective at extracting meaningful cross-domain semantics from M4, making it unsurprising that domain-independent methods, such as KernelSynth, produce data that is better suited for training forecasting models that perform well on the M4 test set.

Additionally, the M4 dataset includes relatively few samples in the Weekly/Daily/Hourly categories (359/4227/414 series, respectively, as reported in the original M4 paper [1]), with the majority of the data coming from unknown domains. For instance, the Hourly subset consists entirely of time series without any domain information. This creates two challenges for our approach: (1) the limited number of samples makes it difficult to learn clear patterns, and (2) text descriptions intended to provide domain-specific context may turn into noise in this scenario. In contrast, KernelSynth’s random generation method, independent of the original data, performs better on these subsets, where randomness and sparse samples are more dominant.

Finally, we would like to emphasize that we compared the quality of synthetic data generated by our method with that of the KernelSynth method used in Chronos for both long-term and short-term forecasting tasks. The proposed method demonstrated better performance across most subsets. Notably, in the long-term forecasting task for the ILI dataset, models trained on our synthetic data consistently outperformed those trained on KernelSynth data.

[1] Makridakis, S., Spiliotis, E. and Assimakopoulos, V., 2020. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1), pp.54-74.

I would like to thank the authors for replying to my questions. While I appreciate the efforts to address the concerns that I raised, there remain some significant issues that I believe need to be addressed before this work is ready for publication. For example, the authors have answered one of my questions in ways that directly contradict the contents of the paper. This decreased my confidence in the paper, and as a result, I have lowered my score.

Two of the key issues which I am concerned with--

1. GPU resource utilization: I asked a question about GPU resource utilization because it seemed as though the paper had originally misrepresented the resource efficiency when comparing with prior works. But there appears to be a discrepancy between the author's response, and the paper's contents. The authors attempted to clarify the discrepancy by stating: "As reported in the paper, when trained without text input, it only requires one hour and 20,000 steps on a single 32GB V100 GPU." However, it appears that such a statement does not actually exist anywhere in the paper. I gave the authors two separate opportunities to clarify this point, and unfortunately I am more confused now than when I originally asked the question.

2. Potential data leakage issue: the potential data leakage issue in the web scraping methodology requires further clarification. For example, if you're trying to generate time series similar to a given dataset, then the web scraping approach may inadvertently incorporate information about similar time series, which could give the model an unfair advantage (specifically, data leakage) when compared to methods that don't utilize web data. To ensure a fair comparison, it would be valuable to see e.g. a detailed analysis of how this potential advantage is controlled for.

We believe there may still be some misunderstanding regarding our training pipeline and how our model works. As outlined in the paper and our previous rebuttal, we would like to summarize the process again in hopes of clarifying the web search procedure and the overall data flow.

1. Web Search for Template Generation (only involves training data)
   • Given the training TS data, perform a web search for relevant TS text descriptions.
   • Agents summarize these descriptions to create an initial template.
   • The agent applies the template to each TS training record to generate TS-text pairs.
   • TS-text pairs are used for TS forecasting to validate the template.
   • If performance is unsatisfactory, the agent refines the template and repeats the process.
   • This iteration continues until the template's performance is deemed sufficient.
   • Output the final template and TS-text pairs for training set.
2. Train the Prototype Enhanced Diffusion Model
   • The diffusion model is trained using the TS-text pairs generated from the final template.
3. Conditional generation (only involves the text template)
   • During generation, the fixed template is populated to form the conditioning
   • The trained diffusion model is then used for generating new time series samples

Therefore, the web search is only conducted with the training set to generate the text template. No information leakage occurs, as the test set is never involved in the web search process.

Comments: We appreciate your ongoing constructive suggestions and rigorous evaluation of our work. During the rebuttal phase, we carefully addressed your concerns through detailed responses and additional experiments, resolving several of the issues you raised. We also refined the paper to improve its readability and enriched it with further experiments, making it more comprehensive. These efforts address the concerns raised and provide a clearer and more comprehensive presentation of our work, which we believe should lead to a more positive perception. Considering the steps taken to improve the clarity and depth of the paper, we respectfully request you to reconsider your rating based on this refined understanding.

❓Clarify the contradict information about GPU resource utilization?

💡We have provided detailed clarifications regarding two scenarios: the first being “without text input” and the second being the actual platform used in the main experiment, which includes text input. While the machine is equipped with 8 physical GPUs, but it’s not need using all of them, each experimental run requires only a single GPU is enough.

We apologize for the typographical error and will ensure it is corrected in the next release. However, we would like to emphasize that this issue does not warrant delaying its inclusion in the proceedings, as even the next submission will only be a text correction and does not impact the contribution of our method or the validity of our experimental results. Requiring another review cycle for such a change would unnecessarily consume months.

❓Potential data leakage issue?

💡Thank you for expressing your concerns regarding the web search process.

We want to clarify that we ensure no data leakage occurs from web search. Specifically, the only influence the web search component may have on the inference stage is in the creation of the text template, but the template does not reveal any information about test set.

To clarify, the web search is conducted only during the multi-agent collaboration stage, where it is used to search for text descriptions related to the training data. The text template is generated at his stage before training the diffusion model . During conditional generation, the text template remains fixed and static, with no further changes, regardless of the test set. The conditional generation stage does not involve any web search.

In practice, our agents process each time series (TS) record in the test dataset according to the fixed text template, producing TS-text pairs. It is important to note that the test dataset is does not overlap with the training set, meaning the text template cannot contain any information from the test set, preventing any potential data leakage.

To investigate whether the multi-agent framework leaks unintended information to the model, we further examined prompt templates. We asked GPT-4 whether it could link the exact data with the template and also verified this through human checks. We concluded that 0% contained any information that directly describes a time-series (witho ut populating it with actual data from the training set). To further demonstrate that the text template does not include specific information about the test set, we provide example templates below. These templates should clearly illustrate that they are not tied to any specific domain or individual TS record, allowing you to directly observe that the template does not relate to any test data. We will also include these templates in the appendix.

Thank you for your kind advice. We really appreciate your suggestion and have already revise them in the new version of the papers.

❓ The experiments in the paper do not mention the number of runs over which the results were averaged and what is the standard deviation between runs?

💡 Thank you for your kind reminder and suggestions. We reported 3 times averaged results in new version of the paper now. please refer to Table 3. Due to time constraints, we have not yet had the opportunity to conduct more robust tests on all the baseline models. But we will include these results in the main page at a later time.

❓ Missing the explanation of what makes Bridge w/o Text better than other baselines?

💡 Our work includes two main contributions: (1) validation of what kind of text is useful and (2) a new model for time series generation. Therefore, the motivation of the new diffusion model itself takes into account how to achieve better generation quality in the absence of text input. When not using text, we use semantic prototypes as latent representations of commonalities in time series data and train the model to decode from combinations of these latent representations to generate specific domains. For example, in time series generation, the fundamental properties important to time series align with its decomposition, specifically trend and seasonality. We conducted additional experiments to validate this point. As shown in Figure 4, different semantic information exhibits completely distinct representations. For example, Prototypes 6, 7, and 13 display markedly different trend patterns. Prototype 6 demonstrates a sigmoid-like curve, useful for representing gradual changes with plateaus. Prototype 7 shows a gradual upward trend, while Prototype 13 depicts a sharp transition from a low to a high state. Please also refer to ZPw4.

Comments, Suggestions And Typos: Thank you for your suggestions. We have addressed all the issues in the revised version. We carefully checked for any spelling errors and added an explanation about the bridge w/o text. Please give us an opportunity to be present in the proceedings.

Thank you for your kind advice. We really appreciate your suggestion and already revise them in the new version of papers.

❓ In what ways do the LLM prompts succeed and fail?

💡 First, it is difficult for LLMs to strictly adhere to the given prompt. This also applies to LLMTime ([2310.07820] Large Language Models Are Zero-Shot Time Series Forecasters). This shows that directly using LLM as backbone is still an open challenge. Second, similar to the findings of Chenglei’s work ([2210.09150] Prompting GPT-3 To Be Reliable), excessive contextual information is more often detrimental than beneficial for prompting LLMs at this stage, as it makes it difficult for the model to extract key information, thereby reducing output quality. As shown in Table 1, the initial detailed descriptions actually misled the LLM. Third, Although aligning multimodal information has been shown to be effective (Image-based time series forecasting: A deep convolutional neural network approach - ScienceDirect), directly aligning TS or decomposed TS to LLM does not significantly improve the performance, as shown in Appendix A.

❓In what ways do the text succeed and fail to describe the time series (e.g. sinusoidal, trending upward, extreme values, ...)?  

💡 Here is a summary about the influence of different text components

• Background information on the data is useful: this may be because the LLM encountered related information during the pre-training stage.
• Mean, maximum, minimum, and variance are useful: these define the range of values.
• Overall trend is useful: this can help prevent the LLM from fluctuating within a specific range.
• The length of the generation is very useful: this can be a good way to limit LLM from generating too short time series.
• Specific trends are not useful: the LLM is still unable to understand subtle differences between time steps.

Comments, Suggestions And Typos: We appreciate your detailed feedback and the constructive suggestions. We have carefully follow your advice to revised paper in order to improve the clarity and readability. To address the concerns about comprehensibility, we have:

• We rephrase the section 3 and section 4 to streamline explanations and use more specific language to clearly describe each component of our method. More Detail could be find in Appendix A.3, A.4 and A.5
• We redraw the Figure 1, 2 to provide clearly describe of our methods. We keep cite each of figure to explain the sentence to help reader follow the interactions between components more easily and understand the overall scheme at a glance.
• In addition, we have clarified terminology and performed a thorough grammar and spell check to correct any errors. We also addressed the discrepancy in the results mentioned in the abstract and introduction, now ensuring consistency between sections.

Since none of the comments outline serious issues with the papers’ methodology or findnigs, please give us the opportunity to be present in the proceedings.