Comment 1:

Major issues

Response:
We thank the reviewer for their insightful comments. We address each of the points raised as follows:

• Incomplete Literature Coverage:
  We appreciate the reviewer pointing out the omission of key references such as UniTime. We have incorporated this important work, along with other relevant multimodal forecasting models, into the updated version of the paper. The related work section has been expanded to provide a more comprehensive overview of the field and to better position our contribution in relation to existing approaches.

• Limited Comparative Analysis:
  We appreciate the reviewer’s insightful feedback regarding the need for broader comparisons with other multimodal forecasting models. To address this concern, we have expanded our comparisons to include additional multimodal models, particularly in scenarios where descriptive text is provided alongside time series data. Notably, we have included comparisons with GPT-4TS, TimeLLM, and purely time-series-based models (Table 5), and our method outperforms these models in the tasks considered, demonstrating its high performance even outside the scope of instruction-based tasks. Additionally, we present a detailed evaluation of our method against UniTime and Qwen4MTS in Table 2 (Page 9) to further address the reviewer’s concerns.

• Insufficient Dataset Description:
  We apologize for the lack of detail regarding the datasets. We have taken care to include all relevant dataset details—such as sample counts, history length, and forecasting horizon—in the updated manuscript. A detailed analysis is presented in Section I of the Appendix.

• Simplistic Experimental Instructions:
  While the instructions considered in our work may seem simple, they represent an essential first step toward more complex scenarios where complete document instructions can be provided. Our research serves as a foundational effort, demonstrating that even with simple cases, several challenges must be addressed using existing algorithms like TimeLLM. We show that these challenges can be effectively managed with a specifically designed architecture.

  Although there is room for improvement in handling instructions, our paper already considers scenarios where the test text instructions differ from those used during training. These instructions are somewhat complex, combining several base instructions, and our methods demonstrate good generalization capabilities (Table 3). This success suggests promising perspectives toward the ultimate goal of tackling any instruction, regardless of complexity.

• Circular Evaluation:
  We appreciate the reviewer’s feedback regarding the evaluation datasets and the potential for circular reasoning. To address this concern, we highlight that in Table 3, we provide an assessment of the generalization capabilities of our model. In this evaluation, the training and test instructions differ significantly, which we believe is a fair and robust way to evaluate the model's ability to handle instructions adequately.

  This approach ensures that our model is tested on scenarios not explicitly covered during training, providing a more reliable measure of its performance in real-world applications. We are confident that this evaluation demonstrates the model's generalization capabilities and addresses the reviewer’s concerns about the reliability of our evaluation setup.

Comment 2:

Minor issues

Response:
Thank you for your detailed feedback on our paper. We appreciate your time and effort in providing these valuable comments. We will take each of your suggestions into account in the updated version of the paper.

• Regarding the inconsistent use of closing quotes (") instead of opening quotes (“) in multiple locations, including but not limited to lines 197, 203, and 213, we will ensure that the correct quotation marks are used throughout the manuscript.
• We agree with your suggestion to use textual citations rather than parenthetical citations for the references in lines 117 to 128. This will enhance the readability and flow of the text.
• Additionally, we will provide appropriate citations for the original dataset sources.

Thank you once again for your constructive feedback. We look forward to addressing these points in the revised version of the paper.

Dear reviewer: I wonder if our response can solve your concerns! Thank you!

Comment 1:

Given the synthetic data generation process, how can the authors ensure that there is no data leakage between the text data and forecasting targets? Could the authors provide a detailed explanation of the data generation process to address this concern.

Response:
We thank the reviewer for raising this important question. While the concern about data leakage is valid in many contexts, it is not a central issue in our case. The primary goal of our work is to assess the model's adherence to specific textual instructions rather than predict the target based solely on the time series data. To clarify, consider three samples with identical context length: a deterministic machine learning model would typically produce the same forecast for these samples. However, by adding textual instructions that specify a particular scenario or condition, we introduce a new layer of information that the model must adhere to. This is not a data leakage problem but rather a way of interacting with the model through different hypothetical scenarios.

The compliance rate is explicitly defined to measure how well the model follows these instructions while preserving the underlying time series structure. Thus, the model’s ability to follow instructions is the focus, rather than predicting targets based solely on historical data. This being said, while the focus of the paper is on text-instructed problems, we also perform experiments in the Appendix on other types of data where the text describes the task (e.g., domain, forecasting type, features). In these cases, the dataset contains no leakage, and our proposed algorithm outperforms the state-of-the-art.

Comment 2:

How practical is the proposed approach in real-world scenarios where textual instructions may not always be available or may be ambiguous? Could the authors discuss the potential limitations and challenges in deploying TITSP in practical applications?

Response:
We appreciate the reviewer’s thoughtful question. As discussed in our response to the first comment, the primary purpose of our approach is to evaluate how well the model adheres to specific textual instructions in controlled scenarios. While textual instructions are central to this evaluation, we acknowledge that in real-world applications, such instructions may not always be available or could be ambiguous.

One potential limitation is the reliance on clear, actionable instructions, which may not always be feasible in dynamic or unstructured environments. Additionally, the model’s performance may be affected by the quality and specificity of the textual input. However, our framework is designed to handle a wide range of instruction formats and adapt to different hypothetical scenarios, making it flexible for practical deployment. We also envision that the model could be augmented with supplementary mechanisms (e.g., user feedback loops or clarification prompts) to address ambiguity in real-world use cases.

Comment 3:

Has the model been tested on any other multimodal time series analysis tasks beyond forecasting? If not, what are the potential challenges in applying TITSP to other tasks?

Response:
We appreciate the reviewer’s question. While our model has primarily been tested for forecasting tasks, extending it to other multimodal time series analysis tasks such as classification presents some challenges. In classification, the output is often constrained to predefined labels, which limits the flexibility needed to explore different hypothetical scenarios through textual instructions. This makes it difficult to leverage the full potential of our approach in such tasks.

However, as demonstrated in the Appendix, our framework can be extended to scenarios where instead of instructions, we incorporate other types of additional information related to the task at hand. In these cases, our model still outperforms existing methods, suggesting that the framework has potential beyond forecasting, even for tasks with more constrained output spaces. Furthermore, we believe that imputation tasks could be a natural extension of our framework, as it can easily accommodate missing data by conditioning on other available information, showing that our approach is adaptable to different problem settings.

Dear reviewer: I wonder if our response can solve your concerns! Thank you!

For convenience, we also provide table 5 in the paper about the test results.

Table: Comparison of Compliance Rate (CR) and MSE for TITSP, Time-LLM, Qwen4MTS, UniTime, and Llama-3.1-8B across various instructed actions with highlighted best (in bold) and second-best (in underlined) results.

Question 1:

How would the proposed model perform without access to textual inputs or under noisy conditions? If textual instructions are incomplete, inconsistent, or contain noise, how would the model's performance be affected? This scenario is particularly relevant in high-stakes areas like finance, where decision-making often involves dealing with imperfect information. What measures have been taken to ensure robustness against these issues, which are common in real-world data?

Response:
We appreciate the reviewer’s question on robustness under imperfect or noisy textual inputs. In “what-if” scenarios, even if expert instructions are incomplete or contain intentional inaccuracies, the model’s primary goal is to follow these inputs accurately. This is often precisely what an expert wants—simply to test various hypothetical scenarios by observing how the model behaves under different instructions, rather than to ensure perfectly accurate instructions. Our model’s high compliance rate shows that it reliably adheres to these inputs, enabling experts to evaluate potential outcomes and behaviors without needing to formalize each scenario mathematically within the framework.

Although handling incomplete or inconsistent text is not the main focus of our work, we recognize its relevance. In the Appendix (Table 5, Page 18), we include experiments comparing our model's performance with and without textual inputs. These results show that even in cases without explicit instructional text, our method outperforms purely time-series models, highlighting the value of informative text for forecasting accuracy. This additional evaluation demonstrates the model’s ability to handle various text input scenarios, further affirming its robustness and versatility.

Question 2:

How does the proposed framework address interpretability in practice? The paper claims that incorporating textual instructions enhances interpretability, but there are no concrete demonstrations of how this contributes to meaningful insights for domain experts. Could you provide explicit examples or user studies that validate this claim? Without such evidence, how can the claim of improved interpretability be substantiated?

Response:
We thank the reviewer for raising this question on interpretability. The interpretability of our framework primarily comes from its capacity to directly link predictions to textual instructions. For example, if a linear growth is predicted, it can be traced back to specific input instructions, providing clear insight into why a particular behavior was forecasted. Additionally, attention map visualizations (see Figure 21, Page 27) reveal that the model highlights relevant keywords from the instructions, further demonstrating its focus on critical components of the input. This not only makes the reasoning process transparent but also allows experts to verify that the model is attending to meaningful terms.

Our framework’s generalization ability also contributes to interpretability, as it shows the model’s capacity to apply learned associations to new contexts, indicating it understands the core instruction beyond specific examples. While we acknowledge the value of explicit user studies, these elements collectively provide substantial interpretability by aligning the model's outputs directly with expert input and highlighting the key instructions that guide predictions.

Thank you for your precious comments!

Comment 1:

Technical Contributions are Incremental
The proposed approach lacks significant technical innovation. Integrating LLMs with time series is an incremental step rather than a groundbreaking contribution. The use of cross-attention and VQ-VAE offers no substantial improvement beyond established techniques.

Response:
We appreciate the reviewer’s feedback. While it is true that certain components of our architecture, such as cross-attention and VQ-VAE, are established techniques, our contribution lies in the development of a novel methodological framework. This framework includes a tailored data pipeline, innovative architecture design, and a comprehensive evaluation approach, all specifically geared towards integrating text-based instructions with time series forecasting.

We believe this is a significant contribution because it provides a structured approach to applying language models in the context of time series, which is a growing area of interest with wide applicability in fields like supply and demand forecasting. The ability to define and manipulate hypothetical scenarios through textual instructions opens new avenues for adaptable and context-sensitive forecasting models. We are confident that this framework will be valuable to the community, as it sets a foundation for future work in this space.

Comment 2:

Poor Structure and Clarity
The paper is poorly organized, with unclear explanations and an incoherent flow. The motivation and rationale for the proposed method are inadequately communicated, and critical components like AutoPrompter are explained in a convoluted manner, hindering comprehension.

Response:
We are sorry to hear that some aspects of the paper were unclear. We greatly value the reviewer’s feedback and would be very interested in a constructive discussion to better understand which specific parts of the motivation and rationale were difficult to follow. In the related work, we have added a paragraph to highlight the main motivation and differences of our work compared to state-of-the-art methods, especially those targeting instruction-based forecasting.

Regarding the reviewer’s concern with the explanation of AutoPrompter, we would like to clarify its purpose: AutoPrompter serves as a bridge that translates the time series data into the text embedding space. By quantizing the time series space, we map it into a compressed semantic space, which may have contributed to some of the complexity in the explanation. We have added additional clarifications in the updated version of the paper (Page 5) to ensure this concept is more accessible and the overall flow is clearer.

We appreciate the reviewer’s insights and hope the revised manuscript will address these concerns effectively.

Comment 3:

Inadequate Experiments, Superficial Related Work, Numerous Typos and Lack of Polish, Insufficient Practical Insights

Response:

Inadequate Experiments:
We acknowledge the reviewer’s concern about the reliance on synthetic datasets. While we agree that real-world data is crucial for evaluating practical applicability, synthetic datasets were used primarily to demonstrate the model’s capacity to handle controlled scenarios where the impact of specific factors can be isolated. We have now included benchmarks against state-of-the-art approaches (Llama-3.1-8B-instruct, Qwen4MTS and Unitime in Table 2). The results are highly stable and reproducible, with substantial performance margins over competitors, which reduce the necessity for statistical significance testing.

Superficial Related Work:
We have expanded the related work section to better differentiate our approach from prior research, particularly in the integration of text and time series. References such as Unitime have been added to strengthen the justification for our originality.

Insufficient Practical Insights:
The interpretability of our framework lies in facilitating interaction between expert users and the model through hypothetical scenarios. For example, the model generates forecasting scenarios based on textual instructions about supply and demand conditions, enabling experts to evaluate potential outcomes. This is particularly useful in fields like supply chain management, where generating and testing "what-if" scenarios through textual inputs offers clear practical benefits.

Dear reviewer: I wonder if our response can solve your concerns! Thank you!

For convenience, we also provide table 5 in the paper about test results

Table: Comparison of Compliance Rate (CR) and MSE for TITSP, Time-LLM, Qwen4MTS, UniTime, and Llama-3.1-8B across various instructed actions with highlighted best (in bold) and second-best (in underlined) results.

Question 1:

For Table 4, can you provide the same results, but for your model instead of only for TimeLLM? It would make it more obvious whether your model succeeds on those tasks with incorrect textual information.

Response:
We thank the reviewer for this insightful suggestion. As part of our ongoing experiments, we aim to address this by evaluating our model under the same conditions. Specifically, for the same base time series (same context length), we provide multiple different instructions and observe that our model achieves a high compliance rate, demonstrating its ability to follow instructions accurately. In contrast, TimeLLM exhibits lower compliance, highlighting the importance of the instructions. We appreciate the reviewer’s input, and we have now included these results in the updated manuscript for even more models (Qwen4TS and UniTime) (see Table 2, page 9).

Question 2:

For the real-world dataset, was the textual information always constant (as shown in Section B.3) for each dataset? This would allow a fine-tuned model to fully ignore it, since it could bake said information in its weights anyway.

Response:
We thank the reviewer for raising this important point. In our experiments, the format of the textual prompts varied across datasets, ensuring that the model was exposed to different types of instructions and did not simply memorize a single format. However, within each dataset, the prompt format remained consistent to ensure a fair evaluation of the model's ability to handle the specific instructions. This approach prevents the model from "baking" the textual information into its weights and ensures it adapts to diverse instructions. We have clarified this in Section B.3, page 19 of the updated manuscript, where we add another prompt format for an additional dataset (Traffic).

Thank you for your precious comments!

The following are our responses to your concerns.

Comment 1:

There seems to be a mismatch between the described technique used to apply the modification (equation 3), and the examples shown (figure 3). According to the equation, the data in the forecast window should be a pure affine function, without any of the noise shown in figure 3.

Response:
We thank the reviewer for highlighting this point. Equation (3) indeed describes a pure affine function; however, to ensure an increasing trend in certain time series, we allowed the slope $A$ to vary within the forecast window. This deliberate choice introduces some noise, as shown in Figure 3, but it demonstrates the model’s ability to adapt to evolving trends. For clearer examples without slope variation, please refer to Figure 10 in the Appendix (page 20). We have clarified this in the revised manuscript on page 4, where a comment is added to clarify this important point.

Comment 2:

While the model is tested against other multimodal text+timeseries models, it should also be tested against pure LLM approaches: just plugging the text and the history in a prompt for GPT-4 or Llama 3, and looking at the generated output. While such an approach won't scale to long series, recent work has shown it to be surprisingly decent at forecasting under textual instructions. See: LLM Processes by Requiema 2024 for a slightly more complex approach, but there may be more appropriate references for the more direct one.

Response:
We thank the reviewer for this suggestion. Although LLMs have shown some capability with time series data, they are fundamentally designed for language tasks and often struggle with numerical accuracy, as highlighted by several studies. This limitation motivated our dual-channel approach, where time series and text are processed in specialized frameworks, leveraging an expert model for each modality.

In Table 2 (page 9), we conduct an experiment by directly prompting Llama-3.1-8B-Instruct to perform these tasks. The results show good understanding of simple instructions but significant failures in most tasks. This approach also demonstrates instability, as the output may be challenging to directly utilize due to the mixture of numerical values and textual content. The designed prompt is shown in Appendix I.

In the Appendix (see Table 5), we present an experiment comparing our dual-channel method with GPT4TS—a purely LLM-based model for time series (for descriptive text instead of instructions). Despite GPT’s strong backbone (compared to Qwen used for our approach), our method outperforms it, confirming that dual-channel designs are more effective for multimodal tasks. Additionally, as the reviewer suggested, we conducted a new experiment focused on instruction-based tasks, which is the main focus of the paper. Here, our model also demonstrated superior compliance rates compared to Qwen4TS, underscoring the advantages of dual-channel methods for instruction-based text. These results are now included in the revised manuscript in Table 2 (page 9).

Comment 3:

Hyperparameters and training curriculum for the timeseries portion of the model are missing.

Response:
We thank the reviewer for pointing this out. The missing experimental details regarding the hyperparameters and training curriculum for the time series feature extractor are now included in the updated version of the manuscript. These details are provided in Section I of Appendix, where we outline the specific settings and the training procedure used for this part of the model, as well as the experimental setup for training UniTime and Qwen4MTS for the new additional experiments.

Dear reviewer: I wonder if our response can solve your concerns! Thank you!