---

W4: The cooperation between texts and numbers is well-designed. Generally, LLMs including GPT are not good at dealing with numbers, and they are not sensitive to the symbols in numbers.

There have been many recent works that show that LLMs can perform surprisingly well forecasting tasks zero-shot. For instance, Gruver et al. (2024) and Requeima et al. (2024) showed that accurate forecasts could be obtained through zero-shot sequence completion. Our work extends such evaluations of LLMs to the relatively new problem setting of context-aided forecasting, and shows that they are especially useful in this problem setting. Moreover, as we point out in Section 5.3, LLM baselines are good forecasters also when compared to quantitative forecasting models in a purely numerical, no-context setting. Our results confirm that LLMs are surprisingly strong at forecasting, but further research would be needed to understand the surprising performance of LLMs in forecasting, in efforts to improve these models.

References:

Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.

James Requeima, John Bronskill, Dami Choi, Richard E Turner, and David Duvenaud. LLM processes: Numerical predictive distributions conditioned on natural language. arXiv preprint arXiv:2405.12856, 2024.

---

Questions:

---

Q1: can you just open-source all the related-materials and I think it's better to let all people to judge if it has enough value and easy to use.

We plan to release the code and datasets used for the benchmark and all experiments in the paper, under an Apache-2.0 license upon acceptance. Anonymized code for all tasks and experiments is available here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/.

---

Q2: how do you choose the best-quality text and how would you access this quality?

We wrote the text of all the tasks ourselves. To ensure their quality, we used peer review to make sure that the texts are high-quality and also important for the forecasting task. Please see our answer to W1 for the full task creation process.

---

Q3: Is there any better way to cooperate the numbers in prompt to evaluate the effect of the benchmark?

Thank you for your interesting question. The scope of our work is to establish a high-quality benchmark to evaluate models that jointly use text and numerical history for forecasting. Exploring how to best leverage numerical information and text is an interesting research direction that we leave for future work, that this benchmark enables quantifying progress on.

---

Q4: text brings more calculation, but the effect of the texts is well discussed. How to balance and choose the most effective one?

All tasks in the benchmark are designed such that the text is absolutely necessary to be taken into account. We write the context ourselves to ensure its relevance (see our answer to W1 for the full task creation process). There are tasks that require the model to choose the most relevant text from the given textual context, as opposed to providing the filtered, relevant text directly. These tasks are tagged under the Retrieval from Context, (see tasks under “Retrieval:context” in the benchmark visualization website). Rightly, we find that when contexts are already filtered to only have the relevant context, models perform better at the task.

For example, we compare two tasks on predicting the Unemployment Rate of a county. For the UnemploymentCountyUsingSingleStateData task, we give context relevant to the variable (Unemployment Rate of the State in which the county is) - this task is visualized here. In the UnemploymentCountyUsingExpliciteMultipleStateData task, in addition to the context about the relevant state, the context includes unemployment rates of 2 other states - task visualized here. This figure visualizes the difference in performance in these tasks for 3 different models. We find that models perform much better when only the relevant county’s data is provided, as opposed to the data from multiple counties.

Thank you very much for your thoughtful review. We are delighted that you consider our work to be 'huge'. We hope that the responses below will alleviate some of your concerns, and that you will consider raising your score.

---

W1: lack of discussion of noise in texts. The texts are complete but the quality filter is not well designed. In general, we need more well-designed texts which is really useful

We would like to point out to the reviewer that all tasks in the benchmark were manually designed, from scratch, by the authors of this work (without resorting to external annotators or LLMs) , to ensure quality. Note that this is a key novelty of our work. Further, all tasks in the benchmark were filtered for quality through a rigorous peer review process by a committee with significant expertise in time series forecasting (co-authors of this work).

Additionally, please see point W1 of our response to reviewer kJ9d, where we describe a newly-performed LLM-based analysis of the relevance of the context for each task.

We detail the full task creation process in the benchmark below, for your reference:

First, we identified high-quality sources of public time series data from various application domains (see Section 3.1). Second, we established the categorization for sources of context (Section 3.2) and capabilities (Section 3.3) as a framework to guide the creation of new tasks and ensure their diversity. Third, team members separately created the tasks, each time selecting a data source, implementing a time series window selection strategy (e.g., short or long history), brainstorming about realistic sources of context and capabilities based on the forecasting task, writing code to generate such context by instantiating a template, and finally, if required, writing code to modify the time series data to reflect the context (e.g., some future spikes). Then, the tasks were peer-reviewed by a committee with significant expertise in time series forecasting (co-authors of this work). The creator of each task was not allowed to participate in its review. The review ensured that the text was of a high quality, that it without doubt enabled a better forecast, and that the context source and capability tags were well-assigned. If a task was deemed of not high enough quality, it was either returned for revisions, or rejected.

The code for all tasks and experiments is available here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/. An example task can be found here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/cik_benchmark/tasks/montreal_fire/short_history.py, where the time series window selection occurs from L94-112 and the context generation occurs from L114-158.

We will add the same discussion to our paper in the next revision.

---

W2: what is the importance of texts? are they hidden in time-series itself? For me, time-series is the sampling result of a complex system, and even you have already done a lot and try to give more complete one, but the system is hard to define. As a result, the time-series itself may contain more information than the texts.

The benchmark is designed such that the provided numerical information alone is insufficient to forecast properly. Thereby, additional information in the form of text is required to solve the tasks, and models are expected to leverage both the history and the additional text information when forecasting. Note that this additional information is not derived from the time series itself, but rather from auxiliary information sources (in the case of intemporal context), or information on other variables (for covariate context), scenarios that condition the future states of the system on (for future context), statistics about the time series that are beyond the available short history (in case of historical context), and causal information about provided covariates (in case of causal context). Thereby, the information conveyed in the text is not hidden in the time series, and cannot be derived from the time series.

---

W3: besides model evaluation, the benchmark is hard to use as in real world, people may prefer to use simple model and may lack of texts.

The focus of our benchmark is context-aided forecasting, a very new problem setting. We agree with the reviewer that our benchmark can be seen as complementary to all other benchmarks such as that of Godahewa et al. (2024) and Qiu et al. (2024), which all focus on the canonical purely-numerical forecasting setting.

References:

1. Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero Manso. “Monash time series forecasting archive.” arXiv preprint arXiv:2105.06643, 2021.

2. Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo et al. "Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods." arXiv preprint arXiv:2403.20150, 2024.

Dear Reviewer RToE,

We want to thank you for the time you took to review our work. We appreciate the suggestions and the detailed feedback, and we have carefully worked to address any concerns.

As the discussion period comes to a close, if you feel that your concerns have been addressed, we would greatly appreciate any change to your score. If you require any additional clarifications, we are happy to provide them.

Thank you again for your time and for providing thoughtful comments.

Kind regards,

The Authors.

W1: The reader is supposed to assume that all provided textual information contributes meaningfully to the forecasts [...] evaluating the relevance of the textual context [...] through methods such as LLM-based or human assessments would strengthen the claims

Indeed, it is crucial for readers to be able to assess the quality of the tasks. This was the motivation for providing the website alongside the paper, and examples in Appendix B. Nevertheless, we appreciate your suggestion and, following your recommendation, we further assessed the quality of the context using an LLM-based critique. The critique was built by prompting GPT-4o with the historical and future numerical data, as well as the context, and asking it to assess if its estimation of future values would either improve or degrade based on the context (see this file for the code; the prompt used is given in lines 18-66). We ran this critique on 5 instances from each of the 71 tasks and report results in this figure. In short, for all tasks, the critique believes that the context enables better forecasts (significantly better for most tasks).

The output of the critique, including justifications of its score for each task instance, is attached to our response as a CSV here. A detailed presentation of the critique and these new results will be included in the appendix of the revised paper.

We hope this resolves your concerns regarding analytical rigor in evaluation. Please let us know if you would like us to deepen the analysis.

---

W2: some covariate information appears to include future events, which a causal forecaster would not typically access (e.g., “Speed From Load” in Appendix B.2). This raises concerns about causal consistency, as there is no mechanism for systematically separating different types of contextual data other than through manual or LLM editing. Such limitations could present challenges for users who want to avoid incorporating future or irrelevant covariate information in their experiments.

The mentioned future events are either scenarios a user would like to simulate (e.g., in UnemploymentCounty tasks such as this task, or this task) or control variates that do not break causal consistency (e.g., in SpeedFromLoad, knowing that the load is set to a certain value in the future does not leak useful information for predicting the current speed). In either case, tasks containing this type of context are clearly tagged as containing ‘Future information’ or 'Causal information’, and these components of the context are defined in separate variables in the code (see e.g. line 75 here). Thereby, in these cases, a user can easily remove them from the context for evaluation for their experiments.

---

W3.1: Lacks clarity regarding the historical context length and forecasting horizon

Please see this image, for histograms depicting the distribution of lengths for the context, numerical history and target length of a set of five instances for each task in CiK. We measure the length of the natural language context in characters, and the numerical sequences in floats.

We will add this information to the paper in the next revision.

W3.2: The reliability of the benchmark results hinges on the sample size, yet no information about the number of samples for the datasets is provided.

This information is available in Section 5.1 (Evaluation Protocol) of the main text. Forecast distributions are estimated with 25 samples from each model and, for each of the 71 tasks, we consider 5 random instances to evaluate the models. This was done in order to make the evaluation of the benchmark sufficiently accurate, while being reproducible and affordable.

Q1: Does the covariate information always imply the availability of the future values of the covariates, or are there examples with covariate information provided only for the history time series?

There are tasks where the covariate information is only available for the history. Please see this example taken from the benchmark report. In this task, the covariates: the presence of water bodies and the number of incidents in the city's other boroughs, are only provided for the history.

Suggestions for the authors:

Again, thank you for your efforts in reviewing our work and for these suggestions which will definitely improve the quality of our contribution. We implement each of these and comment on them below:

1. Include a discussion on the manual curation process with information on the data sources and the selection of relevant context from them.

Please see our response to W4.

2. Include benchmark descriptions mentioning the sequence lengths, prediction horizon, and number of sequences present in each benchmark to build confidence in the robustness of the work.

Please see our response to W3.

3. Provide some ideas for separating different contextual information in the text.

The various sources of contextual information are exemplified in Fig. 3 and defined in the text in Section 3.2. Moreover, each task is tagged with its context sources in the visualization website that accompanies the benchmark. We also give concrete examples of domain, context source, and capability assignments in Appendix B. Further, the various components of the context are defined in separate variables in the code (see e.g. line 75 here for an example) for clarity. Please let us know if this provides clarification. We are more than happy to discuss this further.

4. Highlight the efforts taken to ensure that any contextual information paired with the time series is actually correct/relevant. Do a similar task to highlight the relevance of the said "region of interest."

We include a new LLM-based analysis of the relevance of the context for each task. Please see our response to W1. The regions of interest are highlighted only for tasks where context refers to a set of timesteps, for which we thoroughly verify the code for each task manually to ensure a consistency between the highlighted timesteps and the timesteps mentioned in the context text. For examples, see lines 70 to 84 in this task in the code where the exact same variables are used in the context and to consider the region of interest during evaluation.

Thank you for your feedback and for emphasizing the originality, relevance, and clarity of our work. We appreciate the depth of your review. Please find answers to your comments and questions below, and do let us know if you would like further clarification on anything else. We hope that these explanations will resolve your concerns, especially regarding soundness and transparency, and that you will consider increasing your score.

We first respond to W4 and Q2 together, and then move to addressing the other weaknesses and questions thereafter.

---

W4: Perhaps the most notable contribution of the paper is its manual curation of the dataset. However, this process remains underexplained.

We apologize for the lack of clarity here. The word “curate” was poorly chosen in the contribution section. We hereby clarify the dataset creation process.

All tasks were manually designed, from scratch, by the authors of this work without resorting to external annotators or LLMs. First, we identified high-quality sources of public time series data from various application domains (see Section 3.1). Special care was taken to find data sources that are continuously updated to facilitate future benchmark updates. Second, we established the categorization for sources of context (Section 3.2) and capabilities (Section 3.3) as a framework to guide the creation of new tasks and ensure their diversity. Third, team members created the tasks, each time selecting a data source, implementing a time series window selection strategy (e.g., short or long history), brainstorming about context types and capabilities required to solve the forecasting problem, writing a code to generate the context (e.g., calculating statistics of the series beyond the observed numerical history), and finally, if required, writing code to modify the time series data to reflect the context (e.g., introducing some spikes in future values).

Then, the tasks were peer-reviewed by a committee with significant expertise in time series forecasting (co-authors of this work). The creator of each task was not allowed to participate in the review. The review ensured that the text was of high quality, that it undoubtedly enabled a better forecast, and that the context source and capability tags were well-assigned. If a task was deemed of not high enough quality, it was either returned for revisions, or rejected.

The code for all tasks and experiments is available here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/. An example task can be found here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/cik_benchmark/tasks/montreal_fire/short_history.py, where the time series window selection occurs from L94-112 and the context generation occurs from L114-158.

We will add the same discussion to our paper in the next revision.

---

Q2: While I believe it would be handled in a discussion of the manual curation process, I wanted to know if the entire manual curation for so many datasets was done by the authors who would be credited for the paper or if any form of crowdsourcing was utilized for the manual tasks. Were the annotators paid fairly for their efforts if any crowdsourcing was utilized? Was any LLM used during the manual curation process?

The above answer should clarify that no annotators, crowdsourcing, or LLMs were used to either delegate or automate the creation of tasks in this benchmark. It was manually created by time series experts to ensure its quality and relevance, and all are credited as authors of the paper.

Dear reviewer,

Thank you again for your feedback; the manuscript has undoubtedly improved based on your suggestions. Please find below all relevant changes, which are in blue in the paper for visibility.

We have added a section on the task creation process in appendix A.2., and added references to it in multiple parts of Section 3.

We have added a section in appendix A.3., on the LLM-based critique on the relevance of context, to further validate the quality of the benchmark, and added an reference to it in Section 3.

We have added an overview of the distribution of the lengths of the natural language context, numerical history and prediction horizon in appendix A.7, and added an reference to it in Section 3.

Finally, we have added a reference to the open-source code for all tasks and experiments available here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/ in Section 3.

---

We would also like to point out that the paper has several other new additions, such as the inclusion of TimeGEN (a foundation model), ETS and ARIMA (statistical models) in the benchmark, a study on the impact of having both relevant and irrelevant information in the context in Appendix C.7. etc.

---

We would like to make sure your comments and concerns have been addressed in the updated manuscript. Please let us know if you need any additional clarifications.

Dear Reviewer kJ9d,

We want to thank you for the time you took to review our work. We appreciate the suggestions and the detailed feedback, and we have carefully worked to address any concerns.

As the discussion period comes to a close, if you feel that your concerns have been addressed, we would greatly appreciate any change to your score. If you require any additional clarifications, we are happy to provide them.

Thank you again for your time and for providing thoughtful comments.

Kind regards,

The Authors.

We thank the reviewer for the positive evaluation of the work, and for highlighting the clear presentation, the diversity of models and prompting techniques, the importance of the introduced RCRPS metric and the rigorous evaluation. We are pleased that you consider that our work provides a good foundation for an important question in the research field. We are glad to clarify the points brought up in the review.

---

W1: From the experimental results, we can see that text provides a good auxiliary role, but this method should be limited to models with LLM as the backbone.

Indeed, for a model to perform well on the CiK benchmark, it must effectively process natural language context. Currently, the top-performing models are large language models (LLMs) that are directly prompted to generate forecasts. This surprising result may stem from their ability to understand context, their enhanced reasoning capabilities, and their training for sequence completion—a task similar to time series forecasting.

However, future methodologies need not be restricted to models with LLM backbones. Several time series models can utilize context provided as numerical covariates. If there is an effective way to convert natural language into numerical representations (such as text embeddings), it could be possible to leverage these models to generate accurate context-aware forecasts, even without an LLM backbone. While it remains to be seen how the field will address this challenge, one certainty is that the CiK benchmark will provide the community with a reliable means to measure progress toward this goal.

---

Q1: Are all the time series is univariate?

Yes, all tasks in the benchmark require forecasting univariate time series. This choice was made for simplicity and, since we wanted to focus on evaluating the ability to use natural language context to improve forecasts. Introducing additional challenges, such as modeling multivariate dependencies, would have complexified the result analysis, by entangling failure to use context with failure to model multivariate dependencies or challenges in assessing the quality of multivariate forecast distributions (see Marcotte et al., 2023 for a discussion of pitfalls of multivariate metrics). Moreover, existing benchmarks, such as the datasets in the Monash Time Series Forecasting Archive (Godahewa et al. 2021) are well-suited to evaluate multivariate forecasting and can be viewed as being complementary to CiK. Nevertheless, we agree that including multivariate tasks is a natural extension to CiK and mention it in the “Future Work” section of the discussion.

References:

1. Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, and Nicolas Chapados. "Regions of reliability in the evaluation of multivariate probabilistic forecasts." In International Conference on Machine Learning, pp. 23958-24004. PMLR, 2023.

2. Godahewa, Rakshitha, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. "Monash time series forecasting archive." arXiv preprint arXiv:2105.06643 (2021).

---

Q2: In this benchmark construction, have you tried very refined text rather than information at a specific time step or overall? How did it perform?

For some tasks in the benchmark, the context only applies to specific timesteps in the prediction region, and the context explicitly refers to these regions. Two such tasks are the Electricity Consumption Increase task (shown in App. B.2; visualized here) and the ATM Maintenance task (shown in App. B.3; visualized here). In both these tasks, the context was manually crafted and the time series was modified at the specific timesteps where the context would apply. In such tasks, we highlight the region where context is necessary, as the region of interest (as shown in dark green shade in the figures) which is taken into consideration in the evaluation metric (as explained in the Region of interest (RoI) paragraph of Section 4). More example tasks from the benchmark report: Example 1, Example 2, Example 3.

The benchmark also contains tasks where the context refers to all timesteps, such as containing constraints that apply to all timesteps (example in App. B.1; visualized here), is descriptive and contains intemporal information about the variable (examples in App. B.4 and visualized here; another example in App. B.5 and visualized here) or containing causal information (referring to dependencies between variables and covariates). These tasks ensure that the context is necessary for accurate predictions, by limiting the history to be insufficient for the prediction horizon. More example tasks taken from the benchmark report: Example 1, Example 2.

---

Q3: Regarding retrieval, if I use the time series segment corresponding to a specific text as the retrieval "text", will the performance be better? Because this is more direct.

All tasks in the benchmark are designed such that the text is absolutely necessary to be taken into account. We write the context ourselves to ensure its relevance (see our answer to W1 for the full task creation process). There are tasks that require the model to choose the most relevant text from the given textual context, as opposed to providing the filtered, relevant text directly. These tasks are tagged under the Retrieval from Context, (see tasks under “Retrieval:context” in the benchmark visualization website). Rightly, we find that when contexts are already filtered to only have the relevant context, models perform better at the task.

For example, we compare two tasks on predicting the Unemployment Rate of a county. For the UnemploymentCountyUsingSingleStateData task, we give context relevant to the variable (Unemployment Rate of the State in which the county is) - this task is visualized here. In the UnemploymentCountyUsingExpliciteMultipleStateData task, in addition to the context about the relevant state, the context includes unemployment rates of 2 other states - task visualized here. This figure visualizes the difference in performance in these tasks for 3 different models. We find that models perform much better when only the relevant state's data is provided, as opposed to the context also containing data from other states.

Dear reviewer,

Thank you again for your feedback; the manuscript has undoubtedly improved based on your suggestions.

We have added an explicit note on the univariate nature of tasks in our benchmark, and suggest multivariate tasks as a natural extension in Section 7.

We have added a section in appendix C.7. on the impact of relevant and irrelevant information in the context, to study if models perform better on context that has already been filtered to only contain relevant information. A reference to appendix C has been added in Section 5.3.

---

We would also like to point out all other major changes in the paper below.

We have added a section on the task creation process in appendix A.2., and a section on the LLM-based critique on the relevance of context in appendix A.3., to further validate the quality of the benchmark.

The paper has several other new additions, such as the inclusion of new models: TimeGEN (a foundation model), ETS and ARIMA (statistical models) in the benchmark, an overview of the distribution of the lengths of the natural language context, numerical history and prediction horizon in appendix A.7. etc.

Finally, we have added a reference to the open-source code for all tasks and experiments available here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/ in Section 3.

---

We would like to make sure your comments and concerns have been addressed in the updated manuscript. Please let us know if you need any additional clarifications.

Dear Reviewer 6GZP,

We want to thank you for the time you took to review our work. We appreciate the suggestions and the detailed feedback, and we have carefully worked to address any concerns.

As the discussion period comes to a close, if you feel that your concerns have been addressed, we would greatly appreciate any change to your score. If you require any additional clarifications, we are happy to provide them.

Thank you again for your time and for providing thoughtful comments.

Kind regards,

The Authors.

Thank you for your thorough evaluation, which recognizes our manuscript’s clarity, the novel direction that our benchmark opens for multimodal prediction, the new RCRPS metric, and the robustness and real-world relevance of our benchmark. We hope our responses clarify your concerns, especially regarding soundness, and that you will consider revising your score. If not, please let us know and we will do our best to clarify.

---

W1: Missing Information on Context Annotations

We apologize for the lack of clarity in describing how tasks were created. In particular, the term “curation” used in the contributions section was poorly chosen. We clarify the dataset creation process, including the generation of textual contexts, below.

We, the authors, designed each task ourselves by handwriting context to accompany manually chosen sources of time series data. In particular, we did not use any external annotators for this work. First, we sought publicly available time series data across various domains. Then, we developed the framework around sources of context (Sec 3.2) and capabilities (Sec 3.3) to encourage the creation of a diverse, comprehensive set of forecasting tasks. Each task was created separately by team members, who would 1) select a time series data source and brainstorm about a context-aided forecasting task pertinent to that data source, 2) implement a selection strategy for a window according to the task idea, 3) brainstorm about context types and capabilities that could help address the particular forecasting problem, 4) write a template for the context and any accompanying code to instantiate the template (e.g. statistics on past values to be included in the text), and finally, if the task required it, 5) modify the time series to reflect the context (e.g. adding a spike to the ground truth in the forecast region that would be only predictable by the textual context).

Tasks, when created, were peer-reviewed by a committee of time series forecasting experts who are co-authors of this work. To ensure the validity of this peer-review, the original creator of a given task did not participate in the committee. This review process ensured that forecasting experts, when faced with a task for the first time, could validate whether the context 1) could improve the forecast accuracy if used properly, 2) was accurate and comprehensible, and 3) that the context sources and capabilities assigned to a given task were appropriate. If the task was deemed of not high enough quality, it was either returned for revisions or rejected.

To further illustrate how task instances are generated, the reviewer can inspect the code for all tasks and experiments here: https://anonymous.4open.science/r/context-is-key-forecasting-E391. An example task can be found here: https://anonymous.4open.science/r/context-is-key-forecasting-E391/cik_benchmark/tasks/montreal_fire/short_history.py, where the time series window selection occurs from L94-112 and the context generation occurs from L114-158. Additionally, please see point W1 of our response to reviewer kJ9d, where we describe a newly performed LLM-based analysis of the relevance of the context for each task, which indicates that the context enables better forecasts (significantly better for most tasks).

We will add the same discussion to our paper in the next revision.

We hope that this resolves any outstanding concerns. Otherwise, please let us know and we are happy to expand.

W2: Limited benchmark novelty

Thank you for sharing these additional references, we will revise the related work section to include them. While we do agree that these works are related to CiK, we reiterate a key difference that supports the novelty of this work: CiK is the first benchmark where the tasks are carefully designed such that the text context contains information that is crucial to accurate forecasts. Previous works assembled large collections of time series and text, but the crucial nature of the text is not guaranteed. We detail the exact differences between CiK and these works below:

For example, [1][2] include 1) contextual information that is programmatically scraped and filtered through, e.g., regex, and 2) a large volume of possibly irrelevant contextual information, such as the majority of earnings calls audio and transcripts. While such automated approaches to dataset construction are useful for generating a large dataset in a scalable manner (e.g., for training purposes), the added value of the contextual information is uncertain, hence one cannot reliably evaluate context-aided forecasting with it.

This is the issue that we aimed to avoid when creating CiK, where all tasks are meticulously handcrafted using real-world data to ensure reliability. Hence, we believe our work is complementary to related work, and addresses an important quality gap in existing benchmarks, introducing novel elements such as a rigorous framework for manually generating context-aided forecasting tasks with various sources of context and capabilities required to leverage it, broad domain diversity (e.g., compared to the single domain focus in [1, 2]), and a new metric specifically designed for context-aided forecasting whose relevance has been emphasized by other reviewers.

In our next revision of the paper, we will extend the related work section by including and discussing the differences between CiK and the two additional references you provided. We will also include the exploration of more scalable approaches to high-quality benchmark design for context-aided forecasting in the Future Work section.

---

W3: Ambiguous Task Type Annotations

Tasks are categorized according to three characteristics: domain, context sources, and capabilities. Examples of tasks and justifications for their domain, context source, and capability categorizations are given in Appendix B to help the reader's intuition.

Domains are fully determined based on data provenance and are outlined in Appendix A.1. Context sources, defined in Section 3.2 and exemplified in Fig. 3, refer to various aspects of the underlying temporal process from which the context draws information. Finally, capabilities, defined in Section 3.3, reflect high-level capabilities that are required to translate the context into correct forecasts. Exact definitions of the capabilities can be found in Appendix A.4. It is true that capabilities are subject to debate and that achieving a perfect categorization is perhaps not possible. We used the peer review process previously described to arrive at these conclusions (which will be clarified in the paper). However, we view these as being informative as opposed to prescriptive and they serve two purposes: 1) they provide us with a conceptual scaffolding for designing new tasks by highlighting cognitive capabilities to test and 2) they offer structured dimensions for analysis, allowing us to evaluate model performance and identify limitations across different aspects of a forecaster's capability.

As for your question regarding the public safety (or MontrealFire) tasks, these were not classified as instruction following since the context is purely descriptive of the situation, rather than giving explicit instructions like “suppose that” as in the ElectricityIncreaseInPredictionTask or “consider that” in the CashDepletedinATMScenarioTask. We hope that this explanation, combined with the definitions and examples in the appendix adds clarification and resolves your concerns. Please let us know.

W4: Unexplained Results Discrepancies

Figures 4 and 5 show results that seem contradictory for Mixtral, but are compatible and due to a phenomenon we have observed: adding context improves the model’s RCRPS for most tasks but greatly worsens it for a minority of tasks (see Sec. 5.4, significant failures). The improvement on the majority of tasks is thus visible in Figure 5, with an increased number of tasks for which the model with context becomes better than all quantitative baselines. But the significant performance degradation (in absolute terms) on the few other tasks dominates the aggregated RCPRS value shown in Figure 4.

To better illustrate this effect, we have generated a figure which shows the impact of adding context to Mixtral-8x7B-Inst on the RCRPS (the lower, the better). While adding context (blue) increases the frequency of RCRPS values close to 0, it also adds a fat tail of RCRPS higher than 2. The former is why adding context improves the model win rate, while the later explains why adding context worsens its aggregate RCRPS.

Thank you for pointing out this pair of results. We believe it motivates reporting both the aggregate RCRPS metric and the win rate across tasks, which offer complementary perspectives on the results. It also strengthens our claim for the need for future work on more robust models for context-aided forecasting (Sec. 5.4). We will also include a discussion on this set of results in the appendix in the next revision of the paper.

---

W5: Limited model variety

We acknowledge that evaluating on a diversity of model sizes is important, which is why we evaluated multiple sizes of models (8B, 70B, 405B) and reported an efficiency-performance analysis.

Following the reviewer’s suggestions, we experimented with the following models with Direct Prompt: Qwen-2.5-0.5B-Instruct, Qwen-2.5-1.5B-Instruct, Qwen-2.5-3B-Instruct, Qwen-2.5-7B-Instruct, Mistral-2.5-7B-Instruct, Falcon-2.5-7B-Instruct.

and the following models with LLMP: Qwen-2.5-0.5B, Qwen-2.5-1.5B, Qwen-2.5-3B, Qwen-2.5-0.5B-Instruct, Qwen-2.5-1.5B-Instruct, Qwen-2.5-3B-Instruct.

The updated results are attached in this figure. We explain them below.

Results with the Direct Prompt method:

We find that with the Direct Prompt (DP) method, Qwen-2.5-7B-Instruct especially seem to outperform many other models as per the average RCRPS metric, and also obtain significant improvements in performance when provided with the context. The models achieves the third best score with the Direct Prompt method, significantly outperforming other, more expensive models. We thank the reviewer again for suggesting us to run this model.

We found that Mistral-7B-Instruct does not output accurate forecasts and further suffers from significant failures when provided with the context (worsening its performance), similar to Mixtral-8X7B-Instruct. This indicates that the specific model family used indeed may have an impact on the robustness of the model for context-aided forecasting with the Direct Prompt method.

All other models (Qwen-2.5-0.5B-Instruct, Qwen-2.5-1.5B-Instruct, Qwen-2.5-3B-Instruct, Falcon-2.5-7B-Instruct) fail to follow the template required in the output, which impedes proper evaluation. Instead, despite these models being instruction-tuned, the models produced unrelated information in response to the Direct Prompt prompt, for example, see the output of the Qwen-2.5-0.5B-Instruct model here. This may indicate that the Direct Prompt method would need models of a certain size to work with, to leverage their instruction-following capabilities to output valid forecasts.

Results with the LLMP method:

We found that all tested models (Qwen-2.5-0.5B, Qwen-2.5-1.5B, Qwen-2.5-3B, Qwen-2.5-0.5B-Instruct, Qwen-2.5-1.5B-Instruct, Qwen-2.5-3B-Instruct) achieve mediocre performance both with and without context, much worse than previously tested models. Notably, all these models except Qwen-2.5-3B and Qwen-2.5-3B-Instruct degrade with context and obtain significant failures, which may indicate that the LLMP method may need models of a certain size to work well for context-aided forecasting.

We tried to run Qwen-2.5-7B, Qwen-2.5-7B-Instruct, Mistral-2.5-7B-Instruct and Falcon-2.5-7B-Instruct however these models took up a high amount of compute and time to run, therefore we could not finish them in time for the rebuttal. We will add them to the camera-ready and analyse them wherever required.

We will add all these results to the next revision to the paper.

Q1: How the textual contexts were annotated, including any guidelines, annotator expertise, and inter-annotator agreement (IAA) metrics?

Please see our response to W1 for the details.

---

Q2: Do the authors envision methods to automate or partially automate this process, such as using existing NLP techniques to generate context?

The process was completely manual. We believe that the manual task creation process ensures a high quality benchmark. Please see our response to W2 for the details.

---

Q3: Could the authors provide explicit definitions for each capability and clarify the criteria used to categorize tasks?

Please see our response to W3 for details on the task type annotations. For a full description of the task creation process, please see our response to W1.

---

Q4: Could the authors clarify whether this discrepancy suggests limitations of the CRPS metric or other factors in the benchmark design?

Please see our response to W4 for more details. In short, we believe that this set of results actually credits the RCRPS as a metric that captures the quality of a forecast (how far the forecast is from the ground truth, in an absolute sense), as opposed to a task-wise win rate, which captures in how many tasks the model wins over the other models. Especially in the case where models fail significantly and these failures dominate the aggregate RCRPS (see Sec 5.4), the win rate provides a different view and shows the models still are useful and perform well across many tasks.

Annotation Quality

• Inter-annotator agreement (IAA) is not an established practice in multimodal forecasting. IAA is used in only a single paper unrelated to forecasting [1], out of 7 discussed related works [2-8], including two papers put forward by the reviewer [7,8] and two papers on multimodal forecasting published at the NeurIPS [2] and EMNLP [3] conferences, both of which rely on manual curation. Furthermore, in [1] the reported IAA is motivated by finding agreement between inherently subjective author preferences between two answers that they manually designed for a task in a meta-evaluation benchmark on human preference datasets. However, no clear instructions or criteria for evaluation are reported, apart from objectivity. Therefore, our formalized task creation and peer review validation process (Appendix A.2) not only shares elements with that put forward in the suggested paper that uses the IAA [1], but in effect supersedes it because tasks were not admitted to the benchmark without the unanimous agreement of all co-authors (excluding the task creator).
• The use of IAA as in [1] is not scalable, because it is human-based, despite the reviewer claiming that lack of scalability is a limitation of our work.

---

Novelty and Relevance

• It’s unclear why the reviewer lists two strengths of the benchmark as being “uniquely requir[ing] essential textual context for forecasting, marking a new direction in multimodal prediction” and being “robust, with real-world tasks”, while subsequently insisting on the importance of “low-quality or noisy data, which are far more common in real-world applications [no citation provided]”. We welcome any constructive feedback with references to the appropriate literature.
• “... the authors argue that high-quality contexts are crucial for accurate forecasting”. This is misreported. We argue that high-quality contexts are crucial for assessing how well models perform context-aided forecasting, to confidently attribute failures to models instead of poorly-defined tasks.

---

Unfounded Instruction-Following Definition

• Despite the reviewer claiming insufficient motivation for our purely informative definition of instruction following, the structured approach in [1] commended by the reviewer for its objectivity offers no support or motivation for their definition of instruction-following. Further, they use the IAA for establishing agreement on preferences, not for defining instruction-following.
• The validity of our benchmark is unaffected by this definition, which we use informatively for result interpretation.

---

Appropriateness of the RCRPS

• As the RCRPS is derived from the CRPS metric, it accounts for distributional errors for each task instance. It is trivially true that the aggregate RCRPS misses distributional aspects by the definition of metric aggregation. Such aggregations are standard in the literature [2,3], and are not aware of alternatives for the complementary averaging and the win-rate aggregations we present, to provide a more concise view of model performance. We welcome constructive feedback and references in this direction.
• “...RCRPS as a comprehensive evaluation metric…” This is misreported. We introduce the RCRPS as a novel metric based on the well-established CRPS metric, but augmented specifically for context-aided forecasting. We also report the average rank, win rate and visualizations of model failures, to provide complementary views into model performance.
• Please provide a citation for the inappropriateness of the win rate.

---

Larger Models

• The reviewer’s subsequent criticism on the omission of larger models (after their initial request for smaller models) is unfounded. Not only do we evaluate models from diverse families up to 405B parameters, but we also evaluate a diversity of model sizes and families that goes well beyond the diversity exhibited by the related literature, including the small models specifically requested by the reviewer. Existing publications either do not evaluate open-source models exceeding 8B parameters [2] or do not run forecasting evaluations on models outside the closed-source GPT family [3].

Response to Reviewer 7GBq's Comments

We respectfully disagree with the reviewer, as our benchmark and evaluation’s rigor, reliability, and reproducibility are supported by established practices in the related literature. More precisely:

• The reliability of our benchmark is supported by a triplet of strategies: the rigorous peer review process similar to that of the reviewer’s proposed reference [1] and described in Appendix A.2, the LLM-based critique that reviewer kJ9d requested, and the objective evaluation of forecasts against a numerical ground truth, including the additional results that the reviewer asked for.
• The reproducibility of our evaluation is streamlined by the code and data releases, as is standard in the literature (https://anonymous.4open.science/r/context-is-key-forecasting-E391). We went a step further by providing a task visualization interface, to ease the task inspection by the community (https://anon-forecast.github.io/benchmark_report_dev/).
• The rigor of our evaluation methodology exceeds or matches that of any previous work in multimodal forecasting [2-8]: we make use of a scoring rule that evaluates the entire predictive distribution rather than focusing solely on summary statistics (unlike MSE and MAE [2,3]) (Section 4), we discuss and motivate all aggregation and normalization choices (Section 5.1, Appendices A.4 and E.1), we report and analyze examples of model successes and failures (in appendix C.3, C.4 and C.5), and we provide different views for assessing model performance (notably reporting results by context type, capability, with/without context).

In the following comment, we counterargue the reviewer’s points and stress that they do not constitute grounds for rejection: