MIRAI: Evaluating LLM Agents for International Event Forecasting

Comparison of Retrieval Strategies with Agent: Both baseline approaches—CAMEO context augmentation and RAG methods—employ static, predefined retrieval strategies that are fixed for all queries and executed only once per query.

In contrast, our agentic approach enables dynamic, multi-step information gathering and reasoning. The agent can replicate the baseline retrieval strategy by fixing certain API parameter values, for example, the agent can use function call get_events(head_entities=[s, o], tail_entities=[o,s]) to get the retrieved context as RAG Event-Only, and use function call get_news_articles(text_description='(t, s, ?, o)') to get the retrieved context as RAG News-Only; its capabilities extend far beyond these static approaches through its flexible parameter settings for each function call and multiple steps per query.

This multi-step, adaptive approach represents a fundamental shift from static retrieval to dynamic information gathering and reasoning, though it introduces higher requirements for the LLM's planning ability in:

• Automatically selecting optimal information-gathering strategies
• Integrating and reasoning over information of different formats
• Adjusting strategies based on intermediate findings and current context

Key Findings and Analysis: Our experiment results reveal several important insights:

• Performance of the RAG Baselines RAG demonstrates improved precision over Direct IO (15.4% Pre in second-level) when using either event (32.4% Pre in second-level) or text (19.5% Pre in second-level) information source independently, with event data contributing more significantly to recall (43.9% Rec in second-level). When combining the two information sources, RAG achieves higher precision (36.4% Pre in second-level) but with a lower recall than RAG Event Only (from 43.9% to 38.8%), leading to a lower overall F1 score (from 33.2% to 32.1%), this suggests that a simple combination of both information in the context not effectively and collaboratively contribute to a better forecasting performance.

• Comparison and Insights for Agent
  ReAct agents exhibit similar performance patterns with RAG when using different information sources (changed by the type of API functions available for the agent). Specifically, event data also contributes to high precision for the agent (62.8% Pre at first-level and 44.2% at second-level), outperforming RAG baselines. The structured event data consistently provides stronger signals for forecasting across both approaches, likely due to its standardized format.

  However, agent baselines generally obtain a lower recall than RAG baselines. Meanwhile, while ReAct with full API access achieves higher average F1 scores in second-level relation prediction than its partial access performance(31.1% than 30.7% and 12.5%), the benefits of combining information sources aren't consistent across all relation hierarchies and methods, similar to RAG, suggesting substantial room for exploring more effective information integration strategies.

  This reveals both promises and challenges of the agent's more flexible retrieval approach: RAG's predefined and fixed retrieval strategies can often yield stable performance, agent's dynamic and multi-step retrieval allows for flexible information gathering and integration, yet higher requirements for planning and reasoning sometimes also lead to relatively lower performance.

These observations underscore the core purpose of our benchmark: not just to compare current methods but to encourage the development of more advanced agentic forecasting approaches. The current performance patterns suggest significant opportunities for improving agent architectures, particularly in:

• Developing more robust and automatic planning strategies for multi-step information gathering
• Improving information integration capabilities across different information sources, formats, and temporal scales

O5. Having the RAG baseline would be very insightful in knowing if one needs the API interface.

Thank you for this valuable suggestion for incorporating RAG-based methods into our dataset’s evaluation. We have now added RAG evaluations to provide additional context. We also want to note that retrieval is inherently a critical tool for LLM agents [2, 3]. RAG represents a specific type of augmented LLM, and the key difference is that RAG always performs retrieval augmentation for generation while agents automatically choose when to use the tool (see our detailed discussion below).

Follow the suggestion, we conduct additional experiments with different RAG methods. The result table is as follows:

Table: Evaluation results of GPT-4o-mini on the 2024-02 test split using different non-agentic methods and the ReAct agent with Single Function action type. The best-performing score is highlighted in bold and the second-best is in italics.

Methods and Experimental Setup: Besides the Direct IO / QAand three ReAct agent with different tool-use that we already implemented in the paper, we add the following baselines:

• Direct QA with Augmentation (for comment Q4)

  • QA with CAMEO: We provide the CAMEO ontology in an ordered dictionary format mapping relation codes to their names and detailed descriptions. This is closer to the QA-format the authors mention, and a more fair comparison as the model can refer to the output vocabulary without needing to memorize CAMEO codes.

• RAG Methods
  Following recent work, we implement three RAG variants:

  • RAG Events-Only: Following GPT-NeoX-ICL[4], we explore rule-based approaches for retrieving historical facts. Using the 'Pair' and 'Undirectional' setting, given a query event $(s, ?, o, t)$, we retrieve historical events $(s, r \in \mathcal{R}, o, <t)$ and $(o, r \in \mathcal{R}, s, <t)$. Events are sorted by recency with a cap of 30, aligning with the default cap of the get_events API function.
  • RAG News-Only: Following TCELongBench[5], we employ BM25 retrieval to fetch the most query-relevant news articles before the query date. The top 15 news articles are retained, matching the default cap of the get_news_articles API function.
  • RAG All: Combines both retrieved structured events and textual news articles.

[2] Tool Learning with Large Language Models: A Survey
[3] A Survey on Large Language Model based Autonomous Agents
[4] Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning.
[5] TCELongBench: Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding.

O1. Some figures are hard to read, especially in black and white (2, 4, 5, 7, 8)

Thanks for the suggestion. We have revised these figures using more distinguishable color schemes, hatching patterns, line style, and markers in the updated manuscript.

O2. Human evaluation of the dataset is done in the appendix. Is it enough to only evaluate 51 events? Besides, 82% seems quite low.

Thank you for raising this important point about our human evaluation process. The 51 events evaluated correspond directly to the ground truth events derived from our human forecasting queries, encompassing 103 associated news articles. This selection ensures a diverse and meaningful subset of the dataset for evaluation.

While we acknowledge that expanding the evaluation scope would provide additional insights, the current sample size reflects the resource-intensive nature of this process. Each event requires careful analysis of multiple news articles within their temporal and contextual nuances. Furthermore, accurate application of the CAMEO ontology is a non-trivial task, necessitating meticulous attention to detail to appropriately classify real-world events.

Our evaluators, while equipped with the CAMEO ontology, were not domain experts in political science, which likely limited their ability to achieve the precision that specialized experts could provide.

It is worth noting that previous work [1] evaluated raw GDELT event data using GPT-4 on a sample of 100 randomly selected news articles from the Middle East, reporting a precision of only 42.5%. In contrast, the 82% accuracy achieved in our human evaluation demonstrates the effectiveness of our data cleaning process and highlights the quality of our dataset. While there is room for further improvement, these results provide strong evidence for the validity of our methods.

[1] SCTc-TE: A Comprehensive Formulation and Benchmark for Temporal Event Forecasting.

O3. The authors include the confidence intervals but do not discuss them. In many cases, there are overlaps between the baseline, making the conclusions unclear.

Thank you for highlighting the need to discuss confidence intervals in our results. We acknowledge that overlapping confidence intervals between models warrant careful interpretation. We have conducted additional statistical analysis to provide more rigorous support for our key conclusions from the paper:

1. Our first main finding that "Code Block benefits stronger LLMs but hurts weaker models" is supported by clear patterns even when considering confidence intervals:

   • For GPT-4o-mini, the adoption of Code Block shows consistently higher mean values with smaller standard deviations in second-level relation prediction tasks, indicating both improved performance and increased stability.
   • In contrast, smaller open-source LLMs demonstrate consistently decreased performance with Code Block, with the negative impact remaining significant even when accounting for confidence intervals.

2. Our second main finding that "GPT-4o-mini outperforms other models" is supported by statistical evidence. While there are some overlaps between confidence intervals (e.g., between GPT-3.5 and GPT-4o-mini), additional statistical testing validates this conclusion. For instance, a Welch's t-test comparing second-level F1 scores between these models yields p=2.45e-06 < 0.05, confirming that GPT-4o-mini's superior performance is statistically significant.

This additional statistical analysis provides a clearer and more rigorous interpretation of our results.

O4. Adding the human evaluation in Table 2 is only one line, and it would be insightful.

In response to your comment, we have incorporated the human forecasting performance results into the main paper (page 7 Table 2). Additionally, we note that the human evaluation was conducted on a subset of 51 test events due to resource constraints and the time-intensive nature of expert evaluation. This is also noted in the caption of Table 2.
Moreover, for fair comparison, we have run all LLM agent experiments on this same subset, with results presented in Table 6 of the Appendix D.4. The results show that human performance surpassed that of LLM agents in most metrics, especially in recall. This highlights significant room for improvement in LLM agents.

We are grateful for your support and suggestions, for which we have included additional experiments and discussions accordingly. Please find our responses below:

W1. The paper heavily relies on the appendix. Previous work and human evaluation are in the appendix.

Thank you for this suggestion. We have streamlined the Appendix by:

1. Removing the full API documentation and implementation details and providing a link to anonymous github repo.
2. Retaining only the key list of data classes and functions defined in the API that directly support the paper’s contribution in the agentic environment and the code-based tool use setting.

For related works, as our paper primarily introduces a new benchmark designed to facilitate the development and evaluation of agent forecasting methods, the primary competitors in the related work are existing benchmarks in this domain. Therefore, in the main content of paper, we made comparison to these most relevant benchmarks in Table 4 and its following section. By comparing our work with these existing benchmarks, we demonstrated how our benchmark is unique and addresses current gaps in the lack of dynamic data updates, diverse information sources, and agentic evaluation. Extended related work in the appendix focus more on the background of previous non-agentic forecasting methods and agentic/ code-based LLM work.

For human evaluation, as suggested, we have incorporated the human forecasting performance results into main paper (page 7 Table 2). Additionally, we note that the human evaluation was conducted on a subset of 51 test events due to resource constraints and the time-intensive nature of expert evaluation. This is also noted in the caption of Table 2.

W2. The technical contribution of the paper could be clearer. The baselines are coming from previous works.

Thank you for suggesting we clarify the technical contributions of our paper. We would like to emphasize that as a benchmark work, our primary contribution is not in developing novel data processing tools or baseline methods, but rather in constructing the first comprehensive benchmark for evaluating and advancing LLM agents' capabilities in event forecasting. Our benchmark makes several key technical contributions:

1. We design and implement the first agentic environment specifically for temporal forecasting, featuring carefully constructed APIs that enable systematic evaluation of agents' abilities in information integration, tool use, and reasoning. This fills a critical gap in the field where no previous benchmark has specifically targeted the evaluation of LLM agents' forecasting capabilities.
2. Our benchmark uniquely combines structured events and unstructured news within an agentic framework, allowing us to assess how effectively agents can gather and integrate diverse information sources. While we leverage existing data sources, our technical contribution lies in creating an interactive environment that enables agent-driven information exploration and integration—a capability not present in traditional static datasets.
3. We implement a dynamic pipeline that supports regular updates and ensures contamination-free test splits. This technical design choice directly addresses a critical challenge in LLM evaluation: maintaining benchmark relevance as models evolve while preventing data contamination.

You may refer to our global rebuttal comments for a comprehensive discussion of our contributions.

W3. The scope of the task is very narrow: Predicting the future interaction between two countries.

We respectfully disagree that the task scope is narrow. Our benchmark evaluates forecasting across the comprehensive CAMEO ontology, which provides a hierarchically organized framework encompassing the full spectrum of international interactions: The relation space is rich and diverse, including a progression of relation types from cooperative to conflictual interactions, providing a nuanced space for evaluating models' ability to forecast shifts in international dynamics.

Additionally, our experimental results demonstrate that even this foundational task formulation poses significant challenges for both traditional and agentic methods. The performance analysis has already yielded valuable insights about current LLM agents' capabilities and limitations in temporal reasoning, tool utilization, and information integration.

While we acknowledge the potential for more complex task formulations (e.g., multi-party events, intra-country dynamics) and consider exploring them as future work, we believe establishing strong benchmarks for fundamental bilateral interactions is crucial before advancing to more complex scenarios. By constructing MIRAI as a focused yet comprehensive evaluation framework, we aim to drive systematic progress in agentic forecasting methods while maintaining clear metrics for measuring advancement.

Dear Reviewer SzJ2

We greatly appreciate your thorough review and the insightful comments you've provided. Here is an overview of our responses to your feedback:

• Adjustment of Appendix: We have streamlined the appendix by removing extensive API details and moving them to a GitHub repository, while keeping essential information in the paper. Human evaluation results have been moved to the main document for visibility.
• Highlight of Technical Contribution: We've clarified that our core contribution is the establishment of an agentic benchmark for event forecasting, focusing on the design of an interactive environment with diverse information sources and a dynamic data pipeline. We summarized our contribution in our global comments.
• Scope of the Task: We clarified that the task of hierarchical multi-relation forecasting is broad in its diversity, challenging for current models, and provides valuable insights. We see this as a foundational step before scaling to more complex forecasting.
• Figure Readability: Figures have been revised for better readability, especially for black and white printing.
• Human Evaluation: We justified the current evaluation scope, emphasizing the resource-intensive nature and complexity of the task. We compared to prior benchmarks, the accuracy highlights the effectiveness of our data cleaning process, demonstrating the quality of our dataset. We also moved human forecasting performance to Table 2 of the maim paper.
• Confidence Intervals Discussion: We've added statistical analysis to clarify the significance of our experimental results, addressing the issues with overlapping confidence intervals.
• Inclusion of RAG Baseline: We've evaluated multiple RAG baselines, which reveals that, unlike RAG’s fixed retrieval strategies, agents dynamically adapt their information-gathering and reasoning, offering greater flexibility, but also coming with higher challenges in multi-step planning and information integration.
• Minor Corrections: We've fixed the citation error and updated the color scheme in Figure 8b for better intuitiveness.
• News Articles: We highlighted the use metadata and titles with plans for further review of news content if full texts are needed.
• Balancing Test Split: We've explained how our balanced sampling maintains fairness and diversity in the test set while still reflecting real-world event distribution.
• Impact of Model Parameters: We've included experiments with different model sizes from the same family to demonstrate the influence of parameter count on performance.
• Direct IO Baseline: We've discussed how providing the CAMEO ontology affects performance, suggesting that model knowledge and instruction-following are critical, not just the format of the output.

We hope these clarifications and additional experiments address your concerns and provide a stronger foundation for our work. If there are any further points you would like us to address or clarify, please let us know. Thank you again for your time and effort in reviewing our work.

Sincerely,
Authors

Dear Reviewer SzJ2,

I hope this message finds you well. We recently submitted our rebuttal and would like to kindly request your feedback on our responses.

We understand that your schedule is demanding and greatly appreciate the time and effort you dedicate to the review process. Your insights are invaluable to us, and we are eager to address any further questions or concerns you may have.

Thank you for your attention to this matter. We look forward to your response.

Best regards,

Authors

Dear Reviewer SzJ2,

Thank you again for taking the time to review our paper. We greatly appreciate your thoughtful feedback and your support and recognition of our work, particularly regarding the comprehensive description of the data and code, extensive experiments, insightful analysis, and clear overall writing with many illustrative examples.

In response to your feedback, we have conducted extensive experiments with augmented Direct QA and various RAG methods, and studied the effect of model parameter sizes. We have also provided detailed clarifications to each of your concerns in our rebuttal replies, with our paper revised accordingly. We hope our responses have comprehensively addressed your concerns.

With the discussion deadline approaching, we would appreciate your attention to our rebuttal and any further feedback you may have, as it will give us the opportunity to provide more details before the author-reviewer discussion session ends and help us continue to improve our work. Thank you again for your time and valuable comments!

Best regards,
Authors

We sincerely thank you for your constructive feedback. We have carefully considered each concern you raised and present detailed clarifications below:

W1. Only one concrete example is provided and discussed in the paper.

We would like to clarify that we provided comprehensive examples throughout the paper:

• In the main paper, Figure 1 provides a demonstrative example with discussions provided in lines 151-157.

• In the footnote of page 1, we further provided an interactive agent demo with MIRAI at https://drive.google.com/file/d/1kKvYdAYv5hed-sbF_QE1moP-dQbfYoXI/view.

• In Appendix F "Additional Forecasting Examples of LLM Agent", we detailed the entire LLM outputs showcasing the agent ReAct framework with code block (Appendix F.1) as well as with single-function (Appendix F.2). Each example includes the full context, agent's step-by-step reasoning, and final predictions.

  We placed these detailed additional examples in the Appendix to maintain focus in the main paper while providing interested readers with comprehensive documentation.

Specifically, these examples have well demonstrated the strength and uniqueness of our MIRAI benchmark and using LLM agents for forecasting. For exmaple, in the first step of the agent in example Appendix F.1, it generates the following thoughts:

"""To forecast the relations that Australia will take towards China on November 03, 2023, we need to gather historical data on the interactions between these two countries. This includes examining the frequency and types of relations observed in the past, as well as any recent trends or significant events that might influence future relations.

We will proceed with the following steps:

1. Retrieve Historical Events: Gather events involving Australia (AUS) and China (CHN) to understand the context and types of relations that have occurred in the past.
2. Analyze Relation Distribution: Analyze the distribution of different CAMEO codes in these events to identify the most common types of interactions.
3. Examine Recent Trends: Focus on recent events to detect any emerging trends or shifts in relations.
4. Contextual Analysis: Review news articles to gain insights into the context and underlying reasons for these relations.

Let's start by retrieving historical events between Australia and China."""

This example highlights how the MIRAI benchmark's rich, agentic environment enables LLM agents to:

• Automatically Plan Forecasting Strategies: The agent crafts a tailored approach based on the specific query, demonstrating an ability to dynamically adjust its strategy.
• Analyse and Integrate Diverse Information Sources: It leverages a variety of data from structured event to statistical distribution, and to textual news, showcasing its capability to synthesize different information through MIRAI's rich database for detailed analysis. This is also supported by the comprehensive APIs defined in MIRAI's agentic environment.

In the follow-up steps in this example, by adhering to the outlined steps, the agent not only gathers and analyzes data but also engages in reasoning to its forecasting answer, which is crucial for predictive accuracy.

In our revision, we additionally included two concrete examples with full reasoning traces of GPT-4o-mini based LLM agents in Appendix F.3 and F.4. We also make all of its reasoning logs and results, using Single Function or Code Block on the 2024-02 test split, publically available under this anonymous drive folder: https://drive.google.com/drive/folders/1fIVmv5EE-1qxvD8QRcj9K3QIIRpeTQPx?usp=sharing

W2. The work should have been also compared with RAG-based (Retrieval-Augmented generation) approaches. It is unclear how MIRAI's approach differs from RAG-based methods.

Thank you for this valuable suggestion for incorporating RAG-based methods into our dataset’s evaluation. We have now added RAG evaluations to provide additional context (see detailed results below). However, we respectfully note that our manuscript's core contribution is not proposing a forecasting method, but rather a benchmark dataset with an agentic environment for evaluating LLM agents' forecasting capabilities. We had explained our scope in the manuscript (lines 10-14, 21-25, and 90-96). Our goal is to establish a foundational evaluation framework that can drive future developments in this emerging field for agent-based approaches. Please refer to our global comment for a detailed discussion on our contribution.

We also want to note that retrieval is inherently a critical tool for LLM agents [1, 2]. RAG represents a specific type of augmented LLM, and the key difference is that RAG always performs the retrieval augmentation as part of generation while agents automatically choose whether and when to use the tool (see our detailed discussion below).

[1] Tool Learning with Large Language Models: A Survey
[2] A Survey on Large Language Model based Autonomous Agents

Follow the suggestion, we conduct additional experiments with different retrieval augmented generation (RAG) methods. The result table is as follows:

Table: Evaluation results of GPT-4o-mini on the 2024-02 test split using different non-agentic methods and the ReAct agent with Single Function action type. The best-performing score is highlighted in bold and the second-best is in italics.

Methods and Experimental Setup: Besides the Direct IO (Direct QA) and three ReAct agents with different tool-use that we already implemented in the paper, we add the following three RAG baselines following recent work:

• RAG Events-Only: Following GPT-NeoX-ICL[3], we explore rule-based approaches for retrieving historical facts. Using the 'Pair' and 'Undirectional' setting, given a query event $(s, ?, o, t)$, we retrieve historical events $(s, r \in \mathcal{R}, o, <t)$ and $(o, r \in \mathcal{R}, s, <t)$. Events are sorted by recency with a cap of 30, aligning with the default cap of the get_events API function.
• RAG News-Only: Following TCELongBench[4], we employ BM25 retrieval to fetch the most query-relevant news articles before the query date. The top 15 news articles are retained, matching the default cap of the get_news_articles API function.
• RAG All: Combines both retrieved structured events and textual news articles.

Comparison of Retrieval Strategies with Agent: RAG methods employ static, predefined retrieval strategies that are fixed for all queries and executed only once per query.

In contrast, our agentic approach enables dynamic, multi-step information gathering and reasoning. The agent can replicate the baseline retrieval strategy by fixing certain API parameter values, for example, the agent can use function call get_events(head_entities=[s, o], tail_entities=[o,s]) to get the retrieved context as RAG Event-Only, and use function call get_news_articles(text_description='(t, s, ?, o)') to get the retrieved context as RAG News-Only; its capabilities extend far beyond these static approaches through its flexible parameter settings for each function call and multiple steps per query.

This multi-step, adaptive approach represents a fundamental shift from static retrieval to dynamic information gathering and reasoning, though it introduces higher requirements for the LLM's planning ability in:

• Automatically selecting optimal information-gathering strategies
• Integrating and reasoning over information of different formats
• Adjusting strategies based on intermediate findings and current context

[3] Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning. EMNLP 2023.

[4] TCELongBench: Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. ACL 2024.

Key Findings and Analysis: Our experiment results reveal several important insights:

• Performance of the RAG Baselines RAG demonstrates improved precision over Direct IO (15.4% Pre in second-level) when using either event (32.4% Pre in second-level) or text (19.5% Pre in second-level) information source independently, with event data contributing more significantly to recall (43.9% Rec in second-level). When combining the two information sources, RAG achieves higher precision (36.4% Pre in second-level) but with a lower recall than RAG Event Only (from 43.9% to 38.8%), leading to a lower overall F1 score (from 33.2% to 32.1%), this suggests that a simple combination of both information in the context not effectively and collaboratively contribute to a better forecasting performance.

• Comparison and Insights for Agent
  ReAct agents exhibit similar performance patterns with RAG when using different information sources (changed by the type of API functions available for the agent). Specifically, event data also contributes to high precision for the agent (62.8% Pre at first-level and 44.2% at second-level), outperforming RAG baselines. The structured event data consistently provides stronger signals for forecasting across both approaches, likely due to its standardized format.

  However, agent baselines generally obtain a lower recall than RAG baselines. Meanwhile, while ReAct with full API access achieves higher average F1 scores in second-level relation prediction than its partial access performance(31.1% than 30.7% and 12.5%), the benefits of combining information sources aren't consistent across all relation hierarchies and methods, similar to RAG, suggesting substantial room for exploring more effective information integration strategies.

  This reveals both promises and challenges of the agent's more flexible retrieval approach: RAG's predefined and fixed retrieval strategies can often yield stable performance, agent's dynamic and multi-step retrieval allows for flexible information gathering and integration, yet higher requirements for planning and reasoning sometimes also lead to relatively lower performance.

These observations underscore the core purpose of our benchmark: not just to compare current methods but to encourage the development of more advanced agentic forecasting approaches. The current performance patterns suggest significant opportunities for improving agent architectures, particularly in:

• Developing more robust and automatic planning strategies for multi-step information gathering
• Improving information integration capabilities across different information sources, formats, and temporal scales

W3. Typos line 088: ".. is able to better utilizes and benefits... " -> ".. is able to better utilize and benefit... "

Thank you for catching this grammatical error. We made the correction in the updated revision.

Q1. How LLMs would perform if the news were used as a direct source instead of following the ReAct strategy?

We have investigated how LLMs perform with direct news usage through both RAG News-Only and ReAct News-Only approaches, as shown in the above table. Our analysis reveals several key findings:

When using news as the only information source, RAG News-Only achieves better performance (47.2% precision, 23.2% recall, 25.4% F1 at first-level relations) compared to ReAct News-Only (41.5% precision, 16.8% recall, 20.2% F1). This suggests that for unstructured news information, a fixed but focused retrieval strategy might be more effective than flexible but complex agent reasoning.

However, both methods show relatively lower performance compared to their event-based counterparts (RAG Event-Only: 50.5% F1, ReAct Event-Only: 41.7% F1 at first-level relations), indicating that structured event data provides stronger signals for forecasting. Notably, when given access to both information sources, the agent method demonstrates potential for effective information integration (achieving 46.3% F1 at first-level prediction with full API access), though the challenges in reliably combining different information types suggest opportunities for improving future agentic approaches.

Dear Reviewer uDbL,

Thank you once again for your thoughtful feedback and for the opportunity to further clarify our work. We hope our responses and additional experiments have addressed your questions comprehensively. Specifically::

• Comprehensive Examples: We have clarified the presence of various examples in our paper, including a demonstrative example in Figure 1, an interactive agent demo, and extensive examples in Appendix F. We've also made additional data available publicly to further illustrate our methodology.
• Comparison with RAG Methods: We've included comparisons with multiple RAG-based methods. Our experimental results, detailed in the table provided, show how our agentic approach complements and differs from RAG strategies, particularly in terms of dynamic information handling and reasoning. We've reiterated that our primary contribution is the benchmark itself, aimed at fostering future research in agent-based forecasting.
• Grammatical Correction: We appreciate your attention to detail and have corrected the typo on line 88 in our revised manuscript.
• Direct Use of News: We explored the performance implications of using news directly as a source, both through RAG and our ReAct agent approaches. Our findings suggest that while RAG might perform better with its predefined retrieval strategy, the agent method shows promise in automatic planning and integrating multiple data types, highlighting areas for future improvement.

We hope our clarifications and additional experiments have addressed your concerns comprehensively. If there are any further points you would like clarification on, please do not hesitate to let us know.

We look forward to any further feedback and sincerely appreciate the time and effort you've put into reviewing our work.

Sincerely, Authors

Dear Reviewer uDbL,

I hope this message finds you well. We recently submitted our rebuttal and would like to kindly request your feedback on our responses.

We understand that your schedule is demanding and greatly appreciate the time and effort you dedicate to the review process. Your insights are invaluable to us, and we are eager to address any further questions or concerns you may have.

Thank you for your attention to this matter. We look forward to your response.

Best regards,

Authors

Thanks to the authors for the thorough response. Having reviewed the authors' response I stand by my score for this work.

Methods and Experimental Setup: Besides the Direct IO / QA, that we already implemented in the paper, where the LLM directly provides answers using only its internal knowledge without chain-of-thought, we add the following baselines:

• Direct QA with Augmentation

  • QA with CAMEO: We provide the CAMEO ontology in an ordered dictionary format mapping relation codes to their names and detailed descriptions. This is closer to the QA-format the authors mention, and a more fair comparison as the model can refer to the output vocabulary without needing to memorize CAMEO codes.

• RAG Methods
  Following recent work, we implement three RAG variants:

  • RAG Events-Only: Following GPT-NeoX-ICL[1], we explore rule-based approaches for retrieving historical facts. Using the 'Pair' and 'Undirectional' setting, given a query event $(s, ?, o, t)$, we retrieve historical events $(s, r \in \mathcal{R}, o, <t)$ and $(o, r \in \mathcal{R}, s, <t)$. Events are sorted by recency with a cap of 30, aligning with the default cap of the get_events API function.
  • RAG News-Only: Following TCELongBench[2], we employ BM25 retrieval to fetch the most query-relevant news articles before the query date. The top 15 news articles are retained, matching the default cap of the get_news_articles API function.
  • RAG All: Combines both retrieved structured events and textual news articles.

Comparison of Retrieval Strategies with Agent: Both baseline approaches—CAMEO context augmentation and RAG methods—employ static, predefined retrieval strategies that are fixed for all queries and executed only once per query.

In contrast, our agentic approach enables dynamic, multi-step information gathering and reasoning. The agent can replicate the baseline retrieval strategy by fixing certain API parameter values, for example, the agent can use function call get_events(head_entities=[s, o], tail_entities=[o,s]) to get the retrieved context as RAG Event-Only, and use function call get_news_articles(text_description='(t, s, ?, o)') to get the retrieved context as RAG News-Only; its capabilities extend far beyond these static approaches through its flexible parameter settings for each function call and multiple steps per query.

This multi-step, adaptive approach represents a fundamental shift from static retrieval to dynamic information gathering and reasoning, though it introduces higher requirements for the LLM's planning ability in:

• Automatically selecting optimal information-gathering strategies
• Integrating and reasoning over information of different formats
• Adjusting strategies based on intermediate findings and current context

We sincerely appreciate your support and constructive feedback on our MIRAI benchmark.

First of all, we’d like to emphasize that the main contribution of MIRAI is to provide a benchmark dataset to evaluate all kinds of LLM forecasting systems, including Simple QA format (the Direct IO in our experiment), LLM with tool-use, with and without fine-tuning, etc. In our rebuttal, we added recurrency-based baselines, fine-tuned TKG baselines (as shown in Response Table 1) and RAG baselines as shown below.

Please find our point-by-point responses to your questions below:

W1. Are LLMs in your approach demonstrably better than in other task formulations like QA, considering prior methods or prompts?

Thank you for this valuable question. In our submission-version, we have already evaluated direct question answering (direct IO) and question answering with zero-shot chain-of-thought prompting (ZS-COT) in experiment Table 1. Our results show that allowing LLMs to interact in an agentic environment with diverse information sources improves performance by providing richer context for predictions.

To further address the raised concern, we additionally include more baseline methods that are used in previous QA-based studies, particularly augmenting context of direct IO (i.e. direct QA) and QA with retrieval augmented generation (RAG) methods. The result table is as follows:

Table: Evaluation results of GPT-4o-mini on the 2024-02 test split using different non-agentic methods and the ReAct agent with Single Function action type. The best-performing score is highlighted in bold and the second-best is in italics.

Dear Reviewer AZdo,

Thank you again for your support and thoughtful feedback. We deeply appreciate the time you have dedicated to evaluating our work. Below, we provide a summary of our additional experiments and clarifications, hope these have addressed your questions comprehensively:

1. Comparison with QA-based Task Formulations: In addition to direct question answering (Direct IO) and with zero-shot chain-of-thought prompting (ZS-COT) we evaluated, we've expanded our evaluation to include more baselines from the QA domain, including Direct QA with Augmentation and various RAG methods. These comparisons reveal that, unlike RAG’s fixed retrieval strategies, agents dynamically adapt their information-gathering and reasoning, offering greater flexibility, but also coming with higher challenges in multi-step planning and information integration

2. Fine-tuning Baseline LLMs: We acknowledged the potential of fine-tuning baseline LLMs but have clarified the complexities involved, particularly for agentic systems. We've cited recent works exploring this area and emphasized that MIRAI serves as a platform for such innovations. We will include a discussion on this in our revision to encourage further research.

3. Comparison with Related Benchmarks: We've clarified that our main competitors are benchmarks rather than methods, specifically comparing with TempLongBench and GDELT-2015. We've highlighted how MIRAI fills gaps in these benchmarks by introducing dynamic updates, diverse information sources, and an agentic environment with tool-based reasoning.

4. Performance of Agentic Design Compared to Prior Approaches: We've reiterated the findings from our initial experiments and added comparisons with RAG and other QA methods, showing improvements in contexts where agentic reasoning provides additional value.

Additional Points:

• Dynamic Information Use: We've demonstrated that our agentic approach allows for more automatic and adaptive information retrieval, which is not with static, one-shot retrieval methods used in previous works.
• Benchmark Contribution: We've underscored that our primary contribution is the benchmark itself, promoting new research directions in agentic forecasting.

We hope these responses and experiments in our rebuttal have been helpful. If there are any further questions or areas you feel need clarification, please let us know. We are eager to refine our work based on your feedback. Thank you once again for your time and the detailed review of our work.

Sincerely,
Authors

Q2. What motivated the choice of the current metrics, and why were more commonly reported metrics, like MRR and Hits@k, excluded?

Thank you for this important question about metric selection. We'd first point out that the expected outputs for the two types of metrics are different:
Set-based Metrics:
Models output a discrete set of predicted relations for each query. We evaluate these predictions using:

• Precision: Proportion of predicted relations that are correct
• Recall: Proportion of actual relations that are predicted
• F1 Score: Harmonic mean of precision and recall

Ranking-based Metrics:
Models output an ordered list of relations with associated scores. Following TKG conventions of time-aware filtering, we implement:

• Mean Reciprocal Rank (MRR): Average reciprocal of the first correct relation's rank
• Hit@10: Proportion of queries where at least one correct relation appears in top-10 predictions

With this understanding, we acknowledge that agent-based methods also can support ranking-based output and evaluation, but necessary prompt modifications need to be applied. In added experiments, we implemented the ReAct ranking-based variants as presented in the last 6 rows of the table. For React (Rank): we instruct the agent to generate an ordered relation list; For React (Rank w. Prob): we additionally instruct the agent to generate probability scores for ranked relations. We tested on different list lengths k=10, 30, all, where only k=all is valid to be used for MRR calculation, while all configurations support Hit@10 evaluation.

We prioritize set-based metrics as our primary evaluation criteria for LLM Agent-based methods mainly considering the current LLM model capabilities. Our primary consideration is that current LLMs are better suited to generating discrete predictions through natural language reasoning than producing comprehensive ranked lists. This is evidenced by our experimental findings:

• List Length Sensitivity:
  ReAct agent's performance deteriorates with longer list requirements, with k=30 and k=all performing worse than k=10 in ranking metrics (Hit@10: 25.7% for k=10 vs 12.0% for k=30).
• Prompt Sensitivity:
  Performance varies between pure ranking and probability-weighted ranking (Hit@10: 25.7% vs 26.8% for k=10, and MRR: 13.9% vs 12.6% for k=all), suggesting that ranking outputs are sensitive to the prompt formulation and output format.

Given these challenges, we opted for metrics that more directly and reliably assess the agents' ability to predict discrete events without the confounding factors introduced by list generation and ranking.

Q3. The authors commit to updating the dataset split monthly. How long is this commitment expected to last, and will it be sustained indefinitely?

We commit to maintaining monthly dataset split updates for at least the next year through our fully automated pipeline. The update process requires minimal human intervention, as the pipeline is designed to handle data collection, processing, and splitting autonomously. As long as the data format from CAMEO remains consistent, the pipeline can continue to generate updates reliably.

To ensure sustainability, we have made the automation scripts publicly available in our repository, allowing researchers to independently verify and execute the update process. Beyond this, we plan to continue updates indefinitely as long as computational resources and data availability permit. To ensure sustainability, we have also documented the entire update process and will actively maintain the codebase, making it resilient to potential API or data source changes.

Q4. What is the rationale behind applying higher thresholds to the test data (100 daily mentions, 5 news articles) than to the training data?

The different thresholds between training and test data serve distinct methodological purposes. The higher test thresholds (100 daily mentions, 5 news articles) enable precise and robust evaluation of our benchmark by:

1. Ensuring test events have sufficient contextual information for reliable ground truth labeling
2. Reducing noise from potential false positives or misreported events
3. Allowing thorough verification across multiple independent sources The higher test thresholds align with real-world applications where stakeholders focus on detecting and analyzing significant international events.

Meanwhile, the lower thresholds for training data are deliberately chosen to maximize the dataset's utility. First, they create a comprehensive training set that captures both major and minor events, enabling models to learn the full spectrum of international relations. Second, by providing a broader training set, we enable future researchers to either use the complete dataset or apply custom filtering criteria based on their specific research needs, enhancing the dataset's long-term value to the community.

Recurrency's Comparison and Insights for Agents: Comparing with the ReAct agent (Set Prediction, the setting we used in the main paper) in row 9, we find that although the LLM agent could achieve much higher forecasting precision, it obtains much lower recall than the Recurrency baseline. We conducted a detailed analysis of this behavior by manually going through the reasoning traces generated by the LLM agent in the test set. One possible reason is that the agent has a strong tendency to select only a subset of the most frequent historical events in its prediction. For example, for query (2024-02-01, PSE, ? , EGY), it uses function calls like get_relation_distribution(date_range=DateRange(start_date=Date("2023-01-31"), end_date=Date("2024-01-31")), head_entities=[ISOCode("PSE")], tail_entities=[ISOCode("EGY")])’, and obtained a full frequency list as: {CAMEOCode(code='042'): 32, CAMEOCode(code='192'): 18, CAMEOCode(code='040'): 13, CAMEOCode(code='043'): 12, CAMEOCode(code='046'): 8, CAMEOCode(code='080'): 6, CAMEOCode(code='036'): 4, CAMEOCode(code='010'): 4, CAMEOCode(code='190'): 3, CAMEOCode(code='073'): 3, CAMEOCode(code='030'): 3, CAMEOCode(code='084'): 3, CAMEOCode(code='020'): 3, CAMEOCode(code='172'): 2, CAMEOCode(code='014'): 2, CAMEOCode(code='070'): 2, CAMEOCode(code='044'): 2, CAMEOCode(code='086'): 1, CAMEOCode(code='013'): 1, CAMEOCode(code='051'): 1}. It then has a further step of checking recent news articles, and obtains its final prediction as 040, 042, and 192, which are the top three frequent relations.

The effectiveness of simple temporal recurrency heuristics underscores the importance of incorporating more historical pattern analysis in the future development of forecasting agents, in particular, improving its recall of capturing a greater proportion of true relationships between countries.

[1] History Repeats Itself: A Baseline for Temporal Knowledge Graph Forecasting. IJCAI 2024.

[2] Translating embeddings for modeling multi-relational data. Neurips 2013.

[3] xERTE: Explainable Reasoning on Temporal Knowledge Graphs for Forecasting Future Link. ICLR 2021.

We sincerely thank you for your thoughtful and detailed feedback, for which we have included additional discussions and experiments accordingly. Please find our experiment results in the table and detailed responses for each of your questions as follows:

Comparison with Heuristic-based Method:

W1&W6&W7. Comparisons with simple heuristic baselines to contextualize the agents' performance in the MIRAI benchmark (W1). Include a heuristic baseline, such as predicting relations based on historical data (modification from [1]), to improve evaluation. (W6, W7)

Thank you for raising this valuable suggestion of including the heuristic-based baseline, specifically the Reccurrency [1] model. The main idea in this paper is ‘history repeats itself’: events that happened in the past are likely to re-occur in the future.

First of all, in our agent API design, we already have functions inspired by such re-currency heuristics. Specifically, we’ve defined functions like count_events, which count the number of events in the knowledge graph based on customizable conditions on date, entities, and relations. To provide more statistical support for the agent, we also include functions like get_entity_distribution and get_relation_distribution which return a frequency-ranked list of the most commonly interacted entities or happened relations.

Secondly, we appreciate your suggestions on directly including the recurrency-based baseline to compare with LLM Agent, so we can better understand the limitations of existing agent design. Thus, we added the Reccurrency [1] model as one baseline. The original work introduces three settings: strict recurrency, relaxed recurrency, and their combination. While the original work and its scoring functions are specifically designed for link prediction, we adapt the strict recurrency variant for relation prediction. For a query event $(s,?,o,t)$, we compute scores for all relations $r \in \mathcal{R}$ using:

$\phi_{\Delta}((s,r,o,t),G) = \Delta(t, \max\{k \mid (s,r,o,k) \in G\}) \text{ if } \exists k \text{ with } (s,r,o,k) \in G, \text{ else } 0,$

where $\Delta(t,k) = k/t$ measures temporal proximity.

Cutoff Dates: To fairly compare the performance with LLM agents and study the effect of cutoff date, we evaluate the recurrency model on 4 different cutoff dates, 2023-06, 2023-08, 2023-10, and 2023-12, where 2023-06 uses around half of the training data in MIRAI; 2023-10 is the knowledge cutoff date of GPT-4o-mini, the best-performing LLM we tested; 2023-12 is the knowledge cutoff date of Mistral and LLama, the open-sourced LLMs we tested. The cutoff date for the Recurrency model determines available historical events, e.g., 2023-10 means using only events before 2023-10-31 for score computation.

Evaluation Metrics: For evaluation, the Recurrency model supports a full ranking list for all relation types with their prediction scores. Therefore, for ranking-based metrics, we apply MRR and Hit@10. We follow TKG forecasting conventions of time-aware filtering [2-3] for the ranking-based metrics. For set-based evaluation, we acknowledge that there are an infinite number of possible thresholds to select the set of relations with their score above the threshold. In the experiment table, we report the set prediction result with the threshold set to be 0 - this has an intuitive meaning: we predict the relations that happened at least once between the two countries.

Performance of the Recurrency Model: The experiment results of the Reccurency (strict) model are shown in row 5-8 of the table. We observe that the Recurrency baseline demonstrates strong performance in ranking metrics (17.8% MRR and 43.2% Hit@10 with 2023-12 cutoff), leading other models; it also shows consistently high recall (86.0% Recall at first-level relation, and 80.1% recall at second-level relations with 2023-12 cutoff), suggesting that international events indeed often follow repetitive patterns.

Table 1: Evaluation results on the 2024-02 test split for relation prediction using TKG-based methods and LLM agents based on GPT-4o-mini. The best-performing score is highlighted in bold, and the second-best is in italics.

Dear Reviewer ZMPf,

Thank you again for your detailed and constructive feedback. We greatly value the time and effort you invested in reviewing our work, and we hope that our additional experiments and discussions addressed your concerns comprehensively. Specifically:

1. Heuristic Baseline Comparisons: We included the Recurrency [1] heuristic model and provided a detailed analysis compared to LLM agents (see Table 1). This addition highlighted the strong recall of heuristic methods, emphasizing areas for future agents.

2. TKG Baseline Analysis: We have detailed the training settings used for TKG models, incorporated additional comparisons with agents, and explored the effects of training cutoff dates. The updated analysis reinforces the strengths and areas of improvement of LLM agents as compared to TKG methods.

3. Metric Selection and Extensions: We provided results on ranking-based metrics (MRR and Hit@k) for the ReAct agent with necessary modifications to a ranking list output. We underscored the trade-offs between set-based and ranking-based evaluations for current LLM methods limited by current model capacity, explaining our rationale for prioritizing set-based metrics.

4. Task Focus on Relation Forecasting: We clarified our focus on relation prediction as the primary task due to its alignment with the structured nature of CAMEO relations and the goal of capturing fine-grained dynamic shifts in international relationships.

5. Rationale for Test Data Thresholds: We clarified that the higher thresholds applied to the test data were designed to ensure precise and robust evaluation by focusing on the contextual-rich and cross-verified international events; while the lower thresholds for training data enable models to learn from a broader spectrum of event types.

6. Appendix and Human Baselines: We streamlined the appendix by linking to the full API documentation and included human forecasting performance directly in the main paper (Table 2).

7. Future Dataset Updates: We have clarified our commitment to maintaining monthly dataset updates is supported by an automated pipeline. Additionally, we have documented and open-sourced this process to ensure its sustainability and usability for the broader research community.

We would like to inquire if there are any questions about our rebuttal, for which we're happy to provide additional information and further clarifications. We deeply appreciate your time and effort in reviewing our work.

Sincerely,
Authors

Dear Reviewer ZMPf,

Thank you again for taking the time to review our paper. We greatly appreciate your detailed feedback and your recognition of key strengths of our benchmark work, such as the clarity and structure, the contamination-free test sets, insightful visuals, and the in-depth analysis across diverse aspects. Your thoughtful comments are invaluable to us.

In response to your feedback, we have conducted extensive experiments with heuristic and TKG baselines, and agentic methods with both set-predictions and ranking-based metrics. We have also provided detailed clarifications on our contribution (global comment) and on your concerns in our rebuttal replies with our paper revised accordingly. We hope our detailed responses adequately address your concerns.

With the discussion deadline approaching, we would appreciate your attention to our rebuttal and any further feedback you may have, as it will give us the opportunity to provide more details before the author-reviewer discussion session ends and help us continue to improve our work. Thank you again for your time and valuable comments!

Best regards,
Authors