ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

Thank you for your valuable feedback! We address your questions and concerns below.

Weaknesses

> Overall concerns- Forecasting based on questions is a highly subjective problem. The questions and their respective responses could vary depending on the demographics, education background, understanding of the domain, etc of the forecasters. Does the authors have any openly accessible user-study report highlighting the biases and how they are being addressed. This follows for both - human forecasters as well as "superforecasters"

This is a valid point. We have updated our draft to include Appendix L describing the demographic information of our survey respondents, mentioned the potential issues we see regarding the impact of demographics on both questions and forecasts, and referenced this section from a footnote in the main text.

> I'm wondering how does the 1000 questions designed for evaluation covers forecasting of an open world set? Or is it only targeted to a few domains?

Of the 500 standard questions in the LLM dataset, 250 come from forecasting websites that can cover any subject matter. New questions come in daily and are informed by world events and hence are diverse and cover a broad set of domains.

The other 250 questions come from datasets. We’ve selected diverse data sets (ACLED, FRED, Wikipedia, …) to ensure coverage of many different domains.

Indeed, one reason we categorize the questions in the ForecastBench question bank by topic is to achieve a broad representation of domains. When sampling from the question bank to generate the question set, we sample questions by category within each source to ensure topic diversity.

A detailed breakdown of categories, with the number of questions available in the question bank per category, is available in Table 15 in Appendix C.2.

> The question examples shown in Figure 4&5 seem at a very high-resolution. Although, it is appreciated that the source of the questions is provided, but "is 2024 going to be hotter than 2023" - What parameter is being considered here for comparison? is it mean temperature? Can LLMs infer these for a new undefined concept?

Before responding to the query about this question, we want to note that the vast majority of our questions have clear resolution criteria. Though there are a few exceptions (as you note in your question), we have closely examined our dataset and ensured they are a very small minority of questions and do not impact our mainline results.

Regarding this specific question, you correctly point out that the resolution criteria is not clear. That has not prevented human forecasters from converging on a response (the crowd forecast as of 2024-11-13 is 98.9%, when the question was asked the crowd forecast was 76%). Indeed, in an active market, the crowd forecast should incorporate all pertinent information (information about recent events, short-run changes in weather, and even the resolution criteria). If a question has ambiguous resolution criteria, that should be reflected in the crowd forecast.

We have replaced this question in the text with a more representative question. You can see the updated question in Figure 2 of Appendix D.2. And we plan to impose additional filters to catch/remove questions like this in the future.

> Methodology relies on online links and APIs, thus limited in its coverage of questions.

To ensure coverage by source and topic, we specifically draw questions from a broad range of sources, including databases of geopolitical events, economic indicators, weather time series, sports records, and current events. We therefore think that our benchmark contains the broadest coverage of any set of forecasting questions constructed in the literature, and we plan to continue to expand the coverage of the question set over time.

To more specifically answer one of your points, the ‘online links’ we rely on are themselves forecasting platforms that write and curate thousands of forecasting questions, updating the available questions and topics regularly. By relying on these sources, we ensure coverage of key real-world events in our benchmark.

The tradeoff here is that we have a dynamic, ever-expanding source of questions that changes with current events and remains relevant. This automatically-updated system requires minimal maintenance.

Currently, we draw from a large enough set of online databases and APIs to ensure robustness of our benchmark to API instability and downtime. If you have suggestions for how to improve or modify our setup to ensure broader coverage of domains, we’re happy to address them.

This paper does not propose new advanced machine learning methodology or demonstrate novel ML techniques.

This is a silly, effortless feedback. Lot of progress in our field came from good benchmarks (examples: ImageNet, adversarial robustness evaluations circa '18, to more recent LLM evaluations such as chatbot arena, AlpacaEval and many others). If anything, I would say developing good benchmarks is at least, if not even more, important. For example, there were tons of papers claiming to tackle distribution shift pre-ChatGPT using various sophisticated new methodology, but comprehensive benchmarks like WILDS showed that most of the interventions don't generalize, and the best solution is to just finetune on OOD data (if you have them) most of the time.

Also the work is clearly very well documented so not sure what the criticism about non-reproducibility is about..

> Weakness 1b. Relatedly, it's never really shown why it's important for this to be a "dynamic" benchmark, despite this being a key selling point. Avoiding data leakage is indeed important, but it's surprising that this doesn't allow for any dedicated experiment or discussion around how often questions are resolved during the benchmarking.

When we say dynamic, we mean a benchmark that is regularly updated so that there is a constant set of fresh, unresolved questions. Current forecasting datasets used to evaluate LLM forecasting capabilities rely on questions that have already been resolved. This has two major problems. First, they rely on the reported training cutoff of LLMs: they choose forecasting questions that resolve after training cutoff dates to evaluate forecasting ability; but these dates are not precise, and researchers cannot be sure these dates are accurate for closed-source models. Second, these datasets become outdated as groups release new models with training cutoffs after the resolution dates of the forecast questions.

More generally, there's a well-established problem with data leakage in ML datasets. For example, Zhang et al. (https://arxiv.org/pdf/2405.00332) showed that on fresh GSM8K questions, model performance dropped by 13% for some families of models. This is no different in forecasting; a recent study claiming LLMs are superhuman at forecasting real-world events (‘LLMs are Superhuman Forecasters’, by Phan et al.)) has been removed from arxiv after other researchers showed that the claims did not hold up when evaluated on a new set of questions.

Since ForecastBench constantly adds questions about events that have not yet occurred, it is impossible for AI systems to ‘cheat.’ This is different from traditional ML datasets where the answers to the test set are either published or held by a third party. Our benchmark is valuable because it allows us to always be confident about the results of LLM forecasting capabilities. We hope we also set an example for the field to follow as it attempts to more accurately assess and compare models.

In addition to the benefits above, we lastly highlight that a dynamic benchmark ensures that our question sets remain relevant to new events in the future, as the focus of world geopolitical, cultural, and economic events changes.

> Weakness 1c. it's surprising that this doesn't allow for any dedicated experiment or discussion around how often questions are resolved during the benchmarking.

It is not clear to us what you mean by “how often questions are resolved during the benchmarking.” Data questions resolve at every forecast horizon once new data is available. Market questions resolve when the question is resolved on the platform they’re sourced from. Please let us know if this clarifies your understanding.

Thank you for your valuable feedback! We address your questions and concerns below.

Weaknesses

> Weakness 1a. The main weakness of this work is the limited experiment section. Given the main contribution is the curation of a new dataset and resultant evaluation, it is unsatisfying that the result is only one page of experiments with minimal interpretation. The result is it's unclear if this benchmark is useful beyond this experiment.

A key topic of research is whether LLMs have achieved human parity in general reasoning, and if they have not, when or if they will. Forecasting is a useful testbed of LLM reasoning ability, just like math and coding. To produce an accurate forecast, a person or AI system must synthesize information, ideally from both live and preexisting knowledge, reason, avoid overconfidence, guard against biases, combine information, and quantify beliefs. Among the 17 models we test (across the 7 baselines outlined in Section 5.1), only Claude-3.5 Sonnet performed as well as the general public (Brier score of 0.111), and both Claude-3.5 and the general public perform significantly worse than superforecaster-level (Brier score of 0.091, lower is better). Our benchmark thus provides valuable insights into the capability of AI systems.

We also address a key issue in LLM evaluation: the tendency for models to overfit to static datasets. For example, Zhang et al. https://arxiv.org/pdf/2405.00332 showed that on fresh GSM8K questions, model performance dropped by 13% for some families of models. This is no different in forecasting; a recent study claiming that LLMs are superhuman at forecasting (‘LLMs are Superhuman Forecasters’, by Phan et al.)) has been removed from arxiv after other researchers showed that the claims did not hold up when evaluated on a new set of questions.

Here, since our benchmark is dynamic--automatically adding fresh forecasting questions about events that have not happened yet--we avoid potential leakage problems. We hope we set an example for the field to follow so we can more accurately assess models. Lastly, we recently analyzed the scaling trends of LLMs to observe how forecasting performance scales with more training compute. Figure 1 shows a negative log-linear relationship between training compute and forecasting accuracy (Brier score). This relationship was marginally statistically insignificant ($r = -0.64$, $p = 0.06$) because of the small number of tested models (only 9 models we tested had reliable estimates of training compute). But, we use this pattern to project out the log-linear relationship, finding that if performance continues to improve at these rates, LLMs could match superforecaster performance when training compute approaches $1 \times 10^{27}$ FLOP, though there is a large confidence interval (bootstrapped 95% CI: $$7.08 \times 10^{26}$,$3.64 \times 10^{30}$$ .

We also analyzed the relationship between Chatbot Arena scores (see Chiang et al., 2024) and Brier scores, for which we could compare all 17 models tested. Here we found a strong and highly statistically significant relationship (r = -0.69, p = 0.002), showing that models with higher Arena scores also tend to produce more accurate forecasts. The calculated linear relationship implies that LLMs will match superforecaster performance when the Arena score approaches 1398 (bootstrapped 95% CI: [1335,1581]).

We have added a paragraph entitled “LLM performance and forecasting accuracy” to Section 5.2 with this new analysis.

Thank you for reviewing and engaging with our work! We address your questions and concerns below.

Weaknesses

> It may be worth justifying the adopted method in this benchmark for LLM to make forecasts, since LLM reasoning methods (e.g. RAG, etc.) can have great influence on the final performance

The choice of methods and prompts indeed significantly affects LLM forecasting performance.

For our initial evaluation, we adopted the methodology from Halawi et al. (2024), a recent paper that established effective practices for LLM forecasting, including both prompting strategies and question filtering criteria.

While this approach helped demonstrate how different reasoning methods and model choices influence forecasting accuracy, its primary value lies in establishing a consistent baseline for measuring improvements in LLM forecasting capabilities over time; maintaining the same methods and varying only the model used will allow us to do this.

That said, we’re excited to see teams experiment with their own approaches by submitting forecasts on ForecastBench.

Questions

> What if we allow LLMs to access different tools to actively acquire relevant information in a feedback loop? In this way the LLM may construct more context for more accurate answer.

This approach, which aligns with the agentic LLM paradigm, is indeed a promising direction. A competitive forecasting system would likely benefit from accessing relevant tools, reasoning iteratively, updating its memory, and refining outputs through a feedback loop. We anticipate improved performance with this paradigm for two reasons: (1) it closely mirrors human processes for forecasting, and (2) it has demonstrated superior results on various benchmarks, including SWE-Bench and Web-Bench (Wang et al. https://arxiv.org/pdf/2407.16741), CYBench (Zhang et al. https://arxiv.org/pdf/2408.08926), and others.

While we recognize the potential of this approach, developing such a system would require substantial effort and resources. For the scope of our current work, we believe our existing baselines, particularly Halawi et al. (2024), provide a strong foundation for comparison. We have thus left the exploration of this agentic paradigm to future research directions.

Please let us know if you have any other questions or concerns we can address.

Thank you for reviewing and engaging with our work! We address your questions and concerns below.

Weaknesses

> The evaluation of unresolved questions relies on comparisons with crowd predictions, potentially complicating future assessments of whether models outperform the crowd baseline.

In many markets, the crowd forecast approaches the eventual market resolution as the resolution date approaches. Hence, evaluating unresolved market questions against the crowd forecast is the best estimate we can obtain of their accuracy before the question is resolved.

Once a question has been resolved, the forecast is compared to ground truth. We can always compare model performance to the crowd forecast at any point in time if we so desire. Indeed, in every forecasting round we run a dummy forecasting baseline called the “imputed forecaster,” that forecasts the crowd forecast on the forecast due date and 0.5 on dataset questions

As a corollary, on the ForecastBench website we allow viewers to sort the leaderboard by performance only on resolved questions.

> The number of expert predictions was limited, with only 39 experts contributing, leading to an average of 8 expert predictions per question. This relatively small sample size may affect the robustness and generalizability of the evaluation results.

We agree that the number of expert predictions was limited by our small sample of experts. But we also believe that this provides a realistic measure of the level of state-of-the-art human forecasting ability. A small number of firms and think tanks provide expert forecasting in geopolitical and economic domains where human forecasters with proven track records answer policy-relevant questions. It is not possible or cost-effective to get 10+ forecasters to submit forecasts on even a handful of questions. In the private market, doing this for even one question would cost thousands of dollars. So, our goal with the expert human forecaster comparison group was to provide a realistic measure of accuracy from the current top-level human forecasters who might be asked to answer questions like these. We felt that getting around 5-10 of these forecasters per question accomplished our goal there.

Questions

> Have you investigated the impact of the prompt date on model predictions? Specifically, could questions asked further in advance of the resolution date introduce greater uncertainty in LLM predictions, potentially affecting prediction accuracy?

We expect this to be the case and plan to evaluate long-term forecasting accuracy of both LLMs and humans in follow-up studies. This is one reason the questions sourced from datasets are important: for these questions, we ask for forecasts at 8 intervals, from 7 days to 10 years into the future. Over time, we’ll be able to assess and compare forecasting ability across time horizons amongst benchmark forecasters and LLM baselines. We can also explore whether alternative prompting strategies reduce any obvious patterns of bias in the elicitation of forecasts over different time horizons relative to the prompt date.

We have clarified this point in Section 3.2 in the Forecast horizons paragraph.

> How is the quality of questions released for evaluation determined?

We follow the method outlined in Halawi et al. (2024) to filter out low-quality questions, prompting gpt-3.5-turbo-0125 with a question validation prompt that includes a quality rubric. We have added the question validation prompt we use to Appendix F figure 10.

> Have you considered evaluating model predictions by examining the logits of the answer tokens? Although this may be more feasible with open-source models, some providers do offer token-level logit outputs. Incorporating logits could provide insights into model calibration and further enhance prediction reliability.

We have not done this yet but agree that this would be a worthwhile addition to a follow-up paper.

Please let us know if you have any other questions or concerns we can address.

Thank you for reviewing and engaging with our work! We address your questions and concerns below.

Weaknesses

> Table 2 is a bit big / dense and hard to parse. The paper could benefit from using more visual plots to convey the results

We agree that it’s quite dense. In our updated version of the paper, we streamlined the text and were able to split up the table (now Table 2 and Table 3) while staying within the page limit. We have also added plots (Figures 1a and 1b) showing the relationship between performance on ForecastBench and training compute/Chatbot Arena scores.

Questions

> Is there anything easy (cheap) we can try to improve the capabilities of these models to forecasting beyond RAG-like things? I guess finetuning open-source models would be ideal but that is probably infeasible.

Yes. Recent work by Halawi et al. (2024) finetuned a reasoning model (from GPT-4) precisely to reason and make judgmental forecasts. Their result shows that even with finetuning and RAG, LLMs still did not surpass or reach human performance, but using such methods helped improve performance quite a lot (decreased Brier score by 0.03 with respect to the baseline).

However, our work aims to establish a benchmark and a set of baselines. We believe it is possible to improve upon our baselines via finetuning using more advanced LLMs other than GPT-4, but we will leave this to future work.

Please let us know if you have any other questions or concerns we can address.

Thank you for your valuable feedback! We address your questions and concerns below.

Weaknesses

> This is an interesting and useful dataset, but reads like a tech report or class assignment; the level of technical detail is superficial, and there is little to no exploration of what we learn about LLMs from this.

A key topic of research is whether LLMs have achieved human parity in general reasoning, and if they have not, when or if they will. Forecasting is a useful testbed of LLM reasoning ability, just like math and coding. To produce an accurate forecast, a person or AI system must synthesize information, ideally from both live and preexisting knowledge, reason, avoid overconfidence, guard against biases, combine information, and quantify beliefs. Among the 17 models we test (across the 7 baselines outlined in Section 5.1), only Claude-3.5 Sonnet performed as well as the general public (Brier score of 0.111), and both Claude-3.5 and the general public perform significantly worse than superforecaster-level (Brier score of 0.091, lower is better). Our benchmark thus provides valuable insights into the capability of AI systems.

We also address a key issue in LLM evaluation: the tendency for models to overfit to static datasets. For example, Zhang et al. https://arxiv.org/pdf/2405.00332 showed that on fresh GSM8K questions, model performance dropped by 13% for some families of models. This is no different in forecasting; a recent study claiming that LLMs are superhuman at forecasting (‘LLMs are Superhuman Forecasters’, by Phan et al.)) has been removed from arxiv after other researchers showed that the claims did not hold up when evaluated on a new set of questions.

Here, since our benchmark is dynamic -- automatically adding fresh forecasting questions about events that have not happened yet -- we avoid potential leakage problems. We hope we set an example for the field to follow so we can more accurately assess models.

Lastly, we recently analyzed the scaling trends of LLMs to observe how forecasting performance scales with more training compute. Figure 1b shows a negative log-linear relationship between training compute and forecasting accuracy (Brier score). This relationship was marginally statistically insignificant ($r=-0.64$, $p=0.06$) because of the small number of tested models (only 9 models we tested had reliable estimates of training compute). But, we use this pattern to project out the log-linear relationship, finding that if performance continues to improve at these rates, LLMs could match superforecaster performance when training compute approaches $1 \\times 10^{27}$ FLOP, though there is a large confidence interval (bootstrapped 95% CI: $$7.08 \times 10^{26}$,$3.64 \times 10^{30}$$ . As we continue to add more models, we'll be able to refine our scaling trends.

We also analyzed the relationship between Chatbot Arena scores (see Chiang et al., 2024) and Brier scores, for which we could compare all 17 models tested. Here we found a strong and highly statistically significant relationship ($r=-0.69$, $p=0.002$), showing that models with higher Arena scores also tend to produce more accurate forecasts. The calculated linear relationship implies that LLMs will match superforecaster performance when the Arena score approaches 1398 (bootstrapped 95% CI: $1335,1581$ ).

We have added a paragraph entitled “LLM performance and forecasting accuracy” to Section 5.2 with this new analysis.