Consistency Checks for Language Model Forecasters

We thank the reviewer for recognizing our evaluation approach, research artifacts, and writing!

Limited breadth of data domains: Dependency on NewsAPI and Manifold and Metaculus for some forecast questions, while practical, may limit the breadth of testing topics and introduce data biases

We expand the NewsAPI-derived dataset by spanning reference classes of questions. This expands the distribution of topics, but does not resolve all biases. This is described in Appendix J.2. We will release our code to make it easy to create consistency evaluations on any dataset of base forecasting questions. We defer creating better forecasting question datasets for future work.

I would assume that you chose events that resolved "between May 1, 2024, and August 15, 2024." (L203), because that would be after the knowledge cutoff, right. But are we absolutely 100 percent sure that we know the actual knowledge cutoff of these models (since OpenAI loves switching things behind the scenes in the API without letting users know?)

That is true; we picked data after the stated cutoff date for all the models. In cursory inspections of our data, we did not observe forecasts of 0% and 100% probability that would be proof of data leakage.

We additionally thank the reviewer for noting the phrasing issue, which we have now fixed in the updated submission.

We thank the reviewer for recognizing that our work explores an important topic in a high-effort way. Below, we answer the reviewer’s main questions.

For instance, maybe a certain class of models (perhaps larger or trained in certain ways) are better forecasters – and many of these features are observable anyway, so why not just use these instead of consistency?

A good forecaster, intuitively, requires (1) strong reasoning; (2) lots of knowledge about the world (either in weights or retrieval); and (3) a robust and consistent way of converting the above into probabilities. There are benchmarks for (1) and (2), but not that many for (3). In addition, we think here is value in doing an outside view on evaluation, instead of relying on measurements of different skills that we think estimate the forecasting accuracy correctly.

The metrics themselves seem overly and possibly unnecessarily complicated. The rationale for these were also not provided clearly. What about something simpler, such as distances between scores that involve S? Perhaps I’m not understanding the complexity of scoring when arbitrary predicates are used for consistency.

Thank you for the comment; the rationale for the exact metrics we use is indeed not explained well within the main body. We will update the paper before publication. The key reason why we expend effort to do metrics properly is principled aggregation and comparison of inconsistency scores.

In the standard Bayesian setting, one’s beliefs are consistent or not: there either is a Dutch book (a way to bet against the forecaster’s beliefs to get infinite profit) or there is not. In practice, a forecaster’s beliefs (even on the same question) are never perfectly consistent across runs. If an election model has a presidential candidate at 48% with one random seed and 50% on the other, this is not a reason to discard it as completely flawed. If we discarded every single Dutch-bookable forecast, even very good forecasters might be discarded as inconsistent.

A good metric should have answers to the below:

• Is a Negation forecast tuple of (P, not P) = (0.5, 0.6) better or worse than a Negation forecast tuple of (P, not P) = (0.89, 0.01)? (We think the latter is much worse in many applications, such as forecasting tail risk.)
• Is a Negation forecast tuple of (P, not P) = (0.5, 0.6) better or worse than a Cond forecast tuple of (P, P | Q, P and Q) = (0.2, 0.5, 0.2)?
• How to compute a single-figure inconsistency score over a batch of questions, without certain logical checks (or probability regions) unfairly dominating the score?

We do not exactly understand what “distances between scores that involve S” means. Prior work (Fluri et al.) just came up with some metric for each score that normalizes to $0, 1$ . There are two major issues with this approach:
(1) Most simple metrics will be linear and fail the intuition on the (0.5, 0.6) vs (0.89, 0.01) example above.
(2) It’s unclear how to compare and aggregate scores from different consistency checks.

To tackle these issues, our approach ensures that all consistency scores share a common “unit”. For example, in the arbitrage metric, to aggregate inconsistencies, we sum up the profit made by an arbitrageur across questions.

I'm a bit confused by the focus of the study itself, which is to use logical consistency to evaluate a forecaster before resolution of the events themselves; presumably, the point of this is to evaluate forecasts sooner. But what is the value of using consistency as an early marker?

There is no way to directly measure anything about a forecaster’s performance over future events. We either need to hope for generalization from backtesting on past events, or we need some metrics to apply to unresolved predictions.

The literature review seems too sparse. For instance, there are several papers on consistency in general, and probably many others on forecasting with LLMs.

Thank you for the advice! We improved the related work section now with several more papers. The modifications are in red. We welcome suggestions on other papers that we should comment on.

How large are the datasets created? Could the authors elaborate? I probably missed this information.

This is in Section 3.3: “We run each of these forecasters on 5000 tuples in total (for each of the 10 checks, we use 200 tuples from scraped questions and 300 from NewsAPI questions), ...”.

We additionally thank the reviewer for noting the notation issues and typos, which we have now fixed.

We thank the reviewer for recognizing our thorough evaluation, metrics, and writing.

Here, we answer the questions about the choices we made in our experiments with ArbitrageForecaster.

I wonder what's the performance for o1-mini for the negation and paraphrase experiment in Figure 3?

Our estimates of cost constraints and iterating on the ArbitrageForecaster with smaller models were the only factor here, as detailed in Appendix F1. Internally, we might have overestimated the costs a bit, but it is still not a trivial cost. A single call to ArbitrageForecaster with even just depth=r, arbitraging checks= $Neg, Paraphrase$ requires 3^r LLM calls; measuring a single consistency check of each type requires ~21*3^r calls. Assuming 600 input tokens and 600 output tokens (our rough estimate of the expected hidden reasoning length) and performing the experiment on 200 questions, this amounts to a total token cost of ~400million tokens, or about $3000 with o1-mini for this single plot.

Design choices for ArbitrageForecaster. Appendix F.1 goes into the detailed theoretical motivations for the specific set-ups we experimented with. In summary:

(1) we hypothesized that ArbitrageForecaster will be particularly effective on “symmetric” and “deterministic” checks; thus we studied ArbitrageForecaster instantiated with Neg.

(2) we hypothesized that there would be consistency gains from increasing depth, thus we studied recursive ArbitrageForecaster setups.

(3) We were interested in knowing if the consistency gain after arbitraging on a single check would persist after applying further checks. Thus we studied the case of $Neg,Paraphrase$ applied together.

(4) We hypothesized that ExpectedEvidence may improve ground truth and consistency across the board.

Apart from that, the limited scope of these experiments was due to significant cost barriers to branching on multiple checks simultaneously. Future research could focus on a cost-effective way to implement ArbitrageForecaster: for example, in training, one may consider randomly sampling some subset of checks to arbitrage on for each question.

We thank the reviewer for recognizing our ideas and theory as principled, as well as commending the writing and the execution.

Goodharting and ArbitrageForecaster. The ArbitrageForecaster set-ups we tested do not empirically improve ground truth evaluation or consistency on other checks — this suggests that consistency scoring could indeed be Goodharted. However, note that most of our experiments with ArbitrageForecaster only involved arbitraging on a single check (apart from one experiment with both Negation & Paraphrase). We avoided adding more checks due to the large cost of experiments. It’s easy to imagine how a bad forecaster could Goodhart a single check — simply reporting the same probability for all questions will pass ParaphraseChecker and ExpectedEvidenceChecker; reporting prob=0.5 for all questions will further pass the Negation check — but we expect that being consistent under a variety of checks becomes increasingly difficult without a consistent world model.

Is there any way to ensure contamination-free or low-contamination consistency checking? Can we design an arbitrary search space for consistency checks that is large enough to not be saturated trivially by consistency training (see weakness 2).

Our intuition is that checks with multiple source questions do not seem to be easily saturated by consistency training, as we can always sample new questions to combine into tuples, and the forecaster does not know which questions are in the tuple when answering any particular question.

Other. Due to the GNews API changes described in the paper, we are not at this point able to fully reproduce the Halawi et al. forecasters. We believe it is likely that better LLM forecasters will be built in the near future, and hope that these will then be evaluated on our future-based consistency benchmark. To this end, we will release our code, and will strive to make it easy for future LLM forecasting papers to check consistency on our 2028 events benchmark.