Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions, which we will incorporate into the future version of our manuscript.

Concerns about refusal rate

We appreciate the reviewer’s thoughtful comments regarding the refusal cases of Mistral and Mixtral models. We would like to clarify the following points: as seen in Figure 8, only the Mistral and Mixtral models exhibit notable refusal rates, with approximately 10–30% on TF questions and 1.5–8% on MC questions. In comparison, Qwen-2-7B and Gemma-2-2B show relatively low refusal rates—<5% for TF and <2% for MC—while all other models have near-zero refusal rates for TF and <1% for MC. We count refusal cases as incorrect both to maintain comparability across models and because failing to provide an answer—when a prediction is expected—represents an unsatisfactory outcome from the user's perspective.

Further,

1. We provide an additional plot Fig.S7 that excludes questions the models refused to answer, focusing only on cases where the model provided a definite answer. While this means the results among models are no longer directly comparable, the performance degradation trend remains evident, particularly for TF questions in Mixtral-8x7B. We hope this analysis helps address the reviewer’s concern regarding the impact of refusal behavior on the observed trends. We would be happy to include this plot in a future version of the paper to help clarify concerns related to refusal cases.

2. Our claim of "their ability to forecast future events deteriorates much more rapidly as they move beyond their training data, struggling with temporal generalization" specifically contrasts pre- and post-knowledge-cutoff performance for models with known cutoff dates. The Mistral and Mixtral models mentioned by the reviewer do not disclose such information, and therefore are not used to support the claim here. For the models with known knowledge cutoff (Claude, GPT, LLaMA, and Gemma), refusal rates are minimal—most of them having nearly 0% for TF questions and less than 1% for MC questions—and thus have negligible impact on the observed degradation patterns.

3. Effect of rewording questions: We thank the reviewer for suggesting the rewording experiment, and tested the impact of softer phrasing (e.g., rephrasing “Will...” to “Would it be likely...”) using Qwen-2-7B and Mistral-7B on TF questions. As shown in Fig.S8, this change reduced refusal rates by approximately 10%. However, the overall performance degradation trend still persists, suggesting that refusal alone does not account for the observed decline.

4. On models “not being trained to be oracles”: We believe that model refusals alone do not explain the observed performance decline. As shown in Fig.S7, even when refusal cases are excluded and only answered questions are considered, the downward trend in performance persists.

5. On reluctance vs. performance: While Claude demonstrates the lowest refusal rate on TF questions (Figure 8), and high performance (Figure 3), this pattern does not generalize across models. GPT-3.5, GPT-4, and LLaMA-3-8B also have near-zero refusal rates, yet show varying levels of forecasting accuracy. Thus, we do not find strong evidence that lower reluctance to predict directly correlates with better performance.

Overall, our findings indicate that performance decline is a general phenomenon that persists even when refusals are removed from the analysis.

Others

1. For concerns about the number of human evaluation samples, please refer to our response to Reviewer xifq ("Sample size of human evaluation"). Regarding the shift of newer 4o models since September 2024, please refer to our response to Reviewer xifq ("Distribution of question categories and question types") and Reviewer s4SM ("Choice of models in QA generation").
2. Background of human annotators: The 4 annotators are graduate students from the authors’ institution, majoring in finance, accounting, statistics, and data science, respectively. They were not involved in the research beyond their role in conducting the evaluations.

We sincerely thank the reviewer for the detailed feedback.

Brier score

While we agree that Brier score is valuable to account for uncertainty in binary predictions, we clarify that accuracy remains a valid metric in this setting, revealing a clear performance degradation trend in our experiments. We provide Brier score result in Fig.S4. The increasing trend further confirms the previously identified trend of performance degradation. We also note that Brier score can be sensitive to calibration issues of LLMs, which is an important but orthogonal direction that we leave for future work.

Quality of dataset

We refer the reviewer to Section 3.1 & Appendix A.3 for our quality control process. During the QA filtering step, 7 principles are identified based on common mistakes observed during manual reviews while testing various QA generation prompts, and overall 18.11% TF questions and 24.20% MC questions were filtered out. Further validation, presented in Section 3.3 and Appendix A.4, includes a human evaluation of the filtering process. We find strong agreement between human reviewers and LLM-assigned scores, with an average accuracy of 89.52% across the 7 principles.

We are aware of examples like those the reviewer mentioned and designed our filtering criteria such as “Answerability” to address such issues. While some imperfections are inevitable in any LLM-generated dataset, we note that for final QA pair acceptance, the agreement between LLM and human evaluations achieved 85% accuracy, indicating that the majority of the retained questions are valid and of acceptable quality. Moreover, if the questions were broadly unanswerable, we would not observe clear differences in LLM accuracy across models or the consistent degradation trends over time.

Comparison with forecasting markets

We would like to explain why we chose to focus on LLM-generated questions in this work:

1. Prior work collected data from 5 existing forecasting platforms, sourcing 48,754 raw questions (2015-2024). They note that many questions in the raw dataset were unsuitable, resulting in a much smaller filtered dataset of 5,516 questions compared to our 31,510. Also, in Figure 11, existing platforms offer limited coverage in earlier years (<300 questions per quarter before Q4 2021), making longitudinal analysis difficult. In contrast, our method supports high scalability and retrospective generation, allowing for uniform coverage across the full time range.
2. Concurrent work collected 2,532 questions from GoodJudgmentOpen to study temporal generalization. However, the trend is difficult to discern due to the limited number of data points (ranging from 2 to 8) for each model (Table 3). In contrast, while their bi-monthly accuracy results exhibit significant fluctuations, our dataset presents a clearer trend of monthly accuracy degradation, providing deeper insights into how LLM performance evolves over time.

Finally, while we do not claim that LLM-generated questions are of inherently higher quality, we believe our automatic QA generation approach offers several key advantages. If one sources questions from forecasting markets, the dataset update frequency is dependent on whether there are active users. In contrast, our approach enables daily updates, scalability, and more comprehensive event coverage, making it a valuable complement to human-curated forecasting benchmarks.

RAG results with one-year retrieval window

We conduct the suggested experiment with a fixed one-year retrieval window on a subset of 1,500 TF questions (randomly selected 25 questions for each month). As shown in Fig.S5, our key findings remain consistent: while models can benefit from retrieving more updated articles (RAG cutoff 2024-03), a degradation trend still persists over time. This reinforces the broader conclusion that temporal distance from pretraining continues to impact performance, even with external knowledge augmentation.

Choice of models in QA generation

Our initial version of the dataset was generated before GPT-4o became available. We compared question generation across GPT-3.5 & GPT-4 and the newer GPT-4o & GPT-4o-mini models using the same set of articles. Each model produced 48 TF and 48 MC questions, manually evaluated using the same seven QA filtering criteria. Newer models outperformed the older ones with a 54.55% win rate, highlighting their potential to improve question quality in future dataset iterations.

Others

1. Complete RAG results can be found in Appendix B.5.
2. Number of articles available for retrieval - Fig.S6
3. Paleka (2025) introduces a forecasting dataset to test LLMs’ logical coherence. While “generate-then-verify” via LLMs as a common approach, our main contribution lies in capturing and quantifying degradation patterns.

We appreciate your thoughtful feedback, and hope to address the concerns below:

Sample size of human evaluation

While the evaluation involved 60 questions, we respectfully note that this sample size aligns with standard practices in similar dataset validation studies. For example, TLB-forecast conducted human evaluations using 24 samples, SituatedQA utilized 100 samples for assessing human performance, and FreshQA similarly performed human evaluations on 100 samples. To further mitigate bias, we had 4 annotators per question and conducted inter-annotator agreement analyses, as shown in Appendix A.4.

Distribution of question categories and question types

We provide the distributions of question categories and question types over time in Fig.S1. Additionally, we conducted further analyses by balancing the categories monthly (selecting TF 5,520 out of 16,783; MC 4,680 out of 14,727) and further balancing both categories and question types within MC questions (2,400 out of 5,520). The performance degradation pattern consistently persists, showing that degradation primarily arises from increasing temporal distance rather than shifts in category or question distributions. For difficulty-level analysis, while it would be possible to assign difficulty using language models, we believe such automated measures would introduce unnecessary noise. We would rather see the observed model performance itself effectively reflects the inherent difficulty of the questions.

Evaluation of mid-size LLMs

We selected models based on recent popular choices, covering both open-source and closed-source options. We appreciate the reviewer’s suggestion regarding mid-sized LLMs. Accordingly, we've included Llama-2-13B and Qwen-2.5-14B, and provided an updated plot (Fig.S2) and table. As an earlier-generation model, Llama-2-13B underperforms compared to Llama-3-8B, though it still demonstrates higher performance before the knowledge cutoff than after. For Qwen-2.5-14B, we observe relatively strong performance on MC questions but near-random accuracy (~50%) on TF questions. Interestingly, we note that Qwen-2.5-14B exhibits a strong bias towards responding "No," selecting this answer 91.66% of the time on TF questions.

Table: Yearly Accuracy and YoY Accuracy Change for Llama-2-13B and Qwen-2.5-14B

Contribution to ML research

We believe much progress in ML research benefits from open benchmarks. To name a few, ImageNet, GLUE, SQuAD, …. Specifically, our benchmark provides contributions in two important research directions:
(1) LLM forecasting: Our large-scale, daily-updated dataset reflecting real-world events enables the training and evaluation of models to better support human decision-making.
(2) Continual learning: Our dataset highlights the challenge of maintaining up-to-date knowledge in LLMs. Our analysis demonstrates that even when provided with gold articles, performance degradation persists, emphasizing the necessity of continuous pre-training to mitigate outdated representations. With Daily Oracle, one could explore how continuous pre-training and efficient adaptation can address the performance degradation challenges presented in our work.

Would CoT prompting help?

We randomly sample 25 MC questions per month (1,500 in total) to study how CoT impacts the performance (See Fig.S3). For LLaMA-3-70B, we prompt the model to explicitly generate a rationale before providing the final answer. Compared to directly answering, we observe a slight performance improvement initially, though this advantage diminishes after 2023. Additionally, for DeepSeek-R1-Distill-Llama-8B, we utilize the original prompt without modification; however, since this model is fine-tuned to naturally generate reasoning prior to the final answer, we treat its outputs as CoT results. Its performance similarly matches that of the non-CoT approach.

This observation aligns with recent findings [4], indicating that CoT reasoning primarily benefits tasks involving mathematical, logical, or algorithmic reasoning, with limited gains on other task types. While improved CoT prompt engineering might yield better performance, the results presented here provide baseline insights, leaving further optimization for future research.

[4] Sprague, Zayne, et al. "To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning." arXiv preprint arXiv:2409.12183 (2024).

Thank you for your recognition of our work!