Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

We thank the reviewer for the detailed comments. Below are our responses to your concerns:

Weakness 1- major

Limited Innovations in Data Construction

We provide a comparison table in the general response. In short, our work improves upon the prompts of TCELongBench by significantly expanding its size and scope while incorporating several innovations to improve the quality of forecasting questions. It's important to highlight that our contribution does not focus on the data construction process itself—using LLMs to generate and verify data is a common approach. Instead, our novelty lies in the dataset's expanded size, broader scope, updated frequency, and most importantly, novel analyses of temporal generalization patterns.

Weakness 2 - major

No Human Verification

During the QA filtering step, seven principles are identified based on common mistakes observed during manual reviews of question quality while testing various QA generation prompts. A threshold of score 13 is defined as we only allow one point deduction to ensure the quality. Using this threshold, we generate a total of 37,775 QA pairs, of which 29,988 (79.39%) successfully pass the QA filtering step. For detailed results of the QA filtering process, please refer to Appendix A.3.

We also acknowledge the importance of human evaluations and inter-annotator agreement. While we are planning to design the human evaluation framework and recruit evaluators, we would also like to include our dataset in the Supplementary Material for your reference.

Weakness 3 - major

No Cost Description or Comparison

Thank you for highlighting concerns regarding the feasibility and cost-effectiveness of our data construction and evaluation process. We have conducted a detailed cost analysis:

For data construction:

Token counts (Mean ± SD):

• Article Summary: 1,472.94 ± 444.51 tokens
• QA Generation: 2,350.58 ± 505.53 tokens
• Misleading Choices Generation: 1,410.18 ± 502.58 tokens
• QA Filtering: 2,601.51 ± 500.09 tokens

Cost breakdown:

• GPT-3.5 ($ 0.5/1M tokens): Used for Article Summary and QA Filtering
• GPT-4 ($10/1M tokens): Used for QA Generation and Misleading Choices Generation

Average per-article cost: (1,472.94 + 2,601.51) × 0.5/1M + (2,350.58 + 1,410.18) × 10/1M = $0.0039

Monthly cost: 30 days × 6 articles × 0.0039 = $0.702

For Evaluation:

• Open-book setting: Primary cost driver (3,051.99 ± 336.41 tokens per prompt, including five reference articles)
• Closed-book setting: Minimal cost (~17.3 short questions daily)

In summary, our process demonstrates both feasibility and cost-effectiveness for large-scale implementation, with manageable monthly expenses of less than $1 for data generation.

Weakness 1 - minor

Relatively Trivial Conclusion

We would like to emphasize several key contributions and findings from our study:

• Novelty and scientific verification: While it may seem intuitive that LLM performance degrades after the knowledge cutoff date, our work is the first to scientifically validate this hypothesis through a new, daily updated, continuous evaluation benchmark. This benchmark not only captures but also quantifies this degradation pattern (e.g. the gradual decline even before the knowledge cutoff dates are observed, and the amount of the decline can be clearly measured), establishing a foundation for future temporal generalization studies.
• Additionally, we have expanded our analysis to include more model performance results in both the constrained open-book setting (updated in Appendix B.4, Figures 12-18) and the gold article setting (updated in Figure 5). In both cases, the degradation patterns remain evident.
• Notably, the degradation observed in the gold article setting is particularly surprising. In this setting, where answers are directly accessible, LLMs are expected to achieve consistent accuracy regardless of the publication date of the article. However, our findings reveal that outdated parametric representations hinder their ability to generate correct answers consistently, even in a reading comprehension context.

We appreciate the suggestion of testing LLM’s memorization changes over decades, however, due to time constraints, we leave it as a future item.

Weakness 2 - minor

No Estimate of Human Performance

Thank you for the valuable suggestion. We agree that incorporating a human performance baseline could enhance the study by providing a plausible upper bound. However, since humans already have knowledge of past answers, such a comparison may not be entirely fair. More importantly, we would like to emphasize that the primary objective of our benchmark is to underscore the need for continuous model pretraining rather than focusing on comparisons with human performance. The key insight of our paper is to encourage the development of more efficient methods for continual training. In other words, comparisons to human performance may be less relevant, as our primary goal is to call for bridging the gap between the model’s past performance and recent degraded performance.

Hi Reviewer VuTG, thank you so much for your thoughtful reviews and suggestions. We have addressed your questions and conducted new experiments and analyses as suggested. As the discussion phase is close to the end, we aim to ensure our responses fully address your concerns. We kindly ask you to consider our responses and updates in your final evaluation of our work.

Thank you for your further clarifications and detailed responses. I would like to clarify that my previous comments were specifically addressing the authors’ claim in the rebuttal that "our novelty lies in the dataset's expanded size, broader scope, updated frequency, and most importantly, novel analyses of temporal generalization patterns." As I mentioned earlier, I personally do not consider "expanded size, broader scope, updated frequency" as research contributions, and the claim of "novel analyses" also remains debatable.

On Research Contributions and Novelty:

However, I find some aspects of the authors' latest response to be either incorrect or overclaimed. To better position this submission, I would like to compare its contributions to prior work:

• Compared to [1]: This submission and [1] share very similar dataset construction methods and use the same corpus to build datasets. The primary difference lies in the following:

  • [1] includes human evaluation to validate dataset reliability but does not analyze temporal generalization
  • This submission analyzes temporal generalization and constructs a dataset with a larger temporal span and more data points.

• Compared to [2]: Both this submission and [2] analyze temporal generalization using datasets.

  • [2] introduces and applies multiple measures to assess temporal generalization and biases (including accuracy) and uses diverse data sources, with ground truth obtained via collecting human data rather than automatic generation.
  • This submission primarily uses accuracy as the metric, does not analyze bias, includes two task types (TF and MC), and has a single-domain data source (DailyNews), with all QA pairs generated by GPT. Also, this submission has the advantage of a larger dataset and a broader temporal span, alongside analysis across different QA settings.

As I am not an expert in this specific domain, I welcome corrections if I have missed or misunderstood any points. However, from my perspective, the claims made in the latest rebuttal, such as "the application of such a dataset to assess LLMs’ temporal generalization is both novel and unexplored" and "our work is the first to offer a continuous, quantifiable measure of LLM performance degradation over time" are not accurate. Work such as [2] has already investigated temporal generalization with more diverse measures. The main advantages of this submission are its larger dataset and broader temporal span. Considering these two related works, I find it difficult to view this submission as sufficiently novel or contributive to meet the acceptance threshold.

On Human Evaluation:

As noted earlier, [1] includes human evaluations across six dimensions as well as human performance. The assertion that “our findings demonstrate consistent declines in model performance across all settings, which remain robust despite minor variations in QA quality” is, in my view, unsupported. There is no scientific basis to claim that QA quality does not impact the robustness of observed trends across settings.

On Experimental Conclusions:

The authors also suggest that their conclusions differ significantly from those in [2]. However, I struggle to agree with this claim. Within the same time span (Jan 2023–April 2024), Figure 3 in this submission and Table 3 in [2] show very similar fluctuations in performance. While this submission provides a larger temporal span, the overall conclusion that "LLM performance degrades over time" is consistent across both studies.

References:

[1] Zhang et al. Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. ACL 2024.

[2] Zhu et al. Is Your LLM Outdated? Evaluating LLMs at Temporal Generalization.

Thank you for your recognition of our work. We appreciate the opportunity to address your concerns in detail.

Weakness

A running example, referring to a specific model with a clear cutoff date and a question regarding a piece of news, in order to highlight how the model would perform differently in different situations.

We are sorry for any confusion raised in the evaluation setup. We would love to further provide an example here as suggested, and please check Appendix B.7 (Figure 23) for how Mixtral-8x7B responds to a question under different experimental settings.

Question 1

It seems that the dataset is completely constructed automatically through the usage of LLMs - I was wondering if the authors have performed any manual check to assess the quality of the construction and if they could add details and evaluation metrics about that.

Yes, we have manually reviewed the quality of the questions while testing various QA generation prompts. We also acknowledge the importance of human evaluations and inter-annotator agreement. While we are planning to design the human evaluation framework and recruit evaluators, we would also like to include our dataset in the Supplementary Material for your reference.

Question 2

Are the authors planning to set-up a platform where users could test LLMs against the created dataset?

Thank you for your suggestion regarding a testing platform. Indeed, we plan to establish one, where users could track the performance of various LLMs daily or submit their own forecasting questions.

Hi Reviewer mAPh, thank you so much for your thoughtful reviews and suggestions. We have addressed your questions and conducted new experiments and analyses as suggested. As the discussion phase is close to the end, we aim to ensure our responses fully address your concerns. We kindly ask you to consider our responses and updates in your final evaluation of our work.

We appreciate your thoughtful feedback, and we hope to address the questions below:

Question 1

Did you evaluate the clustering approach? how is your clustering approach different from bert-topic?

Thanks for the question. To clarify, the article selection methodology described in Appendix A.1 is not presented as a novel contribution but rather as a technical implementation detail. However, our article selection methodology does incorporate some key components from BERTopic [1], including TF-IDF for representations, PCA for dimensionality reduction, and DBSCAN for clustering. We implemented careful threshold selection to filter out noise clusters and conducted manual verification of the articles within each cluster.

While we acknowledge the existence of more sophisticated topic clustering algorithms like BERTopic, our primary goal in hot topic selection is to construct representative QA pairs that capture daily trending information. However, article selection itself is not the central focus of our work. Importantly, we update a plot in Figure 22 of Appendix B.6, where we show that even with randomly selected article subsets in the closed-book evaluation, clear performance degradation patterns are observed. This indicates that even a random sampling approach yields a dataset capable of effectively highlighting the temporal generalization challenges in LLMs.

[1] Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794.

Question 2

What's the overlap rate between question and answer?

The overlap rate across our questions is low. For each question type, a maximum of two questions are derived from any single article. To further analyze this, we utilized the Sentence Transformer model 'all-MiniLM-L6-v2' [2] to get the sentence embeddings and then compute cosine similarity scores for the question pairs. Among the TF questions, the average cosine similarity score is 0.23, and for the MC questions, the average score is 0.20. The proportion of similarity scores greater than 0.5 is only 0.68% for TF questions and 0.56% for MC questions. Please also refer to the distribution plots in Appendix A.4, Figure 9.

[2] https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

Question 3

Degradation after cutoff is expected and RAG is commonly used to mitigate the problem. However, in this work the retrieval used to test RAG is quite weak. First, BM25 should be at least replaced with some hybrid or dense approach. Second, 5 top articles may not be enough (how did you choose 5?) and truncating the article at 512 can potentially cut off answers ( it is unclear if the provided information contain the answer. In other words, how do we know if the problem is the retrieval or the model ability to handle such information?

Thank you for your thoughtful comments regarding the constraint open-book setting. We would like to address your concerns about BM25 and provide additional clarification:

• We encourage reviewing our updated comprehensive RAG results for other models in Appendix B.4 (Figures 12-18), which demonstrate RAG's potential to enhance performance across some other models.
• Choice of Retriever: Our use of BM25 aligns with established precedent from prior work [2, 3, 4], providing a valid baseline for our dataset's constraint open-book setting. While we acknowledge the existence of more sophisticated hybrid and dense retrievers, BM25 serves our primary goal of establishing a baseline rather than maximizing performance. Crucially, even with potential RAG improvements, the observed performance degradation pattern persists, emphasizing the need for continual model updating.
• Number of Retrieved Articles: The limitation to 5 articles was primarily driven by computational cost considerations, particularly when implementing RAG with commercial models.
• Impact of 512-word Truncation: We are confident this limit does not significantly impact answer quality for several reasons
  • The gold article setting, which uses the same truncation, achieves ~90% performance, suggesting key information is retained
  • News articles typically front-load critical information in the opening paragraphs, aligning with our truncation approach

[2] Jin, W., Khanna, R., Kim, S., Lee, D. H., Morstatter, F., Galstyan, A., & Ren, X. (2020). Forecastqa: A question answering challenge for event forecasting with temporal text data. arXiv preprint arXiv:2005.00792.

[3] Zou, A., Xiao, T., Jia, R., Kwon, J., Mazeika, M., Li, R., ... & Hendrycks, D. (2022). Forecasting future world events with neural networks. Advances in Neural Information Processing Systems.

[4] Zhang, Z., Cao, Y., Ye, C., Ma, Y., Liao, L., & Chua, T. S. (2024). Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. arXiv preprint arXiv:2406.02472.

Hi Reviewer YiaM, thank you so much for your thoughtful reviews and suggestions. We have addressed your questions and conducted new experiments and analyses as suggested. As the discussion phase is close to the end, we aim to ensure our responses fully address your concerns. We kindly ask you to consider our responses and updates in your final evaluation of our work.

Thank you for your response. Due to the large use of automated methods, I still believe that the lack of human evaluation remains a weakness of this paper. After reading other reviewers concerns (that in part are similar), I believe that this paper would benefit from another round of review, after properly addressing these concerns. Since my other concerns were clarified, I'll raise my score to 5.

We sincerely thank the reviewer for their thoughtful and detailed feedback. Below, we address the raised concerns and suggestions:

Weakness 1

A comparison table that clearly outlines the key differences between their pipeline and previous work like TempLongBench

Thank you for raising concerns. As suggested, we provide a comparison table in the general response. In short, our work improves upon the TCELongBench by expanding the size and scope and introducing several innovations to enhance the quality of forecasting questions. It's also important to highlight that our contribution does not focus on the data construction process itself—using LLMs to generate and verify data is a common approach. Instead, our novelty lies in the dataset's expanded size, broader scope, updated frequency, and novel analyses of temporal generalization patterns.

A flowchart or diagram of the dataset construction process with annotations explaining the rationale for each step

Please see the data construction flowchart in Figure 1(a).

Also, the authors could add their prompts in the appendix for a better understanding.

Please see Appendix C.

Weakness 2

Provide a breakdown of how many questions passed each principle in their quality control process. Show or plot the distribution of the final data's score in each of the designed principle dimensions

Thanks for the constructive feedback. We’ve plotted the distribution as required in Appendix A.3.

Conduct a human evaluation on a randomly sampled subset

We acknowledge the importance of human evaluations and inter-annotator agreement. While we are planning to design the human evaluation framework and recruit evaluators, we would also like to include our dataset in the Supplementary Material for your reference.

Also, conduct human forecasting performance on the sampled subset as a reference. Include inter-annotator agreement scores to demonstrate the reliability of their assessments.

We agree that incorporating a human performance baseline could enhance the study by providing a plausible upper bound. However, since humans already have knowledge of past answers, such a comparison may not be entirely fair. More importantly, we would like to emphasize that the primary objective of our benchmark is to underscore the need for continuous model pretraining rather than focusing on comparisons with human performance. The key insight of our paper is to encourage the development of more efficient methods for continual training. In other words, comparisons to human performance may be less relevant, as our primary goal is to call for bridging the gap between the model’s past performance and recent degraded performance.

Weakness 3

Thanks for the suggestion. Below are some additional results (which can also be found in Appendix B.5) as requested:

Compute and report the average relevance score of retrieved articles to the questions.

We randomly sample 1,000 questions along with their top 5 retrieved articles and use the Sentence Transformer model 'all-MiniLM-L6-v2' [1] to get the embeddings. We then compute the average cosine similarity score between each question and its corresponding articles, yielding an overall average score of 0.49. For the distribution plot of these scores, please see Appendix B.5, Figure 19.

[1] https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

Plot a histogram of the temporal distribution of retrieved articles for different RAG cutoffs.

We provide a plot in Figure 20, Appendix B.5, illustrating the average date difference between the question date and the publishing dates of the retrieved articles across different RAG cutoffs. The results demonstrate that as the RAG cutoff moves forward in time, the BM25 retriever on average would retrieve newer articles from the question date. In other words, our RAG cutoff does impact retrieval behavior.

Analyze the correlation between article recency and model performance.

In Appendix B.5, we present a case study that visualizes the relationship between article recency and model performance. In this analysis, we select the Mixtral-8x7B model with an RAG cutoff of September 2023. We calculate the average date difference as the mean temporal gap between a question and its top 5 retrieved articles. These average date differences are divided into quantiles: Q1 = 176 days, Q2 = 364 days, and Q3 = 626 days. As shown in Figure 21, there is no evident correlation between article recency and model performance. (Those extreme values result from a smaller number of data points available for that specific date.)

Provide a specific case study of a few example questions, showing the full chain of retrieved articles and how they influenced the model's prediction.

We’ve added an example of how Mixtral-8x7B responds to a question under different experimental settings in Appendix B.7 (Figure 23).

Hi Reviewer 2UhV, thank you so much for your thoughtful reviews and suggestions. We have addressed your questions and conducted new experiments and analyses as suggested. As the discussion phase is close to the end, we aim to ensure our responses fully address your concerns. We kindly ask you to consider our responses and updates in your final evaluation of our work.

TCELongBench is a benchmark designed to evaluate LLMs in analyzing Temporal Complex Events (TCEs) - events composed of interconnected news articles spanning extended periods [1]. It includes TLB-forecast, a dataset for the future event forecasting task. While we adopt TCELongBench’s prompts in our QA construction process, our work improves upon it by expanding the size and scope and introducing several innovations to enhance the quality of forecasting questions:

[1] Zhang, Z., Cao, Y., Ye, C., Ma, Y., Liao, L., & Chua, T. S. (2024). Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. arXiv preprint arXiv:2406.02472.