AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval

Q1: The writing of the paper needs to be further improved and all the symbols need to be interpreted.

Thank you for your comment. We have updated the notation explanations in the PDF to enhance clarity.

Q2: The work focuses on a single textual modality, so why mention multiple data sources, multiple modalities in the introduction section?

Thank you for your feedback. In response, we have adjusted our text to more explicitly state our objective of exploring data modalities that are more complex than traditional structured time-series data. The revised sentence is presented below for reference:

However, as the demand for more accurate forecasts in diverse domains has grown, the need to integrate data from beyond the structured time-series modality has become apparent.

Q3: N_q is a subset of D. Why are the elements in N_q n and not d?

Thank you for pointing out this inconsistency. We have updated our notation to use $n$ instead of $d$ to represent each news article, with $n$ serving as an abbreviation for 'news'. Our text has been revised accordingly to reflect this change.

Q4: The dataset used is too small and results on more as well as larger datasets are needed to be validated.

Thank you for your feedback. The Autocast dataset [1] comprises approximately 5700 questions, including both training and testing sets. In comparison, the most similar dataset, ForecastQA [2], contains slightly over 10,000 questions in total. However, we contend that the quantity of questions should not be the primary criterion for assessing the actual scope of such datasets.

While Autocast has fewer questions, it offers a significantly larger context retrieval space. ForecastQA's questions are limited to the time span from January 11, 2019, to November 12, 2019, all within the year 2019. In contrast, Autocast's questions span from 2016 to 2022, necessitating the retrieval and reader module to potentially handle more intricate information that encompasses greater long-term unpredictability.

Furthermore, as highlighted in the Autocast paper [1], the closest benchmark dataset, ForecastQA, has certain limitations in its data. We quote from [1] as follows:

ForecastQA's questions were authored by crowdworkers without any forecasting experience. Consequently, these questions often lack coherence or clarity due to the absence of additional context, as exemplified by queries like "To how many people will the Representative of an internet speak to by September 2019?" or "In July 2019, will an article say there were no volunteers in 2016?"

We believe that exploring our methodology on additional datasets is a promising avenue for future research, and we are committed to pursuing this in our future work.

Ref:
[1] Forecasting future world events with neural networks. NeurIPS 2022.
[2] FORECASTQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data. ACL 2021.

We are thankful for the reviewer's valuable feedback.

Q1. Limited reproducibility.

We acknowledge the reviewer's concern regarding reproducibility. In our study, a significant finding is the contribution of pre-trained GPT-3 in enhancing event prediction by performing relevance re-ranking and text summarization. This is conducted on the original datasets that consist of prediction questions and a large number of news articles. During our implementation, these preliminary steps are executed before the training phase. We then save these enhanced retrieval results and integrate them into our dataloader for training the model. To achieve full reproducibility, it is indeed necessary to share the re-ranking results generated by GPT-3, which essentially constitute our modified training data. However, the substantial size of this data exceeds typical file size limitations. As a commitment to transparency and reproducibility, we plan to make these materials publicly available following the acceptance of our paper.

Q2. The isolated effectiveness of the Alignment Loss component.

Please kindly refer to Table 3 in our paper, where we present ablation studies focused on the impact of disabling the alignment loss in our model.

Additionally, in this rebuttal, we trained the FiD Static baseline with additional alignment loss. The summarized results of these experiments, specifically for the 0.2B model sizes, are detailed in the table below.

This alignment loss is beneficial for the plain FiD Static baseline, which lacks any LLM-enhanced components. However, in the full Autocast++ model, where relevance re-ranking and text summarization are employed, the alignment loss appears to have a diminished role in enhancing performance.

In our additional experiment FiD Static + GPT3, we demonstrate that the baseline performance can be significantly enhanced by using the pretrained GPT3-enabled techniques. We acknowledge that the effect of alignment loss is relatively minor when compared to using pretrained LLM in the retriever and reader.

Hi reviewer mh89,

Can you kindly check the authors' rebuttal to the weaknesses you pointed out? Specifically, the authors have responded to

• The absence of a detailed ablation on the alignment loss in isolation.
• The lack of open-source code.

We are grateful for the insightful feedback provided by the reviewer. Below, we address the reviewer's queries and have also updated our manuscript accordingly, with the changes highlighted in blue in the PDF.

Q1: revamp the baselines considered using the GPT3.

We have expanded our experimental results to include the FiD Static (0.2B model size) baseline, now enhanced with retrieval re-ranking and summarization using GPT-3.

Our findings demonstrate a notable improvement in the FiD Static model when incorporating GPT-3 enhancements in the retriever and reader stages. With these enhancements, the FiD Static baseline now exhibits performance much closer to our model.

It's important to note that the UnifiedQA and T5 baselines are not reliant on retrieval processes (as they do not utilize context from news articles), and thus, are not impacted by this modification.

Regarding the FiD Temporal baseline, it uniquely utilizes daily news articles as context. Given that the average question in the Autocast dataset spans approximately 130 days, this would necessitate processing a volume of articles about 130 times greater than our current setup using the pretrained GPT-3. While this presents an intriguing direction for future research, it was beyond our capacity to fully explore within the limited time frame allocated for this rebuttal.

Q2: Which GPT3 version was considered in the work?

In our experiments, we utilized the gpt-3.5-turbo-0613 API version of GPT-3.

Q3: impact of binning numerical questions and results on the baselines?

Our approach is based on the T5 transformer model [2], and this enables the easy integration of our numerical question binning method with the FiD Static baseline, which is also based on T5. We conducted comparative ablation studies regarding the binning on both the FiD Static and our Autocast++ models, each with a 0.2B model size.

This modification transforms the numerical question prediction from a regression task to a classification one, aligning it with the loss used for T/F and MCQ. Our empirical findings show that this adjustment consistently enhances the accuracy of numerical predictions and does not significantly impact T/F and MCQ performance.

Ref:
[1] Forecasting future world events with neural networks. NeurIPS 2022.
[2] Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 2020.