FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

We thank the reviewer for the thoughtful comments.

1. Regarding the reviewer’s suggestion about additional experiments, we agree that the paper could be strengthened by providing additional results with recent open-source models. We commit to benchmarking some open models (e.g., Llama 2, Mistral, and Zephyr) in the next version of our paper.

2. We have uploaded a new version of our paper where the best results in tables of results are bolded.

We thank the reviewer for the constructive review. We apologize for the delayed response. We spent the majority of the rebuttal period implementing and evaluating the auto-rater suggested by the reviewer. We are happy to report that it works very reliably and can thus be used in place of human evaluation, and we thank the reviewer for directly improving the accessibility and impact of our FreshQA benchmark.

1. The reviewer inquired about the number of human raters and whether future work will follow the same evaluation protocol. All model responses underwent a rigorous evaluation process conducted by two authors using our two-mode evaluation protocol. We believe that human evaluators possess the expertise and common sense required to detect issues like hallucinations, making them more reliable than automated evaluation metrics for assessing LLMs' factuality. However, researchers have the flexibility to adjust their evaluation methods if human evaluation proves challenging. An easily implemented alternative is to use standard metrics like F1/exact match or recall, which assess the overlap between the model response and the ground truth answer(s) (e.g., see You.com's recent evaluation: https://about.you.com/introducing-the-you-api-web-scale-search-for-llms/). Alternatively, researchers can utilize LLM-based automatic evaluation metrics such as FactScore (https://arxiv.org/abs/2305.14251) or our newly developed FreshEval metric (details below).

2. The reviewer asked about implementing an auto-rater for FreshQA. To facilitate future evaluations, we have developed FreshEval, a simple automatic metric that uses few-shot in-context learning to teach an LLM to assess model responses, achieving an average agreement of 96.5% with human evaluations for Relaxed and 96% for Strict (please refer to Appendix B in the updated version of our paper). In each evaluation, the model is conditioned on a given question, a list of valid answers for the question, and a model responses, and is then expected to generate a comment on the correctness of the response, followed by a final judgment (please see Figure 5 and Figure 6 for FreshEval’s prompts, and Figure 7 for its sample output). We are in the process of re-evaluating all models with FreshEval.

3. The reviewer suggested the possibility of reformatting FreshQA into a multiple-choice format, and we appreciate this input. Researchers are encouraged to consider reformatting FreshQA into other formats (e.g., multiple-choice QA, timestamped QA), if it aligns with their specific evaluation needs and objectives. Our dataset serves as an open-ended QA benchmark, allowing us to assess not only the correctness of responses but also the presence of any hallucinations. While adopting a multiple-choice format might streamline evaluation, there are several drawbacks to consider: (1) When presented with multiple-choice questions, models can potentially rely on heuristics to deduce the correct answer rather than genuinely understanding the context; (2) Assessing hallucinations becomes challenging in a multiple-choice setup, as models are simply required to select from predetermined options; and (3) Recent models are designed for open-ended responses, and real-time QA often does not involve presenting multiple choices. Thus, retaining FreshQA's open-ended evaluation can be valuable for assessing models' practical utility in real-world applications.

4. Regarding the reviewer's comment on the predictability of our results, while it is expected that LLMs without access to real-time data would struggle with questions requiring up-to-date knowledge, our findings on questions with false premises were somewhat unexpected. Regardless of model size, all models had difficulty with such questions, and larger models did not improve accuracy ("flat scaling"). To the best of our knowledge, we are the first to evaluate LLMs at scale on false-premise questions and report this flat-scaling pattern. Additionally, we discovered that LLMs can debunk false-premise questions when explicitly asked.

   On the method side, while it may not be surprising that injecting up-to-date knowledge into LLMs can enhance factual accuracy, our work contributes to the discussion on how to effectively integrate external knowledge, like web search results, into LLMs. We delve into the specifics of optimizing this process, offering practical strategies for maximizing LLM performance. In contrast to previous approaches by Lazaridou et al. (2022) and Press et al. (2022), which use only one retrieved evidence at a time, we demonstrate that recent LLMs can handle an increasing number of retrieved evidences inserted into the prompt simultaneously, significantly boosting performance.

5. As for FreshPrompt's compatibility with other search engines, we believe it can readily adapt. FreshPrompt's success depends on the number and order of retrieved evidences, which are typically available from most search engines.

We thank the reviewer for the insightful review.

1. The reviewer is concerned about potential challenges posed by FreshQA’s dynamic nature for consistent benchmarks. While FreshQA may be affected by the ever-evolving nature of information, we believe that it remains a valuable resource for the following reasons:

   • First, FreshQA's dynamic nature mirrors real-life scenarios where information is continually updated. This makes it a suitable benchmark for evaluating LLMs’ ability to handle dynamic and ever-changing information or provide answers based on the most recent updates. These evaluations demonstrate their practical utility in real-world applications, where knowledge updates occur in real-time. Researchers are encouraged to use FreshQA for evaluating models that can continuously learn and adapt to new information.

   • Second, researchers can adapt their evaluation methodologies to account for FreshQA's dynamic nature when needed. One approach is to use various “snapshot” versions of FreshQA that freeze answers at specific time points and include timestamps on questions to indicate when they were asked, similar to prior work such as TimeQA and SituatedQA. Alternatively, researchers can assess performance on a subset of the dataset where the relevant information (e.g., question type) has remained consistent over time. These methods help establish stable benchmarks for different time frames, although they may not fully capture the dynamic nature of real-world information.

2. Regarding the reviewer’s comment on FreshQA’s novelty, below we elaborate on the differences between FreshQA, TimeQA, and RealTime QA, and highlight the unique merits and contributions that FreshQA brings to the field.

   • First, while TimeQA contains mostly synthesized questions, FreshQA exclusively features human-generated natural questions, enhancing the dataset's authenticity and real-world relevance. Additionally, TimeQA is static, with time-stamped questions and fixed answers. Therefore, it does not require a model to access real-time data or browse the Internet for current information. In contrast, FreshQA is a dynamic benchmark where answers may change over time, requiring a model to understand the world's up-to-date knowledge to answer correctly.

   • Second, while RealTime QA emphasizes real-time information updates and how LLMs handle them, it exclusively focuses on news data. It re-evaluates models with approximately 30 new multiple-choice questions about recent events from news websites, annotated weekly by its authors. In contrast, FreshQA features a fixed set of open-ended questions covering a wide range of topics, including arts, music, politics, government, religion, science and technology, environment, transportation, sports, and more. These questions naturally allow for changing answers based on new developments in the world.

   • Third, FreshQA covers a wide variety of question types, including questions that involve old or never-changing world knowledge, questions with false premises that need to be debunked, and single-hop/multihop questions. This diversity of question types is essential for mitigating model bias and preventing overfitting. It guarantees that models' performance remains balanced across different question categories and does not become overly optimized for one category at the expense of others.

   • Lastly, our two-mode evaluation protocol, comprising relaxed and strict evaluations, offers a spectrum for assessing LLMs' factuality. This approach not only provides a deeper and more nuanced understanding of their performance but also quantifies hallucination, a crucial aspect absent in previous QA benchmarks.

3. The reviewer raised a question about FreshPrompt’s novelty. While FreshPrompt is not the first few-shot prompting method that integrates current information from a search engine into LLMs, it introduces several innovations. These innovations involve fully leveraging a search engine by extracting all up-to-date and relevant information (including knowledge from related questions), and efficiently handling an increasing number of retrieved evidences (including casting them into a unified format and reasoning over them to identify the most relevant and up-to-date answer). These contributions have resulted in performance improvements. Our analysis provides insights for future work on search engine-augmented LLMs. We found that both the number and order of retrieved evidences significantly impact the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers has been shown to reduce hallucination.

   Compared to Lazaridou et al. (2022)'s approach, which involves using a single retrieved evidence at a time and requires 50 inference calls to an LLM for each question to generate multiple candidate answers and select the best one, FreshPrompt is more efficient, requiring only a single inference call to an LLM.