GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond

Thank you for your insightful assessment of our work and the constructive queries raised. We are delighted to hear that you found our paper to be of good quality, significance and clarity.

Response #1 - Originality

We appreciate your feedback on the originality of evaluation dimensions. The focus of our work is to provide insights for better understanding the key techniques of leading LLMs, so as to improve their transparency for the community. Therefore, creating new LLM capability dimensions is not our first priority. Nevertheless, as LLMs are getting more and more powerful, we will continue the efforts of adding new evaluation dimensions in GPT-Fathom to assess advanced LLM capabilities, such as long-context understanding, multi-turn conversation, LLM agent, etc. With these new dimensions, we can study the strength and weakness of leading LLMs more comprehensively.

Response #2 - Open-sourcing and costs:

To facilitate the research community for reproducible and systematic LLM evaluation, we have open-sourced GPT-Fathom and will post the link in our paper after the anonymity period. With regard to the effort required to evaluate a new model, our codebase serves as a read-to-use toolkit for evaluating new models. Users just need to implement a completion function, and we provide a step-by-step tutorial in our open-source repo. The evaluation time cost depends on the model size, inference efficiency and available computational resources / API rate limtis.

Response #3 - Sufficiency of capability dimensions:

We believe that the chosen capability dimensions cover a broad spectrum of LLM functionalities. We acknowledge that as LLMs rapidly progress, as mentioned above, we will continue the efforts of adding new evaluation dimensions in GPT-Fathom to cover advanced capabilities. Currently, we have already added evaluations of factuality reasoning and long-context understanding, which is available in our repo (link will be posted after the anonymity period).

We hope these responses adequately address your questions. Your feedback is invaluable to us, and we look forward to further discussions on our work.

Response #7 - Aligned settings and ablation studies

Our "aligned setting" refers to the main evaluation results of LLMs on each benchmark are obtained by the same prompt, same number of "shots", same sampling hyperparameter, and same metrics, etc. Please note that these evaluation settings are aligned within each benchmark, but may differ across different benchmarks. For example, we adopt 1-shot setting for most benchmarks, but follow the most commonly used 5-shot setting for MMLU. In the ablation studies, we do alter the number of "shots" since we need to investigate how it impacts the performance of LLMs.

Response #8 - Clarification questions

Automating Web-version testing: We wrapped up the Web-version ChatGPT (including GPT-3.5 and GPT-4) into a local API for the ease of bulk evaluation, following some open-source projects such as ninja (https://github.com/gngpp/ninja).

Clarification on "pass@k" metric: This is a widely used metric in evaluating the coding capability of LLMs since [1], and we follow the standard definition of pass@k in our work: for each coding problem, k code samples are generated, and a problem is considered solved if any sample passes the unit tests, and the total fraction of problems solved is reported. We have added the definition of pass@k in the "Coding" paragraph of Sec 2.2 in our updated submission. Thank you for your feedback helping us improve the clarity of our work.

Number of "shots" in Tables 5 & 6: We report the results of our ablation study on how CoT prompting and prompt variation impacts LLM performance in Table 5 & 6, respectively, and we keep the number of "shots" consistent with our main setting (as in Table 1) rather than zero-shot. We do have the zero-shot setting in another ablation study on the impact of number of "shots" (Table 4), where we report the zero-shot performance on ARC-c and HellaSwag.

Different benchmarks use different "shots": Our ablation study on the number of "shots" (Table 4) shows that the performance of LLMs struggles to improve beyond 1-shot, particularly for stronger models. This indicates that 1-shot typically works well, which is our primary evaluation setting for most benchmarks (Table 1). However, for some benchmarks, there exists widely adopted number of "shots" in existing works, such as 5-shot for MMLU, 8-shot CoT for GSM8K, 0-shot for HumanEval, etc., and we follow these widely adopted "shots" for ease of comparing our results to existing reported scores. Note that these number of "shots" are also adopted in the paper of PaLM 2, so that we can compare with its performance using the same number of "shots" in Table 1.

We hope our responses above adequately address your concerns and clarify the aspects you highlighted. Your feedback has been instrumental in refining our work and we have made the suggested modifications based on your valuable feedback.

[1] Chen, Mark, et al. “Evaluating Large Language Models Trained on Code.” arXiv, 7 July 2021, http://arxiv.org/abs/2107.03374. arXiv.

Response #2 - Numbers from other papers

We apologize for the confusion about the evaluation settings. We clarify our choice as follows.

We refer to numbers from PaLM 2-L and Claude 2 papers since we currently have no access to their APIs (Google only provides access to smaller PaLM 2 models; Claude 2 only provides API access to organization users). We try our best to align our evaluation settings with these two models, such as the number of "shots", whether CoT prompting is used, and the evaluation metrics. Since PaLM 2-L and Claude 2 have different evaluation settings in their official papers, we separate them into two Tables (please refer to Table 1 and Table 2) and in each table, we evaluate other models with aligned settings on every benchmark for fair comparison.

For our own reported results, we strictly align all the evaluation settings and prompts, with the only exceptions for Llama models, where a few numbers are marked with * where we use optimized prompts. This is because we find that Llama models are quite prompt-sensitive, while OpenAI's models are pretty robust. If we randomly pick a prompt, the numbers of Llama models could be significantly lower than their officially reported scores, which may incur challenges of fairness of our evaluation suite. In fact, we study the model sensitivity in detail in later part of Sec 3.2 (please refer to Table 6), and the original scores without using optimized prompts are listed there.

A direct comparison with numbers from other papers is challenging due to the lack of released prompts and few-shot samples in these papers. This lack of transparency in benchmark settings underlines the necessity of our open-source reproducible LLM benchmark suite. Nevertheless, we do carefully compare our evaluation results with officially reported numbers, and most our results match the reported performance in official papers, within a margin of slight deviation.

Response #3 - Selection criteria for benchmarks and LLMs

At the beginning of Sec 2.2, we describe our criteria for benchmark selection: 1) cover as many aspects of LLM capabilities as possible; 2) adopt widely used benchmarks for LLM evaluation; 3) clearly distinguish strong LLMs from weaker ones; 4) align well with the actual usage experience of LLMs. Following these criteria, we carefully pick benchmarks in our evaluation suite. We admit that there is no "perfect" selection of benchmarks for any evaluation framework, since both benchmarks and LLMs are evolving. We are currently working on latest benchmarks such as SciBench, SummEdits and QuALITY for evaluating advanced reasoning and long-context capabilities of LLMs.

With regard to the selection criteria of LLMs, we also have a brief discussion in Sec 2.1 that our goal is to help the community better understand how OpenAI improves from GPT-3 to GPT-4 and pinpoint the position of LLMs under development. To achieve this goal, we mainly consider evaluating these types of LLMs: 1) OpenAI’s leading models, such as GPT-4; 2) OpenAI’s major earlier models, such as GPT-3 and InstructGPT; 3) other leading closed-source models, such as PaLM 2 and Claude 2; 4) leading open-source models, such as LLaMA and Llama 2. Since LLMs are still rapidly evolving, we continue the effort of evaluating cutting-edge LLMs, such as GPT-4-Turbo from OpenAI. In our updated submission, we have added the evaluation results of gpt-3.5-turbo-instruct-0914, a new model from OpenAI which was just released in 2023/10.

Response #4 - Benchmark categories

For the categories of benchmarks, we investigate many existing works especially the papers of OpenAI models, Claude 2, PaLM 2, LLaMA and Llama 2, which agree on a reasonable categorization of popular benchmarks. For example, BBH is often attached to the "reasoning" category, while GSM8K is clearly associated with the "math" category. We do admit that our categorization of benchmarks is by no means the exclusive approach, but a reasonable and commonly acceptable one.

Response #5 - Addressing open questions

We apologize for the confusion. The key question that the community is currently eager to know is how GPT-3 progressively improves to GPT-4, including technical details such as whether adding code data improves LLM's reasoning capability, which aspects of LLM capability can be improved by SFT and RLHF, how much is the alignment tax, etc. We briefly introduce these open questions in the introduction, and detail them in Section 3.2.

Response #6 - Prompt sensitivity

Your point on prompt sensitivity is well-taken. We acknowledge this as a valuable area for future research. Due to resource constraints, we could not comprehensively benchmark the prompt sensitivity of all models. However, we do clearly demonstrate the existence of this issue, and inspire the research community to address this challenge in the future.

Thank you for your detailed review and valuable feedbacks. We appreciate the opportunity to clarify and expand on certain aspects of our paper.

Response #1 - Research questions and improvement over existing benchmarks

The key research question studied in our work is: How GPT-3 progressively improves to GPT-4? Since OpenAI has not revealed much about their evolutionary path, to answer this question, we systematically evaluate all their major LLMs to break down the impacts of techniques such as adding code data for pretraining, SFT and RLHF. Therefore, instead of just creating yet another LLM leaderboard, the focus of our work is to improve the transparency of leading LLMs (especially closed-source ones) and offer insights for better understanding how their techniques improve LLMs across various capabilities.

Compared to existing works on LLM evaluation, our work improves in the following aspects:

1. Reproducible evaluation under aligned settings: LLMs are known to be sensitive to the evaluation setting, i.e., different settings and prompts may lead to very different evaluation results. Yet, many existing work simply refer to scores from other papers without consistent settings and prompts, which may easily skew the observations. In contrast, our work provides systematic evaluation of leading LLMs across various capability dimensions, all under aligned settings for fair comparison. We provide an open-source LLM benchmark suite as an off-the-shelf toolkit, so that other researchers can easily reproduce our reported LLM performance, and save a lot of efforts when comparing a new model with existing LLMs.

2. Novel insights on the impacts of SFT / RLHF: Our work provides in-depth analysis on how SFT / RLHF affects various LLM capabilities, such as SFT having opposite effects on weaker and stronger base models, and RLHF transforming pass@100 (intrinsic coding abilities) into pass@1 (one-take bug-free coding abilities). These novel insights offer valuable guidance for future advancement of LLMs.

3. Novel challenges of advanced LLMs: Our work discovers novel challenges that leading models have to address, such as the "seesaw phenomenon" in LLM capabilities, where certain capabilities improve, while a few other capabilities clearly regress. We demonstrate the existence of this phenomenon in both leading closed-source and open-source LLMs, inspiring the research community to dedicate more efforts to tackle this challenge.

4. Novel study on model sensitivity: Previous works on LLM evaluation rarely study the sensitivity of model, such as sampling variance and prompt sensitivity. In fact, many existing LLM leaderboards reference scores from other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results. In contrast, we demonstrate that even leading open-source LLMs are susceptible to prompt variations: a minor alteration of the prompt could cause the LLM to fail in solving the problem. We highlight the model sensitivity issue with our experimental results, and urge the community to improve the robustness of LLMs.

To summarize, GPT-Fathom is not just a collection of benchmarking results. We offer novel insights for community-concerned questions and discover novel challenges, aiming to improve the transparency of leading LLMs and inspire further advancements of LLMs.

We hope the clarifications above address your concerns about the research questions and novelty of our work. We appreciate your helpful feedbacks in refining our submission.

Dear Reviewer Fohj, we revised our paper with the comparison of our scores and the official scores (refer to the General Response to Reviewers). Please let us know your thoughts, and we are more than happy to answer any further questions.

Responding to each aspect discussed in your comment.

• "How GPT-3 progressively improves to GPT-4?" is not a research question. "Does adding code data to pretraining improve performance"? is a research question. The research question needs to specific, measurable with a metric, and outcome of the result should be clear. So you are actually addressing three research questions here. What is not clear to me is: have these not been studied before?

1. For the first comment, I was expecting a table with model name as columns and prior benchmark papers as rows, and finally your benchmark results at the bottom. I would like to see the impact of the "aligned settings" on the results compared to prior works. Your contribution hinges on a better benchmarking setup, but I'm not able to discern how much it is better compared to prior benchmarking setups where the prompts, shots, etc. were not aligned.
2. Can you make conclusions with statistical tests? The hypothesis "RLHF improves LLM capabilities" should have a yes or no answer based on the result of this test.
3. Are you the first to discover the "seesaw phenomenon"? Can you confirm this seesaw phenomenon consistently occurs with a statistical test?
4. Just saying the results change with change with prompt is not enough. What is your metric to measure sensitivity? How much sensitivity is too much to declare that LLMs are sensitive?

Thank you for your insightful review and feedbacks. We value the effort and time you dedicated to evaluating our submission and have carefully considered your comments. Here are our responses to the main issues raised.

Response #1 - Evolutionary path

We apologize for the confusion of the "evolutionary path from GPT-3 to GPT-4." To clarify, Figure 1 is not just a timeline of OpenAI's feature announcements. We use Figure 1 to visualize the major OpenAI models that are evaluated in our paper, and highlight the key techniques developed along the path from GPT-3 to GPT-4, e.g., adding code data for pretraining, using SFT / RLHF for fine-tuning. It is well-known that GPT-4 dramatically outperforms GPT-3, however, OpenAI has not revealed much about how they get there step-by-step. We evaluated all the major models between GPT-3 and GPT-4 to decipher their "evolutionary path", i.e., how each technique contributes to the advancement from GPT-3 to GPT-4. By providing a systematic evaluation suite, we aim to shed lights on the impacts of the key techniques used by advanced LLMs and improve the transparency of closed-source models.

Response #2 - Novelty

1. Beyond well-known facts: LLMs are known to be sensitive to the evaluation setting, i.e., different settings and prompts may lead to very different evaluation results. Yet, many existing work simply refer to scores from other papers without consistent settings and prompts, which may easily skew the observations. In contrast, our work provides systematic evaluation of leading LLMs across various capability dimensions, all under aligned settings for fair comparison.

   Our work is not just resonating some well-known facts like proprietary models perform better. Instead, we provide an open-source LLM benchmark suite as an off-the-shelf toolkit, so that other researchers can not only easily reproduce our reported LLM performance, but also save a lot of efforts when they need to compare a new model with existing LLMs across various capabilities. To cover the common use cases, we intend to select popular benchmarks with standardized settings and widely adopted metrics.

2. Novel insights on the impacts of SFT / RLHF: Our work provides in-depth analysis on how SFT / RLHF affects various LLM capabilities, such as SFT having opposite effects on weaker and stronger base models, and RLHF transforming pass@100 (intrinsic coding abilities) into pass@1 (one-take bug-free coding abilities). These novel insights offer valuable guidance for future advancement of LLMs.

3. Novel challenges of advanced LLMs: Our work discovers novel challenges that leading models have to address, such as the "seesaw phenomenon" in LLM capabilities, where certain capabilities improve, while a few other capabilities clearly regress. We demonstrate the existence of this phenomenon in both leading closed-source and open-source LLMs, inspiring the research community to dedicate more efforts to tackle this challenge.

4. Novel study on model sensitivity: Previous works on LLM evaluation rarely study the sensitivity of model, such as sampling variance and prompt sensitivity. In fact, many existing LLM leaderboards reference scores from other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results. In contrast, we demonstrate that even leading open-source LLMs are susceptible to prompt variations: a minor alteration of the prompt could cause the LLM to fail in solving the problem. We highlight the model sensitivity issue with our experimental results, and urge the community to improve the robustness of LLMs.

   To summarize, GPT-Fathom is not just a collection of benchmarking results. We build open-source toolkit for reproducible LLM evaluation, offer novel insights for community-concerned questions, and discover novel challenges, aiming to improve the transparency of leading LLMs and inspire further advancements of LLMs.

We hope our responses above address your concerns and demonstrate the novelty of our work. We appreciate your helpful feedbacks in refining our submission.

I was looking for something like this in the paper. So this is a good addition, it shows you can replicate results. Some suggestions below:

1. Instead of conclusions such as "slight deviation" and "pretty marginal", make conclusions with statistical tests.
2. Is the second table, are you following the same prompts as the official benchmark?
3. Experiments here are not sufficient to conclude prompt sensitivity, that requires a different experiment with its own hypothesis testing.
4. I was looking for a similar table comparing your benchmark with the other open source benchmarks. This would demonstrate that your aligned setting did improve on prior benchmarks. I still do not see conclusive evidence that you are improving on other benchmarks.

Thank you for your insightful suggestions. We respond to your comments one-by-one as follows.

1. We never treat official score as "ground truth", i.e., deviation from that does not mean anything like "error" or "variance". As we claimed above, the official prompts and in-context examples are not released. For most LLMs, different prompts lead to different results, and we don't think statistical test is helpful to compare two results that are supposed to be different already. The really big issue here is the non-transparency and non-reproducible evaluation results. There could be the risk of cherry picking / prompt engineering for better results. A key contribution of our work is the fully open-source and reproducible evaluation toolkit. We just use the official score as a reference for "sanity check", i.e., make sure our implementation produces reasonable results. Although this sounds easy, existing works on LLM evaluation may not pass this line.

   For instance, the latest HELM leaderboard v4.0 (https://crfm.stanford.edu/helm/latest/#/leaderboard), which was just updated days ago, reports that Llama 2-70B achieves 58.2 on MMLU, while the official score is 68.9 and our result is 67.8 as in the second table above. Let's say someone has a new LLM and would like to compare with existing leading LLMs. If this new LLM achieves 62.0 evaluated by HELM v4.0, can we say it significantly beats Llama 2-70B? Probably not, since Llama 2-70B can actually achieve even higher score than 62.0 not only from the official paper but also reproduced in GPT-Fathom. This highlights our contributions on reproducing the evaluation results, and then make our results fully open-source and reproducible.

2. As mentioned above, the prompts used in official scores are not released, and most benchmark datasets have not provided an official prompt for evaluation. We do follow the official prompt if the benchmark happens to provide it, as described in Table 8 of our paper.

3. The tables here are just to compare our results vs. official scores, and are not trying to conclude the prompt sensitivity of any LLMs. In fact, we experimentally study the prompt sensitivity in Table 6. Note that it may be very challenging to "prove" that one LLM is not prompt-sensitive, however, just one counter example (e.g., our case study in Table 6, a slight alteration of the prompt template results in the score of Llama 2-70B on TriviaQA plummeting from 74.0 to 55.5) is enough to demonstrate that one LLM is indeed prompt-sensitive. As for the quantitative metric to assess the level of prompt sensitivity, this could be a great research topic but out of the scope of our work.

4. "Aligned setting" should be a basic yet critical requirement for any LLM evaluation works to ensure an apple-to-apple comparison. For example, the performance on MMLU under 0-shot setting may not be compared to that under 5-shot setting. Our key contribution is not to demonstrate the benefit of "aligned setting", which is purely logical. Instead, we provide a fully reproducible evaluation toolkit that our settings are not only internally aligned, but also verified to produce results that can externally match the official scores of leading LLMs, such as GPT-4.

   To the best of our knowledge, we are the first LLM evaluation work that simultaneously achieve the followings: 1) fully open-source and reproducible; 2) evaluation settings are aligned on every benchmark; 3) evaluation results match official scores of leading LLMs; 4) cover all major OpenAI models from GPT-3 to GPT-4 for retrospective study. These important desiderata show that our work significantly improves over prior LLM leaderboards. As for "a similar table comparing with other benchmarks", it is infeasible to make such comparisons since other benchmarks have not covered all the desiderata listed above.