Efficient Evaluation of LLMs via Branching Preference Learning

Thanks for your review comments.

For Concerns:

Q: The authors missed a lot of key related works

1. We clarify our task and related work (refer to common responses 1, 2, and 4 for more details):

• Our work has broader application scenarios; it is not only for benchmarking model capabilities but also for reward modeling, preference data construction, and correction models [1] (see Sec 5.5 for related experiments). In subsequent experiments, we show our results on Chatbot Arena, which exhibit a strong correlation with human scoring.

• Our work supplements benchmarks like MMLU and GSM8k (as discussed in common responses 1 and 2). While MMLU and similar evaluation tests focus on correctness scoring, many real-world problems don't have strict binary labels (correct or incorrect) and require preference scoring. Our research aims to address the shortcomings of these automated evaluation benchmarks.

• Our work can provide an open-source, automated alternative to replace GPT-4 for evaluations (Like WildBench relies on GPT-4 for evaluation).

• Our contributions (see common response 4 for more details): our aim is not only to provide a better evaluation method for ranking a large number of LLMs but also to assist with tasks that require human evaluation. The core contribution of our paper is transforming the evaluation task into an optimization problem over an evaluation tree, enhancing evaluation capability through preference branch selection.

Q: I think the writing needs improvement.

2. We will make revisions based on your suggestions:

• Create a figure to illustrate how the evaluation task is performed. As described in section 3 (lines 91-95), the preference evaluation task involves taking two AI responses and determining which one better addresses the query (win, lose, and tie).

• Include key conclusions in the captions. The main conclusions are already included in the paper, but if you have specific concerns, please let us know.

Q: just evaluating 5-6 models is far from enough. like model rankings?

3. New experiments. We promise to include these experimental results in the next versions:

The quality and diversity of dialogue data are certainly crucial for training models. While collecting more diverse data is important, it is not the core contribution of our work. Our method can achieve significant performance improvements by incorporating more dialogue data, demonstrating its scalability and effectiveness.

• We conduct Model Ranking, referring to [2] (all data from official github repo). Considering that our approach is more suited for Pair-wise Evaluation rather than Single Evaluation, the evaluation results might be biased. Nevertheless, our results exhibit a strong correlation with human scoring. Experimental setup: We used the DPO model under the OOD setting to score different models. We present our model's scores (Our Score) and the scores from the ChatBot Arena Leaderboard [2] as a reference (Arena Score). Delta represents the change in ranking.

(2) We conduct experiments with other baselines [3]. It is worth noting that our model was trained using only 7K dialogue data, significantly less than the 200K used by PROMETHEUS.

(3) We add experiments on the more general benchmark Rewardbench [4], which covers the AlpacaEval data you mentioned. Please refer to the common response 5 for detailed experimental results. These results effectively validate the efficacy of our method. Our model outperforms many commercial APIs and 70B parameter models. (Importantly, our experiments did not use any human labels, providing strong evidence for scalability.)

The RewardBench evaluation dataset contains following:

• Chat (alpacaeval-easy, alpacaeval-length, alpacaeval-hard, mt-bench-easy, mt-bench-medium)
• Chat Hard (mt-bench-hard, llmbar-natural, llmbar-adver-neighbor, llmbar-adver-GPTInst, llmbar-adver-GPTOut, llmbar-adver-manual)
• Safety (refusals-dangerous, refusals-offensive, xstest-should-refuse, xstest-should-respond, do not answer)
• Reasoning (math-prm, hep-cpp, hep-go, hep-java, hep-js, hep-python, hep-rust)

Our work demonstrates that (1) automated data construction can rival or even surpass human collected data, and (2) automated preference optimization methods can help evaluation models rapidly identify key evaluation criteria, ensuring efficiency in the evaluation process.

Q: Is it necessary to consider evaluations specifically for dialogue settings?

4. Thanks for your question. We will revise any inconsistent expressions in the paper accordingly (see common response 3). Furthermore, our experiments cover a broad range of domains, and we have supplemented experiments on Rewardbench, which should support the claims in our work and demonstrate the effectiveness of proposed method.

[1] LLM Critics Help Catch LLM Bugs (OpenAI 2024)

[2] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (NeurIPS 2023)

[3] PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models

[4] RewardBench: Evaluating Reward Models for Language Modeling (Allen AI 2024)

Thank authors for your responses. I keep my original score unchanged.

Thank you very much for appreciating our idea of transforming the evaluation task into a tree search problem. We hope the following responses can further address your concerns：

Q: The paper is honestly pretty hard to follow. There's a lot of moving parts and it's not explained in an easy to digest way

1. Preference evaluation is indeed a rapidly developing field. We have detailed the task background, related work, and our contributions in the common response, hoping to alleviate your concerns. Additionally, considering that Reviewer YdUv Strengths 2 and Reviewer sKgh Strengths 1 acknowledge our work clear and easy to understand, could you please specify which parts is difficult to comprehend? This will help us make better revisions.

Q: The specific method presented seems a little bit ad-hoc, and could be justified better in the paper

2. We believe our method for decomposing the evaluation task is reasonable, though not the only possible approach. Using a different decomposition method would not diminish our contributions: (1) Human evaluations also rely on criteria, scoring guidelines, and judgments. Recent research [1] has highlighted the need for multiple rounds of negotiating these criteria and guidelines for human evaluators. Another study [2] employed an automated evaluation process. There are certainly better ways to decompose evaluation tasks, and we are open to exploring them further. (2) Our primary focus is not on the decomposition method itself, but on how to avoid relying on human supervision and efficiently identify the key evaluation criterion, as discussed in common response 4.

Q: Looking at Figure 1, it doesn't seem that their method improves all that much over the baseline

3. (1) Our method shows significant performance improvement over the initial model (w/ tie 55.89 vs. 49.64 w/o tie 75.76 vs. 57.02). (2) For the AutoJ and Fennec methods, it's important to note that our settings differ. Our experiments are conducted in OOD scenarios, using training sets samples from large-scale data rather than human collected (Figure 4 illustrates the differences in data domains). Additionally, we have added new experiments in the common response 5 to demonstrate the effectiveness of our method. On the RewardBench, our model outperforms the 70B LLaMA3 and the Prometheus v2, which was trained on 200K evaluation data, while our training data accounts for only 10% of theirs.

Q: The related work seems pretty sparse.

4. As discussed in the common responses 1 and 2, we will add more related work to address your concerns. It's important to clarify that the challenges in preference evaluation (LLM evaluation) include not only accuracy evaluations like MMLU and Math Reasoning but also open-ended generation that requires multiple evaluation dimensions. Our work aims to address multi-dimensional evaluation in complex and diverse user intent scenarios, as mentioned in the abstract (line 4: Particularly, in complex dialogue scenarios involving diverse and intricate user intents). Furthermore, our experiments include Code and Math test data in Table 2, reflecting improvements in these areas. Our new experiments on RewardBench show significant performance improvements in reasoning tasks (nearly 10 points, as referenced in the common response).

Q: Figure 4, the text is really small and hard to read.

5. We will add a Table for Figure 4 to explain the percentage of each category. Figure 4 illustrates the color differences in various domains, indicating significant differences in the constructed OOD data domains, which are more challenging.

Q: The description of the growth / pruning part is pretty confusing and in general a better explanation of what exactly is happening there would be really helpful.

6. I'm not sure what caused your confusion. If you could specify, it would help us make better revisions. In fact, we have attempted a clearer explanation: The purpose of constructing an evaluation tree: (1) to expand the candidate space of the evaluation process and (2) to create SFT data for different evaluation paths and (chosen, reject) DPO data. Our method achieves goal (1) through heuristic sampling and temperature sampling (lines 151-154). Goal (2) is accomplished using two consistency pruning methods, with consistent samples used as SFT data and inconsistent ones as DPO data (lines 158-161).

Q: why both of these criteria need to be met? Isn't the swap part already covered by the consistency evaluation?

7. Agreement (AGR) reflects the consistency between LLM evaluations and human judgments. Consistency (CNS) represents the extent of bias introduced after swapping positions, indicating the stability of the model's evaluations. Please refer to [2].

[1] A Holistic Approach to Undesired Content Detection in the Real World (OpenAI 2023)

[2] Branch-solve-merge improves large language model evaluation and generation （Meta 2023）

[3] Generative judge for evaluating alignment (ICLR 2024)

Thanks for your review comments!

If you could add some more clarity in the paper about how the AutoJ and Fennec settings differ from yours:

We outline the settings and core challenges, highlighting the differences:

• Auto J's approach evaluates two different AI responses using a one-step evaluation process. The dialogue data in its training dataset was human created to match the distribution of the test set, as shown in Figure 4. During training, only the SFT method was used, with no DPO training included.

• Fennec employs a multi-step evaluation, using criteria and scoring guidelines to alleviate the complexity of the evaluation task. The training also involved human selected dialogue data and included only SFT training, with no DPO training.

• In our approach: 1）There is no human selection or labeling cost—both the sampling of dialogue data and preference labels are fully automated. 2）We use a multi-step evaluation process to simplify the difficulty of complex evaluation tasks. 3）Most importantly, we incorporate DPO optimization, which helps the model achieve better performance with fewer branches, enhancing evaluation effectiveness.

• Our method demonstrates that even a 7B parameter evaluation model has significant potential compared to the 70B model used in [1] for evaluation tasks. This advantage is largely due to our task decomposition (similar to Chain of Thought), which breaks down the evaluation process into three steps: generating evaluation criteria, scoring guidelines, and final judgements results. These steps and their intermediate outputs simplify the task, making the 7B model more capable in complex scenarios.

Your method has a large description length, which generally goes against my prior, and makes more skeptical that every aspect of the method is necessary and also muddles the true contribution of the work.

• If possible, could you please point out the details that might be unclear to you? This will help us explain our work more clearly.

• We will also include a new Figure to illustrate how the evaluation task is conducted: The evaluation model receives a user query and two AI responses, and it only needs to assign one of three labels (win, lose, or tie) to indicate which AI response is better, or if they are equal.

• In the Methods section (Section 4), we cover the following: 1) Section 4.1 explains how we collect the dialogue dataset. 2) Section 4.2 shows how we train an initial model. 3) Section 4.3 details how we construct the evaluation tree. 4)Section 4.4 describes how we collect training data based on the evaluation tree. 5) Section 4.5 demonstrates how we train the SFT and DPO models. Among these, only Section 4.2 follows a process similar to other related work. The remaining sections introduce new approaches developed in our work, which may explain some discrepancies with your prior understanding. Even compared to the latest research [1], our work offers novel contributions: We demonstrate that it's possible to improve evaluation model performance without relying on human labeled data, and we also showcase the potential of a 7B model, which is popular in both academia and industry.

I think positional consistency is a pretty random heuristic. I agree that LMs might be sensitive to position and whatnot, but LMs are sensitive in all kinds of ways to prompts, so why focus on just position

• The issue of positional consistency was not proposed by us; please refer to references [2] and [3].

• Given that works [3], [4], and [5] all consider positional consistency and use Agreement and Consistency as evaluation metrics, I do not agree this as a flaw in our work.

• We fully agree with your point that LLMs exhibit more bias beyond positional consistency. However, since positional consistency is the most direct and easily improvable, I believe it is necessary to address these biases during training. In fact, with pirwise evaluation, if positional consistency significantly decreases when positions are swapped, it would be unacceptable for downstream applications such as reward models or model corrections.

Considering that both Reviewer YdUv and Reviewer sKgh also agree our presentation to be clear, we hope to engage in further discussion to address any concerns for you.

[1] Self-Taught Evaluators （Meta 2024,8）

[2] Large language models are not fair evaluators

[3] Generative judge for evaluating alignment

[4] PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models

[5] OffsetBias: Leveraging Debiased Data for Tuning Evaluators

Thank you very much for appreciating the novelty of our work. We hope the following responses can further address your concerns:

Q: If in-distribution evaluation performance is mediocre but out-of-distribution does better, then doesn't the most gain come from a better dataset?

1. We try to explain in detail why counter-intuitive results appeared in the in-distribution (ID) experiments, particularly as it may be related to "the curse of recursion" [1]:

• First, there is a key difference between in-distribution (ID) settings and other (OOD and transfer) settings: the training data in the ID setting consists of high-quality, well-defined domain data collected by humans (including baseline AutoJ and Fennec). Therefore, in ID setting, the initial model, SFT model, and DPO model were all trained using the same Dialogue data. In other words, we iteratively synthesized new training data, and the queries for these data are the same. In contrast, the OOD and transfer settings used different Dialogue data (as described in lines 201-216 and in the Appendix).

• Recent studies on iterative data synthesis have found that continuously using synthetic training data can led to model degeneration because the model tends to converge to a single modal distribution. As noted in [1], recursively using model-generated data can lead to "model collapse". This suggests that the training instability in our ID setting is likely caused by the use of synthetic data.

• This phenomena align with our findings in the paper. In the OOD and transfer settings, where new training data is used, training remains stable. The new data typically constrains the model training process, helping it maintain a diverse distribution and preventing it from converging to a single distribution. Therefore, we recommend using dialogue data with different distributions at the SFT or DPO stages to prevent "the curse of recursion."

• Despite this, the ID experiment results were still impressive, achieving the highest agreement score of 57.18 on Eval-P, a significant improvement over Fennec (note that in the ID experiment, our methods and Fennec used the same training data). All above evidence proves the effectiveness of our method.

Regarding "most gain coming from a better dataset," the direct evidence is that the performance of our Initial model（trained on the our collected data）, is only 49.64, which is significantly lower than Fennec's 55.36. This shows that our data did not yield better results through direct training. However, considering another aspect of data or task diversity, our collected data is certainly better (see Sec 4.1 line 133-135). We sampled from a large-scale dialogue dataset rather than a specific data source. We then apply the K-Means algorithm to cluster the data. Subsequently, we sample data from these clusters, ensuring that the training dataset encompasses a diverse set of dialogue scenario. This validates our motivation: even without human collected or labeled data, the model can achieve comparable or even better performance. This also indicates that we can significantly reduce labor costs for preference evaluation tasks in the future.

Q: Are they overlapping each other or completely different?

2. In our experiments, only a minimal chance exists for the model to output semantically similar criteria. Our prompt is provided in Table 10, and Table 11 shows specific examples. It’s important to note that the idea of automatically generating criteria has been proposed in previous work [2]. Our contribution lies in guiding the model to prioritize criteria with high discriminative capabilities. Table 1 shows that we can achieve better results with fewer inference steps (branches).

Q: Why is each criteria subtree only a binary tree?

3. You are correct that evaluation trees can extend to multiple nodes. However, considering: (1) Computational efficiency, each sample to be evaluated includes 10 criteria (k=10), each criterion includes 2 scoring guidelines, followed by 4 different judgments (see lines 151-157). Thus, there are 80 different subtrees. The purpose of building the evaluation tree is to expand the search space, and the current method already has a high computational complexity, making additional children unnecessary. (2) Scoring guidelines and judgments are sampled by adjusting temperature and swapping response positions. The diversity obtained by adjusting temperature is limited and requires different conditions (criteria and scoring guidelines). (3) Another purpose of constructing the evaluation tree is to obtain preference pairs (chosen, reject) for DPO training. Clearly, a binary tree is sufficient.

Q : It is unclear how to sample multiple evaluation path? By using a high temperature?

4. In lines 151-154, we have stated how different tree nodes are generated, i.e., criteria are ensured to be diverse by adjusting the model’s prompt, while scoring guidelines and judgments are obtained by adjusting temperature (specifically, judgments can be obtained by swapping response positions).

Q: What is the Eval-P benchmark?

5. Eval-P was proposed by AutoJ [3]. We will add the citation at line 199.

We also provide a more detailed background and related work to clarify our work, as well as clearer contributions and new experimental results. Please refer to the common response (Author Rebuttal). We hope this can further address your concerns.

[1] AI models collapse when trained on recursively generated data（Nature）

[2] Branch-solve-merge improves large language model evaluation and generation （Meta 2023）

[3] Generative judge for evaluating alignment (ICLR 2024)

Thank you very much for your suggestions, which have helped us improve the quality of our work. Following your advice, we have conducted new experiments and to provide the following clarifications：

• We have never denied the contribution of data, especially the additional unlabeled data (in the OOD setting) used in our approach. However, it is important to note that this data is unlabeled, making it widely accessible without increasing extra costs.

• These additional data are simply dialogue data. The purpose of the evaluation tree is to generate the criteria, scoring guidelines, and judgments required to train the evaluation model. Therefore, including only dialogue data is insufficient for training an evaluation model. Hence, we cannot consider the new unlabeled dialogue data as the key to improving performance. Our approach requires combining these dialogue data and constructing SFT and DPO data.

• Most importantly: Our proposed method is based on the observation in Figure 1, which shows that increasing the number of branches can enhance evaluation performance. This means that even without the additional data, the initial model can achieve good evaluation performance (though it may require > 40 branches). The SFT and DPO methods are designed to enable the model to efficiently reach the final result (< 3 branches). This is also why we employed branch ensemble in Section 4.4, as branch ensembles generally yield better decision results and provide a higher upper bound on performance. The purpose of designing the evaluation tree is to allow the model to more quickly identify the key evaluation paths, which can be derived from the branch ensemble results (as referenced in Section 4.4). Thus, the evaluation tree is largely aimed at accelerating the evaluation process by sampling fewer branches (yet more crucial). I believe our method's design and experimental results are well-supported by the substantial evidence mentioned above.

• We verified whether model collapse was the reason for the impact on ID performance. By mixing the original training data with the newly synthesized data when training the SFT model according to your advise (thanks again), we observed significant performance improvement, which supports our initial hypothesis in Section 5.7. Continuously training the SFT model and DPO model with the same data affects model convergence. However, compared to the OOD setting, there is still a lack of new unlabeled data, making direct comparisons still unfair. If we obtain more unlabeled domain-specific data, we believe our methods can achieve even better results. Considering the rebuttal time limitation, we may not be able to conduct additional experiments. However, our experiments, along with the common response experiments on the Reward Bench, should be sufficient to address your concerns.

Caption：Our Mix Training setup differs from that in the paper by using both original and synthetic data to train the model, whereas the paper only used synthetic data for training the sFT model. As a result, we achieved an Agreement of 58.41 vs. 55.80 , and a Consistency of 79.69 vs. 74.14 compared to Fennec.

Thank you very much for your appreciation of our work. We hope the following responses can address your concerns:

Q: It seems that no human preference/judgment label is used.

1. You are right that we do not use any human labels (see common response 4). As you can see, many research fields and studies lack sufficient resources (time and money) to employ annotators for labeling preference data. However, the APIs from OpenAI or Claude are also very expensive, and we aim to address these problems by providing open-source evaluation datasets and models to contribute to the community.

Q: how did you obtain different versions of J1?

2. Yes, there are two types of self-consistency here: (1) Consistency after swapping positions (yellow dot J1 and red dot J1) and (2) Consistency due to sampling temperature (tmp>0) leading to different judgments (J1 and J2). Thus, we obtain four evaluation results, and these judgments should be consistent, based on the same criteria and scoring guidelines.

Q: how do you create the ensemble?

3. The idea of using a model ensemble stems from our findings in Figure 1, which show that increasing the number of branches significantly improves the model's inference performance. Therefore, we increase the number of branches during inference (to about 80) to collect the model's preference labels (including 'win', 'lose', 'tie'). We then use the ensemble results from all branches as the final preference labels.（It is worth noting that the ensemble is highly effective because the classification labels are limited, although we still approach this as a generative task rather than a classification task.）

Q: Where is the result of using Auto-J as a judge to create labels for DPO?

4. The results in "w/ GPT-4" are those evaluated using GPT-4 for preference evaluation (rather than dialogue evaluation) and used for training. However, the improvement may not be significant as shown in Sec 5.4. We suspect this is because the preference evaluation task is also challenging for GPT-4, further demonstrate the importance of this area we are exploring.

Q: Auto-J dagger (second row). What is this!?

5. In line 218, we explain the meaning of the dagger (it represents our reproduced results). We will also clarify this in Table 2 to ensure a clearer description.

Q: Can you expand on why AGR and CNS are good evaluation metrics?

6. The purpose of preference evaluation is to measure alignment with human preferences, evaluating whether different AI responses better reflect human behavior or values. Agreement (AGR) [1] reflects the consistency between LLM evaluations and human judgments, serving as a direct metrics. Consistency (CNS) represents the extent of bias introduced after swapping positions, indicating the stability of the model's evaluations. Currently, it is a relatively good until we find more precise and fine-grained metrics.

• To address your observation on "The performance improve over AGR is minor, compared to GPT-4": Firstly, it's important to note that most responses can be judged as either "win" or "lose," while a "tie" typically indicates that an effective discriminative criterion has not been identified (even with human evaluations). If a more suitable criterion or strict scoring guidelines were applied, many of these "tie" labels could be classified as either "win" or "lose." From this perspective, our method effectively identifies more discriminative criteria and scoring guidelines. This allows cases that human evaluators might label as "tie" to be classified as either "win" or "lose." Consequently, this leads to a more noticeable AGR improvement in scenarios "w/o tie" labels. This directly supports our claims and demonstrates the effectiveness of our method.

• "why CNS however is beating GPT-4?" Our evaluation models mitigate position bias by training on data where response positions are swapped. In our DPO training, the consistency of swapping positions of resposne is included in our training objectives and datasets (line 158-161).

Q: can you offer some explanation on why DPO is unstable for in-distribution training?

7. We are very happy to discuss this issue, particularly as it may be related to the "curse of recursion" [2]:

• First, there is a key difference between in-distribution (ID) settings and other (OOD and transfer) settings: the training data in the ID setting consists of high-quality, well-defined domain data collected by humans. Thus, in our experiments, both the SFT and DPO processes use the same dialogue data. In other words, we iteratively synthesized new training data, and the queries for these data are the same. In contrast, new data is used in the OOD and transfer settings (as described in the Experiments and Appendix).

• Recent studies on iterative data synthesis [2] have found that continuously using synthetic training data can led to model degeneration because the model tends to converge to a single modal distribution.

• This phenomena align with our findings in the paper. In the OOD and transfer settings, where new training data is used, training remains stable. The new data typically constrains the model training process, helping it maintain a diverse distribution and preventing it from converging to a single distribution.

• Therefore, we recommend using dialogue data with different distributions at the SFT or DPO stages to prevent the "curse of recursion."

We also provide a more detailed background and related work to clarify our work, as well as clearer contributions and new experimental results. Please refer to the common response (Author Rebuttal). We hope this can further address your concerns.

[1] Generative judge for evaluating alignment (ICLR 2024)

[2] AI models collapse when trained on recursively generated data（Nature）

Thanks for your review comments, as well as your appreciation of the importance and writing quality of our work.

For Concerns：

Q: Potential Biases

We believe that incorporating synthetic data is essential for the future development of LLMs. Of course, reducing bias is also a crucial issue (points 1 and 2), and while we can attempt to mitigate the harm caused by biases, we cannot completely eliminate them (point 3). In our experiments and settings, the impact of bias is relatively small because our evaluation align with human evaluation results (point 4).

1. The use of synthetic data has become indispensable in both research and industry, enhancing performance [1, 2, 3] and supporting theoretical analysis [4]. OpenAI's GPT-4, one of the most widely used models today, excels in both safety and bias, having undergone extensive user testing with no more comparable open or closed-source alternatives.

2. One of our key contributions is exploring how to use LLMs to make improvements without rely on human supervision (line 39). Based on this condition, we did not introduce additional high-quality human-supervised data. Of course, we believe that incorporating such data could further enhance model performance and reduce bias。

3. In fact, both humans and LLMs inherently exhibit bias (though not all biases affect practical use). Addressing how to mitigate bias should be part of a comprehensive discussion, and we will further explore the impact of data bias on evaluation in future work. "Labeling a small annotation set to validate the branch ensemble approach" cannot be effectively done until we clearly understand the types of biases introduced.

4. Given that our experimental metric is "agreement and consistency with human preferences," our results demonstrate that our method achieves judgments more aligned with human evaluations (as our experimental evaluation are based on human evaluation results). This indicates that our approach does not introduce significant bias. If any bias exists, it likely aligns more closely with human preferences rather than originating from synthetic data.

For Questions:

Q: what is the number of branches generated by Initial, SFT, and DPO for evaluation?

1. The numbers in the third column of Table 1 (Branch) indicate the number of branches used in inference or evaluation. （For obtaining training data, we sample 10 criteria, each with 2 scoring guidelines, and each score includes 4 judgments, resulting in a total of 80 = 10 * 2 * 4 branches.）

We have also outlined the background and significance of current task, along with our contributions, in the common response (Author Rebuttal), which we hope addresses your other concerns.

[1] LLM Critics Help Catch LLM Bugs (OpenAI 2024)

[2] The Llama 3 Herd of Models (Meta 2024)

[3] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (MicroSoft 2024)

[4] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws (ICML 2024)