Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models

Dear reviewer WpqT,

We appreciate your comments and review of the paper.

(W1) - (W4) is our response to the “Weaknesses”. Please note that we have updated our draft of the paper, so please refer to the index of Tables and Figures in the updated version.

---

(W1) Dataset Publicity Issues and Construction Details of the Feedback Collection

[Dataset Publicity Issues]

Regarding our dataset, we couldn’t upload our file due to its large size (OpenReview has a 100MB limit) and we didn’t include the link due to the anonymous policy (it is currently open-source at a public website). We will include the link in our camera-ready version. Also, we have uploaded our anonymized version of the code as supplementary material.

[Details for Dataset Construction Process]

Here is a detailed explanation of the dataset construction process in the updated draft (Section 3.1) :

The collection process consists of (1) the curation of 50 initial seed rubrics, (2) the expansion of 1K new score rubrics through GPT-4, (3) the augmentation of realistic instructions, and (4) the augmentation of the remaining components in the training instances (i.e. responses including the reference answers, feedback, and scores). Figure 6 shows the overall augmentation process.

• Step 1: Creation of the Seed Rubrics We begin with the creation of a foundational seed dataset of scoring rubrics. Each author curates a detailed and fine-grained scoring rubric that each personnel considers pivotal in evaluating outputs from LLMs. This results in an initial batch of 50 seed rubrics.

• Step 2: Augmenting the Seed Rubrics with GPT-4 Using GPT-4 and our initial seed rubrics, we expand the score rubrics from the initial 50 to a more robust and diverse set of 1000 score rubrics. Specifically, by sampling 4 random score rubrics from the initial seed, we use them as demonstrations for in-context learning (ICL), and prompt GPT-4 to brainstorm a new novel score rubric. Also, we prompt GPT-4 to paraphrase the newly generated rubrics in order to ensure PROMETHEUS could generalize to the similar score rubric that uses different words. We iterate the brainstorming → paraphrasing process for 10 rounds. The detailed prompt used for this procedure is in Appendix J.

• Step 3: Crafting Novel Instructions related to the Score Rubrics With a comprehensive dataset of 1000 rubrics at our disposal, the subsequent challenge was to craft pertinent training instances. For example, a score rubric asking “Is it formal enough to send to my boss” is not related to a math problem. Considering the need for a set of instructions closely related to the score rubrics, we prompt GPT-4 to generate 20K unique instructions that are highly relevant to the given score rubric.

• Step 4: Crafting Training Instances Lastly, we sequentially generate a response to evaluate and corresponding feedback by prompting GPT-4 to generate each component that will get a score of i (1 ≤ i ≤ 5). This leads to 20 instructions for each score rubric, and 5 responses & feedback for each instruction. To eliminate the effect of decision bias when fine-tuning our evaluator LM, we generate an equal number of 20K responses for each score. Note that for the response with a score of 5, we generated two distinctive responses so we could use one of them as an input (reference answer).

For the prompts used for each step, please refer to Appendix J in the updated draft. We hope the updated version will waive your concern regarding the lack of details.

---

(W2) Detailed Analysis of the Feedback

We strongly agree that language feedback often conveys richer information compared to just looking at the scoring decision for evaluation. We have included the results regarding the examination of the feedback quality from Prometheus and GPT-4, which is shown in Figure 5.

When human annotators assessed the failure cases of Prometheus and GPT-4, `the semantic inconsistencies between the “score decision & feedback” (GPT-4: 2.00%, Prometheus: 2.86%) or the “score rubric & feedback” (GPT-4: 8.00%, Prometheus: 5.71%) wasn’t the main reason.

We are thrilled towards the direction of extracting meaningful insights from the language feedback which could be explored in future work. In this aspect, we found that existing open-source models such as Llama-2-Chat (70B) often generate abstract feedback that is meaningless and GPT-4 might be too expensive to use as it took $8,000 to construct the Feedback Collection.

We hope Prometheus could be used as a reliable and inexpensive source for further research in this direction.

Dear reviewer LTFo,

We appreciate your comments and review for the paper.

(W1) - (W3) is our response to the “Weaknesses”. Please note that we have made significant updates to the organization of the paper (content is same), so please refer to the index of Tables and Figures from the updated version.

---

(W1) Generalization to Other Fields

[Difference Between the Feedback Collection and the Datasets used for Evaluation]

We would first like to emphasize that the datasets used for experiments in Table 3 (Vicuna Bench, MT Bench, Flask) contains instructions that have different characteristics with the training dataset we used (Feedback Collection).

Specifically, the dataset in Table 4 have the following characteristics:

• Vicuna Bench:Relatively short instructions that are mostly consisted of How/What type of questions in addition to some coding and math instructions. (e.g., "How can governments utilize fiscal and monetary policies to combat economic recessions?)

• MT Bench: The 2nd instruction we used in this multi-turn dataset mostly asks for revision or further explanation of the previous response. (e.g., “Now, do the same task again but only use four-word sentences.”)

• Flask Eval: Contains academic NLP tasks such as classification datasets (e.g., MMLU, FEVER) or the train split of instruction datasets (e.g., Self-Instruct, WizardLM) (e.g., "Paraphrase the given text in an academic style. Input: Lots of papers have been published on this topic.”)

In contrast, the Feedback Collection was constructed with the main consideration of a very detailed, realistic situation where a user is interacting with an AI. (e.g., “Suppose there is a friend who is feeling low due to a poor performance on a recent exam. The friend is now seeking advice, encouragement, and a bit of humor to lighten the mood. How would one approach this situation, incorporating wit and a playful tone to uplift the friend's spirit?”)

The point we would like to emphasize is that Prometheus shows a higher correlation with both human evaluators (Figure 3) and GPT-4 (Table 3) in all of these different datasets. This supports that Prometheus could generalize to other instruction evaluation settings.

[Prometheus also works in Coarse-grained Relative Evaluation Setting although it was trained for Fine-grained Aboslute Evaluation]

Also, we would like to emphasize that the human preference dataset experiments in Table 4 (HHH Alignment, MT Bench Human Judgment) is evidence indicating the generalizability of Prometheus.

Specifically, we would like to emphasize the following two points:

• Results show that Prometheus could generalize to coarse-grained criteria (Helpfulness, Harmlessness, Honesty) although it was trained with detailed and fine-grained criteria (Cultural Sensitivity, Humorous, Considering regulation and compliance requirements).

• Results show that while Prometheus was trained in an absolute scoring setting, its evaluation capabilities could also be transferred to a relative scoring setting. Prometheus outperforms open-source reward models that were specifically trained with human preference datasets (StanfordNLP Reward Model, ALMOST) and even GPT-3.5-Turbo. Although it doesn’t get access to both responses, it could manage to make a score decision that gives a higher score to human-preferred responses. Without any generalization, this would be extremely hard.

---

---

(W2) Potential bias of Prometheus

[Absolute Scoring is less vulnerable to Length Bias]

We strongly agree that your concern regarding various biases that could occur during evaluation is very crucial in the field. As you specifically mentioned about length bias, we would like to discuss more about it in detail.

We would like to highlight that in this aspect, evaluation based on Absolute Scoring holds a strong advantage over Relative Scoring since the evaluator LM doesn’t get vulnerable to the length differences or other surface patterns between the two responses during evaluation.

In Figures 13,14 and 15, we have measured whether Prometheus or GPT-4 gave a higher score to longer responses in an absolute evaluation setting and found that it is not the case. The responses that got a score of 1 - 5 all have similar score distributions, supporting the idea that Absolute Scoring might be a good way to mitigate length biases.

[Additional Experiment on Adversarial Dataset]

To further analyze other datasets, we have conducted experiments with the recently proposed LLMBar dataset [1]. This dataset has an adversarial subset where the better response is shorter and the worse response is longer. With the same setting as in our human preference datasets in Table 4, we have conducted additional experiments and obtained the following results:

This result supports that Prometheus is relatively robust to adversarial patterns such as length bias compared to other baselines such as Llama-2-Chat (70B) and GPT-3.5-Turbo-0613.

---

(W3) Dependency on GPT-4 feedback

[Justification on the reliance towards GPT-4]

We strongly agree that the concern you have raised is a very important point not only for our work but also for the overall community in general since a lot of recent work heavily relies on data augmented from GPT-4.

Nonetheless, augmentation based on GPT-4 has the following advantages relevant to the strengths you have mentioned:

• GPT-4 shows remarkable performance in terms of Evaluation, and previous work [2,3,4] has also highlighted that GPT-4 could closely emulate human evaluators.

• For performing fine-grained evaluation with detailed criteria, the only source we could rely on was either GPT-4 or human evaluators since other LLMs could not flexibly ground on the given score rubric. While we approximately spent $8,000 to create the Feedback Collection, collecting the same amount with only humans will cost at least 10x, especially because the level of expertise required for such annotations is very high.

• While our construction process heavily relies on GPT-4, it also opens the door for other researchers to build upon our work and catch up since they could fully utilize the open-source dataset and model instead of paying another $8,000 through OpenAI API access.

On the other hand, we think that exploring different strategies such as mixing with human feedback or other types of model feedback (when it exists) is a worthwhile future work to explore.

[Difference of the Language Feedback Pattern between Prometheus and GPT-4]

Lastly, we would like to highlight that although Prometheus was trained on data from GPT-4, it shows quite a different feedback pattern compared to that of GPT-4.

In Figure 5, when inspecting the quality of the feedback, we observed that human annotators determined that GPT-4 generated relatively abstract and generic feedback, while Prometheus generated either too critical or too optimistic feedback.

Hence, even though the training data was generated by GPT-4, the results show that they can behave differently. This may not be a conclusive answer to your concern, but we hope it clears some of it.

---

References

[1] Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T. and Chen, D., 2023. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641.

[2] Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. and Zhang, H., 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685.

[3] Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. and Hashimoto, T.B., 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387.

[4] Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J. and Seo, M., 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928.

Dear reviewer B7Vr, We appreciate the review and comments on the paper.

Before addressing the other issues, I think the problem of the missing figure might be due to your browser. I had the same problem before when using Safari. The draft size is quite large and Safari doesn't seem to render it very well.

Could you try another browser or environment to check if the problem holds? Currently, Figure 2 and Figure 4 are included in the draft.

Dear reviewer B7Vr,

We appreciate your comments and review for the paper.

(W1) - (W4) are our responses to the weaknesses you have mentioned. Please note that we have made significant updates to the organization of the paper (content is same), so please refer to the index of Tables and Figures from the updated version.

---

(W1) Details of Model Implementation

Our explanation of the model implementation is included in Section 3.2 and Appendix F. Using the llama-recipes repository, we have fine-tuned Llama-2-Chat (7B & 13B) to obtain Prometheus using 8 A100 (80GB) GPUs with FSDP. The training is similar to Chain-of-Thought Fine-tuning, where we fine-tune to sequentially generate the feedback and the the score.

The hyperparameters we used during training and inference are included in Table 8 and 9.

---

(W2) Explanation of the Experimental Methodology

Since our main objective is to obtain a open-source evaluator language model that could closely emulate GPT-4 and human evaluators, we mainly divide our experiments into two different phases. Further details are included in Appendix B.

• We first check how the score is similar with either human evaluators (as shown in Figure 3) and GPT-4 (as shown in Table 2 and 3). Specifically, after parsing the score decision, we compare with Pearson, Spearman, and Kendall-Tau correlation with the opponent.
• Next we check how the other component that Prometheus generated, which is the language feedback has good quality. This is shown in Figure 4 along with an analysis in Figure 5. Specifically, we ask human evaluators to compare the language feedback and choose among Prometheus and GPT-4.

---

(W3) Explanation of our Contribution

The main contributions of our work are as follows:

• We introduce the FEEDBACK COLLECTION dataset specifically designed to train an evaluator LM. Compared to previous feedback datasets, it includes customized scoring rubrics and reference answers in addition to the instructions, responses, and feedback.
• We train PROMETHEUS, the first open-source LLM specialized for fine-grained evaluation that can generalize to diverse, real-world scoring rubrics beyond a single-dimensional preference such as helpfulness and harmlessness.
• We conduct extensive experiments showing that by appending reference materials (reference answers, fine-grained score rubrics) and fine-tuning on feedback, we can induce evaluation capability into language models. PROMETHEUS shows high correlation with human evaluation, GPT-4 evaluation in absolute scoring settings, and also shows high accuracy in ranking scoring settings.

---