Benchmarking and Improving Generator-Validator Consistency of Language Models

Concerns with the PromptPrioritization task.

To contextualize this concern, we first detail PromptPrioritization. In PromptPrioritization, the generator is given a task of “write about the benefits of {TASK}” while adhering to a given persona. Half of the examples purposely include contradicting tasks and personas, where the correct behavior of the generator should be to refuse the task because it contradicts its persona, e.g., “I’m sorry, I am a {PERSONA}, and so I cannot write about the benefits of {TASK}.” These contradicting tasks and personas were obtained by prompting text-davinci-003 to generate statements that a persona would never write. Note that PromptPrioritization broadly tests GV-consistency: the generator is asked to follow a task under a persona, and the validator is asked if the generator achieved the task under the correct persona.

Reviewer pfQS raises the concern that PromptPrioritization may not actually test GV-consistency, because the validator response that’s marked as GV-inconsistent may not actually imply inconsistencies between the generator and validator. This can arise, though rarely, in examples with contradicting tasks and personas, such as the one in the Appendix referenced by pfQS, where the persona is “an advocate for gun control” and the task is “write about the benefits of the 2nd amendment.” The persona and task are intended to be contradictory, but could simultaneously be satisfied with text like “the 2nd amendment is good because it helped establish precedence for amending the Constitution.” Given this text, the validator may predict that the persona is “an advocate for the 2nd amendment,” which is marked as GV-inconsistent, but does not necessarily imply inconsistency between the generator and validator. We agree that this can happen, but it occurs extremely rarely in the data — we manually inspected 40 examples with contradicting tasks and personas from PromptPrioritization and found that it was only possible to reasonably simultaneously satisfy both 22.5% of the time (11.25% of the total examples). The remaining 89.75% do appropriately test for GV-consistency, so PromptPrioritization still tests GV-consistency, albeit with some noise. Further, while it is technically possible to satisfy the 11.25% of near-contradicting examples, we find that language models do not do so in practice, and instead clearly contradict their given persona. We thank pfQS for raising this and have updated the paper to clarify this detail in Appendix E.

The format for close-book QA “introduces unnecessary complexity [over] simply judging whether the provided answer is correct or not.”

We agree that the suggested setup of “judging whether the provided answer is correct or not” is also reasonable, and we’ve added new experiments under this setup to Appendix E. The results are in the below table, and we find that under this setup, consistency fine-tuning still increases GV-consistency, while improving validator performance and barely changing generator performance. In the table, we use the notation X —> Y (delta) to denote a value X before consistency fine-tuning, Y after consistency fine-tuning, and delta as the improvement Y - X.

However, we maintain that the original multiple choice setting in the paper of selecting the correct answer from a set of potentially misleading possibilities is quite natural, as well: It mirrors a common use case of language models, i.e., answering multiple choice questions. Furthermore, we include this multiple choice setting to add diversity to the experiments, as other tasks (e.g., Arithmetic) already consider validators that determine if the answer is correct or not.

We thank Reviewer pfQS for their review and feedback. Reviewer pfQS primarily raises concerns with the task and evaluation design. We have strengthened the paper by adding additional experiments based on pfQS’s suggestions and clarifying areas of misunderstanding. We summarize the changes to the paper below. Please let us know if there are any remaining questions or concerns. Thanks!

“Assessing the quality of [the generated] data.”

Based on pfQS’s and 5N9m’s comments, we have added new analysis of the generated data used in consistency fine-tuning to Appendix A of the paper. Specifically, we analyze the accuracy (or F1 score) of the generated data before and after consistency-filtering, i.e., what fraction of the generator / validator responses in the generated data are factually correct. We also analyze the consistency rates, i.e., what fraction of the data is GV-consistent. We use the delta in these performance metrics as indications of how much consistency-filtering impacts the quality of the data. We analyze the arithmetic and QA tasks, since these tasks have a known ground-truth answer, which allows for automatically computing generator / validator accuracy (or F1 score).

In the arithmetic task, we find that Alpaca-30B generates middling-accuracy data before consistency filtering: the generated generator data has 66% accuracy, and the validator data has 72% accuracy. However, after filtering for the consistent data, the accuracy rises to 83% for the generator and 91% for the validator, indicating that consistency filtering can significantly improve quality. Additionally, (unsurprisingly) consistency filtering significantly increases the consistency rate of the generated data from 62.9% to 100%. Importantly, model size plays a role here, where the smaller Alpaca-7B model generates low-accuracy data before consistency filtering (only 39.3% for the generator and 40.1% for the validator), and consistency filtering brings a smaller, though still significant improvement in accuracy (increasing to 42.7% for the generator and 57.6% for the validator). Consistency filtering still significantly increases the consistency rate from 50.4% to 100%.

In the QA task, we see a similar trend, where (1) consistency filtering improves performance metrics of the generated data and also the consistency rate; and (2) the final performance metrics are dependent on model size. Recall that QA tasks the generator with generating a correct and incorrect answer to a question and the validator must choose which answer is correct. In this task, we cannot compute accuracy of the validator on the generated data, since both answers from the generator may be incorrect. (Note that this differs from the results in Table 5, which uses examples from TriviaQA with ground-truth answers, whereas here we’re analyzing generated examples.) Hence, we analyze only the word-wise generator F1 score.

For the Alpaca-30B model, we find that consistency filtering increases generator F1 score from 0.49 to 0.51; and the consistency rate from 79.9% to 100%. For the Alpaca-7B model, we find that again the base generator F1 score is lower, and consistency filtering increases it from 0.39 to 0.41. Consistency filtering again significantly increases consistency rate (from 68.5% to 100%).

Overall, these results indicate that consistency filtering can substantially improve data quality in both generator / validator performance metrics, as well as in consistency, though the filtered data is imperfect and still contains significant noise. We further note that a significant advantage of the consistency-filtering approach is that it is unsupervised and does not require human intervention on the generated data.

Considering “both [factual correctness and GV-consistency] would provide a more comprehensive and meaningful evaluation”

We thank pfQS for the suggestion. As pfQS notes, the submission reports both factual correctness (Table 5) and GV-consistency (Table 3), but does not simultaneously consider both. We have added new plots to Appendix I, which plot both factual correctness and GV-consistency simultaneously. For factual correctness, we use the mean of the generator and validator performance (e.g., accuracy for arithmetic and F1 score for QA). These plots more clearly show that all iterations of consistency fine-tuning Pareto dominate the base Alpaca model and self training. Further, these plots indicate that the multiple iterations of consistency fine-tuning form a frontier in many tasks, allowing users of consistency fine-tuning to choose between trading off between additional improvements in consistency vs. factual correctness. We suggest that future comparisons also use these plots to compare models along both GV-consistency and correctness simultaneously.

Confusion about the validator evaluation.

We apologize for the missing validator evaluation details, which we have added to Section 5 and Appendix H of the paper and detail below:

• Arithmetic and PlanArith: We compute the right answer by evaluating the mathematical expression. We obtain an incorrect answer by prompting text-davinci-003 to output an incorrect answer and verify that the answer is indeed incorrect. We evaluate the validator by randomly sampling between the correct and incorrect answer and asking the validator if the provided answer is correct or not.
• HarmfulQ and PromptP: We generate generator outputs by prompting text-davinci-003. We choose not to evaluate the validator on generations from Alpaca models because the text-davinci-003 generations tend to be higher quality and this also helps avoid evaluating on a distribution that’s unfairly similar to the training dataset collected for the consistency fine-tuning pipeline. We obtain a correct binary label for the validator via crowdsourcing on Amazon Mechanical Turk, using the majority response from three humans.
• QA. We evaluate the validator on examples from the TriviaQA dataset. We generate incorrect answers by prompting text-davinci-003 and verify that the answer is indeed incorrect.
• Style. We evaluate the validator on the Yelp dataset from Li et al., 2018. Each example includes a Yelp review stylized once positively, and once negatively.

Writing concerns: We have also updated the paper to fix the formatting issues raised by the reviewer and inconsistency between the abstract and Section 6.3, which both had a stale number.

We thank reviewer 5N9m for their thoughtful review and helpful suggestion. 5N9m suggests analyzing the consistency finetuning data to understand when it may help the generator/validator. We add this analysis to Appendix A of the paper and summarize below. Please let us know if there are any follow-up questions. Thanks!

Analysis of the consistency-filtered data.

We analyze the generator/validator performance of the generated consistency fine-tuning data, e.g., what fraction of the generator/validator responses in the generated data are factually correct. We consider the performance before and after filtering for consistency (i.e., the fine-tuning data for the SelfTrain baseline and the fine-tuning data for consistency fine-tuning respectively). We analyze the arithmetic and QA tasks, since these tasks include a known ground-truth answer, allowing for automatically computing generator/validator performance.

Overall, our analysis suggests that consistency fine-tuning improves generator performance exactly when the consistency-filtered data has much higher generator performance than the pre-filtered data, and analogously for the validator. When the gap between pre- and post-filtered data is smaller, consistency fine-tuning also appears to improve the generator/validator less.

The table below contains the results for Arithmetic on the Alpaca-7B model. We find that generator accuracy is only 3.4% higher on the consistency-filtered data than on the pre-filtered data. However, we find that validator accuracy is significantly higher (17.5%) post- vs. pre-filtering. Additionally, consistency finetuning barely changes the generator accuracy, slightly decreasing from 39% to 37%, but significantly improves validator accuracy, improving from 54% to 62%, suggesting that larger improvements from pre- to post-filtering translate to larger improvements after fine-tuning.

We see a similar trend for the Alpaca-30B model for Arithmetic in the below tables. Both generator and validator accuracies are significantly higher post-filtering. We also find that consistency fine-tuning significantly improves both generator and validator accuracies over the base model and SelfTrain.

We observe similar trends in QA. Recall that in QA, the generator is tasked with answering a question with a correct and incorrect answer, and the validator must choose which answer is correct. For the generated data, validator accuracy is ill-defined, since both answers from the generator may be incorrect. Hence, we only analyze the word-wise F1 score of the generator, and no performance metric for the validator. (Note that Table 5 in the submission computes validator accuracy on QA by using examples from TriviaQA with a ground truth correct answer and a generated incorrect answer, in contrast to the generated data, which may not have the correct answer).

The generator F1 scores for both the 7B and 30B models barely change between pre- and post-consistency filtering. This may explain the results in Table 5, where consistency fine-tuning does not improve generator F1 compared to the base model or SelfTrain.

Importantly, even when consistency fine-tuning does not improve generator or validator performance, it still improves GV-consistency.

We thank reviewer bsD6 for their time and review. Reviewer bsD6 primarily asks for additional insights and analyses into generator-validator consistency. We have added several new experiments and additional discussion in the paper to address these, summarized below. Please let us know if there are any additional questions or concerns, during the discussion period. Thank you!

How do “different model sizes and initial generator/validator performances” impact consistency and generator/validator performance after consistency fine-tuning?

To answer this question, we have added new experiments with the Alpaca-7B and Alpaca-13B models on additional consistency tasks. In the below table, we report the following four metrics before and after consistency fine-tuning: (1) the generator performance (i.e., accuracy for Arithmetic, PlanArithmetic; F1 score for QA; and success rate for PriorityP); (2) the validator performance (i.e., accuracy); (3) the mean of the generator and validator performances (mean GV performance); and (4) the GV-consistency. We use the notation X —> Y (delta) to denote a value X before consistency fine-tuning, Y after consistency fine-tuning, and delta as the improvement Y - X.

We find that consistency fine-tuning significantly improves consistency across all tasks and all model sizes. We also find that consistency fine-tuning improves the mean of generator and validator performance across all tasks and model sizes, though it does not necessarily always improve generator or validator performance individually. Furthermore, the magnitude of improvements to the mean GV performance tends to increase with model size.

We have added these new experiments to Appendix C of the paper.

“Why [do] even powerful LLMs suffer from [generator-validator] consistency?”

As noted in prior work (Elazar et al, 2021), the training objectives for large language models do not explicitly enforce forms of consistency, including GV-consistency. More specifically: maximizing likelihood of the pre-training data does not directly optimize for consistency; in fact, the pre-training data may even contain inconsistent information from subjective content, such as one restaurant review stating “Starbucks had tasty food” and another stating “does Starbucks have good food? No!” In principle, the instruction-following step of model training (e.g., instruction-tuning or RLHF) could help improve consistency via careful design of the reward model or instruction-tuning data. However, gathering GV-consistent data has not been a priority in current instruction-following training pipelines, as this work is the first to highlight this type of inconsistency. We have added this discussion to Section 3 of the paper.

“Why [does] GPT-3.5 outperform GPT-4 and davinci-003 on GV-consistency score?”

To shed light on this question, we analyzed the consistency scores of these models across our tasks. As shown in Table 2 of the submission (and reproduced in the table below), GPT-3.5 achieves the highest average GV-consistency primarily due to the prompt safety task (i.e., PromptP), where it achieves the highest GV-consistency by a wide margin (79.6 vs. 52.0 and 68.0 for GPT-4 and davinci-003 respectively). If we exclude this task, both GPT-4 and davinci-003 achieve higher average GV-consistency than GPT-3.5. To understand GPT-3.5’s superior consistency in PriorityP, we qualitatively analyzed responses of all 3 models in this task. We find that GPT-3.5 frequently generates abstentions of the form: “As an agent with the following persona […], I cannot provide an answer to the question advocating for the benefit of […],” which leads to high GV-consistency, since it is easy for the validator to predict the correct persona with these responses. In contrast, neither GPT-4 nor davinci-003 generate any abstentions, leading to lower GV-consistency in these tasks.

The RLHF data for these models is not public information, so it is difficult to identify the cause of GPT-3.5’s abstention behavior in PriorityP with certainty. However, we speculate that it may arise from GPT-3.5 receiving more safety or prompt injection fine-tuning data to more conservatively accommodate its earlier public release via ChatGPT, though we do not know why GPT-4 does not exhibit similar abstention behavior. We have added this analysis and discussion to Appendix B of the paper.

Why does “consistency fine-tuning perform much better than CC-FT?”

The trend that data filtering (used by consistency fine-tuning) performs better than CC-FT is not new in this paper. For example, Dubois et al. 2023, observe that data filtering via BinaryFeedME outperforms binary reward conditioning (i.e., CC-FT). The exact reason behind this performance gap is still unknown, though we speculate that it may stem from CC-FT using up some of its capacity to model multiple classes, while models trained on filtered data do not need to model the classes that have been filtered out. The key insight behind consistency fine-tuning is that fine-tuning data for GV-consistency can be generated without human supervision — and this insight is agnostic to the exact choice of fine-tuning method on this data. Hence, we believe that detailed analysis comparing various fine-tuning methods to comprehensively answer this question lies out of scope of this paper. We have updated Appendix D to clarify this point.

Reference:


Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models, 2021

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A simulation framework for methods that learn from human feedback, 2023