RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models

We thank the reviewer for helpful comments.

Here we address the reviewer’s two stated weaknesses:

1. “The method shows limited generalizability beyond the stated task”. Here we would like to point out that on two (out of three) OOD tasks, SWORDS and Hypernymy generalize quite well (the first two rows of Table 4). In particular, training on lexical-level tasks generalizes to complex and practically useful question answering tasks (TriviaQA). We are currently expanding the set of tasks and our understanding of generalization of the G-V gap for a forthcoming paper.

2. “Hard to know if problems (or solution) will generalize to larger models”. We chose some of the best performing “small” models; note that Llama-3.2-3B is the largest of the two text-only Llama-3.2 models, and actually outperforms some larger models (e.g., Llama-2 13B). The G-V gap is also still a problem for larger models, e.g., for Gemma-2-9B (with correlations of 74, 38, 12 for SWORDS, TriviaQA, and LAMBADA, respectively) and for Llama-3-8B-Instruct (with correlations of 63, 31, and 10 for SWORDS, TriviaQA and LAMBADA, respectively).

We thank the reviewer for their feedback which we will use to clarify certain points in our paper. We appreciate the reviewer’s comments that the RankAlign objective is interesting and works well at reducing the G-V gap.

We address the three specific critiques below:

1. “The metric is conceptually flawed”

We feel there is a misunderstanding here. Briefly, the reviewer is saying that:

(A) Our metric is measuring the inconsistency of examples like “Poodles are a kind of__” (LM: “dog”) and “Do you think poodles are mammals?” (LM: “Yes”), which are in fact already consistent.

and

(B) Our metric does not, but should, account for cases like “Olives are a kind of __” (LM: “dog”) and "Do you think olives are dogs? (LM: “Yes”) (e.g., “My recommendation for this is to reformulate the metric so it only captures the olive examples I provided but not the poodle examples.”)

For point (A), we agree that saying poodles are dogs and asserting to poodles being mammals is consistent, if we treat consistency as a binary metric following previous work (Li et al., 2023). However, this is not what we were trying to illustrate in Figure 1. In the figure, “dog” was being shown as the most likely next token, whereas our focus is on the token “mammal”, which is ranked much lower (meaning much lower generator probability as shown in the figure). A very low probability for “mammal” is inconsistent with generating “Yes” to “Do you think poodles are mammals?” with high validator confidence, because the validator is asserting something that the LM is unlikely to ever generate. In our view, the high validator score should imply a higher generator probability as well.

For point (B), we argue that our metric should actually not treat this case as inconsistent. In this case if the LM strongly prefers both “dog” (generator) and “Yes” (validator) then they actually agree (are consistent), even though both are wrong.

2. “Metric and Method Validation”, i.e, we should provide experiments showing that our evaluation of the G-V gap correlates with previous metrics like G-V consistency.

We disagree with this point. Our goal in this work is to introduce a new measure of G-V consistency, which we believe is conceptually useful, and improve on that. We argue that validating our metric based on comparing to past notions is not appropriate. G-V consistency from (Li et al., 2023) only measures completions where the generator has high likelihood (where the next token(s) are decoded greedily) and the validator disagrees with those (e.g., the “olive” example in Figure 1). However, our metric considers the entire spectrum of generator likelihoods and much broader set of candidate completions. In essence, we argue for a more holistic notion of consistency, and our method is designed to improve on this holistic notion.

We included Table 15 in the Appendix mainly to show the reader that our implementation of Consistency FT does improve G-V Consistency, in line with (Li et al., 2023). Our intention was not to comment on the relation between G-V consistency and log-odds correlation, and we will make that clearer in the camera ready version of the paper.

3. “Significance”

(a) Sampling: we did not address sampling from the Language Model because we did not need LM sampling in our paper given how we defined the G-V gap. We evaluate the G-V gap with respect to a given dataset of (generator, validator) prompts and answers (e.g., “an olive is a __” and “fruit”). We then look at the log-odds for “Yes” (for the validator) or for the answer (for the generator), so we do not have to sample from the LM. We argue that this actually makes our metric more useful to the community, because it looks at a wide range of possible outputs, restricted only by the dataset under consideration, and so with RankAlign one can focus on evaluating the LM, rather than the decoding method.

(b) Paraphrase variation: consistency over paraphrase variations is a different notion of consistency than our G-V consistency defined here. Relatedly, we did evaluate generalization of our method under paraphrase variation in our paper; see section 5.2, under “Generalization across prompt formats”. However, since the main goal in this paper is to reduce the G-V gap, we see further experiments with paraphrase consistency as outside the scope of the current work.

We thank the reviewer for their detailed feedback.

Here we address the two weaknesses identified by the reviewer:

1. RankAlign does sometimes cause a drop of generator accuracy. We note however, that validator accuracy is hardly affected for Hypernymy and SWORDS. We believe our method shows promise for certain focused settings, and future work can build on it to develop methods that maintain or improve both generator and validator accuracy as well.

2. We have a few hypotheses for the differing performance across models. Different models have different base capabilities, as shown by their different base performance on the tasks and how they benefit to different extents from SFT. The validation threshold of each model is also different, as shown by the different R@0 numbers. Intuitively, these factors have an impact, but their relationship is complex and we do not have a simple, evidence-grounded story to convey, so we leave this for future work. Also in the future we can apply our method to other open LMs to determine how characteristic the drop in accuracy is for Gemma-2-2B.

We thank the reviewer for their detailed feedback on our paper.

The evaluation data is somewhat limited.

We selected 4 tasks that we felt showcased a range of inference types. These include lexical semantics tasks (hypernymy, synonymy), a task involving world knowledge (TriviaQA), and next word prediction (LAMBADA). SWORDS has moderately long input passages. We agree that extending our evaluation to longer outputs would be useful. However, paraphrase variation (different ways of expressing the same content) makes it difficult to concretely evaluate an LLM for the G-V consistency property as we’ve defined it here. Our work represents a first step towards formulating and evaluating the G-V gap in this way, and we believe that future work can extend it with a focus on long-form outputs.

Do you think RankAlign can support thinking/reasoning models?

To evaluate the G-V gap for a model with CoT, in principle we would need to marginalize over the CoT generation to find the probability of generating an answer given the input prompt. This could be approximated by sampling, but the quality of this approximation is unclear and we don’t see an easy way to conduct rigorous evaluation in this setting. We think this would also be a great step for future work!