Great Models Think Alike and this Undermines AI Oversight

We are glad you found our metric useful, experiments well-designed, evidence sufficient, and motivation clear. Given these positive points, we were a bit surprised by the recommendation. We hope the new analyses and clarifications below address the concerns.

(W1)

The tasks that are used for measuring similarity appears to only be based off of accuracy. However, various tasks do not directly use hard ground-truth labels

Our metric uses ground truth labels to prevent the inflation of similarity scores as we illustrate with new plots. In applications where ground-truth labels are not available, one can still use the observed agreement $c_{obs}^p$ as computed by our metric. Evaluating open-ended tasks is an exciting direction of future work [L392-404].

(W1)

I believe a diversification in the number of tasks that are being evaluated for can better demonstrate the benefits of the method

We appreciate the reviewer's suggestion regarding task diversification. To further confirm the broad relevance of our method, we extend our analysis to AlpacaEval, open-ended chat responses which do not have objective ground truth. We find that MMLU-Pro benchmark similarity correlates strongly ($r=0.75$ to $r=0.87$) with judge preferences on AlpacaEval in new plots. This reinforces conclusions in Sec 3, so we will include it in App. B of the revision. We believe our experiments across MMLU-Pro (14 categories), BBH (23 categories, Sec 5), and 15 NLP tasks (Sec 4) are comprehensive, as noted by Reviewers RC4b and qUdH.

(W2)

In Table 2, the correlation scores can vary quite a bit based on the model size… better understand the relationship between the metric and model size.

To check whether model size is a possible confounder, we extend our multiple regression results by including model size as a predictor -- new plots. We find that similarity remains a significant predictor of judgement score when controlling for both model size and accuracy. Furthermore, the effect size of model size when controlling for similarity and accuracy is close to 0, confirming that our focus on model capability rather than characteristics is a sound approach, consistent with prior work [1].

(W3)

I feel that claims about model similarity have been extant for an extended period of time. I do not particularly believe that this work offers a significantly novel perspective on this front”

While there might have been a common feeling in the community about models being similar, our work is the first to quantify it comprehensively (as highlighted by reviewer qUdH) at scale — showing a concerning increase in similarity with improving capability by analyzing 130 LLMs from various developers. Our analysis of LLM-as-a-judge extends existing self-preference results showing mitigations like using separate judges [4] are not enough. In Sec 6, we discussed previous work that explored model differences, and we will also cite theoretical results like [3] in our revision, which study the effects of algorithmic monoculture on fairness.

(W3)

many of the claims remain speculative in some part

We are happy to clarify our claims and provide additional support in our revision. Could you please indicate which specific claims you found speculative?

(W3)

while the metric the authors introduce can be useful, it doesn't provide a direct solution towards mitigating the problem of AI oversight

We acknowledge your observation, our metric does not directly mitigate the impact of model similarity on AI oversight. Our paper's primary goal was to establish a robust framework for quantifying this effect, which is a crucial first step in addressing the issue. Specifically, our findings (1) shed light on the limitations of LLM-as-a-judge systems, popular in many leaderboards [L195-204], (2) offer insights into open problems in weak-to-strong generalization, as highlighted by reviewer qUdH, and (3) reveal an inverse scaling phenomenon [2] where increasing capabilities exacerbates these issues. By quantifying the problem, we set the stage for future research on potential mitigation strategies, as discussed in Sec 7.

We thank you for the valuable feedback, which has made our paper stronger! Please let us know if there are any further questions or concerns we can resolve to increase your support for our work.

References

[1] Huh, Minyoung, et al. "Position: The platonic representation hypothesis." ICML 2024.

[2] McKenzie, Ian R., et al. "Inverse scaling: When bigger isn't better." TMLR (2023).

[3] Bommasani, Rishi, et al. "Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?." NeurIPS (2022)

[4] Decentralized Arena via Collective LLM Intelligence, https://de-arena.maitrix.org

We are grateful for your strong support of our work. We are glad you that you found our claims nuanced and well-supported by comprehensive, rigorous experiments.

On your question about how we compute the “elicitation $\cup$ complementary” ceiling estimate in Table 3, we should indeed have made this clearer in the main paper itself. We take the union of samples either the strong model fine-tuned on ground-truth labels (elicited) or the weak supervisor (complementary) get right. This is likely an overestimate of what can be realised from weak-to-strong training, but we think that is a lesser harm for a ceiling estimate. The previously proposed ceiling of ground-truth elicitation on the strong model ignores potential gain from complementary knowledge, which can be significant as indicated in Figure 4 and 12. We think such underestimation is worse for a ceiling estimate, as it might explain concurrent work [1] reporting >100% PGR, beating the “ceiling” of ground-truth elicitation on the strong model. Thus, we avoid estimating the ceiling by methods that can introduce any suboptimalities. For example, if we sequentially finetuned on ground-truth labels and then the weak supervisor's predictions or vice-versa, this could lead to catastrophic forgetting of the first part of training instead of fully combining their knowledge as desired.

We agree with you that many important details about the weak-to-strong setup are in the Appendix. This is due to the limited space for the initial submission. In the camera ready, we will utilise the extra page available to shift setup details for the weak-to-strong experiments currently in Appendix C.1 and C.2 back to Section 4.

We are happy to discuss any further questions about the weak-to-strong and other experiments!

[1] Shi, Junhao, et al. "How to Mitigate Overfitting in Weak-to-strong Generalization?." arXiv (2025).

We are glad you liked the motivation and experimental design. We hope our new analyses address your concerns.

(W1)

Comparison with alternative similarity metrics (e.g., KL Divergence, JSD, RSA)

Divergence metrics have desirable information theoretic properties, but we did not find any way to adjust the effect of higher accuracy inflating the similarity reported by these metrics [L88-99 (R), Table 1]. We re-plot Figs. 6-8 with JSD, a symmetric and bounded variant of KL, to demonstrate this -- new plots, e.g., the second plot studies 90% accurate models with independent errors, where $JSD > 0.75$, but as desired, $k_p \sim 0$.

App. A4 discusses design choices for $\kappa_p$ by comparing with many alternate similarity metrics, including why we use probabilistic overlap instead of JSD [L1029-1031]. Here, we will also include that alternative priors, if known, can be used to compute chance agreement $c^p_{exp}$ on ``tasks where wrong answers aren’t uniformly distributed'', thanks for this suggestion. Finally, we focus on functional similarity metrics and not model-dependent representational similarity analysis (RSA) [L60-80 (R)].

(W2, W3)

Some confounders are not sufficiently controlled …, including: training data, architectural similarities; Limited casual evidence in similarity trends

We agree deeper analysis of the causal factors is an important next step [L386-390 (R)]. We tried our best not to make causal claims in the paper, and will rephrase any we overlooked.

We will acknowledge training data overlap as a potential reason for our observations. Unfortunately, training data mixtures are not known even for open-weight frontier models to control for this. As requested, we now exclude ``same-family pairs'' for Fig. 2 in new plots. Judge scores remain highly correlated with similarity $r \in$ 0.89, 0.95 $$. Further, new table controls for model size as requested by reviewer nzM3. Similarity still has high partial correlation.

For similarity trends with increasing capability, we already excluded same-family pairs in Fig. 5. In Sec 5.2 we mention how switching to state space models (Mamba) does not reduce similarity. App D.2 has more details and observations, showing sample-hardness has a small effect on this trend, while instruction tuning exacerbates it. We are happy to demonstrate further controls if you have any suggestions.

the benchmark datasets are limited. In section 3, there is only dataset MMLU-Pro

In Sec 3, we were constrained to tasks which can be evaluated as both free-response and MCQ [L199-203]. This is why we cannot use benchmarks like BBH in Sec 5 and the 15 NLP tasks in Sec 4. MMLU-Pro itself is quite diverse, and new plots show consistent results across its 14 domains ranging from law to engineering. In fact, could this diversity lead to similarities on MMLU-Pro predicting judgements on more open-ended tasks? New plot shows MMLU-Pro similarity has high correlation with LM judge based elos on the chat response dataset AlpacaEval 2.0 [1].

(Evidence for W2S)

conclusions about complementary knowledge are entirely dependent on the κp metric

We forgot to add a pointer in our main paper to Fig 13 which shows that the complementary knowledge conclusion consistently holds with 1 - JSD, though $\kappa_p$ explains more variance ($r=0.77$ vs $0.85$) and has better normative grounding for analyzing model similarity trends as discussed in Sec 2. We will mention the ``selection criteria for weak vs strong models'' is model size, consistent with Burns et al. (2024) and Scherlis et al. (2024) in the revision.

(W4)

no evaluation of the qualitative nature of mistakes.

Qualitative analysis of mistakes is an interesting related, yet complementary direction [L360-369 (R)]. We respectfully disagree that this implies a ``lack of insights to connect to/improve the real LMs applications''. Our quantitative measurement of similarity provides a necessary foundation for applications like evaluating qualitative descriptions of model differences [L431-437 (R)], debiasing LLM-as-a-judge based leaderboards [L195-205], and many exciting directions for future work discussed in Sec 7.

For 3.2 Q1, provide the table where the data is from.

New tables have the data of all 351 model pairs studied. To make this data easier to interpret, we reported it as scatter points in Fig 2. In the final version, we will link our PyPi package and interactive tool to aid readers in exploring similarities of chosen model pairs.

Thanks! Your detailed feedback has helped us greatly improve the paper. We hope this increases your support for our work.

[1] Dubois, Yann, et al. "Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators." COLM (2024)

We appreciate that you found our metric well-motivated (with comparisons to alternatives), experiments comprehensive, and observations insightful.

The proposed similarity metric seems to be tailored for MCQ tasks. It would be insightful if some discussions on possible extensions to other tasks (e.g., regression) could be provided, even just as future directions.

Indeed, our metric as described in the main paper is designed to address a key challenge in MCQ tasks—specifically, the lack of coherent classes needed to compute the marginal distributions for inter-annotator agreement metrics like Cohen’s κ [L976-979]. You might find Appendix A.3 interesting, where we extend the metric to classification, along with a brief discussion of challenges for exact match settings, for which we only provide a discrete version.

On the other hand, for free-response tasks like creative writing, one could use both perplexity and model-based metrics, which have their own challenges as discussed in L392-403. That said, we ran experiments where we compute similarity on MMLU-Pro MCQs and plot Elo ratings assigned to evaluated models by making an LLM-judge pick between open-ended chat responses on AlpacaEval -- New Plot. The strong correlation shows initial evidence that MCQ similarity might transfer across tasks as a predictor.

For tasks like regression, for the observed agreement, $c_{obs}$, one could measure a distance metric over the two models' predictions, aggregating across samples. Once again, models with lower error would have lower distance in prediction, so the challenge lies in defining chance agreement, $c_{exp}$, for a model with a given error. We would have to make appropriate assumptions about the distribution of errors, such as gaussian errors, based on which $c_{exp}$ can then be computed. Thanks for this interesting question; We will add this discussion to Appendix A.3. We are excited about adapting the metric for other tasks in future work!

The negative correlation between weak-to-strong generalization and model similarity is arguably surprising. It would be helpful to provide more insights or possible explanations for this phenomenon.

The result can seem surprising if we view weak and strong models purely through the lens of accuracy. This is where we think our framing of model similarity (or difference) at a sample level is insightful! Lower accuracy does not imply that the knowledge of weak models is a strict subset of stronger ones. Rather, weak models can have complementary knowledge, and we hypothesise the transfer of this also contributes to weak-to-strong generalization. Model similarity provides a way to measure complementary knowledge in terms of the difference in samples they get right. Lesser complementary knowledge to transfer might be the explanation for the seemingly surprising trend of lower weak-to-strong generalization when model similarity is higher (negative correlation). We tried to motivate this in L248-260. We will be sure to utilise the extra page allowed in the revision to expand this section, by including content currently in Appendix C.1 and C.2.

“Regarding the definition of "observed error overlap" on the right of line 87, it may be worth remarking why "the fraction of samples on which both models are correct or both models are wrong" is a more reasonable metric than things like "the fraction of samples on which the two models agree (in the multi-class setting)". Is the binary nature of the definition crucial here?”

In L87, we are stating the definition used for error consistency defined in Geirhos et al. (2020). Our own metric modifies this to measure “the fraction of samples on which two models agree”, just as you proposed. We agree it’s a better definition, as it distinguishes differing mistakes [L129-133]. As you correctly noticed, our metric $\kappa_p$ is equivalent to error consistency (Geirhos et al.) for binary classification [L894-896, Appendix A.1]. Great minds do think alike ;)

Thanks for your question and suggestions. We hope our response increases your support for our work and we are happy to discuss further!