Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning

Thanks for the review! We highly appreciate the effort and will respond to specific comments.

Clarifying the statistical significance and effect size of our findings

Reporting statistical significance at different levels is a very valid constructive point. We will adopt it in the final publication.

We thank the reviewer for this important suggestion. To better demonstrate the practical relevance of our findings, we will report the coefficient of determination (R2) as a standardized measure of effect size.

The Martingale Score (the unstandardized regression coefficient, β1) is itself a direct measure of the effect, but supplementing it with R2 will clarify the proportion of variance in belief updates explained by the prior belief. For instance, in the Gemini 2.0 Flash-NoPrompt-CoT-Forecasting setup, the R2 is 0.0589, indicating the prior belief accounts for approximately 6% of the variance in the update.

Most importantly, we argue that the practical relevance of the Martingale Score is most clearly validated by its strong correlation with prediction accuracy (the Brier score), as shown in our Figure 4. This direct link between a higher Martingale Score and a tangible drop in performance confirms its practical importance. We will add the R2 values to our results table and clarify this discussion in the final manuscript.

Clarifying the correlation between Martingale Score and Brier score

We thank the reviewer for their sharp observation, as positive correlation does not universally similarly apply to all setups.

Regarding your concern on outliers being overly influential - we conducted three other series of regressions:

• 1. (Light-handed) We remove the 10 most significant outliers (out of 78 points in total) as measured by Cook's distance and re-do all regressions.
• 2. (Heavy-handed) We remove the 30 most significant outliers as measured by Cook's distance and re-do all regressions.
• 3. (Conservative): Statistical threshold (4/n Cook's distance)

• The table below summarizes the results, specifically concerning the linear coefficient of the Martingale Score (the dependent variable being Brier score) when under the Debate regime.

• Ignoring the “+ Reasoning + Interaction” the row given its extremely low statistical power and high p-value, overall, removing outliers does not change the sign of the coefficient. This can be confirmed in one case with statistical significance, and another with marginal significance.

Debate x Martingale Score Interaction Effects Across All Approaches (n=78)

Note: Light, heavy, and conservative methods show robust results consistent with no outlier handling.

The discrepancy among setups on their martingale-brier scores relations

While our results vary across setups, this is expected as the ability to validate the Martingale Score depends on the quality of the experimental conditions.

Our most convincing results come from the setups best suited for this analysis. For example:

• Chain-of-Thought (CoT) is more trustworthy than Debate due to a significantly larger sample size.

• The Forecasting domain is more reliable than OpenReview because its ground-truth signal is objective and less noisy than the subjective decisions of a few human reviewers. [1]

The primary challenge is finding data with reliable ground truth that the LLM has not seen during training. Forecasting is the ideal domain for this purpose. The strong results obtained in this domain provide the core validation for the martingale score as a meaningful, unsupervised measure of belief entrenchment. In the following research, we hope to find more domains where martingale scores can meaningfully predict either ground truth labeling or collective voted judgments if there is no ground truth. Two domains we plan to run: OpenReview but to assess the relation between martingale score and citation number (as this more closely aligns with collective voted judgments) and Reddict r/AITA (also collective voted judgments) [2]

Other suggestions

Also, we very much appreciate that you spend time identifying typos and making suggestions about writing and figures. All the suggested changes will be reflected in the final manuscript.

Reference

[1] Beygelzimer et al. Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

[2] Cheng et al. Social Sycophancy: A Broader Understanding of LLM Sycophancy

I still have issues with Figure 4 that haven't been addressed. You claim positive correlations between Martingale and Brier scores, but I think you might only be able to do so for one particular case. Yet, as you said, "the practical relevance of the Martingale Score is most clearly validated by its strong correlation with prediction accuracy (the Brier score)".

We'd like to note that we see weakly positive results in CoT+OpenReview as well, in addition to the strongly positive results in CoT+Forecasting. We believe the weaker results for OpenReview can be explained by issues with the OpenReview task itself - more on this below.

We would like to first clarify some issues with Debate and OpenReview, and why the presentation of Figure 4 may cause confusion; then we hope to address the concern with regard to the generality of the Martingale Score.

Issues with Debate and OpenReview

• We believe CoT is a much better testbed than Debate is, and we should focus most of our attention on CoT. Martingale is to evaluate the belief entrenchment of a given reasoner, whereas in debate two debaters intentionally present opposite views. Even if the two debaters are entrenched by their own parametric beliefs, there are always views from both sides present to the judges. Naturally, belief entrenchment, even if exists, is hardly observable in such setups, and thus Martingale scores for debate tend to be dominated by stochastic noise.
• OpenReview has very noisy ground-truth signals (i.e. acceptance decisions). The NeurIPS 2021 review consistency experiment [4] demonstrates that there is shockingly large randomness in the acceptance decisions, e.g. spotlight papers have >50% chance of being rejected if independently reviewed again. Hence OpenReview may not be a great testbed for Martingale Score either.
• There are 4 plots in Figure 4. Admittedly, only Forecasting + CoT setups have achived results that strongly support that claim "the practical relevance of the Martingale Score is most clearly validated by its strong correlation with prediction accuracy (the Brier score)". However, we'd like to note that forecasting is NOT merely a dataset within a task, but an ML task in itself, covering a diverse range of real-world subjects.

The Versatility and Unique Position of Forecasting as a Task

• Using LMs for event forecasting is, in itself, a machine learning task that has seen a long line of research papers ([1,2,3] to name a few) and tightly connected with the general world-modeling ability of LLMs.
• As an application of the Martingale score, we explored a new path to evaluating performance on such a task: testing inference-time cognitive bias by incentivise sound forecasting reasoning, instead of incentivising memorization of related facts. Forecasting is unique because no related facts to be memorized if LLM training cut-off dates were earlier than the dates when forecasting questions are resolved.

Martingale Score is Generalizable

• We still hold the belief that Martingale Score is generalizable. Our Forecasting dataset includes a diverse range of domains including economy/business, environment/climate, elections/geopolitics, technology/AI, and more. LM-based forecasting is one general methodology to deal with many problems in different problem domains.
• Forecasting tasks are about incorporating evidence in light of uncertainties. Such a capability is central to general reasoning, and it is precisely what Martingale Score measures.

---

[1] Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

[2] Approaching Human-Level Forecasting with Language Models

[3] Evaluating LLMs on Real-World Forecasting Against Expert Forecasters

[4] Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

Sample size

Prior to removing outliers, it's 84 in total. 21 in CoT+Forecasting, and likewise for the remaining three setups.

Full results

• Table 1 (CoT, both domains): Positive. With-outlier-removal results and no-outlier-removal results consistently agree in sign and are positive. Consistent with Figure 4.
• Table 2 (CoT, OpenReview only): Weakly positive. Light and heavy both consistently (100%) positive, while medium leans (75%) negative. No-removal leans positive but without stat significance.
• Table 3 (debate, both domains): Ambiguous. Heavy outlier removal is consistently (100%) positive, while medium outlier removal leans (80%) negative. Light outlier removal gives mixed results.

As rows in the tables, we include all regression setups that receive non-trivial $R^2$ in Figure 5. Bold means marginally significant, bold+italic means significant. 'Light' outlier handling removes 3-5 samples according to a $4/n$ Cook's distance threshold; 'medium' always removes 10; and 'heavy' always removes 30.

As a next step, we will look into the cause of ambiguity especially in the debate case.

---

Background information:

• In our final comment (an appendix), we present 5 tables: 1-3 described above, and 4-5 being legacy results that we share for transparency. For reproducibility, we are happy to share the raw data (n=84) upon request in a new comment, but are omitting them here due to excessive length.
• We produced table 4 containing results for CoT, and table 5 containing CoT+OpenReview. Notably, table 4 presents much more positive and consistent results than those in our rebuttals.
  • Re the question on misleading results: We hope these positive & consistent results show that we weren't attempting to mislead. In fact, our decision to only report debate results was a mistake when interpreting the request - we had an internal miscommunication when performing regressions to obtain results for the request here and Reviewer G3H9's questions, the latter specifically asking about debate. The full results in table 1-5 demonstrate our transparency.
• Outside of table 4, both table 5 and the old one in our rebuttal suffer from an extreme lack of statistical power, and it's hard to reach confident conclusions from them. This is because each data point (i.e. Martingale score for a setup) is computed from only 100 trajectories, so for high-variance domains like OpenReview, and few-step reasoning modes like debate, the Martingale score ended up highly comtaminated with noise.
• As such, over the past few days, we performed a full re-computation of our data with 6x OpenReview trajectories-per-sample and 7.5x debate trajectories-per-sample (i.e. we independently sample more reasoning trajectories for each question), and then carried out regressions. Now each of the debate or OpenReview samples in regression is backed by, on avergage, 6x or 7.5x reasoning trajectories.
  • Table 1-3 are results of regressions performed on this dataset with strengthened statistical power.

Table 1: Full Martingale-Brier Causal Relations for ChainOfThought (OpenReview + Forecasting)

---

Table 2: Full Martingale-Brier Causal Relations for ChainOfThought (OpenReview only)

---

Table 3: Full Martingale-Brier Causal Relations for Debate (OpenReview + Forecasting)

---

Table 4: Legacy (Underpowered) Martingale-Brier Causal Relations for ChainOfThought (OpenReview + Forecasting)

---

Table 5: Legacy (Underpowered) Martingale-Brier Causal Relations for ChainOfThought (OpenReview only)

Outside of table 4, both table 5 and the old one in our rebuttal suffer from an extreme lack of statistical power

You should've identified beforehand that you were underpowered. It's fine, we all miss things, but now the issue is that including more trajectories increases power, but also can be seen as p-hacking.

with 6x OpenReview trajectories-per-sample and 7.5x debate trajectories-per-sample

What is the rationale behind 6x vs 7.5x?

We'd like to note that we see weakly positive results in CoT+OpenReview as well,

Again, I disagree. See my comments above under Table 2.

in addition to the strongly positive results in CoT+Forecasting

This I believe is correct because of controlling for Domain. Still, would you mind sharing this by itself? Table 1 includes both Forecasting and OpenReview.

We believe CoT is a much better testbed than Debate is, and we should focus most of our attention on CoT.

I might agree with this, but then the question is, why do you include Debate at all? If we have to ignore it, then you are only evaluating CoT (quite minimal evaluation) and could also be seen as cherry picking. To me this is a sign of a poor evaluation design.

OpenReview has very noisy ground-truth signals (i.e. acceptance decisions). [...] Hence OpenReview may not be a great testbed for Martingale Score either.

Again, this shows me that there is a flaw in your evaluation. Otherwise, this just leaves us with only your results for Forecasting and CoT, where you have the largest positive correlation.

There are 4 plots in Figure 4. Admittedly, only Forecasting + CoT setups have achived results that strongly support that claim "the practical relevance of the Martingale Score is most clearly validated by its strong correlation with prediction accuracy (the Brier score)". However, we'd like to note that forecasting is NOT merely a dataset within a task, but an ML task in itself, covering a diverse range of real-world subjects.

I'm sorry, but this sounds like an ad-hoc justification. If that were true, then why would you need to include all other setups?

We still hold the belief that Martingale Score is generalizable.

I might agree with this, and I really like the core idea of the paper. However, your evaluation is flawed, and it's not answering this question convincingly.

---

In your rebuttal, you addressed my most urgent concerns, but missed something that I'd like to know:

Why does part of the prompt in “The Prompt for Belief Measurement” in Appendix B.2 constantly repeat?

First of all, thank you for such detailed responses. I hope we have enough time before the deadline to discuss everything. I'm going to summarize the discussion so far for the other reviewers and AC.

Summary up until now: My main concern during the initial review was the evaluation, especially issues of nonlinearity and heteroskedasticity in Figure 4. The authors in their rebuttal provided results handling outliers in three conditions (1) light (up to 10 outliers removed), (2) heavy (up to 30 outliers removed), and (3) conservative. However, they did not report results for all setups due to a misunderstanding on their side. Recently, they provided a lot more results with different conditions (1) light (up to 3 outliers removed), (2) medium (up to 10 outliers removed), and (3) heavy (up to 30 outliers removed). Besides changing the criteria for outlier removal, they also found that they were underpowered and included more trajectories in the analysis (with different criteria per setup). The outliers are computed from the total number of points (n=84), which does not guarantee balanced removal across setups (each setup has n=21).

One of the issues I see with removing outliers based on all points, is the imbalance it creates. For example, no outliers are removed for Debate on Forecasting or OpenReview, except for the few in heavy removal. However, looking at Figure 4, I can see that Debate would benefit from removing some outliers.

Under heavy, among the 30 outliers removed, 9 is in Forecasting + CoT, 15 are in OpenReview + CoT, and 6 are in Debate.

What do you mean by "6 are in Debate"? Does that include both OpenReview and Forecasting?

The imbalance here becomes a problem, and heavy might be too heavy to even consider these results very useful. For Forecasting + CoT, you're removing almost 50% of points, and for OpenReview + CoT you're removing over 70%. At that point, the regression is meaningless.

What is the justification for changing the outlier removal criteria? That is, going from light, heavy, and conservative to light, medium, and heavy?

Table 1

The Basic modeling in Table 1 (CoT for OpenReview and Forecasting) shows positive correlation (0.632) when no outliers are removed. It decreases a bit for light (0.509, i.e., removing 3 points), and flips the sign for medium (-0.105, i.e. removing 10 points) and heavy (-0.354, i.e. removing 24 points).

By default, it's not very promising, so it's great that the authors controlled for other variables. When controlling for domain, the consensus is better (except for heavy, but I'd ignore that). My takeaway is that one of these two setups does not have a positive correlation, which you can see below as I discuss Table 2.

Table 2

The Basic modeling (CoT + OpenReview only) goes from 0.120 (no outliers) to 0.604 (light, removes 2 outliers) to -0.282 (medium, removes 6 outliers) to 0.109 (heavy, removes 15 outliers). My takeaway is that there is no meaningful correlation. My main reason is that the medium outlier removal is the one that makes the most sense and it flips signs.

Table 3

The Basic modeling (Debate for both OpenReview and Forecasting) shows no difference for light and medium outlier removal. The reason behind it is the imbalance in the outlier detection, since no outliers are being removed in this case. There are some slight changes in heavy outlier removal, but only 6 points were removed.

However, when controlling for domain and other variables, correlations flip. My takeaway is that these correlations are not meaningful.

---

Let me address some of your comments on your results:

Table 1 (CoT, both domains): Positive. With-outlier-removal results and no-outlier-removal results consistently agree in sign and are positive. Consistent with Figure 4.

I think this is only true for CoT on Forecasting. Please see what I said above under Table 1.

Table 2 (CoT, OpenReview only): Weakly positive. Light and heavy both consistently (100%) positive, while medium leans (75%) negative. No-removal leans positive but without stat significance.

I think this is not true. Already, no-removal is not significant, heavy removes too many points (over 70%) to be a useful signal; and medium flips.

Table 3 (debate, both domains): Ambiguous. Heavy outlier removal is consistently (100%) positive, while medium outlier removal leans (80%) negative. Light outlier removal gives mixed results.

To me, this is not ambiguous at all. The outliers are not removing anything (or almost anything), so is expected for Basic to not change at all. However, when controling for other variables correlations reliably flip.

To level the sample size gap between CoT and Debate, we have multiplied the former's sample size by 5x, making the two equal in statistical power. This changes the numbers in Table 1-2.

For transparency, we include pre-expansion results too in Table 1a-2a at the end. We acknowledge the double-testing concern associated with expanding the sample size, which we hope to mitigate with transparency on the previous results.

Table 1: Forecasting-Only ChainOfThought (post-expansion: consistent with pre-expansion)

Table 2: OpenReview-Only ChainOfThought (post-expansion: mixed coef signs pre-expansion -> consistent coef signs post-expansion)

Table 3: Forecasting-Only SelfDebate

Table 4: OpenReview-Only SelfDebate

---

Table 1a: Forecasting-Only ChainOfThought (pre-expansion)

Table 2a: OpenReview-Only ChainOfThought (pre-expansion)

We'd like to thank the reviewer again for the thoughtful feedback!

Besides changing the criteria for outlier removal

Quick clarification: we did not change the criteria. The new modes light/medium/heavy are simply a rename of the old conservative/light/heavy modes respectively; you can verify this by checking the definition of light/medium/heavy in our response. We apologize for the confusion!

One of the issues I see with removing outliers based on all points, is the imbalance it creates.

We agree with this concern. Please see the new Table 1-4 for per-quadrant results, with the imbalance issue fixed by restricting samples to only those in the quadrant. We also adjust medium and heavy modes to percentage-based (15% and 30% respectively) rather than fixed-count, to avoid removing too many samples, while aligning with the original removal counts on the full dataset.

Light mode remains based on a $4/n$ Cook's distance threshold.

What do you mean by "6 are in Debate"? Does that include both OpenReview and Forecasting?

Yes. And upon your request (which we agree with), we present regression results for each of the 2x2 quadrants in the new Table 1-4.

Table 1 / Table 2 / Table 3

We agree with you that much of the inconsistency is due to (1) mixing Forecasting and OpenReview in the same table, and (2) outlier imbalance causing unreasonably many removals.

We have tried to address these issues in the new Table 1-4. The verdicts there seem much clearer. Notably, within each quadrant, significant results now always come with the same sign.

• Forecasting-Only ChainOfThought (Table 1): Positive.
• OpenReview-Only ChainOfThought (Table 2): Leans positive; only partially/marginally significant. (note the multiple-testing issue here; that said, even the signs of insignificant results are consistent)
• Forecasting-Only SelfDebate (Table 3): Leans negative; only partially significant. (note the multiple-testing issue here)
• OpenReview-Only SelfDebate (Table 4): Positive.

including more trajectories increases power, but also can be seen as p-hacking.

We acknowledge that it's a valid concern. That said, testing 2 times results in a Bonferroni correction factor of 2 (a conservative estimate given the non-independence between tests; the actual factor is somewhere between 1 and 2). Given that most of our $p<0.05$ results indeed satisfy $p<0.025$, such a correction isn't fatal to our conclusions. We will add such a note to our results in the body of the paper.

What is the rationale behind 6x vs 7.5x?

Upon examination of the legacy data, we heuristically chose to multiply all OpenReview sample counts by 2x, and all Debate sample counts by 5x (the latter to compensate for the, on average, 4-5x fewer reasoning steps that debate trajectories have). The two factors multiply, so the OpenReview + Debate setup receive 10x.

When summed across OpenReview setups, total sample size becomes (2+10)/2=6x that the original sample size, and for Debate it's (5+10)/2=7.5x. These are "on average" statistics, as we mentioned in our previous response.

In our new Table 1-2, we have leveled the gap between CoT and Debate by multiplying the former by 5x too. Now all Forecasting data points are backed by 1000 trajectories each, and all Debate data points are backed by 2000 trajectories each.

This I believe is correct because of controlling for Domain. Still, would you mind sharing this by itself? Table 1 includes both Forecasting and OpenReview.

Sure. Please see the new Table 1 attached.

I might agree with this, but then the question is, why do you include Debate at all?

We started out hoping to demonstrate the Martingale score's cross-task, cross-reasoning mode applicability by having this 2x2 quadrant.

We only realized the issue of noise and statistical power in Debate and OpenReview later on in our analysis, and by that point, removing those results would actually be cherry-picking. For transparency, we decided to include them in the body of our paper too.

Why does part of the prompt in “The Prompt for Belief Measurement” in Appendix B.2 constantly repeat?

Apologies - there was a formatting error when we created the code block. As you can see, it contains multiple string variables:

• belief_eval_judge_prompt: The prompt itself; the first string in the block; variable name blocked out by accident.
• belief_eval_judge_prompt_with_traj: The prompt itself, but with the per-trajectory evaluation variation. Namely, the judge model reads an entire reasoning trajectory at a time, and assigns belief scores to each step all at once. This reduces within-trajectory random noise in judge assessments. This is the default prompt design we use in our results.
• additional_info_interlude / additional_info_item / additional_info_ending: These are add-on items to the prompt, where you pass things like background information related to the topic. We do not use these by default.

Thank you so much for the feedback!

1. Difference between truth-seeking and truthfulness/correctness:
   1. Truthfulness/correctness on closed-ended questions: Language models have been shown to hold parametric beliefs (beliefs stored in parameters) related to the question at hand [1]. When we measure model accuracy on common-sense logic or knowledge-based QA tasks, we are primarily testing the extent to which (i) the parametric beliefs match reality and (ii) the model honestly reports its parametric beliefs.
   2. Truth-seeking on open-ended questions: Truth-seeking concerns the model’s ability to form new, accurate beliefs at inference time through extended reasoning, which is distinct from parametric beliefs. Frontier mathematics and scientific research questions, forecasting, academic peer review, and value reflection are examples of open-ended domains where this kind of extended reasoning is key. Since belief entrenchment is less likely to occur in mathematical reasoning, we use the other three, more subjective domains as our tasks.
2. Clarifying the utility of Bayesian reasoning: In Bayesian epistemology, evidence can be both external and internal. The capability of incorporating external evidence deals with search evidence to answer certain query; whilst the capability of incorporating internal evidence, Not all evidence is external to the reasoner; by coming up with novel arguments/counterarguments, logical deduction, etc, also helps LLMs (and humans too) to be better Bayesian reasoners. The language model can obtain evidence without external information, and therefore needs to perform Bayesian updating based on that evidence.
   1. Example: Suppose that the language model is asked to forecast the match result between team A and team B in FIFA 2025, two teams that never played against each other before. The model may have memorized all past FIFA results in-parameter and already know the base win rates of A and B respectively, but by recalling the style of A and B and think about which style is advantageous in a head-to-head match, it will be able to update its belief, even though no external input is involved in the process.
   2. Such discussion is rooted in Bayesian epistemology literature. Its precise term is “The Problem of Logical Omniscience”. It says that assuming a reasoner can only reassess their belief based on external evidence would equalize a reasoner to be “a logical omniscient”, meaning that if they hold a belief, they instantly know all the logical consequences of their belief or automatically believe all the theorems derived from their belief. Humans are obviously not like this, nor are LLMs [2].
3. Whether training already ensures calibration: Empirically, this is shown to not be the case [3]. Theoretically, this is likely because (i) models learn their belief updating strategies from human demonstrations in pre- and post-training datasets, while the humans themselves exhibit a wide range of cognitive biases including belief entrenchment [4]; and (ii) many post-training datasets train the model to do things entirely different from truth-seeking belief updating, e.g. unquestioningly follow user instructions, which has been shown to break the distribution-matching property of the cross-entropy loss in pretraining-stage autoregression [5].
4. Clarifying what questions we aim at with the martingale metric:
   1. The forecasting questions that are resolved after the LLM knowledge cut-off date are examples of questions we think believe entrenchment hurts performance, similar to confirmation bias that hurts people’s judgment. While the model can’t answer those forecasting questions in full just with parametric belief, its parametric belief (e.g. “Team A has a historical win rate of 70% while team B only has 50%, so it seems like A has an advantage...”) serves as an informative starting point for reasoning (“...but B has a good shooter while A’s goalkeeper is subpar”). We are therefore interested in whether the parametric belief unduly dictates later belief updates, aka belief entrenchment.
5. On Figure 2: Thank you for the feedback. We are replacing it with more informative infographics about how exactly the Martingale metric is computed.

Reference

[1] Wentao Zhu et al., Language Models Represent Beliefs of Self and Others

[2] Robert Stalnaker, The Problem of Logical Omniscience

[3] Andrea Yaoyun Cui et al., Do Language Models Have Bayesian Brains? Distinguishing Stochastic and Deterministic Decision Patterns within Large Language Models

[4] Aileen Oeberst et al., Toward Parsimony in Bias Research: A Proposed Common Framework of Belief-Consistent Information Processing for a Set of Biases

[5] Robert Kirk et al., Understanding the Effects of RLHF on LLM Generalisation and Diversity

Here is our new results to address reviewer's early concern on LLM judge calibration. We would welcome reassessment if the reviewer finds the newly acquired results informative. Thank you!

We would like to to start by noting that LLM calibration has been a notoriously difficult problem. Previous research has demonstrated that LLMs overall are poorly calibrated (in particular, they have strong tendency to overstate self-reported confidence; and token prob can go extremely high for binary problems, which does not represent true confidence) [1]. Findings reported in literature stays consistency with our experience of self-reported confidence and token probability.

Instead, we move on to the "LLM-as-judge" approach as a scalable alternative to avoid the problem of over-stated confidence and extreme token probability. However, LLM-as-judge approach has its own biases (e.g., they may repeat human falsehood; or they may prefer their own answers) [2,3]. Hence, we conduct cross-judge consistency evaluation. Specifically, we construct pairs of judge (e.g., GPT-4o VS Gemini-2.5-pro, GPT-4o VS DeepSeek-v3) and see how much their belief evaluation correlates to each other. Note that we've acquired GPT-4o data for all batches but only a few batches for each of other judges (from 3 to 48 batches), so we set up GPT-4o as default judge for all comparison.

Moreoever, we use small batch of human-evaluation data to validate LLM judge (i.e., human-llm consistency evaluation). Specifically, we construct pairs of human-LLM judges (e.g., human evaluator 1 VS GPT-4o). Note that the amount of human evaluation data is a lot smaller than that of LLM's, due to limited time. Nonetheless it's large enough (e.g., 195 and 173 belief samples) to give us a robust intuition of human-llm agreement on belief evaluation.

Full results can be seen below:

Overall, we have Sample Size: 31,992 total belief evaluations across 4,538 problems; all judge show large positive correlation with GPT-4o; all results are statistically significant (P<0.001). Among them, one of the human judges achieves highest correlation with GPT-4o, and all LLM judges achieve higher agreement with GPT-4o than the other human evaluator. As a comparison, NeurIPS 2021 experiments shows ~0.58 correlation between paper acceptance decisions made by two committees [4].

Reference [1] Yudi Pawitan, Confidence in the Reasoning of Large Language Models

[2] Stephanie Lin, TruthfulQA: Measuring How Models Mimic Human Falsehoods

[3] Arjun Panickssery, LLM Evaluators Recognize and Favor Their Own Generations

[4] Alina Beygelzimer, Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

Thanks reviewer for raising constructive comments, pointing out weakness and limitations, and asking thought-provoking questions. The following is our attempt to address those comments and questions.

Experimental setups and processes and the computation of martingale score (Weakness 1, 2, 3, 5, Q3)

Thanks for pointing out all the weaknesses and asking questions about experimental setups.

• The unclear description of experimental setup and missing out of martingale score calculation in practice:
  • As mentioned in section 4.1, we use Ordinary Least Squares (OLS) method to approximate martingale score in practice. Still, the detail here may still appear inadequate to reviewers and broader audiences. We suspect it might be because we did not link the definition of martingale score to the experiment setup. In our experiments, we calculate the martingale score per setup, where we include 99 questions (n=99; e.g., forecasting questions), b_prior is calculated when reasoning starts, which reflects our best attempt to capture a parametric belief of LLMs; b_posterior is calculated after inference or certain amount of reasoning steps; and b_update is the difference between both.
• The experimental procedure:
  • Key experimental details requested: There are around 200 questions per domain, 1 trajectory per question, and ~ 5 reasoning steps per trajectory. We use 200 different problems per setup for each Monte Carlo Estimation. In other words, calculate expectation of model belief across questions. Thanks for pointing out the missing details here, and they will be added into the final manuscript.

The concerns over judge models and token probability (W4, Q4)

We thank the reviewer for this insightful comment. Our use of an external judge model is a deliberate methodological choice to address the well-known challenges of reliably eliciting beliefs from LLMs. Our approach is validated by new, large-scale tests showing high consistency across different judge models.

Of course. Here is a revised rebuttal that is more concise, incorporates your supervisor's feedback, and addresses the status of the cross-judge validation.

We thank the reviewer for this insightful comment. Our use of an external judge model is a deliberate methodological choice, made because direct belief elicitation from LLMs is notoriously unreliable. Our approach focuses on measuring expressed belief, and while ensuring cross-judge consistency is an important validation step, the core method remains a pragmatic solution to the challenges of probing a model's "true" belief state.

The Unreliability of Direct Belief Elicitation

We initially attempted to measure belief using direct methods like token probabilities. This approach was unworkable because LLMs consistently produce extreme and poorly calibrated probabilities for binary questions (e.g., 99.9%) [1]. This is a recognized challenge in the field of ML calibration. These extreme values do not reflect a model's true epistemic state; in our experiments, models expressing near-certainty would often reverse their positions easily. This makes token probability an unsuitable tool for measuring nuanced belief updates.

The LLM-as-Judge Approach

To overcome this, we use an external judge model. This shifts the focus from an unreliable "internal belief" to the model's expressed belief as interpreted by an independent observer. The primary concern here, which the reviewer rightly notes, is the reliability of the judge itself. We agree that a comprehensive, cross-judge reliability analysis is an important step to fully validate this evaluation paradigm.

However, the core logic of using an external judge is sound. We use a courtroom analogy: the reasoning LLM is the "suspect" asked to think out loud, and the judge model is the independent observer assessing that monologue for bias. This allows us to measure entrenchment in the expressed reasoning—which is most relevant for human-AI interaction—while sidestepping the thorny question of "what does an LLM truly believe?"

This focus is suited for studying how models update on internal evidence—the arguments and logical deductions they generate themselves. This allows us to explore a key aspect of their behavior related to the "Problem of Logical Omniscience" in Bayesian epistemology, justifying the use of an external judge to assess the reasoning process.

The question of whether sufficient parametric knowledge recall is sufficient for practical use (comment 3).

• While the intuition that a model with sufficient parametric knowledge makes it a good Q&A assistant, such a model may be inadequate to the needs of the real-world, as the knowledge production in real-world is a dynamic process whereas the parametric knowledge, however good it might be, is static. When deployed in the real-world, such a model faces practical limitations. LLMs deployed in the real-world are expected to give user evidence-based answers (evidence acquired from search or user-interactions), rather than only share prior-conforming ones. In contrast, LLMs that suffer from belief entrenchment may experience performance degradation because of prior-conforming behaviors (as shown in forecasting).

The scope of domain where the metric is applicable (limitation 3)

• While it is tested on Bernoulli distribution, martingale metric is intended to be an unsupervised, domain/distribution-agnostic metric. It is tested in forecasting questions and peer review because in these two problem domains there are ground truth labels to validate martingale scores. Similar to confirmation bias that harms human judgment which is not domain-specific, belief entrenchment hurt LLM performance, too, as we demonstrated in forecasting, As research progresses, we hope to find ways to validate martingale score in subjective domains, or domains where ground truth labels simply do not exist (Two domains we plan to run: OpenReview but to assess the relation between martingale score and citation number (as this more closely aligns with collective voted judgments) and Reddict r/AITA (also collective voted judgments).

The question of whether evidence LLMs see is already biased (Q2)

• As mentioned above, in Bayesian epistemology evidence can be either internal or external, and for practical reasons we primarily focus on internal evidence in our current experimental setups. To directly answer the reviewer's question: the internal evidence (the reasoning steps) are sampled from the prior and its context, and if it is biased, then this is evidential that belief entrenchment exists: model may always output prior-conforming reasonings to answer user questions.

Conceptual discussion on Bayesian rationality (comment 1)

• The request for “more literature to support the optimality of Bayesian rationality” is fair. We thank the reviewer for pointing this out. Bayesian reasoning is regarded as optimal because it provides a quantitative framework to incorporate both prior and new evidence in forming beliefs and it is able to handle uncertainty [2]. Conceptual arguments (e.g., “Dutch Book Argument” [3]: The claim that "Bayesian reasoning is optimal" and that "effective reasoning requires Bayesian rationality") and ample empirical evidence support the effectiveness of Bayesian reasoning (e.g., financial modeling, jury decision,medical diagnosis) [4] We thank reviewer for this valuable comment.

Assigning intuitive labels to Martingale Score (Q5)

We thank the reviewer for pointing out those important omissions, or otherwise lacking clarity of experimental setup in our paper submission and confusion with figure 1. And thanks for pointing out typos in the manuscript. All those overlooked details will be added into the final publication.

References:

[1] Yudi Pawitan & Chris Holmes, Confidence in the Reasoning of Large Language Models

[2] C Howson & P Urbach, Scientific reasoning: the Bayesian approach

[3] Dutch Book Arguments, Stanford Encyclopedia of Philosophy

[4] Aaron M. Ellison, Bayesian inference in ecology

Thanks for your earlier comment. In the following official comment we would like to share our judge-consistency and human-evaluation validation to address the concern over judge bias. We would welcome reassessment if the reviewer finds the newly acquired results informative.

We explained our rational of using LLM as judge models to overcome the problems with self-report and token probability [1]. As the reviewer rightly pointed out, the LLM-as-judge approach may introduce judge bias in the process of evaluating belief entrenchment. Here we include a note on the conceptual side of belief evaluation and our empirical results of testing different judges.

On the conceptual side, We treat (assume) the judge model as a reliable & independent observer of the reasoner's “monologue” and assessing whether the reasoning process suffers from belief entrenchment. Using a court analogue here: with this method, the judge model, just like a "court clerk", simply asks the “suspect” to "think out loud" about whether they are guilty. If the suspect's internal monologue (the CoT) is a biased stream of "evidence" aimed at proving their innocence, a judge who answer how the binary questions are resolved faithfully based on the records of monologues, will be one-sided. By doing so, we can evaluate whether model + LLM judge as a multi-agent systems, suffer from belief entrenchment. And of course, here it comes to the question of judge bias. If belief entrenchment is a judge bias rather than model's, then changing judge models would remove such artifacts. However, in our experiments, judge evaluations are strongly correlated.

In our cross-judge consistency and human-LLM agreement anslysis, all judges (LLMs or humans) show large strong correlation with GPT-4o.

Cross-judge consistency evaluation: we construct pairs of judge (e.g., GPT-4o VS Gemini-2.5-pro, GPT-4o VS DeepSeek-v3) and see how much their belief evaluation correlates to each other. Note that we've acquired GPT-4o data for all batches but only a few batches for each of other judges (from 3 to 48 batches), so we set up GPT-4o as default judge for all comparison.

Human-LLM agreement evaluation: we use small batch of human-evaluation data to validate LLM judge (i.e., human-llm consistency evaluation). Specifically, we construct pairs of human-LLM judges (e.g., human evaluator 1 VS GPT-4o). Note that the amount of human evaluation data is a lot smaller than that of LLM's, due to limited time. Nonetheless it's large enough to give us a robust intuition of human-llm agreement on belief evaluation.

Full results can be seen below:

Overall, we have Sample Size: 31,992 total belief evaluations across 4,538 problems; all judge show large positive correlation with GPT-4o; all results are statistically significant (P<0.001). Among them, one of the human judges achieves highest correlation with GPT-4o, and all LLM judges achieve higher agreement with GPT-4o than the other human evaluator. As a comparison, NeurIPS 2021 experiments shows ~0.58 correlation between paper acceptance decisions made by two committees [4].

Reference

[1] Yudi Pawitan, Confidence in the Reasoning of Large Language Models

[2] Alina Beygelzimer, Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

Thank you so much for the thoughtful feedback!

1. Experimental setup details: There are around 200 questions per domain, 1 trajectory per question, and 2-20 reasoning steps per trajectory. We use 200 different problems per setup for each Monte Carlo Estimation. In table 1, it can be seen that almost all CoT entries have statistically significant Martingale scores. For Debate, while only some of the individual entries are statistically significant, on aggregate, we are able to show it significantly reduces belief entrenchment compared to CoT (Figure 7a).
2. The interpretation of t-test results in table 1: The t-tests attempt to reject the null hypothesis that no belief entrenchment exists (i.e. the Martingale score is zero at the limit of infinite samples). When p>0.05, it should be interpreted as that no credible evidence for belief entrenchment exists.
3. The interpretation of positive and negative Martingale Scores: When the Martingale score is negative with p<0.05, it should be interpreted as that the model tends to be overly “self-critical” - it tends to update against its prior belief - the opposite of belief entrenchment. This is observed in only 2 out of 108 setups, and is very rare. As a result, we believe it is of limited value to study these rare cases in depth.
4. An intuitive explanation of figure 5: The regression in Fig 5 answers the question “Does a higher (absolute value of) Martingale score causally worsens accuracy?”, where every sample is the (Martingale score, accuracy) pair of a cell in Table 1. To tell causality from mere correlation, we add control variables to the regression and thereby remove confounders.
   1. Taken together, we have these variables in our regression:
      1. Explanatory variable: abs(Martingale)
      2. Explained variable: accuracy, i.e. Brier score
      3. Control variables: Domain (forecasting/openreview/CMV), Reasoning Mode (CoT/debate), Prompt (No/CT/PC).
   2. In Fig 5, from left to right, we gradually add more control variables. We watch for (i) the increase in R2 (lower subplot), indicating how much of the accuracy variance has been captured by our control variables; (ii) the decrease in p-value (black text labels in upper subplot) of the regression coefficient on abs(Martingale); and (iii) the sign of the regression coefficient on abs(Martingale).
      1. It can be seen that, when we have at least Domain + RM as controls, we have highly non-trivial R2 (0.27-0.4), p-values < 0.05, and the coefficient on abs(Martingale) is consistently positive.
      2. This suggests that, after adjusting for confounders (domain & reasoning mode), larger absolute value of the Martingale score (aka more deviation from the Martingale property) causally harms accuracy.
5. Re “the positive correlation claim in Figure 4 appears weak and potentially overstated”:
   1. For CoT on Forecasting:
      1. Robustness: rho = 0.63; this is a consistent correlation
      2. Effect size: A 0.1 increase in the Martingale score corresponds to approximately a 0.15 increase in Brier score, which is a huge drop in accuracy, larger than the difference (0.101) between human crowd wisdom and dice-rolling. [1]
   2. For CoT on OpenReview:
      1. Robustness: rho = 0.20, and after removing outlier rho=0.53; while this is smaller than in forecasting, this is likely due to the extreme noise in peer review labels [2].
      2. Effect size: Similarly large compared to in forecasting.
   3. For Debate: It’s worth noting that for most of the Debate entries in Table 1, their Martingale scores are not statistically significant. As such, we have no evidence suggesting belief entrenchment exists in such cases. This would be consistent with the observation that the regression lines are only weakly upward-sloping, as they are dominated by noise.

Reference

[1] Danny Halawi et al., Approaching Human-Level Forecasting with Language Models

[2] Alina Beygelzimer et al., The NeurIPS 2021 Consistency Experiment

Thank you for the thoughts. We really appreciate your feedback!

1. Focus on diagnosis over mitigation: While our experiments cover only diagnosis, we believe our contribution extends to mitigation too. A key difficulty for comprehensively correcting model cognitive biases is that doing so requires ground-truth information, which is unavailable in almost all domains of interest (domains involving subjective judgments). By proposing the Martingale score, a fully unsupervised & domain-agnostic metric for belief entrenchment, we demonstrate both its promise as an evaluation metric, and, as a corollary, its potential as a training signal.
2. Nonlinearities: We have experimented with ways to handle outliers in our regression, which is a primary cause of nonlinearities in data. We also tried quadratic regression. However, non-linear regressions in general lack a single coefficient or value that reliably points to “causal contribution of prior to belief updates”; this makes the design of the Martingale score difficult. We also have a relatively small sample size, which risks overfitting when using parameter-rich models of regression.
3. Does the Martingale Score provide any insight into why debate helps?
   1. We think your hypothesis turns out correct! Analysis of 78 reasoning trajectories shows that instead of making AI reasoning fundamentally better, debate mostly just makes belief updates more conservative. This shows that removing belief entrenchment is still a hard open problem, and we are glad that we unveiled this problem with our evaluations.
   2. Specifically: When we looked at bias scores regularized by posterior variance vs belief change variance (Var[delta]), both give similar model fits (R² around 0.19-0.27). But the Var[delta] models reveal something interesting - debate's main job is keeping belief changes from going wild rather than actually improving calibration.
      • Here, the motivation for regularizing with Var[delta] is that, we want to regularize the amount of reasoning/updating that gets down, to rule out the possibility that debate merely reduces updating.
   3. The correlation patterns support this conclusion. In forecasting tasks, chain-of-thought reasoning shows a strong bias-accuracy relationship (ρ=0.63, p=0.002); basically, when it's biased, performance tanks. But, for self-debate, it's nearly flat correlation (ρ=0.06, p=0.80). This suggests that debate doesn't actually eliminate the bias; instead, it's the bias measurements being overwhelmed by stochastic noise, likely due to insufficient sample size as a result of our compute budget.