Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliability

1. Line 23: LL strategies and models – It’s unclear what the LL strategies refer to.
2. The use of “and” connections feels excessive, making the sentences overly long and harder to follow, even though they could be broken down more effectively. Most of the sentences in the introduction span more than three lines, which impacts readability.
3. Figure 1 is mentioned in Line 46, but there’s little explanation or clear connection between Figure 1 and the content in the text.
4. Line 86: It’s unclear what “this” is referring to. The sentence structure in 86-87 should be adjusted to convey the point more clearly. The connection with previous content is weak, making it harder to understand what problem the authors are addressing.
5. Line 94: It would be better to replace “this” with something more specific for clarity.
6. Line 98: There should be a reference for no-regret optimization.
7. This paper can be better started by defining the key elements, conditions, and concepts behind Perfect Bayesian Equilibrium and Perfect Bayesian Nash Equilibrium.
8. Figure 2 doesn’t sufficiently explain the notations and what they represent in the caption or the text.
9. Line 159: The notation for Information set ‘I’ is confusing, as it overlaps with the notation used for incorrect signaling.
10. The lack of references for many of the equations in the paper affects overall readability, and Line 206 should reference Eq. 1, not Thm. 1.
11. In Figure 3 (b) and (c), it’s unclear what the x and y axes represent. Additionally, as someone with red-green color blindness, I find the figure difficult to interpret. More explanation in the text under “Searching & Convergence Behavior” would be necessary for full comprehension.
12. The layout of Table 2 is confusing. While it’s clear that ECG and BDG outperform G by Imp.%, it’s unclear what InC.% is being compared to or what the down arrow indicates.

My main concerns about the work are regarding the presentation and framing. Mainly, the paper is quite hard to understand and it is not clear what the contributions are wrt. related works:

• Some of the terms are fairly unclear and could benefit from an earlier intuitive explanation, for example, "calibrated ambiguity" (L73). Also, for example, the authors mention NLP metrics and perplexity in L260-L264 but never introduce them (for the latter, this would be easier if also P_LM was introduced properly, which also is called P_LLM at other places).

• The paper is not very accessible to an audience that is not familiar with game-theory, for example, the term strategy profile is never really formally introduced. The paper also lacks a good description of what a language model actually is and what it means for it to be used as a generator or discriminator. In my opinion, the paper would benefit significantly from aligning notations and terminology between what is common in the LLM and game-theory community.

• The paper is unnecessarily written in complex language at many stages, for example, L76-79.

• In L138 you mention that you "propose improvements to address limitations". However, it is not clear what limitations are meant and especially, whether those are limitations of Jacob et al. 2024. If they are, it would be very important to make clear how they are addressed and how they arise in their method. There is only a short comparison of how the methods differ in Table 1 which to me is not sufficient.

• Overall, I am having troubles identifiying based on the main paper what the contributions of the authors are in terms of the algorithm wrt. to Jacob et al. 2024 and what is more of a formalization of their method. This needs to be made clear in my opinion.

I admit that I could not make sense of a large amount of the text in this paper—I find the writing extremely vague and confusing. The developments in the appendix (e.g., Appendix C) were much clearer to me. Some examples from the main text:

First, the introduction is full of vague expressions like the follows:

• "dynamic and rigorous verifier"
• "our work focuses on actual consistency in correctness and calibrated ambiguity in reliability"
• "Instead of absolute reliability, we aim to identify false negative and false positive samples that exhibit fluctuating or unstable behavior under calibrated confidence and disambiguity metrics" these may be interpretable in a way that is technically correct, but the reader is ill-equipped to guess at their meaning. What is the difference between consistency, correctness, reliability, and calibrated confidence? What do you mean by fluctuating or unstable behavior? (fluctuating or unstable as what varies?) Later in the paper when these terms appear again, including in results and figures, they are not defined or illustrated. A single worked-through example would go a long way.

Next, the math prose is incredibly hard to follow. Consider Definition 4.

Definition 4. (Disambiguity Metric) For a prompt $x$ and a finite set of answer candidate $\mathcal{Y}$, a Disambiguity Metric is a function that projects the prompt $x$ and a candidate $y_i \in \mathcal{Y}$ such that $DA(x, y): x \times \mathcal{Y} \to [0, 1]$. The output of the function is a measurement of the disambiguation of $y_i \in \mathcal{Y}$ to the prompt $x$. The disambiguity metric, combined with the correctness parameter, can detect false-positive and false-negative classifications in the correctness alignment phase.

All I can detect that has been said here is the type signature of the metric. Nowhere can I identify how it is computed, what it is supposed to relate to, or why we believe that it can detect false-positive or false-negative classifications, nor is it clear what model is doing those classifications and how we know they're false. It says the function DA is a "measurement of the disambiguation of $y_i \in \mathcal{Y}$ to the prompt $x$", but what does that mean? I can't figure it out. This is the one part that needed to be written out as mathematics, and it is not, unless I'm missing something.

The first six pages of the paper are like this. The experimental section is a bit better in this respect but not much: the figures have a lot of content which is not explained, key metrics like "consistency" are not defined as far as I can tell. I also actually just couldn't figure out what the verifier policy is supposed to be or how it is initialized, until I read Appendix C.

While I feel only weakly confident in my ability to assess the strength of the contribution behind this paper, I feel quite confident that it is not ready for publication because of the issues in the paper's writing and communication. I would look to the work this is building on — Jacob et al. (2024), https://arxiv.org/abs/2310.09139 — as an example of a way of writing a similar paper with more clarity.

1. The paper is extremely difficult to read and follow
2. The BDG framework’s game-theoretic complexity could hinder broader understanding and practical application.
3. The paper lacks introduction to many game-theoretic concepts used, which could be difficult to readers without a game-theory background