Measuring, Evaluating and Improving Logical Consistency in Large Language Models

W2: The evaluation sets are small, and it is not sure if the findings will be applicable in general.

• We would like to emphasize that in Section 3, we conducted experiments on three diverse and well-established datasets: SummEval, NovelEval, and CaTeRS. These datasets encompass a wide range of topics (summarization, document reranking, temporal/causal event ranking) and levels of subjectiveness, ensuring that our findings are applicable across various tasks. This broad applicability supports the generalizability of our proposed metrics.

• We also contend that in the era of LLMs, the diversity and quality of test sets often outweigh sheer quantity. Many recent, highly-regarded benchmarks, such as HumanEval [1] (164 instances) and LLMBar [2] (419 pairwise annotations), are not particularly extensive in size. In contrast, our datasets include substantially larger pairwise comparison scales:

  • SummEval: 100 source texts × 16 × 15 (16 summaries for each text) = 24K pairwise comparisons,
  • NovelEval: 21 queries × 20 × 19 (20 documents for each query) = 8K pairwise comparisons, and
  • CaTeRS (after filtering): 70 instances × 6.2 × 5.2 (on average 6.2 events for each instance) = 2.3K pairwise comparisons.

W3: The datasets also come with "candidates" to select. That means LLMs only need to select an answer from candidates. It is not clear if this connects to the concept of logical consistency.

• We acknowledge that our evaluation approach involves selecting answers from given candidates. However, this paradigm is a common and effective practice for assessing LLMs’ internal understanding, as evidenced by its widespread use in popular benchmarks such as MMLU [1], ARC [2], and StrategyQA [3].

• In our study, we specifically evaluate LLMs’ judgments to probe their logical consistency based on the fundamental rules of transitivity, commutativity, and negation invariance. While these rules may not encapsulate the entirety of human logical reasoning, they are essential and widely recognized tools for assessing logical consistency in systems.

• Additionally, our approach aligns with existing literature, as noted in Section 6 and expanded upon in Appendix D, where we provide further theoretical justification for the importance of transitivity and other logical properties. Many previous works [4,5,6] in NLP and LLM have established that such evaluations are robust indicators of logical consistency.

In conclusion, we assert that probing LLMs’ logical consistency through their judgments on well-defined logical properties is both a sound and commonly accepted methodology. This approach ensures that our evaluation connects meaningfully to the concept of logical consistency.

References

[1] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

[2] Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., & Chen, D. Evaluating Large Language Models at Evaluating Instruction Following. In The Twelfth International Conference on Learning Representations.

[3] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.

[4] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.

[5] Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021). Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9, 346-361.

[6] Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021). Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9, 1012-1031.

[7] Jang, M., Kwon, D. S., & Lukasiewicz, T. (2022, October). BECEL: Benchmark for consistency evaluation of language models. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 3680-3696).

[8] Asai, A., & Hajishirzi, H. (2020, July). Logic-Guided Data Augmentation and Regularization for Consistent Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5642-5650).

We sincerely thank the reviewer for their time and effort in reviewing our paper. We greatly appreciate the insightful feedback provided, as we believe it will significantly contribute to improving the quality of our work. Below, we address and clarify the concerns raised:

W1: This paper mostly explores the empirical behavior of LLMs given 3 performance metrics, and it mentions possible usage for fine-tuning. However, we don't see how to leverage the findings.

We would like to address the potential misunderstanding of our paper’s scope and contributions. Our work systematically examines logical consistency in LLMs from both interpretative and application-oriented perspectives. We believe these contributions, detailed below, provide both theoretical and practical insights:

1. Interpretation and Analysis (Sections 2 and 3):

   • We introduce three novel metrics (Commutativity, Transitivity, and Negation Invariance) to measure logical consistency in LLMs. These metrics are rigorously formalized, allowing us to evaluate LLM behavior across a wide range of models and settings.

     • Unlike the existing benchmarks, our metrics are not designed to assess accuracy in knowledge or reasoning but provide a deeper lens into the logical consistency within LLMs. This information is vital for providing guidance to downstream applications where consistency, not just correctness, is crucial.
     • Furthermore, we demonstrate how our metrics correlate with reliability measures, offering a new dimension to evaluate LLM behavior.

   • Each metric addresses a distinct application need:

     • Commutativity helps address positional bias when making judgements.
     • Transitivity is essential in applications that operate on local comparisons but a reliable global ranking is desired to maintain, such as sorting and optimization algorithms.
     • Negation Invariance provides guidance for tasks that involve flexible relation expressions.

2. Practical Applications and Impact (Sections 4 and 5):

• We leverage these metrics in our novel data augmentation framework to quantify the change of logical consistency in LLMs. Notably, our experiments demonstrate that the framework enhances logical consistency without compromising alignment with human preferences.
• In Section 5, we present a concrete downstream application to highlight the broader implications of logical consistency. The example illustrates how logical consistency impacts the efficiency and performance of high-level downstream algorithms in a completely orthogonal way to accuracy.

By addressing logical consistency in both interpretative and practical contexts, our work lays a foundation for future advancements in logic-dependent algorithm designs and model selection processes. We hope this clarification helps underscore the broad applicability and value of our findings.

Q1(a): The measures depend on the sampling. How do the scores vary?

We would like to thank the reviewer for bringing this concern up. We believe the reviewer is referring to the calculation of transitivity Equ.1, where we sample M sub-graphs from the original relation graph to estimate the transitivity. We would like to claim that setting M=1000 (Line 190) should give statistically accurate and stable estimation. We have included a formal justification in Appendix H.

Q1(b): What are the items (xi or xj) that we put in the equations?

We would like to clarify that ($x_i$, $x_j$) refers to the pairwise comparison by LLMs. As discussed in Line 110-124, the LLMs are used to produce judgements between any two items with in the item set $X$, and the items could be different things depends on the tasks. For SummEval, the items are summary candidates, for NovelEval, the items are retrieved documents, and for CaTeRS, the items are event. The prompt templates used to compare items were included in Figure 7.

Q2: “Are the experiments only conducted in multi-choice datasets that require LLMs to rank the candidates available?”

• We would like to clarify that none of the datasets used in our experiments are classified as "multi-choice" datasets. Instead, each dataset involves multiple items per datapoint with varying characteristics, and the LLM judgments are produced through pairwise comparisons. We did not explicitly ask the LLMs to rank the candidates, instead we evaluated the consistency of LLMs based on all the pairwise judgements.

• This approach does not limit the generalizability of our framework. Our methodology is applicable across a wide range of domains, as discussed in detail in W2 and W3, where we elaborate on the diversity of topics and the relevance of the selection-based evaluation format.

Q3: “The paper doesn't show the details about the instruction-tuning settings.”

• We would like to emphasize that all training configurations for the instruction-tuning experiments are detailed in Appendix E (as referred by line 453). Additionally, information about the Summary with Feedback dataset used in these experiments is provided in Appendix B (as referred by line 446).

• To address the reviewer’s concerns further, we have now included the exact instruction prompt templates utilized for instruction-tuning in Appendix F. We hope this addition ensures full transparency and reproducibility of our methodology.

Once again, we thank the reviewer for their time and detailed feedback. If the reviewer has any further questions or suggestions, we would be more than happy to address them. We hope that, in light of our overall contributions and the responses provided, the reviewer might consider reevaluating the score.

Keeping aside the issues around imprecise description of the work around logic, the content of the paper shows empirical results around the predictive behavior of various LLMs. Although there is a gap in stating that such findings will lead to the internal understanding of LLM around logical reasoning, they still produce useful information. It depends on the interpretation, but the main takeaway would be summarized in Table 1 over K=5.

The next main contribution of this paper would be extracting fine-tuning data from the noisy partial order and aligning the LLM behavior to improve the consistency on the preference rankings. This also connects to the impact on the performance of ranking task in Table 4, where the results indicate that the higher consistency model performs better.

Question: Is this improvement still observed by improving the consistency of a fixed model? Since Table 4 shows five different models, it would be clear if we see that the improved model after fine-tuning in Section 4 indeed shows the improved performance in the Table 4 setting.

Logical consistency is not defined in this paper. The referred papers [4, 5, 6] also don't define it. In the rebuttal, it was mentioned that many previous works [4,5,6] in NLP and LLM have established that such evaluations are robust indicators of logical consistency. However, I couldn't find such statements. [4, 6] don't mention logic anywhere. [5] mentions logic when it mentions a comparator operator.

[1, 2, 3] are datasets for the natural language tasks. After rephrasing the format as question-answering and limiting it to multiple-choice questions, we have a universal task that covers various tasks. This doesn't imply that "it is a common and effective practice for assessing LLMs’ internal understanding." Indeed, none of the three papers claimed anything about the internal understanding of LLMs.

Without a definition of logical consistency, it is unclear how to quantify undefined concepts. The implicit assumption in this paper is having high scores on the three evaluation metrics implies higher logical consistency. It is unclear how such an implication is true. It is unclear how to use this for logic-dependent algorithm design and model selection. It is unclear what a logic-dependent algorithm is.

From what's written, what's measured in this paper is the preference or the order of items that should satisfy a few axioms. When a preference is combined with logic, axioms could be introduced into the knowledge base. For example, such a KB can be seen in ["A logical reasoning with preference," S.K. Das, 1995].

Most places that mention "logical" in this paper refer to the partial order. When this paper claims "logical consistency," it is about consistency with respect to the partial order extracted from LLM over the items. In other words, claims about " logical consistency evaluation " need to be carefully examined to see if they can extend beyond checking the order of preference.

The relationship examined in this paper is the preference/order between two items. The function F could be seen as a predicate such as Preferred(x, y), with x and y being any objects in the knowledge base. The predicate means that x is preferred to y. The paper's main contribution is designing F as prompts for LLM, shown in Appendix F. Figures 7 and 8 show that the predicate is written in natural language sentences, and the objects are two outputs from LLM for a certain task.

It would be more precise to say "LLM as a natural language prompt of a binary preference relation" than "LLM as logical operators." This paper would be closer to measuring/improving the prediction of LLM on the ranking of multiple candidates from various NLP tasks.

It is unclear that "this framework broadly applies to any task where logical relations are relevant." If that is the case, the authors should prove or empirically show it.

Concern 5: Scope of Relations and Logical Consistency in Rankings

From what's written, what's measured in this paper is the preference or the order of items that should satisfy a few axioms. When a preference is combined with logic, axioms could be introduced into the knowledge base. For example, such a KB can be seen in ["A logical reasoning with preference," S.K. Das, 1995].

We appreciate the reviewer’s suggestion regarding preference axioms. However, we would like to emphasize that our scope is to extend the logical consistency evaluation of transitivity, commutativity, and negation invariance from local 2-3 items to arbitrary number of items. This inherently leads to the construction of full or partial rankings when logical relations are coherent.

Previous works have defined these logical consistency rules as constraints in NLI and QA. Our contribution lies in generalizing them to multiple items and general domains with logical relations, and developing metrics that quantify the degree of consistency achieved. Thus, the rankings emerge as a consequence of applying logical consistency rules, rather than as an input assumption.

Concern 6: Definition of Logical Operator

It would be more precise to say "LLM as a natural language prompt of a binary preference relation" than "LLM as logical operators." This paper would be closer to measuring/improving the prediction of LLM on the ranking of multiple candidates from various NLP tasks.

We appreciate the reviewer’s comment and would like to address the suggestion. We are somewhat unclear about the reasoning behind the statement. In the context of a sorting algorithm, the symbol ‘>’ is commonly understood as a logical operator, as it is expected to make logically consistent judgments when comparing two numbers.

Similarly, when using LLMs within logic-dependent high-level algorithms, we expect them to exhibit logical consistency when making pairwise judgments between items. This aligns with the behavior typically associated with logical operators.

If the reviewer has a different interpretation of whether ‘>’ qualifies as a logical operator, we are open to rephrasing it as a "logical comparator" to ensure clarity and alignment with their perspective.

General Response:

We sincerely appreciate the reviewer’s thoughtful feedback. It seems there may be differences in how logical consistency is understood and defined. We acknowledge that as the field of NLP evolves rapidly, variations in terminology and interpretations can arise. However, we want to emphasize that our work builds upon and aligns with established understandings in prior research, which clearly consider transitivity, commutativity (or symmetry), and negation invariance as fundamental logical consistency rules.

We kindly request the reviewer to consider the broader contributions of our work, beyond potential differences in definitions, as we believe our findings and methodologies offer meaningful advancements in evaluating and improving consistency in LLMs, and also demonstrating their impacts.

We sincerely thank the reviewer for their detailed and thoughtful feedback. We greatly appreciate the time and effort you have devoted to evaluating our work. We address the concerns raised below:

Concerns 1: Definition of Logical Consistency

Logical consistency is not defined in this paper. The referred papers [4, 5, 6] also don't define it. In the rebuttal, it was mentioned that many previous works [4,5,6] in NLP and LLM have established that such evaluations are robust indicators of logical consistency. However, I couldn't find such statements. [4, 6] don't mention logic anywhere. [5] mentions logic when it mentions a comparator operator.

We apologize for the oversight in our previous rebuttal, where references [4, 5, 6] were mistakenly cited. The correct references should have been [6, 7, 8]. We would like to emphasize that defining transitivity, commutativity, and negation invariance as rules for determining logical consistency is a well-established approach in prior works. These rules assess the logical consistency of language models from certain perspective, and, as mentioned in previous rebuttal, we have extensively elaborated on this connection in Section 6 and Appendix D of our paper. To further clarify, we explicitly list below the relevant prior works that align with this view:

• BECEL: Benchmark for Consistency Evaluation of Language Models:

  This work discusses negational consistency, symmetric consistency (equivalent to our commutativity consistency), transitive consistency, and additive consistency as forms of logical consistency in Section 2.

• Logic-Guided Data Augmentation and Regularization for Consistent Question Answering:

  Logical rules for consistent QA are covered in Section 3.1, including symmetric and transitive consistency.

• A Logic-Driven Framework for Consistency of Neural Models:

  Annotation consistency, symmetry consistency, and transitivity consistency are analyzed as logical consistency in Section 3.1.

• Consistency analysis of chatgpt.:

  Semantic consistency, negation consistency, symmetric consistency, and transitive consistency are discussed as logical consistency in NLI tasks in Section 1.

• Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge:

  Symmetric, reflexive, and transitive rules are presented as structural constraints for human reasoning in Section 1.

Additionally, on the theoretical side:

• Intransitivity of preferences:

  They explored psychological factors leading to intransitive preferences, illustrating deviations from rational models.

• On the calculus of relations:

  They examined transitivity in relational calculus, foundational to many logical systems.

• The structure of values and norms:

  They provided insights into the implications of violating transitivity in preferences, showing that such violations can lead to logical inconsistencies within decision theory.

• The sorites paradox:

  They proved that apparent violations of transitivity can give rise to logical paradoxes

These works collectively establish the theoretical and practical basis for our approach to defining logical consistency. The detailed citation can be found in the paper.

Concern 2: Definition of Internal Understanding

[1, 2, 3] are datasets for the natural language tasks. After rephrasing the format as question-answering and limiting it to multiple-choice questions, we have a universal task that covers various tasks. This doesn't imply that "it is a common and effective practice for assessing LLMs’ internal understanding." Indeed, none of the three papers claimed anything about the internal understanding of LLMs.

We respectfully disagree with this assessment. The MMLU, ARC, and StrategyQA datasets are widely recognized as benchmarks for evaluating a language model’s internal knowledge and understanding. These datasets require language models to generate answers based solely on their own knowledge and reasoning, without the aid of external information. This evaluation inherently reflects the model’s internal understanding. We would greatly appreciate it if the reviewer could clarify their interpretation of "internal understanding" and provide any specific definitions or references that support their assessment. This would help us better address the concerns raised and refine our arguments accordingly.

To clarify, what we are arguing in previous rebuttal is that the format of multiple-choice tasks effectively captures a model’s internal understanding. Therefore, our setup, which involves LLMs making binary judgments, is a natural adaptation for assessing the consistency of their understanding. We believe this argument is well-supported by both the structure of these datasets and their widespread use in evaluating LLMs.

Additional Experiments

Question: Is this improvement still observed by improving the consistency of a fixed model? Since Table 4 shows five different models, it would be clear if we see that the improved model after fine-tuning in Section 4 indeed shows the improved performance in the Table 4 setting.

We thank the reviewer for suggesting this experimental setup, which indeed provides a fair comparison. However, with the current Summary with Feedback dataset, this specific experiment is not directly feasible. The dataset contains sparse pairwise annotations, and there is no human-provided ranking against which the estimated rankings from PairS can be compared.

To address the reviewer’s concern, we conducted additional instruction-tuning experiments using a different document reranking dataset, MS MARCO (details provided in Appendix B). This dataset allows for a more robust evaluation while maintaining consistency with the experimental setup of Table 3 in the manuscript. We also perturbed 10% of the pairwise comparison labels to test the robustness of the models.

Below, we first present the consistency performance results (equivalent to Table 3) for the fine-tuned model...

The results reaffirm the effectiveness of our proposed data augmentation framework, where the consistency is improved and the alignment to human preference is maintained. Then we show that the Spearman correlations of the predicted rankings by PairS algorithm.

The results also agree with our findings in Section 5, where models with better transitivity can produce overall more accurate rankings, and models with better commutativity rely less on calibration to achieve optimal performance.

Definition of Logical Consistency

In general, logic consistency refers to the set of statements that don't contradict each other.

We appreciate the reviewer’s definition that “logical consistency refers to the set of statements that don't contradict each other,” and we fully agree with this interpretation. Within this broad framework, we argue that transitivity, commutativity, and negation are indeed encompassed as logical consistency rules.

• For example, statements such as “A is better than B” and “B is better than A” clearly contradict each other when evaluated under the same prompt template and criteria.

Similarly, contradictions arising from failures in transitivity and negation demonstrate their relevance to logical consistency.

When you say logic consistency, it is restricted to the consistency over the knowledge base where it has axioms for transitivity, negation, etc as shown in the example paper that introduces the preference to the logic. Those three rules can be introduced to capture the preference, but it wouldn't be fair to say that they are fundamental aspects of logic consistency as we could design any knowledge base that is equipped with different rules.

When the axioms applied to a knowledge base or a set of items are explicitly designed to prevent logical contradictions, we believe it is reasonable to describe this as logical consistency. While logical consistency is indeed a broad term, our work focuses on specific aspects of it, with a well-defined scope centered on transitivity, commutativity, and negation.

If the reviewer finds our use of the term “fundamental aspects of logical consistency” too strong or potentially misleading, we are happy to revise it to a softer phrasing such as “particular/arithmetic aspects of logical consistency.” This adjustment would ensure a clear emphasis that our focus is on certain components of logical consistency, rather than its entirety.

As mentioned earlier, when this paper mentions "logic consistency," it can be replaced by "consistency with respect to the partial order of preference statements", which has a very specific scope. I am not saying that being specific on the scope will hurt the value of the work.

We also acknowledge the reviewer’s point regarding the specific scope of “consistency with respect to the partial order of preference statements” when the concepts of transitivity, negation and commutativity are extended to a set of items. However, we would like to highlight that partial order preference consistency is also widely recognized as a type of logical consistency, particularly in fields such as decision theory, preference modeling, and social choice theory. Our terminology aligns with the understanding established in prior works (listed in previous rebuttal), and we have made our scope explicitly clear from the outset of the paper.

Definition of Logical operator

I think there's a conceptual misunderstanding. When we say a symbol '>' in logic, you could define a predicate such as GreaterThan(X, Y) defined over two objects X and Y, and give a semantic that the predicate serves as a comparator operator. It doesn't sound good to hear that it is a logical operator or logical comparator. Although the field of NLP evolves rapidly, such a basic definition wouldn't change.

We agree there may be a conceptual misunderstanding here. The reviewer seems to be interpreting the symbol ">" in the context of narrow-sense formal or predicate logic, while our usage refers more broadly to arithmetic or quantitative logic.

• The arithmetic logic deals with numerical values and their relationships, typically involving mathematical operations like addition, subtraction, multiplication, and comparison (e.g., greater than, less than). It is a specific subset of logic that deals with quantities and the logical relationships between them.

If this is the root of the confusion, we are happy to clarify and specify the term as "arithmetic logical operator" to avoid ambiguity, and also explicitly mention this in the beginning of the paper.

This is about Section 5.1, which discusses the impact of consistency on the PairS applications.

Table 4 shows the results over five different models and states that "improving the metric" implies "improved performance." When we see the table with five models, there's no clear or strong trend. Phi-3-med is the best in both transitivity and PairS. Mistral is the second best in transitivity but the worst in PairS. Phi-mini is better than Llama-3 and GPT-3.5 in transitivity, but it is worse than Llama-3 in PairS.

One unverified statement is that "fine-tuning" the data augmentation scheme proposed in the paper will improve the performance of the downstream task. It would support the statement in Section 5.1 if that could be shown.

We respectfully request that the reviewer revisit our rebuttal, as we have indeed addressed this point. Specifically, we demonstrated that the model fine-tuned on augmented data outperforms the model fine-tuned on un-augmented perturbed data in the context of the PairS algorithm.

We hope this clarification addresses the reviewer’s concerns, and we remain open to further refining our terminology or phrasing to ensure clarity and accuracy.

W2 & Q1: Lack of analysis on when win-loss rate rank estimation is effective and when it may fail.

We appreciate the reviewer’s observation regarding~ the need for a detailed analysis of when win-loss rate rank estimation is effective and when it may fall short. Below, we provide a comparative analysis of the strengths and weaknesses of different rank estimation methods:

• Win-loss rate does not incorporate transitive information, meaning comparisons against weaker or stronger candidates are treated with equal weight. While this might seem like a limitation, it offers advantages in specific cases. We show the pros and cons in belows:

  • Pros: In the example with pairwise comparisons [A>B, A>C], there is insufficient information to infer the relationship between B and C. The win-loss rate can successfully predict A>B=C. We will show later how ELO and B-T fail to do this.
  • Cons: In the example of [A>B, B>C, C>D], there should be enough information to derive that A>B>C>D. However, the win-loss rate will produce A>B=C>D, as it treats all comparisons with the same importance, regardless of the broader context.

• ELO rating effectively incorporates the transitive information, however, it is highly sensitive to the order of comparisons. When comparison is sparse this will significantly affect the estimated ranking. For example:

  • In the case [A>B, A>C], ELO will predict A>C>B. However, by reversing the comparison order to [A>C, A>B], ELO will predict A>B>C. Since our dataset is static, the order of comparison contains no inherent information. Being sensitive to order will potentially introduce noise to the augmented preference data.

• Bradley-Terry model produces maximum likelihood estimations for rankings, which offers a theoretical advantage. It effectively handles transitive relationships and provides more nuanced rankings. However, being a parametric model requiring optimization, the B-T model struggles to output tied rankings (cannot predict the same score for two items). In other words, it will always produce complete rankings instead of partial rankings, unless some special design on the tolerance. For example:

  • In the case [A>B, A>C], the B-T model will produce A>B~C, where B and C have slightly different scores, even if they should theoretically tie. This lack of tolerance for ties will potentially introduce noise into our augmented data.

In general, the effectiveness of any rank estimation method depends on several factors, including the sparsity of pairwise comparison annotations and the available computational resources. For instance:

• Sparse comparisons: Methods that heavily rely on transitive relationships, like Elo and B-T, might struggle when comparison data is limited.
• Time constraints: Simpler methods like win-loss rate are computationally efficient and may be preferable when time is a limiting factor.

Overall, while methods such as Elo and the B-T model have specific advantages in capturing transitive relationships or producing likelihood-based rankings, their sensitivity to noise and computational complexity make them less practical under certain conditions. By contrast, the win-loss rate provides a straightforward and robust baseline, particularly when tied rankings are acceptable or when computational simplicity is prioritized.

We sincerely thank the reviewer for their time and effort in reviewing our paper. We greatly appreciate their insightful suggestions and believe these will significantly enhance the quality of our work. Below, we address and clarify the concerns raised:

W1 & Q4: Compare the win-loss rate rank estimation with other ranking methods.

We are grateful to the reviewer for highlighting this important point. As mentioned in the first paragraph of Section 4.1, we acknowledge that alternative rank estimation methods might yield better performance. Our choice of the win-loss rate is intended as a proof-of-concept, and we do not claim it achieves the best possible performance.

To address the reviewer’s concerns more thoroughly, we conducted additional experiments comparing win-loss rate with other ranking methods. While the Borda count is a commonly used ranking method, it is not directly applicable to our setting. Borda is typically designed for scenarios involving complete or partial ranking votes, whereas our dataset contains only pairwise annotations. In this context, the Borda count effectively reduces to win-loss count. However, win-loss count does not normalize for the number of comparisons per item, leading to potential unfairness when preference annotations are imbalanced (e.g., some items are compared more frequently than others). This motivated our choice of the win-loss rate, which accounts for this imbalance.

For a more comprehensive comparison, we also evaluated two widely-used ranking methods: the ELO rating system and the Bradley-Terry (B-T) model. Below, we present results from experiments using these methods to refine the data, with all other experimental setups remaining consistent with Table 3 in the manuscript.

Our findings indicate that both ELO and the B-T model achieve slightly better performance than the win-loss rate, with improvements observed in both human accuracy and consistency metrics. However, the performance margins remain relatively small, further supporting our decision to use win-loss rate as a proof-of-concept in the initial study.

Q2: Could you offer more theoretical analysis on why the proposed win-loss rate preference ranking improves LLM consistency?

As discussed in Lines 413–416 and Lines 422–423, the primary motivation behind our data augmentation and refinement framework is to generate consistent, conflict-free preference data. Our hypothesis is that models trained on data with explicit consistency are more likely to exhibit consistent behavior themselves, as they are less exposed to conflicting signals during training. The experimental results validate this hypothesis by demonstrating measurable improvements in logical consistency.

We would also like to emphasize that utilizing data augmentation techniques to enhance specific abilities of an LLM is a well-recognized and intuitive approach in the field. By systematically refining training data to align with the intended task objectives, our method ensures the LLM receives clearer signals for learning consistent preferences, thus improving performance in this dimension.

Q3: "Is it possible to have LLMs analyze why each step or change in the proposed technique works, to ensure they operate in the intended way?”

We are unclear about the specific intent of this question and would appreciate further clarification from the reviewer. Specifically, it is not clear what is meant by “the proposed technique” and why there is a need for the LLM itself to “analyze” the framework. If the question refers to the data augmentation framework, we have already provided a detailed explanation of its workflow and motivations in Lines 370–416 and Figure 6. If the reviewer is seeking additional insights beyond what we have presented, we would be happy to address those concerns with more specifics once clarified.

General response:

We recognize that the reviewer’s primary concerns are centered around the win-loss rate rank estimation. While we acknowledge that more sophisticated ranking methods could potentially lead to improvements, we have conducted additional experiments to compare alternative methods and provided those results to address this point.

However, we would like to stress that the contributions of this work extend beyond the choice of ranking method:

1. Development of Logical Consistency Metrics: We introduced three logical consistency metrics, which provide a novel perspective for evaluating LLMs.
2. Comprehensive Evaluation: We assessed the performance of a broad range of LLMs across different setups, offering valuable insights into their behavior.
3. Proposed Data Augmentation and Refinement Framework: We demonstrated that this framework effectively enhances logical consistency when aligning LLM preferences, a contribution validated by our experiments.
4. Demonstrating the Importance and impact of Logical Consistency: We showed that logical consistency is a critical factor, in addition to alignment with human preferences, for the success of logic-dependent, high-level algorithms.

Once again, we thank the reviewer for their time and detailed feedback. We will revise our manuscript based on the above discussions accordingly. We hope that, in light of our overall contributions and the responses provided, the reviewer might consider reevaluating the score.

W2: “It would be more solid to sample the same amount of perturbed data, augmented data, and augmented & negated data to make the comparison to exclude the data size as a potential factor and discuss the implications of such an experiment.”

We sincerely thank the reviewer for this thoughtful suggestion. We conducted the suggested experiments and provided the results below. Before presenting the findings, we would like to address and clarify potential misunderstandings.

Clarifications:

• In Section 4, we propose a data augmentation and refinement framework to address the challenges of obtaining high-quality, consistent data in the era of LLMs. As acquiring such data through synthesis methods or human annotations is increasingly difficult and resource-intensive, data augmentation is a more practical and cost-effective alternative. Our augmentation framework, which relies solely on the perturbed data, exploits its inherent information without requiring external knowledge.

• Although the datasets differ in size, the comparisons made in Table 3 remain valid. These comparisons effectively demonstrate the strength of our approach in utilizing and expanding the information within the perturbed data. Nevertheless, to address the reviewer’s concerns and further evaluate the implications of data size, we conducted experiments following the suggested methodology: randomly sampling the same amount of data as the perturbed dataset, while keeping all other experimental settings consistent with those in Table 3.

Experiment Results and Observations:

The results of this experiment are presented below. We would like to emphasize the following key points:

1. Information Expansion and Loss:

   • The augmented dataset expands the information contained in the perturbed dataset. Randomly sampling from the augmented data inherently leads to some loss of information compared to the original perturb set.

   • Without employing a sophisticated sampling strategy to minimize this information loss, the comparison may not be entirely equitable.

2. Impact on Performance:

   • Due to the information loss, alignment with human preferences shows a slight decrease. However, the drop in performance is relatively small.

   • Despite the reduced data size, all three consistency metrics still show good improvements compared to the perturbed dataset. This demonstrates the robustness of our augmentation framework, though the performance is naturally lower than models trained on the full augmented dataset.

These findings re-affirm the efficacy of our proposed framework and provide further insights into its behavior under constrained data conditions. We appreciate the opportunity to conduct this analysis and will include these results in the revised version of our paper for added clarity and comprehensiveness.

Q1: “Are transitivity, commutativity, and negation invariance relative to each other, or they are just equally important? Can we rank two LLM's consistency by just one (aggregated) aspect or we cannot easily compare two LLMs, say Phi-3-medium and Gemma-2-9B on SummEval, as their rank for transitivity and commutativity are different?”

We appreciate the reviewer’s insightful question. To clarify, we do not consider the three consistency metrics to be equally important. Their relative importance depends heavily on the specific application scenario:

• Commutativity becomes more critical when addressing positional bias in evaluations.
• Transitivity is essential in applications where calibration is feasible, and accurate ranking of multiple items is a priority.
• Negation Invariance is crucial in contexts requiring consistency between forward and reverse relation expressions, such as “better” versus “worse.”

We do not recommend aggregating the three metrics into a single value, as was intentionally avoided in our paper. Aggregation compromises their interpretability as distinct consistency properties and offers only marginal benefits. By keeping the metrics separate, practitioners can better tailor evaluations to the specific needs of their applications.

We sincerely thank the reviewer for their time and thoughtful feedback on our paper. We deeply appreciate the reviewer’s recognition of our work and the valuable insights they have provided. We are confident that these suggestions will enhance the quality and clarity of our paper. Below, we address each concern raised in detail. We hope that, in light of the following explanations and supplementary materials, the reviewer may consider revisiting their assessment.

W1: “Reason to select transitivity measurements”.

We acknowledge that transitivity in Social Network Analysis (SNA) is a well-studied concept. However, we would like to emphasize that the definition and motivation for transitivity in our framework differ significantly from those in SNA. Below, we outline the key distinctions:

• Motivations:

  • SNA Transitivity: The SNA transitivity is used to reflect the degree to which nodes in a graph tend to cluster together. It is motivated by observations in real-world social networks, where there is a tendency for "friends of friends" to also be friends, leading to the formation of tightly knit groups.

  • Our Logical Transitivity: In contrast, our logical consistency framework uses transitivity to evaluate the degree of self-conflict in relation graphs generated by models or human judgments. Here, transitivity is designed as a measure of logical coherence rather than clustering.

• Definition:

  • SNA Transitivity: In graph theory, transitivity quantifies the extent to which edges form closed triangles (fully connected triplets). It measures the likelihood that adjacent nodes of a given node are also connected to each other.

  • Our Logical Transitivity: Our framework quantifies the extent to which relation graphs are acyclic. Specifically, it measures the probability that a randomly sampled set of K nodes forms acyclic relations.

• Applicability and Scenarios: The differences in motivations and definitions naturally lead to distinct application contexts and methodologies:

  1. Graph Type: SNA transitivity is primarily applied to undirected social network graphs, while our logical transitivity operates exclusively on directed relation graphs.

  2. Scope: SNA transitivity focuses on the closure of triplets, whereas our logical transitivity evaluates logical circulations spanning 3 to K nodes, with K being the size of the sampled sub-graph. Extending SNA measures to higher-order relationships is nontrivial and computationally inefficient.

To illustrate these differences, we provide a detailed analysis of popular SNA transitivity measures:

1. Clustering Coefficient [1]: Global clustering coefficient measures

   $C = \frac{\text{Number of Closed Triplets}}{\text{Number of All Triplets (closed and open)}}$

   and Local clustering coefficient measures

   $C_i = \frac{\text{Number of closed triplets including node i}}{\text{All triplets including node i}}$.

   While applicable to directed graphs, clustering coefficient still measures the ratio between closed triplets and total triplets instead of circular relations and cannot be applied to larger structures.

2. Triad Census [2]: Counts the relative frequency of 16 possible triadic configurations in directed graphs. However, this method is inherently constrained to triads.

3. E-I (External-Internal) Index [3]: Measures the ratio of external to internal ties within a given partition, capturing group closure rather than logical acyclicity.

Novelty of Our Approach

We would like to respectfully clarify that our transitivity measure was developed independently and is not derived from existing SNA methodologies. Instead, it was specifically designed to align with the goals of logical consistency. As we have shown, none of the existing SNA transitivity measures can be straightforwardly extended to measure the logical transitivity of an arbitrary number of nodes.

We sincerely appreciate the reviewer’s insightful suggestion, as it provides a valuable opportunity to further elucidate this distinction. To enhance the clarity of our contribution, we will include a detailed discussion of these differences in the supplementary material.

References

[1] Stanley Wasserman, Katherine Faust, 1994. Social Network Analysis: Methods and Applications. Cambridge: Cambridge University Press.

[2] Batagelj, V., & Mrvar, A. (2001). A subquadratic triad census algorithm for large sparse networks with small maximum degree. Social networks, 23(3), 237-243.

[3] Informal networks and organizational crises: An experimental simulation. David Krackhardt, Robert N. Stern - Social Psychology Quarterly, 1988, DOI:10.2307/2786835

Q2: “How to choose K in a real application?”

We thank the reviewer for this excellent question and would like to emphasize the following considerations regarding the choice of $K$.

1. Robustness of Transitivity to $K$:

   The transitivity metric $S_{tran}(K)$ is robust to variations in $K$. It is uncommon for a model to perform well on $S_{tran}(3)$ but poorly on $S_{tran}(5)$. To illustrate this, we have added a new figure (Fig. 9) in Appendix G. Additionally, the figure also shows that larger values of $K$ expand the value range of $S_{tran}(K)$, making comparisons between models more distinct. For example, when $K < 5$, the $S_{tran}(K)$ of Gemma-2-9B and Phi-3-medium are very close to each other, which starts to diverge more evidently when $K >8$.

2. Dependence on Prior Knowledge:

   Choosing K can depend on prior knowledge about the general consistency performance of current LLMs. For example, we observe that the consistency performance improves progressively from earlier LLMs, such as LLama-2 and Zephyr-7B-beta, to more advanced models like Gemma-2 and Llama-3. This trend suggests that considering the evaluated LLMs' expected consistency levels can inform an appropriate choice of $K$.

3. Task-Specific Considerations:

   As discussed in Lines 301–303, consistency performance varies depending on the task. For objective tasks, consistency tends to be higher, which means task-specific nuances should also guide the selection of $K$.

In summary, there is no definitive “gold standard” for choosing $K$. Similar to selecting the $N$-gram size in BLEU-$N$, the choice depends on the models and tasks under evaluation. We recognize the value of this discussion and have included these considerations in the Appendix G.

Minor: “How to extract LLM's preference accurately using the prompt listed in Appendix F?”

We would like to thank the reviewer for observing this. We do have a format control instruction (“Please only answer with A or B.”), which is now updated in Fig. 7. To extract the preference, we start from the beginning of the response and try to match with char ‘A’ or ‘B’. We do notice they are tokenized differently in different LMs’ tokenizer, and we carefully specify the corresponding token for each tokenizer.

We are aware that tokenization for these characters can vary across different LLM tokenizers. To address this, we carefully specify the corresponding token for each tokenizer to ensure accurate extraction. This adjustment improves the consistency and reliability of preference extraction.

Once again, we thank the reviewer for their time and detailed feedback. We hope these clarifications address the reviewer’s concerns and welcome any additional feedback to further improve our work. Additionally, we kindly ask the reviewer to consider increasing their score in light of the explanations provided.

Thank you, authors, for the detailed response. You have addressed all of my questions thoroughly, and I’m satisfied with the results. The additional experiments and explanations are commendable and make me lean toward a score of 7. However, since ICLR only allows scores of 6 or 8, I will maintain my score at 6 for now.

We sincerely thank the reviewer for their time and thoughtful feedback on our paper. We deeply value their insights and believe these suggestions will significantly enhance the quality and clarity of our work. Below, we address each concern raised:

W1: “In section 3.1 of the article, there is no explanation on how to use the three datasets to probe the logical consistency of LLMs. How the SummEval dataset is used to measure the negation invariance of LLMs?”

We appreciate the reviewer highlighting this point. The evaluation of the three datasets is conducted using the consistency quantification methods outlined in Section 2. To clarify, we have added a detailed explanation in Appendix F (highlighted with red text), which explicitly describes how the consistency metrics are applied to each dataset, including measuring the LLMs’ negation invariance performance on the SummEval. We hope these revisions address the reviewer’s concerns.

W2 (a): “In section 4, the paper proposes a data refinement and augmentation method for improving the logical consistency of LLMs, which can enhance logical consistency without compromising alignment with human preferences. Human preferences are highly correlated with commutativity, and the paper does not mention that human preferences and logical consistency are contradictory relationships. In fact, human preferences should be consistent with logical consistency. Why does the paper emphasize that alignment with human preferences is not compromised?”

We respectfully request further clarification regarding this concern, as there might be potential misunderstanding of our paper. Our paper does not claim or suggest that “human preferences and logical consistency are contradictory relationships.” In fact, we do not believe this statement is well-supported. Additionally, the reviewer’s assertion that “human preferences should be consistent with logical consistency” appears self-contradictory and is also not stated in our paper. We would appreciate further elaboration on the specific questions raised here.

W2 (b): “Why does the paper emphasize that alignment with human preferences is not compromised?”

This statement, found in lines 530-533 [1], specifically refers to the experimental results in Section 4, as presented in Table 3. We emphasize that “human preferences are not compromised” because the results demonstrate that our data augmentation framework improves logical consistency while maintaining the same level of accuracy in human preference alignment compared to models trained on perturbed data. We believe this comparison is both fair and well-supported by the data presented.

W3 (a): the paper is only exploring and improving the logical consistency of LLMs in a new application scenario.

We respectfully disagree with this observation. Our evaluation framework is not limited to a single application scenario but is broadly applicable to any task where logical relations are relevant.

• For instance, it can be utilized in scenarios involving preference rankings of response candidates, relevance judgments over retrieved documents, temporal or causal relationships among events, hierarchical relationships among entities, or entailment relationships between statements, as discussed in Lines 111–116. This generality is one of the key advantages of our framework, compared with previous works.

W3 (b): Concerns about first order relations. “it seems that the paper is only exploring and improving the logical consistency of LLMs in a new application scenario, without solving the problem of "first-order relations."

We would like to clarify our use of the term “first-order relations.” In our paper, this refers to the logical relationships between two or three statements, such as the transitivity of a triplet. Unlike previous works, which are mostly constrained to first-order logical relationships, our framework addresses the logical consistency across an arbitrary number of statements or items. This capability is novel and represents a significant step forward.

• To avoid potential ambiguity, we have revised the corresponding paragraph (Line 517) to explicitly define “first-order relations.”

Reference

• [1] Line 530-533: “To improve the logical consistency of LLMs, we then proposed a data refinement and augmentation framework, which effectively reduces logical inconsistencies at the source: directly in the data. Our experimental results showed that models trained on this refined and augmented data achieve improved logical consistency without compromising alignment with human preferences.”

Q1: In section 3.3, the article calculates the correlation between Transitivity consistency and self-agreement, and finds that they are highly correlated, and therefore concludes that “Transitivity serves as a useful proxy for evaluating the global reliability of LLMs. Is this conclusion reliable? Can Self-Agreement provide a comprehensive measure of model reliability?

We agree with the reviewer’s point that our initial conclusion may overstate the relationship between self-agreement and global reliability. Upon reflection, we have revised the term “global reliability” to “local robustness” to more accurately represent our findings (Line 322). We believe this adjustment better aligns the conclusion with the scope of our experimental evidence.

Q2: “Why are the experimental parts of section 4 only conducted on SummEval?”

We would like to clarify a misunderstanding here. In Section 4, all experiments are conducted on the Summary with Feedback dataset (as shown in Table 3 and Section 4.2), not SummEval. We chose this dataset because of its large size and the breadth of topics it covers, which we believe provide sufficient diversity for evaluating our methods.

However, to further address the reviewer’s concerns, we have conducted additional experiments using a different dataset, MS-MARCO (details explained in Appendix B), with all other experimental setups remaining consistent with Table 3 in the manuscript. (Also perturb 10% of the pairwise comparison labels.)

The results from these experiments confirm the robustness of our findings with the results on summary with feedback, where our data augmentation framework can successfully improve the logical consistency while not loosing alignment with human preference. This experiment reinforces the generalizability of our framework.

Once again, we thank the reviewer for their time and detailed feedback. We hope these clarifications address the reviewer’s concerns and welcome any additional feedback to further improve our work. We will further revise our manuscript according to the discussion. Additionally, we kindly ask the reviewer to consider revising their score in light of the explanations provided.

As the discussion period nears its conclusion, we would like to kindly remind you to review our rebuttal and share your thoughts. We would like to confirm whether our responses adequately address your concerns. If there are remaining issues with the paper, we are open to further clarifications. Otherwise, we respectfully request the reviewer to consider revisiting the score in light of our responses.

To further strengthen the paper, we have conducted additional experiments, which we believe may also be of interest:

1. Beyond the above experiments on MS-MARCO, we demonstrate that models trained with augmented MS-MARCO data achieve improved performance using the PairS algorithm as well (aligned with Table 4 in the paper).

2. We have also explored alternative ranking estimation methods, including the ELO rating system and the Bradley-Terry model, comparing their performance against the Win-loss rate.

These additional results consistently support our conclusions and further validate the effectiveness of our data augmentation framework.

We sincerely appreciate the time and effort you have devoted to reviewing our work and look forward to your feedback.

Best regards,

Authors

Q2: Is there a way to measure the prompt sensitivity.

We agree with the reviewer that prompt sensitivity is a critical dimension of LLM robustness. While our primary focus in this work is more on logical consistency, we acknowledge that prompt sensitivity can intersect with this topic and is an important consideration. Although we did not explicitly measure or analyze prompt sensitivity as part of our study, Figure 5 indirectly addresses this issue by showing performance variations across different prompt formulations.

We note that several existing works have specifically explored methods for quantifying prompt sensitivity, which may offer complementary insights into this area. In future work, incorporating a more explicit investigation of prompt sensitivity could provide a valuable extension to our current analysis.

Once again, we sincerely thank the reviewer for their thoughtful feedback. Your suggestions have provided us with valuable perspectives to enhance the robustness and clarity of our work.

We sincerely thank the reviewer for their time and effort in evaluating our work. We greatly appreciate the positive recognition of our contributions, as well as the insightful feedback provided. We believe these suggestions will significantly enhance the quality and clarity of our paper. Below, we address each concern in detail:

W1: “In the introduction, line 048, the author(s) says that consistency of its predictions is the foundation of a reliable and trustworthy system. While I certainly agree that this is one of the foundations, I don't think consistency alone is the foundation. What about a very consistently wrong system? One that consistently predicts falsehood? Certainly it is internally consistent, but it fails to accurately describe the world. I would say that calibration (or truth tracking, or something along those lines) is equally foundational to reliability and trustworthiness. If the author(s) agrees, I suggest to change this accordingly. What the authors mention on line 087 seems relevant for this, too.”

We fully agree with the reviewer’s observation that both consistency and accuracy are critical for the reliability and trustworthiness of a system. As noted in the footnote on line 053, we state that “In the fields of behavioral economics and psychology, systems are traditionally evaluated on two key dimensions: validity and reliability.” We appreciate the reviewer highlighting this point and will revise the statement in line 048 to more accurately reflect our intended meaning by emphasizing the equal importance of both consistency and accuracy as foundational dimensions.

W2: On Page 3, line 150, the autor(s) mentions that if F is transitive, the corresponding relation graph is a DAG. I think this is not entirely accurate: it is a DAG when F is transitive and irreflexive. The author(s) seems to tacitly assume that F is irreflexive, but they should at least mention this somewhere. Also, why do they assume that this cannot be a way to measure internal consistency?

We thank the reviewer for this insightful observation. You are correct that our statement assumes F is irreflexive, although this was not explicitly stated in the paper. For instance, in Figure 3 and Equations 2 and 3, we omit the diagonal elements of the pairwise comparison matrices, and we compute the total pairwise comparisons as $|X| \cdot (|X|-1)$ instead of $|X|^2$. We will revise the manuscript to explicitly clarify this assumption and provide the rationale behind it.

Regarding the suggestion that reflexive consistency could serve as a measure of internal logical consistency, we agree in principle. However, in practical scenarios—such as evaluating causal or temporal orders, or comparing preferences—it is uncommon to compare an item against itself when working with large language models (LLMs). As a result, our evaluation framework assumes F is irreflexive. That said, we appreciate this important point and will include a discussion of reflexive consistency, along with our motivations for adopting the irreflexive assumption.

Others: Line 191, line 228, line 304 and Eqs 2 &3.

We appreciate the reviewer’s careful corrections, and we have revised the paper accordingly.

Q1: Concerns the robustness of the wording of the prompting to the results.

We appreciate the reviewer’s insightful observations. It is indeed true that LLMs exhibit varying levels of sensitivity to prompt phrasing. As illustrated in Figure 5, where each dot in the subplots represents a semantically equivalent but differently phrased prompt, variations in prompt wording can influence LLMs’ accuracy relative to human judgments and their commutativity performance. The results presented in Table 1 are based on a single, carefully crafted prompt instruction (shown in Figure 7) that is designed to be broadly interpretable by most LLMs.

Interestingly, we observed that transitivity values are less affected by prompt variations compared to commutativity. To illustrate this, we computed the range of transitivity values using the same 10 prompt variations from the experiment depicted in Figure 5. The findings show that transitivity is relatively more robust to prompt variations, and more irrelevant to human preference accuracy.