Thank you for your thorough evaluation of our work! We have addressed your main concerns below:

W1: Limitation to only evaluating with BERTScore

We agree that our comparisons with other metrics were not framed clearly. In Section 3, we discuss why we do not choose to approach the problem with n-gram inconsistency measures - these are typically not able to capture semantic similarities. At the end of Section 4.1, we (albeit unclearly), mention that we tested a metric based on bi-directional entailment clustering. We found that it was not able to sufficiently parse out any similarities between responses that were otherwise conceptually dissimilar. We maintain that our verification of the validity of our inconsistency metric in Section 4 provides evidence that any sufficiently robust metric will yield very similar results (up to scaling/inversion). Part of the contribution is to verify that BERTScore is in fact trustworthy for measuring inconsistency in a generalized question-answering setting.

W2: No human evaluation or correlation to human judgment

We kindly refer you to the first part of our general response titled "Further analysis and human evaluation for inconsistency score."

W3: No discussion of possible mitigations

We addressed this in the second part of our general response titled "Why we do not mitigate inconsistency."

Q1: Can the evaluation framework highlight other types of discrepancies?

Thank you for this question! We showed that our metric can reliably distinguish between "safe" and "unsafe" responses to mental health emergencies in an additional experiment (we expand on details in the general response). Thus, we hope future work can build on this evaluation framework and study ways to automatically evaluate free-form responses of agentic systems in various settings. For example, a researcher may define categories under which the responses fall under and subsequently use our evaluation framework and well-defined comparisons to categorize responses to expose conceptual discrepancies. Broadly, we hope future work can use this framework to explore inconsistency-mitigating decoding strategies surrounding LM self-consistency, automatically categorize free-form text into conceptual categories, or further study LM behavior in further high-stakes domains (e.g. law, government).

We hope our new experiments and arguments have addressed your concerns! We look forward to hearing your thoughts and are happy to conduct new experiments to address any further concerns. Again, thank you for your time and constructive feedback!

Thank you for your thoughtful review and constructive feedback! We address your concerns below:

W1: Why not temperature T = 0.0?

We kindly refer you to the first part of our general response under the section titled “Why we evaluate LMs at temperature T > 0.” We will do our best to meet your wishes exploring the expansion of section 6.

W2: Limited Generalizability

We kindly refer you to the first part of our general response under the section titled “Regarding Generalizability Concerns.”

W3: More in-depth analysis on inconsistency, coarse-grained measure

We address this concern in the first part of our general response under the section titled "Further analysis and human evaluation for inconsistency score." Additionally, we address why we do not pursue mitigating inconsistency directly in this work in the second part of our general response under the section titled "Why we do not mitigate inconsistency."

W4: Missing Citations

Thank you for pointing out the missing citations! We will add these to the revised submission. The first and third one seem particularly pertinent to our prompt sensitivity experiments!

W5: Mention of bi-directional entailment clustering

Thank you for pointing this out - we agree this is unclear. We mentioned this to demonstrate that we considered other potential metrics beyond BERTScore for our analysis. We did not include these results in the main text because we found that this metric was unable to sufficiently extract any similarities between texts that are otherwise conceptually dissimilar. Thus, we obtained inconsistency scores that were extremely high across all settings and models (~0.9), making comparison and evaluation meaningless. We found that BERTScore more robustly captured inconsistency, and so we focused our analysis using this metric. We will clarify these points in the revised pdf.

We hope our additional experiments and clarifications have addressed your concerns, we would be grateful if the reviewer re-evaluate our work. Again, thank you for your time and constructive feedback! Please let us know if you would like us to run additional experiments or if you have any additional concerns.

We appreciate your feedback and understand your skepticism regarding the use of non-deterministic AI systems (T > 0) in military settings. We want to clarify our position and provide extensive evidence to support our claims, although they may seem unintuitive for standard LM evaluations. Predictability in military decision-making is universally considered to be a significant vulnerability. Adversaries capable of anticipating actions may exploit consistent patterns to undermine strategies. Military doctrines and strategic studies emphasize the importance of unpredictability to maintain a tactical advantage:

• Game Theory and Mixed Strategies: In competitive and adversarial scenarios, game theory advocates for mixed strategies, which involve randomizing choices to prevent opponents from predicting actions [1][2]. This concept is crucial in military applications to avoid being outmaneuvered by adversaries who might exploit predictable decision patterns.

• Military Doctrine Emphasizing Flexibility and Adaptability: Renowned historical military strategists like Sun Tzu and Clausewitz have underscored the importance of adaptability and unpredictability in warfare to outsmart opponents [3][5]. Modern military doctrines continue this emphasis: The U.S. Army's Operational Art Primer highlights the need for commanders to employ creativity and adaptability, integrating ends, ways, and means across the levels of war [4]. Deception and unpredictability are considered essential for achieving strategic surprise and maintaining operational security [6].

Given these principles, deploying deterministic AI systems with T = 0 could introduce risks due to their predictable outputs in case of cybersecurity failures. In cybersecurity threats or espionage scenarios, adversaries could exploit this predictability to anticipate and counteract military strategies. While our paper focuses on the inconsistency of LLM outputs, understanding the trade-off between consistency and diversity is crucial in high-stakes settings. Setting the temperature T > 0 introduces controlled randomness, aligning with the strategic need for unpredictability.

Nevertheless, we also back up our findings with an extensive range of variations at T = 0. As discussed in our general rebuttal, both types of experiments (T>0 and T =0) are crucial to meaningfully comparing free-form response inconsistency of LM decision-making. Would you prefer if we re-order the experiments in the paper to start with section 6 and then do section 5?

Our research highlights the importance of addressing this inconsistency to ensure reliable yet unpredictable decision-making support. By examining how LLMs behave under different temperature settings, we provide crucial insights into limitations to safe deployments.

We will revise our paper to include these citations and clarify our argument regarding the strategic necessity of employing LLMs with T > 0 in such high-stakes settings. Thank you for your valuable feedback. We believe this clarification strengthens our paper and addresses your concerns.

References: [1] Osborne, M. J., & Rubinstein, A. (1994). A Course in Game Theory. MIT Press.

[2] Myerson, R. B. (1991). Game Theory: Analysis of Conflict. Harvard University Press.

[3] Sun Tzu. (5th Century BCE). The Art of War.

[4] Sweeny, P. (2010). Operational Art Primer. U.S. Department of the Army.

[5] Howard, M., & Paret, P. (1976). Clausewitz: On War. Princeton University Press.

[6] Barlow, R. E. (2006). "Deception and Unpredictability in Military Operations". Naval War College Review, 59(1), 43–53.

Hello! Thank you again for your thoughtful engagement. We understand how you view our claims as contradictory. However, I am afraid there may have been a misunderstanding.

For maximum clarity in this response, we will say external unpredictability when referring to being unpredictable in the eyes of the adversary. We will say internal unpredictability when referring to being unpredictable in the eyes of the institution deploying the LM for use in their own operations. Our arguments in the rebuttal focused on external unpredictability. What we established in our previous rebuttal is that militaries are incentivized to avoid setting T = 0 to avoid being externally predictable as a result of deterministic responses.

What we focus on in our paper is the notion of internal unpredictability (and likely why you see a contradiction in our framing). For example, in (058), we are saying LMs in high-stakes settings should be required to be internally reliable. In (068), we are saying that delegating trust to an inconsistent agent can lead to being internally unpredictable. This is a cause for concern for many reasons [e.g., 1, 2, 3]. So, when we say “ensure reliable yet unpredictable decision-making support” we refer to internal reliability while being externally unpredictable. What we ultimately show in our paper is that LMs are highly inconsistent at T > 0, which, while making them externally unpredictable, also makes them internally unpredictable and unreliable.

So, what we ultimately argue is that if militaries are to deploy LMs, they must find a balance between the desire for external unpredictability and internal predictability/reliability. What we show in the paper is that, in the context of LM automated or augmented decision-making, these notions are inherently at odds. This tension is likely unsolvable (or at least extremely difficult to solve), calling into question the very deployment of LMs into military operations. To reiterate, nowhere in our paper do we make claims that LMs should be deployed into militaries while being simultaneously unpredictable and predictable. What we do show is that LMs, under the premises in which they are likely to be deployed, exhibit behavior that calls into question their internal reliability. This behavior is a cause for concern and can even lead to catastrophic consequences [4, 5, 6]. In fact, the final sentence of our paper (539) says that robust safeguards should be put in place to prevent unintended outcomes that may arise due to the deployment of LMs into military operations.

Therefore, we do not believe there to be a contradiction in our framework. We hope this clears up the premise and motivation of the paper. It would be regrettable for this work to be rejected under a conceptual misunderstanding.

In acknowledgement of your valid questions regarding our conceptual problem definition, we will add an entire section in the appendix of the revised pdf clarifying this our arguments in this discussion to motivate our experimental setup. Additionally, as Reviewer xYfm suggested, we will move section 6 (prompt sensitivity experiments at T = 0) above section 5 (evaluation at T = 1) to better frame the point of the paper.

We are grateful to have engaged in this discussion with you, and hope that our arguments warrant a score increase in your eyes.

[1] William N Caballero and Phillip R Jenkins. On Large Language Models in National Security Applications. arXiv preprint arXiv:2407.03453, 2024.
[2] Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. Escalation risks from language models in military and diplomatic decision-making. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 836–898, 2024.
[3] Lamparth, M., Corso, A., Ganz, J., Mastro, O. S., Schneider, J., & Trinkunas, H. (2024). Human vs. Machine: Behavioral Differences between Expert Humans and Language Models in Wargame Simulations. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1), 807-817.
[4] National Security Archive. False Warnings of Soviet Missile Attacks Put U.S. Forces on Alert in 1979-1980, 2020. URL https://nsarchive.gwu.edu/briefing-book/nuclearvault/2020-03-16/false-warnings-soviet-missile-attacks-during1979-80-led-alert-actions-us-strategic-forces.\ [5] Geoffrey Forden, Pavel Podvig, and Theodore A Postol. False alarm, nuclear danger. IEEE Spectrum, 37(3):31–39, 2000.
[6] EUCOM History Office. This Week in EUCOM History: January 23-29, 1995, 2012. URL https://web.archive.org/web/20160105033448/http://www.eucom.mil/ media-library/article/23042/this-week-in-eucom-history-january23-29-1995.

Thank you for the thoughtful feedback! We address your concerns below:

W1/Q1: Limited generalizability due to constrained evaluation setting

We kindly refer you to part 1 of our general response under the section titled “Regarding generalizability concerns.”

W2: Limited Novelty

We actually considered developing a new metric. However, as we show, BERTScore performs well in detecting inconsistency on decision-making tasks. We believe it is more impactful to the research community to show that an existing metric can generalize into a new task domain rather than develop a novel metric just for the sake of algorithmic novelty. We maintain that novelty comes from the verification that BERTScore is a reliable metric in decision-making/QA settings. Additionally, we are the first to study LM free-form inconsistency in the military domain. Reviewer xYfm recognizes the importance of this work given the societal implications surrounding the adoption of LMs into high-stakes domains.

W3: Lack of comparison with other inconsistency measurements

We agree that our comparisons with other metrics were not framed clearly. In Section 3, we discuss why we do not choose to approach the problem with n-gram inconsistency measures - these are typically not able to capture semantic similarities. At the end of Section 4.1, we (albeit unclearly), mention that we tested a metric based on bi-directional entailment clustering. We found that it was not able to sufficiently parse out any similarities between responses that were otherwise conceptually dissimilar. We maintain that our verification of the validity of our inconsistency metric in Section 4 provides evidence that any sufficiently robust metric will yield very similar results (up to scaling/inversion).

W4/Q2: Choice of temperature 1.0

We kindly refer you to part 1 of our general response under the section titled “Why we evaluate LMs at temperature T > 0.”

W5: Evaluation limited to few models

This is a good point. We'll run our experiments on more models, particularly on those that are open-source.

W6: We do not study inconsistency mitigation

We kindly refer you to part 2 of our general response under the section titled “Why we do not mitigate inconsistency.”

Q1: Why did we choose the wargame scenario?

We choose to focus our analysis on the wargame setting as this is a worrying real-world application of LMs where studying inconsistency will have grounded implications. We maintain the importance of our work in rigorously exposing any pitfalls of LMs given the sensitivity of the domain. As mentioned, we perform further experiments on chatbot responses pertaining to mental healthcare and find that inconsistency and our evaluation framework generalizes to another critical scenario.

Q3: Could this research potentially inform new methods to improve LLM consistency in decision-making contexts?

This is a fantastic question! As mentioned above, much of the contribution of this work is testing a metric that can robustly measure inconsistency, a necessary precondition to mitigating inconsistency. As we did when showing our metric can distinguish between "safe" and "unsafe" responses to mental health emergencies, we hope future work can build on this evaluation framework and study ways to automatically evaluate free-form responses of agentic systems in various settings. We hope future work can use this framework to explore inconsistency-mitigating decoding strategies surrounding LM self-consistency, automatically categorize free-form text into conceptual categories, or further study LM behavior in further high-stakes domains (e.g. law, government).

We hope these new experiments and arguments have addressed your concerns! We look forward to hearing your thoughts and are happy to conduct new experiments to address any further concerns.

Why we do not mitigate inconsistency

We actually considered mitigating inconsistency over the course of the research. However, prior to finding mitigations for inconsistency, it was necessary to either develop or verify a metric that robustly captures inconsistency in general decision-making contexts as, to our knowledge, one does not yet exist for such settings. Thus, we focus our work on finding and verifying such a metric and study how such measurements are affected by temperature and prompt sensitivity. We show that for any method that aims to improve inconsistency, it is necessary to evaluate the performance of the metric at different temperatures and across prompt variations.

While having a metric is a necessary precondition, there are other challenges associated with mitigating free-form inconsistency, particularly in high-stakes applications. In this work, it would not be impactful to, say, fine-tune models to reduce inconsistency because this would require one set of arbitrary strategic preferences over others. These naturally meaningfully differ across individuals, societies, and cultures. Additionally, conducting such fine-tuning would raise severe ethical concerns surrounding the enabling of automated military decision-making. For high-stakes applications in general, our work demonstrates that meaningful evaluations and governance efforts of safety-critical deployment settings must go beyond just capability evaluations and consider inconsistency due to both sampling temperature and prompt sensitivity.

Thank you to all reviewers and chairs for considering our submission for ICLR. The reviews were instrumental in improving our work.

References

[1] Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004–9017. Association for Computational Linguistics, 2023b.
[2] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024.
[3] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the Association for Computational Linguistics, 2018.
[4] Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023b. Universal self-consistency for large language model generation. ArXiv, abs/2311.17311.
[5] Prabhu, Sumanth. "PEDAL: Enhancing Greedy Decoding with Large Language Models using Diverse Exemplars." arXiv preprint arXiv:2408.08869 (2024).
[6] Declan Grabb, Max Lamparth, and Nina Vasan. Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. In First Conference on Language Modeling, 2024.
[7] BERTScore. BERTScore. https://github.com/Tiiiger/bert score, 2020. [Online; accessed 30-September-2024].