Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing

1. Narrow definition of $\boxed{\mathrm{PRIN}}$

We appreciate the reviewer’s suggestion and agree that exploring PRIN beyond multiple-choice tasks is an exciting direction. However, we believe the example provided (e.g., “how many taken” vs. “how many left”) still fundamentally fits within our defined PRIN framework, and it represents a complementary prompt pair where the reverse logically determines the original answer, similar to our setup. Our current work focuses on the cleanest case by separating prompts and candidate answers, but we agree that broader forms, such as compositional or embedded reverse reasoning, are valuable extensions.

---

2. Lack of details/considerations on decoding approaches

We used the default decoding settings for each model — typically sampling-based decoding with their default temperature and repetition penalty — because this reflects the most common usage patterns and provides a more direct and realistic warning to users about how PRIN manifests in everyday applications. While we agree that exploring alternative decoding methods like greedy or beam search (particularly on open-source models) could yield interesting insights, our primary goal here was to assess the inconsistency that users are most likely to encounter. We will clarify these decoding choices in the paper and note this as a valuable direction for follow-up work.

---

3. Limited exploration on paraphrasing prompts

We understand how Table 4 may give the impression that we only tested word-level paraphrasing, but as detailed in Section 4.4, we actually expanded the paraphrasing analysis by generating two additional paraphrased versions using GPT-4, creating a total of five distinct paraphrased prompts, including more substantial, global rephrasings beyond simple word swaps. Our results showed that even under these stronger paraphrasing variations, PRIN remained relatively stable, suggesting that it reflects a deeper systematic issue, not just sensitivity to prompt wording.

---

4. Not examining/comparing to existing compelling approaches

We agree that, compared to reasoning-focused models like o1 and DeepSeek-R1, as well as exploring combinations like CoT + PRIN, would be valuable. However, we faced two practical constraints: (1) o1 requires additional financial costs beyond what we already invested in GPT-4o, and (2) DeepSeek-R1 was not available for approved use in the authors’ country. During our study period. Despite these limitations, we believe our focus on widely accessible general-purpose models establishes a solid foundation, and we see integrating PRIN with advanced reasoning models or combined techniques as promising future work.

1. $\boxed{\mathrm{PRIN}}$ in domains beyond math (e.g., commonsense, factual QA)?

That’s an excellent question. In this paper, we focused exclusively on mathematical domains. Our observations suggest that a model’s ability to handle negation plays a key role in triggering PRIN. Given this, we believe PRIN could plausibly arise in other domains, such as commonsense reasoning or factual QA, where similar linguistic challenges exist. However, due to time constraints, we were unable to conduct direct experiments on datasets from those domains. Investigating whether PRIN manifests more broadly across diverse task domains is a promising direction for future work.

---

2. Other cognitive/linguistic factors related to $\boxed{\mathrm{PRIN}}$?

The concept of PRIN emerged from our observation that model performance systematically diverges depending on the presence or absence of negation in prompts. Our study primarily focused on this negation sensitivity. However, to account for potential confounds, such as unfamiliarity with specific lexical choices, we conducted experiments using multiple reworded versions of the prompts.

As shown in Section 4.2 and Table 5, while LLMs exhibit some sensitivity to prompt formulation, the observed PRIN patterns remain consistent across different rewordings. This suggests that PRIN is not merely an artifact of superficial linguistic variation but reflects deeper issues in how models handle negation. Thus, we did explore one class of linguistic factors, rewording, but have not yet extended our analysis to other cognitive factors. Investigating additional predictors of PRIN beyond negation remains an open and important avenue for future work.

---

3. Is there a capability threshold where the benefits of $\boxed{\mathrm{PRIN}}$ outweigh its drawbacks?

That’s a very insightful point. Based on the results from the three models evaluated in our study, we observed that, unlike the GPT-4 series, models like LLaMA-3 do not consistently benefit from PRIN; in fact, performance can even degrade. While our current experiments are insufficient to pinpoint an exact performance threshold where PRIN shifts from being detrimental to beneficial, our findings suggest that a model’s negation handling capability plays a critical role. With more targeted experiments measuring this capability across a broader range of models, it may be possible to quantify such a threshold. This would offer valuable guidance for both LLM development and evaluation going forward.

---

4. Interaction with few-shot prompting or self-consistency?

While Section 4.5 explores combining PRIN with self-consistency, we acknowledge that deeper interactions with techniques like few-shot prompting remain underexplored. Importantly, few-shot prompting may reduce or mask PRIN by biasing the model toward specific demonstration patterns, potentially hiding its raw inconsistency rather than resolving it.

---

5. Implications of LLM struggles with contradictions for logical reasoning?

We agree that our findings raise important implications: PRIN highlights that even advanced LLMs struggle with maintaining consistent logical complements, suggesting they are currently ill-suited for tasks requiring strict formal logical reasoning without additional safeguards. Applications in fields like law, formal verification, or logic-based reasoning will likely require specialized training, architectural adjustments, or post-hoc correction mechanisms to ensure reliability. We see addressing this gap as a critical avenue for future research.

---

6. Is $\boxed{\mathrm{PRIN}}$ a case of Randomness Inconsistency in discriminative settings, or something distinct?

We argue that PRIN is fundamentally distinct from Randomness Inconsistency. As shown in Section 4.4, models like Llama-3 and Falcon exhibit low randomness inconsistency (i.e., they produce stable outputs across runs) yet still show high PRIN, meaning they consistently fail to align direct and reverse prompts. This suggests PRIN arises not from sampling variability but from deeper issues in how models process negation and complementarity in discriminative reasoning. We see PRIN as capturing a unique failure mode that existing randomness-based metrics cannot explain.

---

7. Do models trained on logical tasks show reduced $\boxed{\mathrm{PRIN}}$ (architecture vs. training)?

While we did not explicitly test specialized models trained solely on logical reasoning tasks, our study includes GPT-4, which, as a top-performing, closed-source model, was very likely pretrained on diverse reasoning and logic tasks. Yet, as our results show, even GPT-4 exhibits notable PRIN, suggesting that PRIN is not just a training deficiency but may reflect deeper limitations in current architectures or learning frameworks. We agree that further testing on specialized logic-constrained models would be valuable and plan to explore this in future work.

Thank you for the response.

I will improve my score to 6 since the authors make some clarifications on some details of the experiments. However my concerns about the implications and generalizability remains. If paper is accepted i suggested more cross-domain evaluations and other prompting methods would strength the papers and benefit future researchers.

1. $**Domain Reliability**$ There is only one concern I have that is a reason to reject which is the validity of the results outside of the math domain. I do think the authors need to add datasets outside of the math domain in order to be able to show the results hold generally for LLMs. There are many multiple choice data outside of the math domain. I would recommend the authors add a few of these to further validate their results. This can be addressed in the rebuttal period.

We fully agree that extending PRIN evaluation beyond the math domain is an exciting and valuable direction. For this initial study, we intentionally focused on math and science domains because they provide clear ground truth for correctness, enabling rigorous PRIN measurement. While we did not include non-math multiple-choice datasets in this version, we believe the core mechanism underlying PRIN (namely, the probabilistic inconsistency between direct and reverse prompts) is model-internal and not domain-specific. Preliminary tests we conducted on non-math QA datasets showed qualitatively similar inconsistencies, although measuring PRIN reliably outside math is more challenging due to the lack of objective correctness labels. We will emphasize this as a key avenue for future work and greatly appreciate the reviewer’s suggestion.

---

2. $**Prompt Reverse is a very confusing name**$ It took me, the reader, a long time to get used to $\boxed{\mathrm{PRIN}}$ as prompt reverse was quite confusing to me. I would recommend something more descriptive since many things could be a reverse prompt. Perhaps "Answer Complement Inconsistency" or something like this.

Thank you for this thoughtful suggestion. We understand that “Prompt-Reverse Inconsistency (PRIN)” may initially sound unfamiliar or confusing. We chose this name specifically to emphasize the inconsistency between the direct prompt (asking for correct answers) and the reverse prompt (asking for incorrect answers), which directly reflects our experimental setup. That said, we appreciate the reviewer’s point that alternatives like “Answer Complement Inconsistency” could offer a more intuitive framing, and we will carefully consider renaming in future revisions or follow-up work to improve clarity.

---

3. $**F1 is not enough for the reader**$ For the research question, we are also interested in looking at precision and recall,l since it tells us much more about the degree of "completeness" of the answer complement/reverse prompt.

We agree that reporting precision and recall alongside F1 could provide a more detailed picture of the completeness and balance between direct and reverse prompt outputs. In this work, we focused on F1 as a summary measure to keep the analysis streamlined, but we appreciate the reviewer’s point and will include detailed precision and recall breakdowns in the next revision or as supplementary material to offer deeper insights into where the complementarity gaps arise.

---

4. $**What about if we just cared about correctness or consistency and not completeness?**$ Related to the above, it would be good to add an analysis that ignored completeness or full overlap or full complement that showed the reader when the "Reverse Prompt/Answer Complement" simply agreed with the Original prompt (intersection >= 1). I would further divide this analysis into when this is correct and incorrect according to the task accuracy metric.

We agree that analyzing partial agreement (e.g., intersection ≥ 1) and linking it to task accuracy would add valuable perspective beyond completeness. We will add it in the next version.

1. Could $\boxed{\mathrm{PRIN}}$ be grounded more formally in logical consistency (e.g., monotonicity, Boolean closure)? Are there known logical frameworks where such behavior is characterized?

Thank you for this thoughtful question. PRIN arises in probabilistic neural language models, not symbolic systems. As shown in Table 1, LLMs like GPT-4 generate responses by sampling from conditional probability distributions, without enforcing strict logical complementarity between prompts (e.g., “correct” vs. “incorrect” answers). This makes PRIN an emergent behavioral inconsistency unique to probabilistic generation, not a violation of formal symbolic rules. While existing logical frameworks don’t yet capture such behavior, we see our work as an empirical foundation that can motivate future formalization

---

2. Do you expect $\boxed{\mathrm{PRIN}}$ to appear in non-mathematical domains (e.g., QA, ethics, or summarization)? Have you observed this elsewhere?

Thank you for this insightful question. While our study focused on mathematical and scientific tasks to ensure clear ground truth, we believe PRIN is a general behavioral property of LLMs and expect it to appear in non-mathematical domains like QA, ethics, or summarization — especially wherever prompts and their reversals (e.g., relevant vs. irrelevant facts, ethical vs. unethical actions) are involved. Preliminary observations outside math suggest similar inconsistencies, though systematically measuring PRIN in open-ended tasks is more challenging due to the lack of clear correctness labels. We see this as an exciting direction for future research.

---

3. Do you believe $\boxed{\mathrm{PRIN}}$ arises from objective misalignment, dataset artifacts, or model architecture? Any empirical ablations on these fronts?

We believe PRIN likely arises from a combination of factors: the probabilistic nature of LLM generation, imperfect alignment objectives (which don’t explicitly enforce logical complementarity), and possibly artifacts in training data where correct/incorrect distinctions are inconsistently represented. While our current work focuses on empirically characterizing PRIN’s behavioral patterns, we agree that ablation studies (e.g., probing fine-tuning objectives, data subsets, or architectural components) are valuable next steps, and we highlight this as promising future work.

---

4. Are there specific patterns in prompts or candidate answer sets that systematically trigger high $\boxed{\mathrm{PRIN}}$?

Please refer to Tables 5 and 6. Our experimental results across the three datasets indicate that answer sets involving complex mathematical expressions, such as those in EquInfer, tend to systematically trigger high PRIN. This suggests that the structural complexity and symbolic density of certain candidate answer sets are strongly associated with increased PRIN values.

---

5. Your negation-based fix works for small-scale tasks—does it generalize to long contexts, real-world tasks, or multimodal prompts?

Thank you for the great question. Based on our negation-based fix experiments using CONDAQA (Ravichander et al., 2022), we observed that, with the exception of high-performance models such as GPT-4, many models failed to reliably follow prompts, even producing empty responses in some cases. Given that CONDAQA is a large-scale benchmark with realistic passages and complex negation phenomena, these failures suggest that our fix does not yet generalize well to longer contexts or real-world tasks, and would likely face similar limitations in multimodal settings.