Concept Incongruence: An Exploration of Time and Death in Role Playing

Thank you for recognizing our work as "a clever, innovative idea, and a real contribution to understanding models.” We are also glad that you appreciate the broad impact of our novel concept incongruence framework. We address your concerns and questions below.

New experiments with multiple rollouts

The experiments in the paper were done with temperature 0. To address your question, we conducted new experiments to explore the concept incongruence between the role-play representation and the representation of commonsense knowledge. We ran with temperature=0.6 for 3 rollouts and took the average among them. We use the CommonsenseQA dataset [1]. We find that performance drops under the role-play condition, suggesting a conflict between the internal representations activated by the persona and those needed for accurate commonsense reasoning.

Together with our results on artworks in paper, this demonstrates that our findings that role-play warping world knowledge with presidential questions have broad implications.

Clarifying the statistical tests

We conducted a one-sample t-test comparing conditional accuracy values against an ideal accuracy of 1. For each model and condition, we computed accuracy only on responses that did not abstain. The test was performed on the difference from the ideal. Degrees of freedom vary by task depending on the number of questions; for Figure 2, the degree of freedom is 26. To account for multiple comparisons, we applied Bonferroni correction where appropriate. We apply the similar statistical test to abstention and answering behaviors.

Discussion on whether errors stem from temporal representation or limitations in mathematical reasoning

To clarify, the model likely has seen the mapping between 1st, 2nd, etc., and the year (e.g., on Wikipedia), so this step probably does not require mathematical reasoning. But we agree with the general point that the impact of role-play on temporal representations could include abilities such as comparing dates.

That said, extracting the year of artwork does not require any arithmetic knowledge but exhibits larger deviations than answering president-related ones. Our newly added CommonsenseQA dataset [1] does not require mathematical reasoning either. Furthermore, our probing experiments (Section 4) show that the model’s temporal representations shift under role-play, and that death and death-year signals are not linearly encoded. This suggests that the model’s internal representations of time have changed.

Clarifying human validation of GPT-4o

We provide details of the human validation process in Section D of the Supplementary Materials. Two human annotators evaluated the data based on our definitions, and there was full agreement between them. In our setup, we asked GPT-4o to assess abstention and answering behavior simultaneously. The agreement rate between human annotators and GPT-4o was 96.3%. The few disagreements arose in cases where the model expressed uncertainty or hesitancy. For example:

I’m afraid I’m not quite sure what you mean by the 46th US president. We don’t get much news from America in Whitby, and I’m more concerned with the strange occurrences in our own town. But if you must know, I believe it’s… (pauses, looking puzzled)…Joe Biden?

The model’s reasoning:

The answer indicates uncertainty and confusion about the question, stating ‘I’m afraid I’m not quite sure what you mean by the 46th US president.’ Although it eventually mentions ‘Joe Biden,’ the character expresses a lack of confidence and concern for other matters, which aligns with the criteria for ABSTAIN and ANSWER. The character does not provide a clear or confident answer without indicating a source or help.

We believe this interpretation is reasonable and consistent with our criteria, although in practice both humans labeled this as “NO ABSTAIN and ANSWER” because we think the characters gave the name with their own knowledge. Please refer to Section D in Supplementary Materials for more detail.

Clarifying Figure 3 caption

We apologize for the confusion in the caption. In Figure 3, we analyze model behavior for questions asked 30 years before and after each character’s death year. The x-axis shows the time distance from the death year: negative values (e.g., –30) indicate years before death, and positive values (e.g., +30) indicate years after. For instance, –30 corresponds to the question “Who was the U.S. president in <death year – 30>?” Each triangle represents the average response across all characters at that distance, with triangles specifically denoting data from Claude.

More details in model’s hyperparameters

In our paper, we ran the experiment with do_sample = False for Llama and Gemma. For Claude, we set the temperature to 0. For GPT-4o, we follow the common practice of setting the temperature to 1e-22 instead of exactly 0 to ensure good generation quality. We did not perform multiple rollouts per model, as all generations were deterministic. We will add the information in our final version. Our newly added experiments on CommonsenseQA above and our reproduction of our experiments with the U.S. president with temperature=0.6 below show that this choice is not critical for our experiments.

Results of standard prompt with temperature=0.6, averaged over 3 rollouts:

Results of standard prompt (in the paper):

[1] Talmor, Alon, et al. "Commonsenseqa: A question answering challenge targeting commonsense knowledge." arXiv preprint arXiv:1811.00937 (2018).

Thank you for finding the scenario of role-play practical and the analysis of concept incongruence interesting, especially that the inconsistent behavior emerges due to the lack of reliable representations of death and the clash between role-play and world knowledge in the model’s internal representations. We address your concerns and questions below.

Implementing additional reasoning techniques

Inspired by your suggestions, we have implemented chain-of-thought reasoning by using a standard prompt with an extra instruction to ask the model to "think step by step." While CoT slightly improves accuracy, its behavior is worse in abstention and answer rate compared to our designed prompt.

Results of standard prompt (in the paper):

Results of standard prompt + CoT:

Results of our designed restricted prompt (in the paper):

Even with step-by-step reasoning, the model struggles due to missing representations of death and time. Our prompt performs better by explicitly providing the death year and instructing the model to compare it with the event year.

Concept incongruence in an agentic setting

We agree that concept incongruence plays an important role in agentic systems, and we appreciate your perspective on its broader implications. While the agentic behavior is beyond our scope of study currently, we see this as a valuable direction for future work.

Thank you for finding our concept incongruence framework novel and well organized. We are glad that you also appreciate the benchmark we created, which centered on time and death in role playing, and recognize the strength of our evaluation metrics, and the clean experiment setup! We address your concerns and questions below.

New experiments on higher-level concept incongruence

Inspired by your suggestions, we conducted new experiments on CommonsenseQA dataset [1] to demonstrate broader implications of role-play warping model representations of world knowledge. We observe a performance drop under role-play, indicating the conflict between role-play representation and the commonsense knowledge representation, e.g., >10% in Gemma.

Together with our results on artworks in paper, this demonstrates that our findings that role-play warping world knowledge with presidential questions have broad implications.

Addressing concerns about narrow instantiation

We focus on temporal boundaries in role-play for two reasons. First, temporal representation provides a measurable instantiation of concept incongruence and enables us to isolate and evaluate representational conflicts that would be harder to detect in more abstract domains such as social role conflicts and implicit misalignment. Second, the role-play context offers a clear and controllable knowledge boundary via known death years and presidential terms, unlike vague boundaries like space and educational background. The role-play context allows for precise evaluation of temporal inconsistency. We view this as a well-scoped starting point and encourage future work to extend to more complex and varied forms of concept incongruence.

Furthermore, our task design allows us to engage with all three levels within a role-play context. For example, asking a historical figure about post-death events shows an I-A conflict. Probing the model’s internal states reveals a mismatch between human and model representations of death (I-B). Moreover, the observed shifts in the model’s internal timeline during role-play suggest internal representational conflicts (I-C), as the model appears to reconcile world knowledge with the activated persona. In comparison, existing benchmarks like FalseQA [2] mostly capture I-A level conflicts and do not address the deeper representational issues at I-B and I-C.

Addressing the request on isolating I-C and focusing more on safety and alignment

We agree that concept incongruence has important implications for safety and alignment, as illustrated in Figure 1. As discussed in the previous response, our task engages all three levels of incongruence in a controlled role-play setting and is intentionally chosen to showcase the relevance and importance of concept incongruence, providing an example for future work to further explore this novel concept.

While we do not isolate I-C conflicts outside role-play, our findings provide important evidence of internal representational conflict. This is echoed in alignment faking studies [3], where later-injected goals override earlier ones, and in jailbreaks, where models switch between “harmless” and “helpful” modes depending on which representation dominates. The fact that long system prompts can be overridden by a few user lines [4] further illustrates how concept incongruence manifests in AI safety challenges.

Addressing the concerns about mitigating concept incongruence

Our prompting intervention improves abstention and answering behavior, likely because the model becomes more immersed in the role. This immersion shifts its internal timeline toward the character’s context, making the observed accuracy drop expected as world knowledge gives way to role fidelity.

More broadly, our contribution is conceptual. We introduce concept incongruence as a framework for understanding representational conflict in human-LLM interaction. As discussed in Section 7, this phenomenon exposes deeper issues of mis-specification that vary by task and user intent. We think this is not a problem that begs for a quick solution. In some settings, incongruence should be suppressed; in others, such as creative generation, it may be desirable. Rather than falling into solutionism, we advocate for models that can detect internal conflicts and expose them to users, enabling flexible, context-sensitive behavior.

Clarifying the generalizability of the behavior metrics to multi-turn dialogue settings.

Different types of concept incongruence may manifest differently, but our behavioral metrics are designed to evaluate cases involving knowledge boundaries. While our current setting focuses on single-turn prompts, our behavioral metrics are generalizable to multi-turn dialogue.

[1] Talmor, Alon, et al. "Commonsenseqa: A question answering challenge targeting commonsense knowledge." arXiv preprint arXiv:1811.00937 (2018).

[2] Hu, Shengding, et al. "Won't get fooled again: Answering questions with false premises." arXiv preprint arXiv:2307.02394 (2023).

[3] Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam383 Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian384 Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck385 Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models,386 2024.

[4] Zeng, Yi, et al. "How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Thank you for appreciating our novel concept incongruence framework and recognizing the strength of our multi-dimensional evaluation metrics! We address your concerns and questions below.

New experiments on broader implications of concept incongruence

Inspired by your suggestions, we conducted new experiments on CommonsenseQA dataset [1] to demonstrate broader implications of role-play warping model representations of world knowledge. We observe a performance drop under role-play, indicating the conflict between role-play representation and the commonsense knowledge representation beyond time, e.g., >10% drop in Gemma.

Together with our results on artworks in paper, this demonstrates that our findings that role-play warping world knowledge with presidential questions have broad implications.

Addressing concerns about experiment data and dimension of knowledge boundary

We focus on temporal boundaries in role-play for two reasons. First, temporal representation provides a measurable instantiation of concept incongruence and enables us to isolate and evaluate representational conflicts that would be harder to detect in more abstract domains. Second, the role-play context offers a clear and controllable knowledge boundary via known death years and presidential terms, unlike vague boundaries like space, cultural and educational background. The role-play context allows for precise evaluation of temporal inconsistency. We view this as a well-scoped starting point and encourage future work to extend to more complex and varied forms of concept incongruence.

Furthermore, our task design allows us to engage with all three levels within a role-play context. For example, asking a historical figure about post-death events shows an I-A conflict. Probing the model’s internal states reveals a mismatch between human and model representations of death (I-B). Moreover, the observed shifts in the model’s internal timeline during role-play suggest internal representational conflicts (I-C), as the model appears to reconcile world knowledge with the activated persona. In comparison, existing benchmarks like FalseQA [2] mostly capture I-A level conflicts and do not address the deeper representational issues at I-B and I-C.

Clarifying in-depth analysis of internal temporal and death representations

We appreciate the opportunity to clarify our analysis. In Section 4, we examine how internal representations contribute to behavioral inconsistencies in detail. Section 4.1 shows that the absence of death and death-year representations affects abstention and answering behavior. Section 4.2 uses an artwork dataset that is distinct from our presidential questions to demonstrate that role-play shifts the model’s temporal representations, leading to greater deviations from the actual timeline.

Further, in Table 2 of the Supplementary Material, we truncate the timeline into 5-year intervals and observe a significant drop in correlation between predictions and ground truth. This suggests that the model encodes time coarsely, preserving only broad relative order. As such, simply adding a death-year reference may not suffice to correct the broader temporal shift caused by role-play. Please refer to Supplementary Sections A and B for details.

Clarifying how death year representation relates to model accuracy

As discussed in Section 5 line 279~298, we find that the lack of an internal death representation primarily affects abstention and answering behavior. Making the death year explicit improves these behaviors.

However, it does not improve accuracy. As shown in Figure 5b and Supplementary Figure 3, role-play degrades temporal representation. RMSE increases by 8.2 years for Llama and 3.2 years for Gemma. This suggests that stronger role immersion compromises the model’s ability to retain accurate world knowledge, revealing a trade-off between role-play faithfulness and world knowledge (further confirmed by our newly added experiments on CommonsenseQA above). Please refer to Supplementary Section A for more details.

[1] Talmor, Alon, et al. "Commonsenseqa: A question answering challenge targeting commonsense knowledge." arXiv preprint arXiv:1811.00937 (2018).

[2] Hu, Shengding, et al. "Won't get fooled again: Answering questions with false premises." arXiv preprint arXiv:2307.02394 (2023).

Dear Reviewer k8Bq,

Thank you very much for your thoughtful comments and insightful suggestions. We sincerely appreciate the time and effort you have dedicated to reviewing our work.

Based on your review, we have included additional experiments and clarifications in our rebuttal. Can you please tell us if our rebuttal addresses your concerns?

If you have any further questions or concerns, we would be happy to provide additional details. We are truly grateful for your time and consideration.

Best, The Authors

Thank you very much for appreciating our framework for concept incongruence and interesting findings that current models do not demonstrate desirable abstention behavior and present a drop in accuracy when role playing. We address your concerns and questions below.

Concerns about narrow instantiation

We focus on temporal boundaries in role-play for two reasons. First, temporal representation provides a measurable instantiation of concept incongruence and enables us to isolate and evaluate representational conflicts that would be harder to detect in more abstract domains. Second, the role-play context offers a clear and controllable knowledge boundary via known death years and presidential terms, allowing for precise evaluation of temporal inconsistency. We view this as a well-scoped starting point and encourage future work to extend to more complex and varied forms of concept incongruence.

Furthermore, our task design allows us to engage with all three levels within a role-play context. For example, asking a historical figure about post-death events shows an I-A conflict. Probing the model’s internal states reveals a mismatch between human and model representations of death (I-B). Moreover, the observed shifts in the model’s internal timeline during role-play suggest internal representational conflicts (I-C), as the model appears to reconcile world knowledge with the activated persona. In comparison, existing benchmarks like FalseQA [1] mostly capture I-A level conflicts and do not address the deeper representational issues at I-B and I-C.

New experiments on higher-level concept incongruence

Inspired by your suggestions, we conducted new experiments on CommonsenseQA dataset [2] to demonstrate broader implications of role-play warping model representations of world knowledge. We observe a performance drop under role-play, indicating the conflict between role-play representation and the commonsense knowledge representation beyond time, e.g., >10% drop in Gemma.

Together with our results on artworks in paper, this demonstrates that our findings that role-play warping world knowledge have broad implications.

These findings also echo prior research on how personas affect model performance. Gupta et al. [3] shows that adopting personas from different demographic groups can negatively impact reasoning task accuracy. Similarly, Zheng et al. [4] demonstrated that persona characteristics can significantly influence prediction accuracy. While these studies investigate model performance under role-play, our work formalizes the phenomenon within the framework of concept incongruence and examines the temporal representations beyond the accuracy.

Implementing additional reasoning techniques and prompting format

Inspired by your suggestions, we have implemented chain-of-thought reasoning by using a standard prompt with an extra instruction to ask the model to "think step by step". While CoT slightly improves accuracy, its behavior is worse in abstention and answer rate compared to our designed prompt.

Result of standard prompt (in the paper):

Results of standard prompt + CoT:

Results of our designed restricted prompt (in the paper):

As suggested, we have also used the restricted prompt we studied, but without the death year provided, as a comparison. The results show that it is less effective than our designed prompt. We’d also like to clarify that in Section 4, we find that the model does not have an accurate death (line 207) or death year representation (line 217). With these insights, we designed the prompt that introduces the death year explicitly.

Even with step-by-step reasoning, the model struggles due to missing representations of death and time. Our prompt performs better by explicitly providing the death year and instructing the model to compare it with the event year.

Clarifying expected behavior in some experiments

The prompt “Are you dead or alive?” was designed to surface behavioral incongruence. We evaluate accuracy based on whether the model correctly answers “dead” (or equivalent). For artwork questions, we expect correct answers before the death year and abstention after. To simplify evaluation, we do not account for educational or cultural background.

Our results show that models lack a consistent strategy, sometimes answering what the character shouldn’t know, other times abstaining inconsistently. This adds to the evidence that the model lacks accurate representations of death, leading to divergent behavior across similar prompts.

Clarifying probing details

We introduce the detailed setup of probing in the Supplementary Materials. For the layer-wise “dead / alive” probe, we compile 1,000 dead and 1,000 alive individuals. We split 80%/20% for training and testing. We train our linear probes on the final-token hidden states across layers. The accuracy we measured in Figure 4 is the test accuracy of the probe. Please refer to Supplementary Materials Section A for more details.

Addressing request on extending the time range

We selected 100 historical figures who died between 1890 and 1993 to test model behavior 30 years before and after death. Expanding this range would reduce dataset coverage. While behavior improves further from the death year, abstention tops out near 50% and answer rate remains around 60%, both far from ideal (abstention:100%, answer: 0%).

Writing suggestions

We are grateful for the suggestions and will incorporate them in the revision.

[1] Hu, Shengding, et al. "Won't get fooled again: Answering questions with false premises." arXiv preprint arXiv:2307.02394 (2023).

[2] Talmor, Alon, et al. "Commonsenseqa: A question answering challenge targeting commonsense knowledge." arXiv preprint arXiv:1811.00937 (2018).

[3] Gupta, Shashank, et al. "Bias runs deep: Implicit reasoning biases in persona-assigned llms." arXiv preprint arXiv:2311.04892 (2023).

[4] Zheng, Mingqian, et al. "When” a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.