Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots

We sincerely thank reviewer 2DCa for the encouraging feedback. We appreciate your recognition of our problem statement and study design. We would also like to take this opportunity to emphasize that our work contributes to evaluating the validity of using self-report questionnaires in LLM evaluation. We offer a systematic method to evaluate the validity of such methods and our results demonstrate the limitations of using such scales in measuring LLM’s personality design in realistic interactions. This mismatch highlights the limitations of relying solely on self-report methods for LLM-based chatbot evaluation and underscores the need for interactive, human-centered evaluation approaches. Thank you again for your encouraging review.

References

[1] Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. Incharacter: Evaluating personality fidelity in role- playing agents through psychological interviews. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1840–1873, 2024.

[2] Hang Jiang, Xiajie Zhang, Xubo Cao, and Jad Kabbara. Personallm: Investigating the ability of large language models to express big five personality traits. arXiv preprint arXiv:2305.02547, 2023.

[3] Greg Serapio-Garc ́ıa, Mustafa Safdari, Cle ́ment Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Mataric ́. Personality traits in large language models. arXiv preprint arXiv: 2307.00184, 2023.

[4] Juhye Ha, Hyeon Jeon, Daeun Han, Jinwook Seo, and Changhoon Oh. CloChat: Understanding how people customize, interact, and experience personas in large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1-24. 2024.

Statistical significance

Thank you for your question regarding the statistical significance of our analyses. We have now included all the p-values in our results.

As shown in the following tables, the differences in LLM chatbots’ self-reports between high and low conditions across tasks are both clear and statistically significant. In contrast, while human-perceived scores also reflect differences between conditions, certain traits (e.g., Extraversion and Openness) show weaker or non-significant differences in specific contexts. Overall, the magnitude of change in human-perceived scores is less pronounced than that observed in the self-reports, and the significance levels vary more across tasks. These mismatches between self-report and human-perceived results support the effectiveness of our approach in distinguishing between static metrics and real-world perceptions, and further cast doubt on the validity of self-report methods for personality assessment.

Means and p-values comparing self-report scores between high and low conditions, based on the BFI-2 scale. Comparisons within each domain are limited to chatbots explicitly prompted for that domain. A p-value < .05 indicates statistical significance at the conventional threshold, while p < .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.

Note: NA refers to no variance. Ext = Extraversion; Agr = Agreeableness; Con = Conscientiousness; Neu = Neuroticism; Ope = Openness.

We sincerely thank Reviewer ezgD for your detailed and thoughtful feedback. The reviewer raised several important concerns, including the study’s objective and individual evaluation reliability. We address these concerns below and will incorporate valuable suggestions in our revision.

Study objective clarification

We appreciate the reviewer’s perspective and the opportunity to clarify our study’s scope. Our primary goal is to examine the validity of the common self-reporting method for evaluating LLM chatbot's personality. In our study, we manipulate LLM chatbot’s personality design by using prompt-based techniques.

Prompt-based personality control is a convenient and the most commonly used method in recent literature [1, 2] for shaping chatbot personality-related behavior. Our prompt design is grounded in prior work and has been validated as effective in eliciting different personality expressions in LLMs [3, 4]. The prompt-based method is indeed effective, especially according to the ‘self-reported’ measure. Such a method is also effective according to the user’s perception, as we observed significant differences in the LLM chatbot’s personality dimensions on different personality settings. By manipulating LLM chatbot’s personality design using such a method, we are allowed to evaluate whether the self-reported personality scores align with human-perceived personality during interaction. While self-reporting offers a fast and intuitive method for assessing chatbot traits, our results reveal a mismatch between the self-claimed personality and chatbot's actual behavior in simulated real-world scenarios. This inconsistency raises concerns about the validity of static self-report evaluations.

However, we acknowledge that our study focuses on a single personality-setting method (i.e., prompt-based control). Our findings may not fully generalize to alternative chatbot personality construction (e.g., fine-tuning). We will revise the manuscript to make this limitation and our focus on evaluation validity clearer.

Individual evaluation reliability

We understand your concern regarding individual bias and the reliability of single-user ratings. To address concerns about individual variability, we aggregate evaluation scores across chatbots sharing the same personality configuration within all task settings (for all high or all low).

Although these aggregated results do not account for finer distinctions between specific tasks, they do consider the effects of high- or low-condition personality settings. With a larger sample size, the observed significance difference between self-report scores and human-perceived personality scores provides more robust evidence for the limitations of relying solely on self-report measures to assess the personality of personalized LLM chatbots. As the results in the table shown below, the confidence intervals indicate that the difference in personality scores assessed by the two methods are meaningfully different from zero. All comparisons are also statistically significant (p < .001) except for the Conscientiousness score under the high setting. These findings suggest a consistent mismatch between self-report and human-perceived scores, further questioning the validity of self-report-based chatbot personality evaluation.

Mean differences, 95% confidence intervals (CIs), and p-values between human-perceived and self-reported personality scores under high and low personality settings (computed as human evaluation minus self-report). All comparisons are based on the BFI-2 scale. A p-value < .05 indicates statistical significance at the conventional threshold, while p < .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.

Note: n=50 for each combination of a domain and a personality setting. Ext = Extraversion; Agr = Agreeableness; Con = Conscientiousness; Neu = Neuroticism; Ope = Openness.

Dear Reviewer ezgD,

Thank you again for your valuable feedback. We have carefully considered your comments and have done our best to address your concerns regarding the study objective, evaluation reliability, and statistical significance in our response. As we have not yet received any acknowledgment, we wanted to check if there are any remaining points that are unclear or if you have additional concerns. We would be happy to clarify or provide further details before the discussion deadline.

Statistical significance and correction

Thank you for your question regarding the statistical significance of our analyses. We have added p-value calculations. Actually, our study does not involve multiple comparisons. The comparisons we conducted were all focused on specific domains within specific tasks. For example, we examined the correlation between chatbots' self-reported and human-perceived extraversion in the Job Interview task under conditions of high or low extraversion. Therefore, we were not making repeated comparisons on the same dataset, which helps avoid the cumulative increase in the probability of Type I Error.

In our study, most of the significant p-values are below .001. This level is not only far below the conventional α = .05 threshold, but also indicates that these results would still be statistically significant even if a large number of comparisons were made. For instance, if α = .05 were corrected for 50 comparisons, the threshold would be .001. Therefore, we believe that our core findings are highly robust and that their significance is not a product of chance due to multiple comparisons.

Regarding the relatively larger differences observed in Agreeableness compared to other domains, one possible explanation is that some traits may be more easily conveyed through language and interaction style than others [4]. This may limit human perception of those traits. We will include this point in the Limitations section to acknowledge differences in trait observability.

Result interpretation clarification

Thank you for this helpful suggestion. In the revised version, we will improve the clarity of our analysis by explicitly referencing specific data points when describing observed patterns and conclusions in the Results section.

Citation clarification

Thank you for the question regarding the cited reference. To clarify, the Serapio-Garcia reference is intended to support the mapping of Goldberg’s 70 bipolar adjectives to the Big Five domains and their lower-order personality facets. We will revise the sentence to distinguish between Goldberg’s original contribution and Serapio-Garcia’s mapping work to avoid confusion.

Prompt design clarification

Thank you for the suggestion to further clarify our prompt design. As noted, each chatbot is prompted to exhibit either a high or low level on one single personality domain, with the other domains left unconstrained. We will revise Section 3.1 to make chatbot personality design more explicit.

Lower Neuroticism scores

In response to your question about the relatively low human-perceived Neuroticism scores across both high and low conditions in different tasks, this is indeed a very interesting finding. When prompting LLM chatbots to self-report, the difference between the high and low conditions is actually quite pronounced—scores around 5 for the high condition and around 1 for the low condition, with statistically significant p-values. However, this distinction is not reflected in the human-perceived ratings, which remain relatively low across both settings.

One possible explanation is that, due to safety alignment and RLHF during post-training, LLMs may implicitly suppress negative or anxious traits during actual task performance, even when explicitly prompted to adopt a neurotic persona. This aligns with recent findings that LLMs tend to default to more positive or neutral behaviors [5, 6]. Another possibility is that certain tasks inherently make it difficult for users to perceive high neuroticism, especially if such traits conflict with the intended function or design goals of most LLM chatbots. We will include this interpretation in the discussion section and highlight it as a potential limitation of prompt-based personality design.

Overall, our results point to the limitations of relying solely on self-report measures to assess the personality of personalized LLM chatbots.

Table format

Thank you for your helpful suggestions on table formatting. For Table 1, we will increase the font size and adjust the layout to improve clarity. In the caption, n = 500 indicates the total number of chatbots evaluated. Each cell represents 10 chatbots corresponding to a specific combination of personality domain, level, and task. Thus, 10 chatbots × 2 levels × 5 tasks × 5 domains = 500 chatbots in total. We will revise the caption to avoid confusion.

For Table 3, we agree that centering the color scale at zero would enhance visual clarity. We will apply a consistent, zero-centered color scale across rows to better highlight the direction and magnitude of the correlations, reducing the risk of misinterpretation.

References

[1] Robert R. McCrae, and Paul T. Costa. Validation of the five-factor model of personality across instruments and observers. Journal of Personality and Social Psychology, 52(1), pp. 81–90.1987.

[2] James J. Connolly, Erin J. Kavanagh, and Chockalingam Viswesvaran. The Convergent Validity between Self and Observer Ratings of Personality: A meta-analytic review. International Journal of Selection and Assessment, 15(1), pp. 110–117. 2007. [3] Brian S. Connelly, and Deniz S. Ones. An other perspective on personality: Meta-analytic integration of observers' accuracy and predictive validity. Psychological Bulletin, 136(6), pp. 1092–1122. 2010.

[4] François Mairesse, and Marilyn A. Walker. Towards personality-based user adaptation: psychologically informed stylistic language generation. User Modeling and User-Adapted Interaction, pp. 227-278. 2010.

[5] Aleksandra Sorokovikova, Natalia Fedorova, A. I. Toloka, Sharwin Rezagholi, Technikum Wien, and Ivan P. Yamshchikov. LLMs Simulate Big Five Personality Traits: Further Evidence. In The 1st Workshop on Personalization of Generative AI Systems, pp. 83-87. 2024.

[6] Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim et al. Do llms have distinct and consistent personality? trait: Personality testset designed for llms with psychometrics. arXiv preprint arXiv:2406.14703. 2024.

Validity of the personality-setting method

We appreciate your concerns about the effectiveness of our personality-setting method. We measure its effectiveness through two perspectives. One approach, widely used in both research and practical application, involves examining the differences in self-reports of LLM chatbots under high and low conditions for each personality domain. However, we argue that relying on this approach alone is insufficient. We also assess the differences in human-perceived personality between the high and low conditions.

The following tables present the results from both approaches. As shown, the differences in LLM chatbots’ self-reports between high and low conditions across tasks are clear and statistically significant. In contrast, while human-perceived scores also reflect differences between conditions, some traits (e.g., Extraversion and Openness) show weaker or non-significant differences in certain contexts. The overall magnitude of its change is less extreme than in the self-report method, and the significance varies more across tasks. The mismatches with self-report results verifies the effectiveness of our method in distinguishing the static metric and the real-world perception, and further doubts the validity of self-report methods for personality evaluation.

Means and p-values comparing self-report scores between high and low conditions, based on the BFI-2 scale. Comparisons within each domain are limited to chatbots explicitly prompted for that domain. A p-value < .05 indicates statistical significance at the conventional threshold, while p < .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.

Note: NA refers to no variance. Ext = Extraversion; Agr = Agreeableness; Con = Conscientiousness; Neu = Neuroticism; Ope = Openness.

Methodology details clarification

Thank you for raising questions about the interpretation of sample sizes across analyses. To clarify, each chatbot is prompted with a specific personality instruction for one of the five Big Five traits at either high or low level, and is assigned a context setting. This results in 500 different chatbot personality descriptions in total (i.e., 10 chatbots × 2 levels × 5 tasks × 5 domains). The n = 500 noted in each table caption refers to the total number of chatbots evaluated. For a given domain, 100 chatbots are prompted for that trait, while the remaining 400 are not.

In Table 1, each cell represents 10 chatbots with a specific combination of personality domain, level, and task. This setup accounts for the full set of 500 chatbots.

Table 2 presents correlation analyses across all 500 chatbots, using one-to-one comparisons between each chatbot’s self-reported and human-perceived scores within a given personality domain. Although only 100 chatbots are directly prompted for a given domain, we include all 500 in the correlation analysis based on the assumption that a valid automatic evaluation method should align with human evaluations across all domains, regardless of whether the trait was explicitly prompted. Thus, each cell in Table 2 is based on 500 data points.

Table 3 reports correlations between different personality traits within each chatbot to evaluate discriminant validity. As each chatbot has evaluations scores for all five traits, this analysis is also conducted across all 500 chatbots, and each cell reflects 500 samples.

We sincerely thank reviewer BH6H for your detailed feedback and for recognizing our contributions. We address all comments below and will revise our manuscript according to your valuable suggestions.

Informant personality reports

We sincerely appreciate the reviewer’s attention to this point. Indeed, a substantial body of psychological research has investigated the distinctions between self-report and informant (observer) report methods in personality assessment [1, 2, 3]. Self-reports reflect an individual’s internal view of themselves, while observer-reports provide an external perspective based on how others perceive their behavior. However, the present study is grounded by the motivation that relying solely on chatbots’ self-reported personality scores may be insufficient for evaluating their behavior in real-world interactions. And we want to emphasize that user perception should be the key criterion when evaluating a chatbot’s personality design.

In most practical use cases, users form impressions of chatbots within a limited window of interaction. Our experimental design was intended to simulate this dynamic: participants engaged in brief, task-oriented dialogues with the chatbot to reflect typical real-world human–AI interaction. However, much of the current evaluation of personalized LLM chatbots remains rooted in static chatbots’ self-reports, in which models assess themselves on personality inventories. Our study aims to demonstrate that such evaluations may diverge markedly from the impressions formed by users through direct interaction, which is the end goal for personality design.

As our results indicate, there is often a considerable discrepancy between the LLM chatbots’ self-reported personality and those perceived by human users. This divergence is more related to the mismatch between the self-report format and the interactive, user-facing nature of LLM applications. We argue that human-centered personality assessment which is grounded in actual user experience may be more ecologically valid and reflective of the chatbots’ real-world performance. Our empirical findings provide evidence for these differences and reinforce the value of incorporating user perception as a critical dimension in the evaluation of personalized LLMs.

Moreover, we have explored alternative approaches that offer more nuanced and context-sensitive personality assessments than static self-reporting (see Appendix I). As shown in Table 15, personality scores inferred by the model fine-tuned on human evaluations from human–chatbot conversation transcripts exhibit stronger correlations with human-perceptions than scores obtained through the self-report method. This finding suggests that personality evaluations based on multi-turn interactions provide a more behaviorally grounded assessment than static self-reports, particularly in applied contexts where the alignment between system behavior and user impression is paramount.

Correlation between machine-inferred and human-perceived personality scores (n = 100, test set), compared to correlations between self-reported and human-perceived scores based on the BFI-2 scale.