PersonaEval: Benchmarking LLMs on Role-Playing Evaluation Tasks

Thank you for the thoughtful and detailed feedback. We reply point-by-point here, to begin the discussion.

W1&W2: Our goal is to assess a fundamental aspect of role-playing: the ability of LLMs to adapt language and communication to align with different audiences. While it is true that the Wired 5 Levels dataset primarily involves adjusting language complexity for varying expertise levels, we argue that this is a critical subset of role-playing. Every persona inherently involves some degree of expertise-based adaptation. If LLMs struggle to differentiate between audiences with clear and structured knowledge-level distinctions, it raises deeper concerns about their ability to handle more complex and nuanced persona shifts involving tone, intent, or behavior. That said, we agree the current setup may conflate language complexity with behavioral adaptation. To address this:

1. We plan to conduct an ablation study to isolate the impact of linguistic complexity, such as controlling for expertise-specific content or randomizing audience indicators, to clarify what LLMs are truly learning.
2. Future work will expand the benchmark to include datasets or tasks that explicitly evaluate personality shifts or behavioral role adaptation, providing a broader view of LLM capabilities in role-playing. We acknowledge that linguistic complexity itself is a well-studied area and appreciate the suggestion to incorporate advanced linguistic feature analysis. We will explore such tools to further disentangle role-specific behaviors from mere complexity adjustments in future iterations of our work.

W3: We appreciate the reviewer pointing out the overstatement in our claim regarding the prevalence of LLM-based evaluation in role-playing research. While it is true that many works utilize supporting metrics alongside LLM evaluations (e.g., Kovač-2024, Lee-2024, Wang-2024), our intent was to highlight that a significant portion of studies rely primarily on LLM evaluations as the central metric. We acknowledge that our phrasing was imprecise and will revise the manuscript to more accurately reflect this nuance. Thank you for bringing this to our attention.

Q: We sincerely thank the reviewer for this valuable suggestion. Reframing the study toward pedagogical adaptation or communication complexity is indeed a compelling direction, given the nature of the Wired 5 Levels dataset. We will carefully consider this perspective for future iterations of this research. That said, in this work, our primary goal is to assess whether LLMs can distinguish between audiences based on expertise levels, as a foundational test of role-playing ability. Role-playing often involves expertise-based linguistic adaptation as one of its core components. If LLMs struggle with even this straightforward task, it suggests broader challenges in simulating more nuanced personas that require behavioral shifts, tone adjustments, or emotional intelligence. We will revise the manuscript to acknowledge the alternative framing and clarify how expertise-based adaptation serves as a stepping stone for evaluating the broader capabilities of LLMs in role-playing.

Thank you for the thoughtful and detailed feedback. We reply point-by-point here, to begin the discussion.

W1: We believe our benchmark serves as a necessary condition for evaluating LLMs as evaluators in role-playing contexts, focusing specifically on their ability to distinguish different levels of expertise. While the educational video domain may not fully capture every aspect of role-playing, we argue that it provides a structured and controlled environment for testing the core evaluative capabilities of LLMs, which can be extended to other domains with further validation. Additionally, we will discuss potential future work, including the incorporation of more diverse datasets to better capture the complexities of role-playing evaluations.

W2: Thank you for the suggestion. While creative writing evaluation differs from our focus on role-playing, we will add a discussion of the Judgemark benchmark to the related work section.

W3: We will reorganize the code to improve clarity and ensure it is easier to reproduce.

W4 & Q1: The Wired 5 Levels dataset was sourced from the Wired 5 Levels video series, where experts explain a concept to five distinct audiences. We manually segment and refine the transcriptions for audiences at different levels, and separate the turns of explainers and audiences. We will cite the relevant papers you mentioned to acknowledge their contributions.

W5: We will reorganize the tables to focus on key metrics that directly support our hypotheses, such as reporting the f-measure for each model.

Q2: While our benchmark specifically evaluates expertise-based adaptation, we argue that this is a core component of role-playing, which often involves adjusting language, tone, and complexity based on the audience. By testing LLMs' ability to distinguish between different levels of expertise, we assess how well LLMs can modulate responses in line with role-specific communication needs. Although it focuses on a subset of role-playing, our benchmark lays the groundwork for evaluating more complex role-play scenarios in future work, offering insights into LLM performance across different domains.

Q3: The primary users of our benchmark are researchers and practitioners interested in evaluating and improving LLMs in the context of role-playing and audience-specific communication. This includes those working on human-computer interaction, natural language understanding, and applications where LLMs must adapt to diverse user personas. Additionally, it is aimed at those developing evaluation metrics for LLMs, offering a structured way to assess language adaptation across varying audience types.

Q4: We plan to support and maintain the benchmark by releasing it publicly and providing clear documentation, including detailed instructions on how to use it for various evaluation tasks. Additionally, we will continue to improve the benchmark by incorporating user feedback, updating it with new datasets and tasks that test broader aspects of role-playing (e.g., emotional tone, intent), and potentially integrating additional evaluation methods. We aim to foster a community of researchers who can contribute to and expand upon the benchmark, ensuring it evolves alongside advancements in LLM evaluation.

Suggestions: We appreciate the reviewer’s constructive suggestions. We plan to incorporate points 1, 4, 5, and 6 into the revised manuscript. Regarding point 7, while we recognize the value of this suggestion, we are somewhat constrained by the limited space in the paper and will address it to the extent possible within these constraints. For point 3, we will revise our wording for greater clarity, while reiterating that the primary focus of our paper remains on evaluating LLMs’ ability to assess dialogues based on audience expertise. We are grateful for the reviewer’s feedback and will make the necessary adjustments accordingly.

Thank you for the thoughtful and detailed feedback. We reply point-by-point here, to begin the discussion.

W1: The ability to accurately classify the speaker’s level is a necessary condition for ensuring that the model can evaluate whether the role-playing output is appropriate, consistent, and well-suited to the target character. This capability is essential for assessing whether the role-playing aligns with the intended role’s complexity, tone, and depth.

Q1: The "Avg Tokens" are calculated based on the entire dialogue. The lower average tokens for the "Child" role arise because both the child's replies and the explainer's responses are shorter, as explanations tailored for children are generally more concise and simplified.

Q2: The "recall" values presented in Tables 2 and 3 can be interpreted as accuracy, as they reflect the overall performance of each model in correctly classifying the roles.

Q3: We have added the full list of the 26 topics in the appendix.

Q4: Due to limited computational resources, we conducted each experiment three times to calculate the mean and standard deviation. For the LLM settings, we used a temperature of 0.7 and top_p of 0.8, with all other parameters set to their default values. We will clarify this in the revised manuscript to ensure reproducibility.

Q5: The pairwise evaluations use the same dialogues but with different questions, such as distinguishing which response is from the child or the teen.

Q6: The pairwise evaluation uses dialogues from the same topic to ensure a fair comparison. Evaluating LLM performance when comparing responses across different topics is an interesting direction and will be explored in future work.

Q7 & Q9: The cases with responses from the same level (e.g., two teen responses with a tie option), would be an interesting extension. We will explore this setting and include an analysis of superficial features as part of our future work.

Q8: The order of the two responses is randomized for each evaluation to minimize positional bias. Despite this randomization, the LLMs still exhibit high contradiction rates, indicating that the issue is not solely due to order effects. That said, we acknowledge the value of further analysis and plan to conduct an ablation study to explicitly investigate the influence of order on the results.

Q10: Our benchmark evaluates LLMs as evaluators in role-playing tasks from a different perspective compared to prior work. Our benchmark specifically assesses the ability of LLMs to classify nuanced roles based on linguistic cues, such as expertise levels. This task introduces unique challenges that may not be addressed in previous evaluations, highlighting areas where current approaches may fall short.

Q11: While simulating conversations at different expertise levels is an interesting aspect, it is not the primary focus of our paper. Our work is centered on evaluating how LLMs assess role-playing performances across different expertise levels, rather than role-playing on different levels.

Typos: We have fixed these errors in the revised manuscript.

Thank you for the thoughtful and detailed feedback. We reply point-by-point here, to begin the discussion.

W1: Our benchmark is designed to assess the effectiveness of LLMs in role-playing evaluation tasks by framing the problem as a classification task. Informative content is well-suited for this purpose, as it provides clear linguistic cues necessary for distinguishing between sentences from different levels of expertise. In contrast, casual conversations lack the structured features required for accurate role differentiation, making them misaligned with our current objective.

W2: Using the five familiar educational levels ensures that the task remains well-defined and feasible for LLMs, as these roles provide clear and structured linguistic distinctions. Expanding to more diverse or unfamiliar user types could introduce ambiguities, making the classification task less reliable and harder to evaluate.

W3: While we aimed to provide a broader context by discussing general LLM evaluation methods, we agree that focusing more on role-playing-specific challenges and prior contributions would strengthen the section. We will revise this part to better highlight the unique aspects of role-playing evaluation and how our work builds upon or differs from prior studies.

W4: Our benchmark focuses on classifying roles based on linguistic cues without restricting the analysis to explanation-focused cues. Both language style and informational depth are available for LLMs to consider during the classification process. We agree that a deeper analysis of how these factors contribute to the model's decisions could provide additional insights and will consider including this in future work.

W5: The user study results highlight the inherent variability in human judgment, which aligns with our observations regarding the challenges in evaluating nuanced differences. The classification task we designed serves as a simplified yet meaningful scenario to evaluate LLMs' ability to assess role-playing performance. By isolating linguistic cues, this task provides a controlled environment to test whether an LLM evaluator can detect and differentiate persona-specific language effectively. Our benchmark served as a necessary condition for role-playing evaluations, as these tasks require assessing coherence and consistency with a persona in specific contexts. Therefore, we argue that high performance in such classification tasks is essential to developing robust LLM evaluators capable of reliably judging role-playing tasks.

Q1: Accurately predicting the speaker’s level is crucial for evaluating the effectiveness of LLMs as role-playing evaluators because it reflects their ability to assess the coherence and alignment of role-playing outputs with the intended roles. This demonstrates whether the model can distinguish nuanced linguistic and contextual differences that define role consistency, which is central to evaluating role-playing performance.

Q2: There were differences in performance across topics. To avoid bias from the single topic, we report the average performance across all topics.