CoSER: Coordinating LLM-Based Persona Simulation of Established Roles

Thanks for your valuable feedback. We have responded to all your comments and questions in detail, and will revise the paper to:

1. Compare authentic v.s. LLM-generated role-playing data with experiments.
2. Experiment with Deepseek V3 and R1 as judges to avoid self-preference bias.
3. Include human evaluation to validate GCA evaluation.
4. Compare CoSER with a AI-driven storytelling method.

Please find details below:

Q1: Comparing authentic v.s. LLM-generated role-playing data with experiments.

We recognize this feedback. As suggested, we ran two experiments to show the superiority of authentic data.

1.GCA Evaluation for Dialogues

We apply GCA evaluation to compare authentic (groundtruth) vs. LLM-generated (GPT-4o) dialogues of 100 samples in CoSER Test, without providing groundtruth to LLM judges as reference. We use Deepseek R1 as judges. Results:

2.Fine-tuning with Dialogues

We fine-tuned Llama-3.1-8B using authentic or LLM-generated dialogues for 200 samples for 4 epochs, using identical settings as CoSER-8B. We evaluate them via GCA on 100 new samples. Results:

Results show authentic dialogues surpass LLM-generated data in both quality and training effects. We will include them in our revised paper.

Q2: The evaluation relies heavily on GPT-4o as LLM judges, which could introduce bias or overfitting to OpenAI's model responses rather than proving general improvements across diverse assessment frameworks.

We acknowledge this feedback. We experimented with Deepseek V3 and R1 as judges to evaluate performance of 7 models:

(Columns are judges)

The results validate GPT-4o's bias towards GPT models: R1 and V3 judges prefer CoSER-70B and Claude to GPT-4o. We will include the results in our revised paper.

Q3: GCA evaluation is not well-validated by human raters. Human assessments of role-playing quality would provide stronger validation than LLM critics.

Thanks for this comment. We have conducted human evaluation on GCA simulation of 7 models. Afterwards, we find that (1) GCA evaluation aligns well with human judges, (2) Human evaluation of LLM role-playing is highly time-consuming, highlighting the needs for automated evaluation. We apologize for omitting the settings for space limit. Please refer to our response to Q2 of Reviewer XqGw for details. The results are:

• Human Evaluation Results

The results generally align with GCA evaluation. One difference is: human judges show less preference for GPT models compared to LLM judges (GPT-4o), similar to Q2.

• Alignment between LLM and Human Judges

(4o, R1 and V3 refers to GPT-4o and Deepseek R1/V3 judges.)

Our results show that GCA evaluation aligns with human judges, with gt, rb, lc, and ds being indispensable components.

Our annotators find that (1) GCA simulation thoroughly reflects LLM’s role-playing abilities; (2) Manual evaluation is highly difficult. It requires careful learning of complex background and abundant dialogues, and takes 15 minutes to evaluate 7 models for 1 case on average.

Q4: More transparent reporting on how rubric-based penalties are assigned would improve reproducibility.

We recognize this feedback. We will improve transparency by releasing the LLM-identified flaws, and adding detailed examples and analysis in our appendix.

Q5: Prior work in digital actors and AI-driven storytelling could be more directly compared or contrasted with CoSER’s approach.

As suggested, we compare GCA simulation with stories generated by HollmWood [1]. We use 30 samples from CoSER Test, apply GPT-4o as the actors/writers, and report average scores assessed by GCA evaluation.

Results show GCA simulation produces more authentic and human-like character interactions compared with HollmWood. We will include the results in our paper.

Reference:

[1] Jing, C., et al. HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing

We sincerely appreciate your thorough review and valuable comments. We have responded to them in detail, and will revise the paper to:

1. Add more related work on persona-based LLMs.

2. Include human evaluation to further validate our models and evaluation methods.

Please find the details below:

Q1: Difference compared with Persona Hub [1]

CoSER differs from [1] in:

1. Goal and Focus:

2. Dataset Quality:

We'll add detailed discussion in our paper.

Q2: A human evaluation would support the high quality beyond LLM critics.

Thanks for your suggestion. As suggested, we conducted human evaluation on GCA simulation of 7 models. Results confirm that (1) CoSER models are preferred by human judges; (2) GCA evaluation aligns well with human judges.

• Settings

  • Models: CoSER-8B/70B, abab7, Llama-3.1-8B, GPT-3.5/4o, and Claude (3.5-Sonnet).
  • Data: 60 samples from CoSER Test.
  • Annotations: 3 annotators scored the models (1-10 scale) across 20 samples, given background context, groundtruth dialogues, and scoring rubrics.
  • Metrics: models' average scores and win rates v.s other models.

• Results

The results generally align with GCA evaluation, confirming CoSER models's superior performance. Annotators noted that GCA simulation better reflects LLMs' role-playing abilities than previous methods. There is one difference: human judges show less preference for GPT models compared to LLM judges (GPT-4o). While LLM judges prefer GPT-4o to Claude and GPT-3.5 to Llama-3.1-8B, the results are opposite for human judges (and also R1 as judge, in our response to Q2 of Reviewer vmSk), which likely stems from "self-preference" bias in LLM judges [5].

We further study alignment between LLM judges and human judges using the annotations.

• Settings:

  • Judge models: GPT-4o (4o), Deepseek V3 (V3) and R1 (R1).
  • Judge methods: Standard GCA evaluation, and ablation variants removing: (i) groundtruth as reference (gt), (ii) rubrics (rb), (iii) length correction (lc), and (iv) dimension separation (ds). Plus BLEU and ROUGE-L.
  • Metrics: We measure how often LLM judges agree with humans when comparing two models, removing model pairs where judges assign similar scores to both models.

• Results

Our results show that: (1) gt, rb, lc, and ds improve LLM judges in GCA evaluation; (2) Reasoning models excel as judges - Deepseek-R1 (77.5%) surpasses V3 (65.1%) and 4o (68.6%); (3) BLEU and ROUGE-L remain effective.

Q3: Missing related work on persona-based LLMs, e.g., [1], [2].

Thanks for this feedback. We'll include more related work on persona-based LLMs in our paper, including [1-4].

Q4: Explore dataset expansion by human feedback.

Yes. Our data pipeline has been iteratively improved with human feedback, and we'll continue to explore dataset expansion.

Q5: Limitations and potential improvements for chunk-based data extraction.

The key limitations are: (1) plot fragmentation: splitting a plot into different chunks and (2) limited context, without long-term knowledge of books. Potential improvements include: (1) iterative, plot-aware chunking (implemented in CoSER) and (2) long-term memory for LLM extractor.

Q6: Where did CoSER approach work best, and where didn't it work so well?

CoSER excels at its high-quality data in rich types and nuanced evaluation. However, its extraction recall remains suboptimal: it doesn't capture all character knowledge. We'll address this in future work.

Reference:

[3] Deshpande, A., et al. Toxicity in chatgpt: Analyzing persona-assigned language models.

[4] Park, J. S., et al. Generative agents: Interactive simulacra of human behavior.

[5] Wataoka, K., et al. Self-preference bias in llm-as-a-judge.

We thank the reviewer for the constructive comments and advice. We have responded to your comments and questions in detail. We hope our responses will help you better recognize the details of our work and findings. Please find the details below:

Q1: Generalizability and Broader implications of our findings, datasets and models.

We believe our work has broader implications in three key areas: (1) alignment and human-like AI, (2) complex instruction following and reasoning, (3) evaluation method for complex open-ended tasks.

1. Alignment and human-like AI

Generally, role-playing represents a form of anthropomorphism and alignment. It is closely related to several key aspects towards human-like AI, including human-like speaking styles, social intelligence, and emotional quotient. Our work brings valuable insights for improving and evaluating these abilities. Specifically, models trained on our dataset demonstrate a speaking and thinking style that is better aligned with humans (as shown in Table 7 and 10).

2. Complex instruction following and reasoning

Role-playing established characters is a highly complex task that requires models to not only behave like humans (as discussed above) but also simulate specific characters, which requires in-depth understanding and adherence to massive background knowledge. This challenges a model's ability to follow complex instructions and constraints, and understand knowledge in long context. Furthermore, decision-making in role-playing represents a sophisticated reasoning task, where LLMs need to consider various constraints in personas (such as their needs, personality traits, and social relationships).

3. Evaluation method for complex open-ended tasks

Our findings on evaluation of role-playing LLMs can generalize to evaluation methodologies for broad complex, open-ended tasks. LLM performance on mathematics and coding tasks can be effectively assessed through verifiable answers. However, nuanced evaluation for subjective, open-ended tasks remains a challenge, such as creative writing, significantly limiting the development of LLMs in these areas.

Many findings from our work have broader implications for LLM-based evaluation for complex open-ended tasks, including: (1) the importance of reference answers for LLM judges; (2) providing detailed rubrics to guide LLM judges; (3) evaluating different dimensions separately to avoid bias and (4) reasoning models' superior performance as LLM judges.

Q2: Fundamental difference between CoSER and previous role-playing works.

Our work differs from previous role-playing works primarily in dataset and evaluation.

• For dataset, as shown in Table 1 and Figure 1, CoSER differs from previous role-playing datasets in two aspects: (1) CoSER extracts authentic, multi-turn, multi-character dialogues from books. In contrast, previous datasets are primarily synthesized by LLMs with little grounding on real plots, and typically focus on simplied scenarios between two characters or one character and one user; (2) CoSER provided comprehensive data types beyonds profiles and dialogues, including characters' actions and thoughts in messages and experiences in key events, which are ignored in previous datasets.

• For evaluation, CoSER is different from previous evaluation methods in that: (1) CoSER thoroughly elicits LLMs role-playing performance via multi-turn, multi-character simulation, while previous evaluations are typically based on single-turn, single-character response. Hence, CoSER provides LLM Judges with comprehensive performance results for evaluation ; (2) CoSER provides groundtruth conversations as reference for LLM judges, while previous evaluations either lack reference or use GPT-synthesized dialogues as reference; (3) CoSER provides detailed rubrics written by experts to guide LLM judges, while previous evaluations simply ask LLM judges to give a score; (4) CoSER explores and mitigates bias in LLM judges, such as length bias and dimension-correlation bias.

Q3: Copyright concerns: althought the authors discussed about this in supplementary G, but I am still not sure about whether releasing processed data would by pass the copyright issue. One way of improving this might be releasing the full processing code/pipeline, and let users process on their own book data.

Thanks for your feedback and suggestion. Following your suggestion, we will seek to protect copyrights and release the full processing code to the public.