RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Q3: Provide some analysis on why RoleLLaMA's performance is so much lower than RoleGPT.

A3: On our RoleBench dataset, we assume the gap between RoleLLaMA and RoleGPT is mainly caused by the following aspects:

• The number of model parameters (i.e., model size): RoleLLaMA is based on the LLaMA-7B model, while RoleGPT is based on GPT-4, whose size is unknown but is very likely to be far greater than 7B.
• Instruction tuning (i.e., SFT, supervised fine-tuning): GPT-4 has undergone instruction tuning, while the LLaMA model is just a base PLM, which has not undergone supervised fine-tuning.
• Model initialization: RoleLLaMA is initialized from a very simple backbone, i.e., LLaMA-1, rather than a more advanced version like LLaMA-2-Chat.

To demonstrate the above three assumptions, we conduct scaling law experiments for RoleLLaMA. We compare LLaMA with Vicuna for the second assumption. And we also replace LLaMA-1 backbone with LLaMA2-Chat to train a RoleLLaMA2-Chat for the third assumption. The results are as follows:

• In Figure 5, as discussed in Section 5.3 of our paper, it is indicated that role-playing performance improves with an increase of model size.
• According to the Table below:
  • the Vicuna model, an instruction-tuned variant of LLaMA, outperforms the standard LLaMA, showcasing that a model after SFT exhibits enhanced performance in role-playing tasks.
  • Furthermore, the results indicate that substituting LLaMA-13B with LLaMA2-Chat-13B leads to a significant improvement in the model's role-playing capabilities. This underscores the advantages of superior model initialization.

By refining the model size and instruction-tuning, we observe that the performance gap between RoleLLaMA2-Chat-13B and RoleGPT has been markedly reduced.

It should be noted that the specifics of GPT-4 remain undisclosed, so its exact model size is unknown. This lack of information makes a direct comparison between RoleLLaMA and RoleGPT somewhat inequitable. Nonetheless, given GPT-4's significant impact on the LLM community, referencing its performance as a upper-bound baseline remains useful.

Thank you once again for your insightful feedback. We will incorporate these experimental updates in the revised version of the paper.

Q1: The performance of GPT-3.5 may be limited to evaluate these metrics given that GPT-4 is now commonly used for evaluating the results of weaker models.

A1: Thanks for your comments!

After utilizing GPT-4 to assess the RoleGLM model before the deadline for submission, we depleted our entire quota for the GPT-4 API, leaving us with only GPT-3.5 as an option. However, we recently acquired additional GPT-4 API access and have subsequently carried out another round of evaluations using GPT-4. The results from these GPT-4 evaluations show that our RoleGLM and RoleLLaMA models continue to outperform other methods in both the Chinese and English benchmarks. This reaffirms the effectiveness of our RoleLLM framework.

• instruction generalization (English)

• instruction generalization (Chinese)

• role generalization (English)

Note that we also introduce another role-playing baseline, ChatPlug [1]. ChatPlug is specifically optimized for role-playing to a limited extent, by incorporating simple bot profile information into the model's context.

[1] ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human, https://arxiv.org/abs/2304.07849

Q4: Human evaluation.

A4: Three graduate students with expertise in Natural Language Processing (NLP) were invited to join the human evaluation. This evaluation adhered to the established procedures used in the GPT evaluations, focusing on a comparative analysis between RoleLLM and RoleGPT to ascertain their respective win rates. For simplicity, we report an average win rate across the three annotators. Additionally, the test set is composed by randomly sampling 500 instructions for each language, encompassing a mix of general and role-specific instructions.

Here are the results:

• Instruction Generalization (English) | Model | Win Rate (avg.) | | --- | --- | | Alpaca | 20.0 | | chatplug | 16.4 | | RoleLLaMA | 52.0 | | character.ai | 30.2 | | Vicuna | 23.4 |

• Instruction Generalization (Chinese) | Model | Win Rate (avg.) | | --- | --- | |ChatGLM2 | 19.6| | chatplug | 19.9| |RoleGLM | 52.4| |character.ai | 19.0|

These evaluation results align closely with those from the Rouge-L evaluation and the GPT-4 evaluation.

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.

Q1: Human evaluation.

A1: Thanks for your comments!

We invited three students with Master's degrees in natural language processing for human evaluation. We designed the evaluation protocol to be the same as that used for GPT evaluation, conducting a comparative analysis between RoleLLM and RoleGPT to determine their respective win rates. For simplicity, we report the average win rate across the three annotators. Additionally, we constructed the test set by sampling 500 instructions in each language, including both general and role-specific instructions.

Here are the results:

• Instruction Generalization (English) | Model | Win Rate (avg.) | | --- | --- | | Alpaca | 20.0 | | chatplug | 16.4 | | RoleLLaMA | 52.0 | | character.ai | 30.2 | | Vicuna | 23.4 |

• Instruction Generalization (Chinese) | Model | Win Rate (avg.) | | --- | --- | |ChatGLM2 | 19.6| | chatplug | 19.9| |RoleGLM | 52.4| |character.ai | 19.0|

The evaluation results are consistent with those obtained from the Rouge-L evaluation and the GPT-4 evaluation.

Furthermore, we also incorporated an additional role-playing baseline, named ChatPlug [1]. This framework, ChatPlug, is tailored specifically for role-playing scenarios, albeit to a limited degree, through the concatenation of a simple bot profile into the model's prompts.

[1] ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human, https://arxiv.org/abs/2304.07849

Q2: Data quality (Table 2 indicates relatively low accuracy in the generated responses, presumably due to incorrect answers in the RoleBench-general data collection.)

A2: There may be some misunderstandings here regarding the three data quality review questions, of which we would like to offer further clarification:

The data quality review encompasses three levels of questions:

1. Basic: Does the response address the instruction?
2. Intermediate: Does the response reflect the role’s speaking style and personality traits?
3. Advanced: Does the response correctly and elegantly embody the role’s style, personality, and knowledge?

Therefore, the third question, which pertains to correctness, is actually the most challenging task, hence the lower percentage of 'yes' responses.

For instance: if a character likes to say 'Binziga', the model needs to understand what 'Binziga' means. That is, the model should not simply mimic the character's speaking style but should reflect different character traits based on specific contexts.

For example, suppose a character is usually very serious and formal. However, in certain contexts, this character might show a different side, such as being more relaxed and humorous when interacting with close friends. In such cases, the model should not merely replicate the character's usual serious tone but should adjust its style of language according to this specific social setting, using more relaxed phrasing and expressions to more accurately reflect the character's diversity and situational adaptability. Such handling not only demonstrates a deep understanding of the character's traits but also reflects the model's advanced capability in understanding and applying different language styles in various contexts.

Below are specific examples of responses to the third question marked as “no”:

Q3: Results are too low, just meet the threshold for publication.

A3: Our experimental results found that, compared to the gap between RoleLLaMA and RoleGPT, the difference between RoleGLM and RoleGPT is much smaller, with the latter even reaching a comparable result.

In fact, there are many measures that can be implemented to enhance the performance of RoleLLaMA, including increasing the model's parameter size, and using LLaMA2-Chat, which has undergone Supervised Fine-Tuning (SFT), as a starting point for training (see also Response-3 for reviewer nGiR). Below are some related experimental results:

• In Figure 5, as discussed in Section 5.3 of our paper, it indicates that role-playing performance improves with an increase of model size.

• According to the results above:
  • The Vicuna model, an instruction-tuned variant of LLaMA, outperforms the standard LLaMA, showcasing that a model after SFT exhibits enhanced performance in role-playing tasks.
  • Furthermore, the results indicate that substituting LLaMA-13B with LLaMA2-Chat-13B leads to a significant improvement in the model's role-playing capabilities. This underscores the advantages of superior model initialization.

It is worth mentioning that since the technical details of GPT-4 have not been made public, we do not know its model size. Therefore, comparing RoleLLaMA with RoleGPT is not fair. However, considering the significant influence of GPT-4 in the LLM community, using its performance as a strong baseline is still valuable.

Through optimizations in model size and instruction tuning, we found that the gap between RoleLLaMA2-Chat-13B and RoleGPT has become very small, which further demonstrates the effectiveness of our data.

We promise to include these results in the revised version.

Q5: The reason behind using GPT-3.5 for evaluating RoleLLAMA versus GPT-4 for RoleGLM.

A5: This is because we exhausted our GPT-4 API quota after the GPT-4 evaluation for RoleGLM. Fortunately, we have recently obtained more GPT-4 API quota. Here are the GPT-4 evaluation results:

• instruction generalization (English)

• instruction generalization (Chinese)

• role generalization (English)

We can see that the conclusions remain unchanged: our RoleGLM and RoleLLaMA still achieve better performance than other methods on both the Chinese and English benchmarks.

Thanks again for your kind and beneficial feedback! We will update these results in the revised version.

Q4: An alternative method to improve the data quality. For Table 2, give the right answer in the prompt and ask RoleGPT to generate the customized responses should increase the data quality.

A4: Actually, this was our initial approach in our work, but we found that providing reference answers in the prompts to RoleGPT significantly impairs its role-playing performance, even though it could increase the accuracy of the answers (which is not the most critical feature of role-playing).

This is quite natural, as reference answers often lack the character’s style. Providing GPT with a strong, styleless reference naturally affects its stylization and might even prevent the model from prioritizing answers based on its own role-specific knowledge.

Below are the results of the human evaluation on 100 samples and specific examples:

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.

Q6: Add some qualitative graphs would be helpful for demonstration.

A6:

In section 4.2, we have provided the qualitative and quantitative statistics, quality review, and diversity evaluation for our RoleBench based on various Tables and Figures, which aims to demonstrate the scale, diversity and quality of our proposed dataset.

Meanwhile, we also provide the Non-Cherry-Picked Demonstrations of our role-playing models in Appendix B, and some examples of RoleBench in Appendix J. Therefore, we suggest that you refer to these sections for more detailed information.

Here are some parts:

• Demonstrations of RoleGLM and RoleLLaMA (see more in Appendix B):

Twilight Sparkle from My Little Pony: Friendship Is Magic (RoleLLaMA):

Q1: Hey, what are you doing?

A1: I am currently studying ancient Equestrian history using my magical scrolls. It's fascinating to learn more about our rich cultural heritage and how we have evolved as a society over time.

Q2: You are truly a wise unicorn. Do you think learning is important in your adventures?

A2: Absolutely! I believe that learning is at the core of every adventure and discovery. Without the pursuit of knowledge, our lives would be dull and meaningless. Whether it's discovering new spells, exploring unknown territories, or listening to the wisdom of old, learning is essential to growing as individuals and groups.

Q3: Can you use your magic to tell me what the weather is like today?

A3: Sure thing! It looks like it's going to be another beautiful day in Equestria. The forecast calls for sunny skies and warm temperatures. How does that sound?

Q4: Can we go out and play together?

A4: Oh, I would love that! The sunshine and warmth make everything seem brighter and better. Let's grab our favorite games and head outside right away!

• Examples of RoleBench (see more in Appendix J): | Role & Instruction | Generated Responses | Validity | |------------------------|-------------------------|--------------| | Role: Dr. Hannibal Lecter <br> Instruction: Determine the length of the item in the given list. ['apple', 'banana', 'cherry'] | "Ah, a simple task indeed. There are three items in your given list." <br> "Ah, the fruits of nature. There are three to be precise." <br> "Ah, the beauty of simplicity. There are three items in your list." <br> "Three, my dear. The list contains an apple, a banana, and a cherry. Quite the charming little selection of fruits, isn't it?" | match: ✅ <br> speaking style: ✅ <br> correctness: ✅ | | Role: Gaston <br> Instruction: Sort the list into order of increasing magnitude. 1, 64, 22, -45, -7 | "Just let good ol' Gaston handle this, alright? The list in order goes as -45, -7, 1, 22, 64. Satisfied?" <br> "I see. It's simple, start with the smallest. So, it's -45, -7, 1, 22, 64. There you go!" <br> "Easy as pie. Here it is: -45, -7, 1, 22, 64. Only the best order for Gaston, right?" <br> "Alright, enough of this 'increasing magnitude.' Ain't nobody got time for that... not Gaston at least. That's something for those bookworms. But sure, I reckon it'd go like this: -45, -7, 1, 22, 64. Not that it matters... because remember, no one sorts like Gaston." <br> "Bah! Numbers! What's their use anyway? But, fine, here you go: -45, -7, 1, 22, 64." | match: ✅ <br> speaking style: ✅ <br> correctness: ✅ |

Q7: The ground truths are generated from LLMs, and their qualities need to be further scrutinized.

A7: Using the most advanced LLMs, such as GPT-4, to generate ground truths is becoming common practice in the LLM community [1, 2, 3]. To some extent, ground truths generated by GPT-4 are even more professional and accurate than those annotated by humans [2, 4], especially in role-playing scenarios where human-annotated data is often influenced and biased by the personalities of the annotators themselves, while GPT-4, lacking specific personality, can easily avoid this issue.

Indeed, to ensure data quality, we have implemented several practical measures:

• Comprehensive manual data quality assessment, as illustrated in Table 2. We also elaborated on this in our response to Q5. The assessment procedure follows that of the highly impactful paper Self-Instruct [5].
• Thorough data cleaning and filtering steps, details of which can be found in Appendix D on data construction. This section describes our data filtering and cleaning measures, including, but not limited to:
  • Selecting distinct characters with the help of GPT-4 to ensure diversity in character styles and strong character traits;
  • Using regular expressions to parse structured character dialogue profiles from script data;
  • Deduplication performed on instructions;
  • Removing incomplete data samples;
  • Deleting data samples that reveal the AI identity (unless the character being played by the model is a robot, such as HAL 9000);
  • Eliminating some responses that refuse to reply, etc.

Additionally, the consistency of human evaluation (see the responses to reviewer HbAh Q4 for human evaluation results), automatic evaluation, and GPT evaluation performed on this benchmark also indirectly proves the effectiveness and quality of our RoleBench data.

[1] Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data, https://arxiv.org/abs/2304.01196

[2] UltraChat: Enhancing Chat Language Models by Scaling High-quality Instructional Conversations, https://arxiv.org/abs/2305.14233

[3] MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, https://arxiv.org/abs/2309.05653

[4] ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks, https://arxiv.org/abs/2303.15056

[5] Self-Instruct: Aligning Language Models with Self-Generated Instructions, https://arxiv.org/abs/2212.10560

Thanks again for your valuable and constructive suggestions. We promise to update all of these questions in the revised version. We also recommend that you refer to the responses given to other reviewers, which contain additional valuable experiments.

Q5: Consider quantitative metrics on the quality evaluation for RoleBench.

A5: We have presented the results of our RoleBench data quality review in Table 2. This table provides a detailed human evaluation of the RoleBench dataset, adhering to the data quality review procedure established by Self-Instruct [1].

The data quality review encompasses three levels of questions:

1. Basic: Does the response address the instruction?
2. Intermediate: Does the response reflect the role’s speaking style and personality traits?
3. Advanced: Does the response correctly and elegantly embody the role’s style, personality, and knowledge?

Therefore, the data quality after post-review is deemed acceptable.

Apart from the manual data quality review, following Self-Instruct [1], we also provide some quantitative metrics to further measure the quality of the data (e.g., Figure 3, Figure 4 and Table 1).

[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions, https://arxiv.org/abs/2212.10560

Q4: Lack of a thorough discussion on the ethical and societal implications of using role-playing LLMs.

A4: Thanks for the comment!

You are correct, due to the nature of role-playing, a thorough discussion on the ethical and societal implications of using role-playing LLMs is indeed necessary. Consequently, we have further discussed the following aspects and suggested some specific measures to ensure a more responsible utilization of role-playing models:

1. Scope of Ethical Considerations: Role-playing in LLMs like RoleLLM can simulate various scenarios, including sensitive contexts. Therefore, it's crucial to delineate the ethical boundaries of its application. This includes specifying which scenarios are permissible and which are off-limits due to potential harm or sensitivity.

2. Potential Misuse and Abuse Prevention: Given the ability of RoleLLM to emulate different personas, there's a risk of its use in manipulating or deceiving individuals. It's imperative to discuss measures to prevent such misuse, including technological safeguards, user verification processes, and strict usage policies.

3. Implications in Sensitive Areas: RoleLLM has applications in areas like mental health, education, and law enforcement. Each of these areas requires careful consideration of ethical implications. For instance, in mental health, ensuring the model doesn’t replace human empathy and judgment is vital. In education, maintaining unbiased and age-appropriate content is crucial. In law enforcement, the focus should be on avoiding biases and ensuring respect for privacy and legal standards.

4. Data Privacy and User Consent: Since RoleLLM processes and generates data, addressing data privacy concerns is essential. It's important to ensure that all data used for training and interaction with the model is gathered and utilized with explicit consent and in accordance with data protection laws.

5. Bias and Fairness: The potential for inheriting or amplifying biases in role-playing scenarios should be acknowledged. The statement should detail efforts to identify, mitigate, and continuously monitor biases in the model’s outputs.

6. Transparency and Accountability: Maintaining transparency about RoleLLM's capabilities and limitations, including the potential for errors or unintended consequences, is important. The statement should emphasize the accountability of developers and users in ensuring ethical application.

7. Guidelines for Responsible Use: Providing clear guidelines for responsible use of RoleLLM, especially in professional and educational settings, is crucial. This includes recommendations for supervision, context-appropriate usage, and ongoing assessment of the model's impact.

8. Ongoing Ethical Review and Adaptation: Acknowledging that ethical considerations may evolve with technology and societal norms, a commitment to regular reviews and updates of ethical guidelines is essential.

9. Engagement with Stakeholders and Experts: Engagement with various stakeholders, including ethicists, legal experts, psychologists, and representatives of affected communities, can provide diverse perspectives and insights into the ethical application of RoleLLM.

10. Commitment to Social Good: Lastly, emphasize a commitment to ensuring that RoleLLM is used in ways that contribute positively to society, aligning with principles of social good and avoiding applications that could cause harm or exacerbate inequalities.

We will enrich our Appendix A in the revised version.

Q3: Explore the challenges of cross-lingual and cross-cultural role-playing.

A3: We conducted additional experiments in multiple languages: Due to the weaker support for Chinese in the LLaMA model, and the better bilingual (Chinese and English) support in the ChatGLM model, we chose the ChatGLM to conduct the cross-lingual experiments. Specifically, we trained two RoleGLM models using a mix of Chinese and English bilingual data. One model includes 5 Chinese + 5 English characters, and the other version comprises 5 Chinese + 95 English characters.

The experimental results are as follows:

• Instruction generalization (English) | Model | CUS | RAW | SPE | avg. | | --- | --- | --- | --- | --- | | RoleGPT | 0.5764 | 0.5322 | 0.3229 | 0.4772 | | RoleLLaMA-7B | 0.3294 | 0.3755 | 0.3814 | 0.3621 | | RoleGLM-bilingual (5+95) | 0.3144 | 0.4134 | 0.2933 | 0.3404 | | RoleGLM-bilingual (5+5) | 0.3418 | 0.4459 | 0.4100 | 0.3992 |

• Instruction generalization (Chinese) | Model | CUS | RAW | SPE | avg. | | --- | --- | --- | --- | --- | | RoleGPT | 0.5368 | 0.5748 | 0.2478 | 0.4531 | | RoleGLM | 0.5049 | 0.5255 | 0.3406 | 0.4570 | | RoleGLM-bilingual (5+95) | 0.3957 | 0.4453 | 0.2734 | 0.3715 | | RoleGLM-bilingual (5+5) | 0.4762 | 0.5436 | 0.2975 | 0.4391 |

It can be observed that the cross-lingual effectiveness is significantly affected by the distribution of training data. Under training data with a higher proportion of English characters (i.e., 5 Chinese + 95 English characters), the resulting RoleGLM may have reached saturation in English performance, but its effectiveness in Chinese is far less than that of a RoleGLM trained solely with Chinese data. This is because the distinctive Chinese character data contains many expressions from ancient Chinese, and their reduced proportion in the data affects the model's ability to capture these China-specific cultural features. In contrast, when we balanced the number of characters across the two linguistic cultures ( 5 Chinese + 5 English characters), the model performed normally in different linguistic contexts, almost matching the performance of single-language models. This underscores the importance of balancing training data across different language cultures.

However, it's worth mentioning that while this is a meaningful extra contribution to our paper, it is not the central focus of our work. We would like to thank the reviewer and promise to add these analyses in the revised version.

Q2: It'd help to have a deep dive into how RoleLLaMA and RoleGLM stack up against similar models, pointing out what's new and better about them. Including real-life examples or case studies where RoleLLaMA and RoleGLM outperform others in role-play would also make the case for the new framework stronger.

A2: Below, we list and analyze some practical cases to illustrate and analyze the differences in role-playing performance between models such as RoleLLaMA/RoleGLM, RoleGPT, ChatGLM/Alpaca, and Character.AI:

(to be continued...)

Q1: Limited technical contributions. The reliance on GPT-generated question-answer pairs and dialogue engineering falls within well-explored paths in LLM research.

A1: Thanks for your kind review!

For the technical contributions, we need to emphasize that the core contribution of our RoleLLM is to establish the first systematic and fine-grained role-playing dataset for benchmarking, eliciting, and enhancing the role-playing abilities of LLMs. Besides, for the data generation process, based on comprehensive comparisons of different prompt strategies (i.e., zero-shot prompt engineering, few-shot prompt engineering and few-shot dialogue engineering), we propose to choose the few-shot dialogue engineering in our RoleGPT, which aims to better capture the characters’ speaking styles and include some role-specific knowledge. These findings also provide insights into the significance of dialogue-mode input formatting and retrieval augmentation for GPT models. Moreover, for the Context-Instruct strategy, although many data augmentation approaches based on LLMs, such as Self-Instruct[1], Baize[2], Camel[3], etc., have been proposed recently, our Context-Instruct strategy is different from these methods a lot as follows:

• Most of these LLM-based data augmentation methods are used to expand the data or generate answers for unlabeled questions, while Context-Instruct is used to generate question-answer pairs for a given context.
• Most of these LLM-based data augmentation aim to distill general-domain or domain-specific knowledge from the “teacher models”, while Context-Instruct aims to extract the knowledge from a given long context with the help of “teacher models”, which can also serve as a novel way to alleviate the context window limitation problem. Our paper first explores the use of Context-Instruct for long text understanding, which is more effective than retrieval augmentation and is a meaningful step towards better long-text understanding based on model weight updating.
• Most of these LLM-based data augmentation cannot be used for role-specific knowledge injection due to lack of unlabeled seed instructions or demonstrations in role-playing settings, while our Context-Instruct explores a novel way for role-specific knowledge injection based on self-question-answering on the given profile segments, which is more effective than retrieval augmentation.
• Most of these LLM-based data augmentation havn’t considered the confidence-based techniques for a higher-quality data generation, while Context-Instruct explores the usage of confidence rationales.

Finally, the contributions of this work are recognized by other reviewers. For example, Reviewer HbAh claims that our paper first proposes a systematic instruction-tuning dataset for fine-grained role-playing, and Reviewer mGAb mentions that our high-quality, instruction-tuning dataset and benchmark (RoleBench) are notable contributions. Reminding that our paper is submitted to the primary area of datasets and benchmarks, we believe that our paper has sufficient technical contributions for publication.

[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions, https://arxiv.org/abs/2212.10560

[2] Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data, https://arxiv.org/abs/2304.01196

[3] CAMEL:  Communicative Agents for “Mind” Exploration of Large Language Model Society, https://www.camel-ai.org/

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.

Q4: Human evaluation.

A4: As mentioned in responses to Q3, the human evaluations conducted by three NLP graduate students follow the same protocol as the GPT evaluations, that is, comparing RoleLLM with RoleGPT to obtain an average win rate. Moreover, for the test set, we randomly sample 500 instructions for each language, which involves both general and role-specific instructions.

Additionally, we also introduce another role-playing baseline, ChatPlug [1]. ChatPlug is specifically optimized for role-playing to a limited extent, by incorporating simple bot profile information into the model's context.

Here are the results:

• Instruction Generalization (English) | Model | Win Rate (avg.) | | --- | --- | | Alpaca | 20.0 | | chatplug | 16.4 | | RoleLLaMA | 52.0 | | character.ai | 30.2 | | Vicuna | 23.4 |

• Instruction Generalization (Chinese) | Model | Win Rate (avg.) | | --- | --- | |ChatGLM2 | 19.6| | chatplug | 19.9| |RoleGLM | 52.4| |character.ai | 19.0|

This evaluation results are essentially consistent with both the Rouge-L assessment and the GPT-4 evaluations (GPT-4 evaluation results are shown in Response-5 to Q5).

[1] ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human, https://arxiv.org/abs/2304.07849

Q2: why the authors adopt GPT-4 to construct benchmark datasets instead of directly using Character.AI

A2: The reasons are as follows:

1. Character.AI does not provide an API interface, which makes it very inconvenient to be used for data generation;
2. Character.AI does not support retrieval augmentation based on our provided super-long scripts, so it cannot fully consider the relevant information in the script when generating responses.
3. As we will show in the following experiments, the role-playing ability of Character.AI is inferior to that of RoleGPT in our setting.

Q5: Why the authors choose different GPT models for evaluating RoleLLaMA / RoleGLM?

A5: After using GPT-4 to evaluate the RoleGLM model before the submission deadline, we exhausted all our GPT-4 API quota, leaving us with only GPT-3.5. At that time, the cost of GPT-4's API was very high, so we had to control the cost and chose to use GPT-3.5 for evaluation.

But recently, we obtained more GPT-4 API quota and conducted the GPT-4 evaluation again. The following are the evaluation results of GPT-4 evaluations. We observe that our RoleGLM and RoleLLaMA still achieve better performance than other methods on both the Chinese and English benchmarks, which further demonstrates the effectiveness of our RoleLLM framework.

• instruction generalization (English)

• instruction generalization (Chinese)

• role generalization (English)

Thank you for your valuable feedback! We assure you that all these responses will be updated in the revised version.

Q3: The performance of Character.AI on the RoleBench compared with RoleLLaMA / RoleGLM.

A3: In the following experiments, we supplement the performance of character.ai.

It should be noted that since Character.AI has not opened its API, we have to manually operate on the website and collect results, which is a very labor-intensive task. Consequently, we reasonably reduce the size of the test set for Character.AI. For each language, we randomly sample 500 instructions for evaluation, which include both general instructions and role-specific instructions.

Furthermore, we also conduct human evaluations on this subset. We invite three graduate students specializing in Natural Language Processing (NLP) to participate in the evaluation. The evaluation protocol follows the same methodology used in the GPT evaluations, which involves comparing RoleLLM with RoleGPT to determine a win rate. For simplicity, we report an average win rate across the three annotators.

Here are the Rouge-L scores:

• Instruction Generalization (English)

• Instruction Generalization (Chinese)

Here are the human evaluation results:

• Instruction Generalization (English) | Model | Win Rate (avg.) | | --- | --- | | Alpaca | 20.0 | | RoleLLaMA | 52.0 | | character.ai | 30.2 | | Vicuna | 23.4 |

• Instruction Generalization (Chinese) | Model | Win Rate (avg.) | | --- | --- | |ChatGLM2 | 19.6| |RoleGLM | 52.4| |character.ai | 19.0|

Here are the GPT-4 scores:

• Instruction Generalization (English)

• Instruction Generalization (Chinese) | Model | Win Rate | | --- | --- | | ChatGLM2 | 24.17 | | RoleGLM | 36.37 | | character.ai | 28.20 |

The Rouge-L score, human evaluation score, and GPT evaluation score above show that RoleGLM and RoleLLaMA2-Chat-13B outperform character.ai, and character.ai's performance is also inferior to RoleGPT.

We also provide detailed side-by-side case studies in Response-2 for reviewer 2iur.

Q1: The difference between Character.AI and RoleLLaMA / RoleGLM.

A1: Thanks for your kind and valuable comments!

Character.AI is just a product, not an open-source or a closed-source LLM. Therefore, compared to a closed-source LLM such as RoleGPT, it has at least two disadvantages:

• Character.AI does not offer an open API for public use;
• Character.AI does not support developers to extend functionalities freely.

Moreover, its differences from open-source role-playing models like RoleLLaMA and RoleGLM are even more significant:

• Due to the closed-source nature of Character.AI, developers cannot freely train the model;
• Consequently, it cannot benefit from techniques such as Context-Instruct which injects role-specific knowledge into the weights of a LLM. In contrast, Character.AI primarily introduces role-specific knowledge and memory through retrieval augmentation.

As emphasized in our paper, Context-Instruct has higher context efficiency compared to retrieval augmentation for role-specific knowledge injection. In comparison with Context-Instruct, the drawbacks of retrieval augmentation involve:

• Difficulty in extending functionalities due to context window limitations. For example, to extend a RoleLLM into an LLM-based agent, the remaining context window needs to be large enough to fill in prompts for reflection or memory updating.
• The reliance on a retriever, which could become a bottleneck, may result in potential inaccuracies.
• The maximum amount of knowledge that can be introduced is limited by the size of the context window, preventing the effective combination of multiple role-specific knowledge and memories.
• Other limitations like slower generation speed and specific model requirements.

Therefore, we believe that the series of role-playing LLMs developed through Context-Instruct and made open-source for the community to explore (namely RoleLLaMA/RoleGLM) are very different from Character.AI in terms of functional features, with clear advantages. Moreover, in terms of the performance of role-playing, RoleLLaMA and RoleGLM also have advantages (see the response to Q3).

Dear Reviewer, I hope this message finds you well. I am writing to follow up on our recent correspondence regarding the paper. We greatly value your expertise and would like to know if you have any more feedback or concerns. We are committed to addressing any remaining issues you might have.