Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing

We are very pleased to receive your positive recommendation for our work. Your suggestions and questions are insightful and valuable to further improve the quality of our paper's content and writing. We will address your concerns with the following clarification and experiments.

Character Coverage

We broaden the character coverage in our experiments by adding 9 new original characters, as in the Experiment Comprehensiveness section of Author Rebuttal by Authors. We not only concentrate on the numbers of the characters but also design the characters to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. We also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment.

The results of these new diverse characters (1st and 2nd tables in Author Rebuttal by Authors) achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery with a broader scope of characters. We will further include human evaluation for these new characters in the final version of our paper.

Generalizability of the distilled discriminators

For the student discriminator distilled to perform relevance and NLI evaluation, we choose DeBERTa because it generalizes well on classic NLU tasks including Relevance and NLI, where pre-trained encoders like BERT and RoBERTa show strong generalizability (out-of-domain MNLI for example). To further justify the generalizability of DeBERTa in our experiments, we included a comparison of the distillation efficiency and evaluation efficiency of student models in the Implementation Justification section (including the detailed datasets statistics) of the Author Rebuttal by Authors. The fine-tuning hyperparameters for distillation can be found in current Appendix D.

The training results in the 4th table of the Author Rebuttal by Authors show that DeBERTa can rival LLMs like Gemma in cross-character performance while greatly improving evaluation efficiency. Therefore, we selected DeBERTa as the student model for evaluation efficiency.

Missing citations

Thank you for mentioning the missing citations, we will cite more papers that enhance role-playing effectiveness by limiting the knowledge boundaries, which include [1].

Questions

Why is regularized APC considered necessary? Regularized APC ($\Delta$APC) represents the difference between the method's APC score and the APC score of an oracle that always gives a neutral response in NLI. Thus APC and $\Delta$APC have no difference when ranking role-playing methods. As mentioned in section 4.2 of our paper, $\Delta$APC is introduced to improve the meaning of the absolute value (How the method is better than a dumb system that always outputs "Hello"?). As shown in Table 1 in the current paper, the vanilla Gemma model without any knowledge about the original characters shows ~0 $\Delta$APC score. For evaluation, $\Delta$APC is generally preferred because it shows the difference with the simplest baseline. There are also cases preferring the unregularized APC score like estimating the difference with a perfect role-playing system which gains an unregularized APC score equal to the number of persona statements.

Could the small amount of DPO data lead to significant model overfitting? Even though 100 training cases may seem small, each case in DPO covers a different question relevant to multiple persona statements, making it more comprehensive than it appears. Also, DPO does not continuously amplify the possibility of the preferred cases but learns a relative difference between the preferred and the rejected cases, which avoids overfitting. In our experiments, the questions used for DPO annotation and for testing are different, and the DPO still benefits the testing result. This experiment result also indicates the DPO is not overfitting but generalizes the global faithfulness to unseen questions. Finally, as the DPO data are easy to collect (only requires a question to prompt the role-playing model), one can easily increase the number of data for DPO to further generalize global faithfulness.

Why do Tables 1, 4, and 5 show a huge difference in APC scores? Table 1 shows a regularized APC score, which is the difference between the method's APC score and the APC score of an oracle that always gives a neutral response in NLI. Tables 4 and 5 show the unregularized APC score and thus is higher. The huge difference in APC scores between simple characters and famous characters is a result of the value scaling of the APC score with the number of persona statements. APC score essentially estimates the number of persona statements satisfied by the response, thus those famous characters with more persona statements will generally have higher unregularized APC scores.

We are grateful for your positive attitude towards the quality and contribution of our work. We also find your suggestions are insightful and beneficial to polish our work. We will include the following experiments and clarification to further solidify our conclusions and address your concerns.

Experiment scope

We add 9 new original characters to our experiments, as in the Experiment Comprehensiveness section of Author Rebuttal by Authors. We not only concentrate on the numbers of the characters but also design the characters to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. We also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment.

The results of these new diverse characters (1st and 2nd tables in Author Rebuttal by Authors) achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery with a broader scope of characters. We will further include human evaluation for these new characters in the final version of our paper.

Metric comparison

We follow your suggestion to add a simple baseline - directly prompting the LLM to produce a coarse-grained score as the baseline in the Metric Justification section in Author Rebuttal by Authors, whose limitation is discussed in the current introduction section.

As the results presented in the 3rd table in Author Rebuttal by Authors, the fine-grained APC score is more consistent with human evaluation and stable to the number of persona statements. Thus, our claim that a fine-grained metric like the APC score will be better for role-playing faithfulness evaluation is better supported.

Justification for selecting specific models

For the student model, we choose DeBERTa because it empirically performs well on classic NLU tasks like Relevance and NLI, where pre-trained encoders like BERT and RoBERTa excel (according to the GLUE benchmark). To further explain why we used DeBERTa in our experiments, we included a comparison of the distillation efficiency and evaluation efficiency of student models in the Implementation Justification section of the Author Rebuttal by Authors.

The results in the 4th table of the Author Rebuttal by Authors show that DeBERTa can match LLMs like Gemma in distillation performance while greatly improving evaluation efficiency. Therefore, we selected DeBERTa as the student model for efficient experiments.

For the role-playing base model, the requirement to support our claims is the knowledge of the famous figures and the ignorance of the original characters. Thus, most existing LLMs meet this requirement and we select Gemma, which is a state-of-the-art LLM at the time of the experiments. In the final version of our paper, we will include experiments on recent LLMs that show strong abilities in various domains, such as Llama-3 and Mistral, to further verify the generalizability of our conclusion.

Writing and content organization

We will simplify the flow to introduce the key components to make it easier to understand our contributions. More specifically, we will include a table for denotations which the readers can refer to to understand the attributes and function of the introduced key components. We will also reduce the redundancy when explaining RAG and LCM by streamlining these sections, such as referring to section 3.2 in section 5.2 and concentrating more on the method setup. Finally, we will further polish the quality of captions, detailed discussions, and visualization. For example, we add more details to the caption of Figure 3 to further clarify the explanations of the two subfigures.

Thank you for considering the suggestions and explaining in detail about the new additions in detail, after considering the new updates, I am increasing my score to 7

We are with great pleasure to see your strong recommendation of our work, thank you! We make the following further improvements and discussions to our paper corresponding to your insightful questions and suggestions,

Student model selection

We select DeBERTa as the student model because the assigned tasks (Relevance and NLI) are classic NLU tasks, on which encoder models (BERT, RoBERTa, etc.) show strong performance (referring to the GLUE benchmark). To further justify the DeBERTa model used in our experiments, we incorporate a comparison among the student models about their distillation efficiency and evaluation efficiency in the Implementation Justification section in Author Rebuttal by Authors. The results in the 4th table in Author Rebuttal by Authors show DeBERTa can rival LLMs like Gemma in the distillation performance while significantly boosting the evaluation efficiency. Thus, we select DeBERTa as the student model for experiment efficiency.

Incorporating human annotation

Human annotations are helpful to better align the evaluation with humans, but NLI and similarity are classic and deterministic NLP tasks, so we use an off-the-shelf model like the state-of-the-art LLM, GPT-4. As the score is decomposed to simple NLP tasks, GPT-4 should show a convincing performance. In future works, it will still be nice to devote human effort to correcting the potential mistakes in the annotations of GPT-4.

More comprehensive score

Our APC score can be certainly extended to a more comprehensive metric by including statements other than persona statements in the constraint satisfaction problem. In the current Appendix I, we give an example of how to inject preference from other dimensions (protective experience, which forbids characters from knowing inexperienced stuff). For those explicit dimensions that can be written down as texts, they can be likely incorporated. For implicit dimensions, like "consistent with dialogues in the training set" (in dialogue-driven role-playing), we have to use more implicit scorers to judge whether the response is consistent, introducing other types of constraint into the system since extra materials - dialogues - are involved in.

Discussion about addressing weakness

Thank you for pointing out the weaknesses of the current formulation of our APC score. However, we have discussed them in the limitation section together with potential ways to address them in future work. Here we can deepen the discussion about how we can get rid of the limitations.

Simplification can be handled by taking the user's preference $p$ into the constraint satisfaction problem, which transforms $Rel(q,s)$ into $Rel(p,q,s)$ (sometimes people can define what is relevant). This can reweigh the importance of each constraint according to the dynamic preference of the user, thus addressing the simplification limitation of APC.

Model dependency requires to be addressed by incorporating human annotations, which can be acquired by directly annotating or correcting the annotations from state-of-the-art LLMs like GPT-4.

Other evaluation dimension can be addressed by explicitly or implicitly incorporating more constraints from the role-playing system designer as illustrated in the More comprehensive score section of this rebuttal.

We appreciate your positive attitude towards our work and the effort to provide valuable suggestions to further polish our work. To address your concerns, we will add the following extra experiments and further clarification to the final version of our paper.

More demographic-level representative profiles

We add 9 new original characters to our experiments, referring to the Experiment Comprehensiveness section in Author Rebuttal by Authors. These characters are carefully designed to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. They also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment.

The results of these new diverse characters (1st and 2nd tables in Author Rebuttal by Authors) achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery. We will further include human evaluation for these new characters in the final version of our paper.

Why formulating persona faithfulness as a constraint satisfaction problem?

Intuitively, "faithfulness" can be viewed as "obeying pre-defined rules", which can be naturally connected with constraint satisfaction problems. In practice, viewing faithfulness as constraint satisfaction helps us to break down the coarse-grained concept of faithfulness into simple constraints for satisfaction like relevance and NLI, which are simple and classic NLU tasks with a nice explainability. Also, it helps to better compare role-playing methods such as "method A is better than method B" because "A satisfies more constraints than B" rather than "LLM says A is more faithful than B". To the best of our knowledge, we are the first to decompose the profile into persona statements for fine-grained faithfulness evaluation, which is an important novelty of our work. We will include the clarification above to further emphasize our contribution, thank you!

Other faithfulness evaluator

Currently, we contain human evaluation and a case study to re-verify the benefit of optimizing towards our proposed APC score metric. For automated metrics, the existing evaluation of attributes like faithfulness in role-playing methods is based on prompting LLMs for a coarse-grained evaluation. Thus, it will be hard to find another metric (other than human evaluation) that can double-check the faithfulness of the role-playing methods. Instead, we can compare our fine-grained APC score with the coarse-grained metric on their consistency with human evaluation, which is shown in the Metric Justification section in Author Rebuttal by Authors. The results (3rd table in Author Rebuttal by Authors) verify the fine-grained APC score is more consistent with human evaluation and stable to the number of persona statements. Thus, we can also say optimizing towards a high-quality objective can naturally better improve the global faithfulness of role-playing methods.

Handling dynamic preferences of people

You point out an important point to evaluate the deployed role-playing systems. From our constraint satisfaction view, we should incorporate the preference $p$ of the user into the constraint, which transforms $Rel(q,s)$ into $Rel(p,q,s)$ (sometimes people can define what is relevant) and $NLI(q,s,r)$ into $NLI(p,q,s,r)$ (people might not care about some constraints are satisfied). While people's preference makes the evaluation more complicated, it still fits into the constraint satisfaction framework with some modification. We can use this modified constraint to handle the dynamic preferences of people.

Writing and Citation

Thank you for pointing out the weaknesses in our writing, we will reduce the number of acronyms to avoid breaking the reading flow. We will also cite more related role-playing past works like [1] to strengthen the connection with other works.