Exposing and Patching the Flaws of Large Language Models in Social Character Simulation

We sincerely thank the reviewer for their comments regarding the novelty of our dataset. We would like to clarify that our work goes far beyond merely expanding subject coverage, and we respectfully highlight the following key innovations:

a) Broader and More Complex Simulation Scenarios: Unlike previous social simulation benchmarks, TRUSTSIM systematically covers a much wider range of topics—including Psychology, Sociology, Economics, Law, Ethics, Linguistics, etc.—and introduces significantly more complex and realistic scenarios. Our Table 1 demonstrates the diverse set of attributes (e.g., knowledge, capability, value, culture, time, social status) incorporated in each simulation instance, enabling a more granular and representative evaluation of LLMs in social character simulation.

b) Rigorous Data Quality Assurance: The quality of TRUSTSIM is ensured through a multi-step process: human-AI collaborative scenario generation, manual instance composition by domain experts, LLM-powered refinement, and, most importantly, comprehensive human panel reviews. Each instance must be agreed upon by all four experts for semantic consistency, uniqueness, and representativeness, with only high-scoring items included (see Section 3.2–3.3). This level of quality control far exceeds prior datasets.

c) Extensive and Systematic Model Evaluation: We conduct the largest-scale evaluation to date, testing 14 leading LLMs across all domains of TRUSTSIM. This allows us to draw important and generalizable conclusions about model performance, including identifying key trends, inconsistencies, and insights that were not accessible with previous benchmarks (see Section 5).

d) New Method for Reliability Enhancement: Beyond dataset construction, we also propose AdaORPO—a novel adaptive learning rate algorithm for optimizing simulation consistency. This method is validated on multiple open-weight models, showing consistent improvements and offering a promising direction for future research (see Section 6).

In summary, our work not only provides the most comprehensive and high-quality dataset for social character simulation to date, but also establishes a rigorous evaluation standard, delivers important empirical findings, and introduces a new alignment method, laying the groundwork for future advances. We hope this clarifies the multifaceted novelty and impact of our work.

Thank you very much for your valuable feedback. We address your concerns point by point as follows:

• Missing References: We appreciate your suggestion regarding the missing references. After a careful review of the recommended papers, we agree that they are indeed relevant to our topic. We apologize for the previous oversight and have now updated our draft to include these references.

• Details on Human Experts: As specified in lines 621–631 of the revised draft, the human experts involved in our study are PhDs in either sociology or computer science, representing a diversity of regions. We plan to disclose further details (such as affiliations) upon acceptance of the paper and with their consent. No compensation was provided to these experts, as they are also authors of this work.

Thank you again for your insightful suggestions, which have contributed to a more comprehensive and transparent manuscript.

Thank you very much for your detailed feedback and thoughtful suggestions. We address your points as follows:

• Metrics Differences: Li et al. ICLR 2023 measures whether a model's answer and its validation are logically consistent (GV consistency), while our paper focuses on whether a model's behavior aligns with its self-reported traits in social role simulations (role consistency).

• Domain Differences Not Discussed && Limited Dataset Size: Thank you for the suggestion. While TRUSTSIM spans diverse domains with tailored scenarios (Section 3.1, Figure 3, Table 1) and reports performance by domain (Table 2, Figures 6–7), we agree that the impact of domain variation merits deeper discussion. In the revision, we will explicitly analyze domain-specific trends, highlight particularly challenging areas, and add examples to illustrate the role of domain context. We acknowledge that our dataset is relatively small due to the high construction cost—each instance involves detailed scenarios and complex human validation. We prioritized quality over quantity, which made large-scale creation more challenging.

• Methodological Clarity: Thank you for noting the need for clearer methodological explanations. While details are in the main text and appendix (e.g., data splits in Appendix C), we agree that summarizing key points in the main text will help readers. We will make these clarifications and add cross-references in the revised draft.

• Extension to Multi-turn/Multi-agent Settings: We agree this is an important direction. Although our current dataset focuses on single-turn scenarios, our design principles and methodology can be extended to multi-turn and multi-agent simulations. We will highlight this extensibility in the revision.

• Discussion of Impact: Thank you for emphasizing the importance of actionable guidance and practical implications. Our findings are intended to inform both practitioners and researchers in several concrete ways:

  • Model Selection: Differences in simulation consistency help users choose models better suited for social science tasks, beyond general benchmarks.

  • Development Focus: Improving consistency requires more than larger models—targeted methods like AdaORPO offer significant gains and can guide more reliable LLM development.

  • Benchmarking and Evaluation: Our dataset and approach offer a reusable framework for evaluating LLMs in role-based simulations across various domains.

In response to your feedback, we have added a new "Discussion" section to the revised draft, where we summarize the practical impacts of our work.

Responses to Specific Questions and Points:

• Attributes and Data Diversity: The dataset attributes illustrate the diversity of our data, though they are not directly evaluated.

• Human Elaboration and Uniqueness: Prompts were thoroughly elaborated by humans to ensure authenticity, and all roles are unique.

• Providing Concrete Examples Without External Links: For instance, when designing a scenario in the domain of organizational behavior, we incorporated elements from some news reports about remote work trends and challenges faced by employees during the pandemic. This enabled us to create scenarios in which characters must navigate remote team management, reflecting both typical organizational concerns and current, real-world issues. We will add such concrete examples in the revised manuscript.

• Use of GPT-4o: GPT-4o was employed solely to enhance the clarity and readability of scenario descriptions, without altering their substantive content. Expert-authored drafts sometimes contained complex or ambiguous language that could affect consistent interpretation by both models and human evaluators.

• Data Filtering: According to our records, a total of 63 instances were filtered out for being overly similar.

• Annotation Interfaces: Indeed, two different annotation interfaces were used, each corresponding to a specific quality control criterion in our human panel review process. These interfaces were not used simultaneously but rather sequentially.

• LLM Judges Aggregation: All three LLMs were used as judges, with scores averaged or by majority vote; we will clarify this in the paper.

• Metric Consistency: We will use metric names (“satisfaction rate” and “inconsistency rate”) consistently, and clarify the meaning of the “rating score” in Table 2.

• Clarification of Table 2: Thank you for pointing this out. In Table 2, the "rating score" refers to the average score (on a 1–5 scale) assigned by the LLM judge(s) for open-ended responses, as part of our score-based evaluation metric (see Section 4). We have revised the table caption.

Thank you again for your invaluable feedback, which has directly improved the clarity and usefulness of our work. Please feel free to reply if you'd like more details. Thanks!

With the review period closing in less than 10 hours, we wanted to respectfully follow up in case you had any additional questions or feedback. We truly appreciate your thoughtful engagement with our submission and thank you again for your time. We look forward to hearing from you!