Thank you for your positive feedback. We're glad you found our work significant, original, and well-written. If you have any further suggestions, we'd be happy to consider them. Thank you again for your review.

Thank you for the review and constructive suggestions. Below, we address the main concerns:

Generalization Capability

OneDayLife is so far the only publicly available corpus suitable for evaluating our system, as it provides extensive, multi-session interactions between agents.

Our framework can be effectively applied to other social interaction simulations for several reasons: a) Generic Agent Design: The modules used in our system are common in LLM agent studies, making integration into future works easier. b) Inevitable Errors by LLM Agents: Advanced LLMs may still produce inaccurate content, so the ability to correct such errors in agent interactions is crucial. The three error types we address are common across various social interactions.

Thresholds

Thresholds were set based on empirical observations. We believe these thresholds should be adjustable according to the simulation's objectives rather than being learned exclusively from a specific corpus.

Human Evaluation

We reported the intraclass correlation coefficients to show the consistency of human evaluation. Annotators were not specifically trained; instead, they received the same instructions and data setup as provided to GPT-4, with procedures for shuffling and anonymizing the content. Annotators were compensated at the minimum hourly wage for 20 hours of work.

NLI-G Explanation

If the current dialogue is consistent, it should receive a high score, showing no significant difference between NLI-G and Prev. Conversely, if the current dialogue contradicts histories, the lower the score, the more evidence is provided, indicating a better evidence retrieval ability. In Fig. 7, each data point represents the mean score for each percentile. While both error-free and erroneous dialogues exist, the average score differences between NLI-G and Prev mainly stem from erroneous examples. Thus, a lower average score suggests NLI-G effectively identifies discrepancies in the dialogue.

Prior Work

We will append the related work section.

Answers

Q2. Memory, history, background, and status constitute the context for utterance generation, originating from GenerativeAgents.

Q3. The annotation was done by the author; We manually selected 25 problematic utterances (containing any repetition, inconsistency, or factual error) and 25 issue-free ones. Each has a single binary label.

Q5. We enforce the model to output in a JSON-like format and retry if invalid. Therefore, 100% of them are valid.

Thank you very much for your further inquiry and interest in our detailed responses!

• Human Annotation: The annotators were asked to score the consistency and factualness of the current dialogue on a scale from 1 to 10. While several works use ICC to measure annotator agreement, we provide only a few recent references as follows:

  [1] Speech Corpus for Korean Children with Autism Spectrum Disorder: Towards Automatic Assessment Systems (Lee et al., LREC-COLING 2024)

  [2] A Prompt Response to the Demand for Automatic Gender-Neutral Translation (Savoldi et al., EACL 2024)

  [3] Jigsaw Pieces of Meaning: Modeling Discourse Coherence with Informed Negative Sample Synthesis (Singh, EACL Findings 2024)

  [4] FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (Riley et al., TACL 2023)

  [5] User Simulator Assisted Open-ended Conversational Recommendation System (Zhan et al., NLP4ConvAI 2023)

  [6] Evaluating the use of large language model in identifying top research questions in gastroenterology. (Lahat et al. Scientific Reports, 2023).

  [7] Inducing Positive Perspectives with Text Reframing (Ziems et al., ACL 2022)

  [8] Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments (Maronikolakis et al., ACL Findings 2022)

  We will include these details in our paper.

• More Detailed Responses to the Original Review:

Prior work discussion

We will append the related work section with the recommended LLM agent papers.

We especially explain the major differences between our work and the two suggested papers, LEGO (He et al., 2023) and ToM for Multi-Agent Collaboration (Li et al., 2023):

• Task: We address errors in long-term open-domain dialogues between multiple agents. In contrast, Li et al. (2023) tackle the goal-oriented task of defusing bombs in multiple rooms, while He et al. (2023) combine LLM outputs for causality explanation generation, assigning different roles to LLMs to analyze cause and effect, mine knowledge, generate explanations, and refine the explanations.
• Approach:

  • We develop an error detection and correction framework for each agent’s utterance, while there is no such mechanism in Li et al. (2023). In addition, our task is more challenging. Our agents’ dialogue history and memory contain information about multiple agents, while Li et al. (2023) agents track only task-related information in their belief states, such as bomb locations and colors. Li et al. deal with only 3 agents and 3 colors of bombs in 5 rooms, with clearly defined states to track due to their goal-oriented nature. In contrast, our task involves complex information propagation in open-domain dialogues between 25 agents, each with distinct and varied personal traits. This complexity is particularly challenging given that language models struggle with maintaining self-identity [1]. For instance, the errors illustrated in Figure 1 of our paper are less likely to occur when the agents are discussing bombs.

    [1] Shuster et al., “Am I me or you? State-of-the-art dialogue models cannot maintain an identity.” NAACL-Findings, 2022.

  • We propose NLI-G and use local models to retrieve evidence from past dialogues for potential errors, whereas He et al. (2023) utilize three additional LLM modules to analyze cause/effect and retrieve world knowledge from Wikipedia.

Elements in Ablation study

Q2. Memory, history, background, and status constitute the context for utterance generation, originating from GenerativeAgents.

• Memory: formed by observation and reflection mechanisms, e.g. "People are interested in fostering creativity and innovation."
• History: previous dialogues of the involved agents
• Background: agent’s predefined characteristics such as personality, age, profession, current projects, etc. E.g. "Abigail Chen is a digital artist and animator who loves to explore how technology can be used to express ideas."
• Status: the agent's action before the current dialogue, e.g., "working on her animation project."

We hope we have addressed your concerns. If you have any further questions, please do not hesitate to let us know.

Thank you for your valuable feedback. We appreciate your recognition of our work's quality, clarity, and originality.

Significance and Purpose of This Work

LLMs are widely used in creating dialogue datasets for training NLP models, facilitating decision-making through simulated human interactions, and creating scripts for entertainment. Our study addresses critical issues for these applications.

Human Dialogue Assumption

Your concerns are valid. Yet the highly frequent repetitions, false statements, and inconsistencies in LLM-synthesized dialogues are unrealistic. Prior research generally treats human-annotated dialogues as flawless. Following your advice, we will discuss these challenges in the paper.

Response to Questions

1. In Fig. 1, different colors indicate different agents, with bold-colored phrases identifying the attribute owner or speaker.
2. One promising application is the development of LLM-powered Non-Player Characters (NPCs) in games. Please also refer to "Significance and Purpose of This Work."
3. We clarify that not all inconsistencies pertain to other agents. NLI-G also detects speaker-specific inconsistencies, as detailed in the "Inconsistency" section on page 4.
4. The NLI model is pretrained on single narrative sentences like “Tom did not want to talk about the subject,” which differs from dialogue utterances like “Eddy: I haven't read much Shakespeare.” Therefore, we transform the format to be similar to the pretrained data, e.g. “Eddy hasn't read much Shakespeare.”
5. We focus on detecting the “harmful” hallucination -- fabrications misalign with the agent’s role. (p.4)
6. We synthesize each dialogue turn by turn and regenerate erroneous utterances immediately instead of correcting one utterance in between the dialogue and leaving the rest as it was.
7. GPT-4 evaluation aligns more closely with humans than other automatic metrics and weaker LLMs.
8. Intraclass correlation coefficient (ICC) is not Fleiss' kappa; the latter measures the agreement between three or more raters and assumes the ratings are categorical and mutually exclusive, which is unsuitable for our data. ICC is the most ideal metric for our situation since the scores are on an interval scale. We additionally calculate Spearman's rank correlation coefficient by turning the scores to ranking, which is 0.437, indicating a moderate positive correlation.

We will add these clarifications and correct typos in the final version.

Thank you for reviewing and recognizing our paper’s "significance in extending prior work to more coherent dialogues across longer periods of time."

Novelty

Our approach is distinct in developing specific modules to manage problems that are not addressed by merely extending dialogue context lengths. Research [1] has shown the information in the middle of long input contexts can be overlooked.

[1] Lost in the middle: How language models use long contexts (Liu et al., TACL 2024)

Evaluation Robustness

The fluency is represented by perplexity, defined as the exponentiated average negative log-likelihood of text, which is not a statistical estimate that comes with a known distribution. Thus, we cannot directly apply statistical tests. Perplexity is a standard evaluation for language models. Lower perplexity indicates a better fit of the model to the data. Besides, "we did not stress on fluency evaluation, as our observations indicate that all generated dialogues are highly fluent" (p.6)

As for consistency scores, the improvements over baselines, while modest, underscore the potential of our framework in a conservative test setup (as described in Appendix C.1 Limitations). However, if we compare the scores pairwisely, we found ours scored higher than origin in 86.9% of examples, demonstrating the effectiveness of our framework.

Ablation study yields mixed results:

As described in Appendix C.1, Our experiment is built on the same context (memory, history, background, and status) from the original dialogue, which may contain inherent problems (repetition, inconsistencies, or factual errors), especially in the last percentile of dialogues. Therefore, removing memory mitigates factual errors stemming from the memory. We expect that these problematic memories will not exist in a complete simulation, as they will reflect the corrected dialogues.

*The agent’s memory is formed by recurrent reflections on their dialogue histories across multiple time points, thus containing many more factual errors than the dialogue history itself. We will add the discussion in the main text.

Robustness of human evaluation:

As mentioned in p.6, we intentionally selected the most challenging subset to rigorously test the model's performance, thereby providing a strong indicator of its robustness. Given our limited budget, this is the best we can do since the human evaluation of the 10% of the full dataset already took 20 hours for each annotator to finish.