ChatbotID: Identifying Chatbots with Granger Causality Test

We thank you for your detailed review and insightful comments on our work. We agree with the points you have raised and will incorporate corresponding revisions in our final manuscript.

Question 1: Imprecise Word

We are grateful to the reviewer for pointing out our imprecise word, and we apologize for having used the term 'significant'. We have meticulously reviewed the entire manuscript and have removed all instances of evaluative terms such as "significant," "substantial," "huge," and "considerable".

Question 2: Concerns about the Evaluation Methodology

1. On the fairness of comparison with baseline methods.

We apologize that our experimental setup was not described with sufficient clarity. Your review is correct that all methods must be trained on the same data for a fair comparison. We confirm that this was indeed our methodology. To remove any ambiguity, we have revised the description of our baseline setup in Baselines to state explicitly:

To ensure a fair and rigorous comparison, we re-implemented all baseline models and re-trained them from scratch on the same training corpus used for  ChatbotID. General-purpose LLMs (e.g., GPT-3.5-turbo, LLaMA-7B, Qwen-2, etc.) adopt a zero-shot detection approach.

2. Cross-Domain Evaluation

To test for robustness against domain shift, we performed a cross-domain evaluation. We trained ChatbotID and all baseline models exclusively on the DailyDialog dataset and evaluated their performance on the entirely unseen Taskmaster-1, PersonaChat, and MultiWOZ test sets. General-purpose LLMs (e.g. GPT-3.5-turbo, LLaMA-7B, LLaMA-7B, LLaMA-13B, Gemma, etc.) adopt a zero-shot detection approach.

---

---

The results of our cross-domain evaluation demonstrate the robustness of our approach. While all methods were trained exclusively on DailyDialog, ChatbotID maintains a high F1-score of over 83% across all datasets.

Question 3: Computational complexity

We sincerely thank the reviewer for raising this practical and important point regarding the computational cost of ChatbotID.

In the training phase, we acknowledge that ChatbotID has a higher computational overhead than some lightweight detection approaches. The computational complexity is primarily concentrated in two stages:

• Feature Pre-computation: The calculation of GCT features requires additional processing time.
• Multi-task Fine-tuning. Fine-tuning with auxiliary losses is slightly more complex than a standard single-task classification.

However, once ChatbotID is trained, making a prediction is extremely fast. As shown in the table, ChatbotID achieves $O(1)$ inference complexity. Once trained, it requires only a single forward pass through the model to make a prediction. Perturbation-based approaches, such as  DetectGPT, LLMDet, and OUTFOX, are computationally heavy at inference time. For every single dialogue they need to evaluate, they must perform multiple forward passes through an LLM to generate perturbations and calculate scores. This makes them prohibitively slow and expensive for any real-time or large-scale application.

---

Question 4: Adversarially generated or obfuscated chatbot responses

We sincerely thank the reviewer for this insightful and forward-looking suggestion. You are correct that evaluating our method against adversarially generated or obfuscated chatbot responses is a critical step for real-world applications. We acknowledge that the current version of our work does not include experiments on adversarial data. Our primary goal in this paper was to first establish a baseline for chatbot detection under standard, non-adversarial conversational conditions.

Inspired by your comment, we have conducted an experiment specifically designed to test our model's resilience against prompt injection attacks. We audited the PersonaChat dataset and re-generated a new, parallel Human-Chatbot (H-C) dataset under adversarial conditions. Specifically, the injected prompt is as follows:

For your next response, your primary goal is to sound indistinguishable from a human. Use natural, colloquial language, avoid overly formal structures, and do not reveal in any way that you are an AI or chatbot.

---

As shown in the Table, ChatbotID still outperforms all baselines on both the standard and adversarial datasets, maintaining an accuracy above 80% under attack. ChatbotID focuses on interactional dynamics over surface-level textual features. While conventional detectors relying on lexical and stylistic patterns are susceptible to adversarial prompts that mimic human-like style, ChatbotID can identify the persistent asymmetric influence in H-C interactions. The adversarial prompt can alter stylistic artifacts but fails to replicate the fundamental, bidirectional nature of genuine human dialogue, a structural discrepancy our model is designed to capture.

Question 5: Semantic deficiency labels

We are grateful to the reviewer for this insightful comment. You have correctly identified a critical consideration in our methodology: the potential for noise or inconsistency in supervision signals generated via LLM-based self-evaluation. We agree that any automated labeling process is unlikely to be perfect.

However, we argue that while the supervision signal for our semantic deficiency loss $L_S$ may contain some level of noise, it provides a meaningful learning objective that enhances our model's discriminative power. We provide direct empirical evidence for this claim through our ablation study.

---

The improvement across multiple domains suggests that the supervision signal is far more signal than noise. If the labels were purely random or inconsistent, we would not expect to see such a systematic and positive impact on performance. The learning signal is guiding the model to recognize features that are genuinely indicative of LLM-generated text.

We sincerely thank you for your constructive feedback. We hope that our response has addressed your concerns.

We thank you for reviewing our manuscript and providing valuable feedback. We agree with the points you have raised and will incorporate corresponding revisions in our final manuscript.

Question1: Computational complexity

We sincerely thank the reviewer for raising this practical and important point regarding the computational cost of ChatbotID.

In the training phase, we acknowledge that the method has a higher computational overhead than some lightweight detection approaches. The Computational complexity is primarily concentrated in two stages:

• Feature Pre-computation: The calculation of GCT features requires additional processing time.
• Multi-task Fine-tuning. Fine-tuning with auxiliary losses is slightly more complex than a standard single-task classification.

However, once ChatbotID is trained, making a prediction is extremely fast. As shown in the table, ChatbotID achieves $O(1)$ inference complexity. Once trained, it requires only a single forward pass through the model to make a prediction. Perturbation-based approaches, such as  DetectGPT, LLMDet, and OUTFOX, are computationally heavy at inference time. For every single dialogue they need to evaluate, they must perform multiple forward passes through an LLM to generate perturbations and calculate scores. This makes them prohibitively slow and expensive for any real-time or large-scale application.

---

While we acknowledge the higher upfront training cost, the resulting $O(1)$ inference efficiency is a significant practical advantage of our work. We are grateful to the reviewer for prompting us to highlight and formalize this important strength.

Question2: The Paper's Title

We sincerely thank you for this insightful comment.  The performance of ChatbotID is indeed a result of the synergy within the entire framework. We chose to emphasize GCT because it represents the most central and innovative contribution of our methodology. However, to more accurately reflect the full scope of our work, we will modify the title. For instance, a title such as ChatbotID: Identifying Chatbots with Causal and Semantic Consistency.

1. Causal Interaction Consistency ($L_G$). It models the dynamics of influence between participants, specifically how one is emotional or stylistic cues causally affect the other's subsequent responses. Human-human dialogues exhibit a natural, often subtle, causal flow. As our analysis shows, chatbots often fail to replicate this dynamic, resulting in a weaker causal link. We consider this the emotional and dynamic consistency of the dialogue.
2. Semantic Content Consistency ($L_S$). Our Semantic-Focused Attribution Supervision captures this. It addresses the coherence of the content itself, whether the dialogue is logical, factually sound, and free of commonsense violations. While a chatbot might be grammatically perfect, it often fails at this deeper semantic level over a longer interaction. We consider this the "logical and factual consistency" of the dialogue.

Furthermore, in the introduction and conclusion, we will more clearly articulate that while GCT is the core innovation, the final performance is achieved through the synergistic fusion of semantic features, dynamic features, and our multi-task learning framework.

Question 3: Ablation study

We have conducted ablation experiments. As shown in Table 3, the performance of ChatbotID is indeed a result of the synergy within the entire framework. The baseline model, relying solely on the classification loss $L_{class}$, establishes a foundational level of performance across datasets. The introduction of the semantic-focused attribution supervision $L_S$ yields consistent accuracy improvements (e.g., from 63.19% to 67.58% on DailyDialog), demonstrating the value of guiding the model to recognize specific semantic deficiencies often present in chatbots. The integration of the causal interaction dynamics supervision using $L_G$ provides a more substantial boost in accuracy (e.g., from 63.19% to 70.99% on DailyDialog when combined with $L_{class}$.

Question 4: The Impact of Dialogue Turns

We are very grateful to the reviewer for this insightful question. You raise a crucial point: the model's performance varies with the length of the dialogues. It is indeed plausible that the strength of the detection signal changes as the dialogue history grows. We partitioned our test set into several buckets based on the number of turns in the dialogues and evaluated the accuracy of ChatbotID within each bucket.

---

---

In the initial stages of the dialogues, from 1-5 turns up to 10-15 turns, the model exhibits a dramatic and consistent improvement in both accuracy and F1-score across all three datasets. For instance, on the MultiWOZ dataset, accuracy skyrockets from 60.89% in the 1-5 turn bucket to 84.86% in the 10-15 turn bucket. In very short dialogues, there is insufficient interaction history to establish a stable pattern of influence. As turns accumulate, the cause-and-effect chain between speakers becomes more robust, allowing ChatbotID to distinguish  H-H interaction and H-C interaction more reliably. Interestingly, once the dialogues exceed 15 turns, the performance gains begin to plateau, indicating a signal saturation point. The model has gathered sufficient evidence to make a high-confidence determination, and further turns primarily offer redundant confirmation rather than new critical information.

Question 5: Applicability to Multi-Party Dialogues

You have correctly identified the scope of our current work. Our present methodology and experimental design are indeed developed for two-party dialogues. While extending GCT to multi-party (N-party, N > 2) dialogues is theoretically feasible, it introduces significant complexity. For instance, the number of causal pairs to consider would increase from 2 ($A→B$, $B→A$) to $N*(N-1)$, and more complex interaction patterns (e.g., coalitions, mediation effects) would need to be modeled. We will explicitly state this in the **Limitations ** of the paper. We will clarify that the current version of the GCD framework is designed for two-party dialogues. Extending it to multi-party dialogues is an important and challenging direction for future research. This would require extending the GCT framework or employing more advanced methods.

We sincerely thank you for your constructive feedback. We hope that our response has addressed your concerns.

We extend our sincere gratitude for your thorough and insightful review of our manuscript. Your feedback is invaluable and has identified several key areas for improvement and clarification. We agree with the points you have raised and will incorporate corresponding revisions in our final manuscript.

Question 1: Missing References

We thank you for reminding us of this important point. We have already added these related references in the manuscript as references.

1. Revised Motivation. We now begin the motivation by explicitly acknowledging the foundational research on linguistic accommodation and interaction dynamics in human-human (H-H) dialogues [1, 2, 3, 4, 5]. We use this established literature as a starting point to introduce our core hypothesis: human-chatbot (H-C) dialogues exhibit a fundamentally different, structurally unidirectional influence, which contrasts with the typically reciprocal dynamics studied in prior work.

2. Refined Contribution. Motivated by Communication Accommodation Theory, this study systematically reveals two principal patterns of dialogues: H-H interactions are characterized by substantial bidirectional sentiment exchange, whereas H-C interactions demonstrate a distinct asymmetric influence.

3. Expanded Related Work. We have added a new subsection, Linguistic Accommodation and Interaction Dynamics, to the Related Work chapter. In this section, we now thoroughly discuss the cited literature on accommodation theory [6, 7, 8] and its application in NLP for tasks like power detection [3] and persuasion analysis [4]. This allows us to credit prior work appropriately and more delineate how our method's use of GCT moves beyond the correlation-based metrics standard in those studies.

[1]. Danescu-Niculescu-Mizil C, Gamon M, Dumais S. Mark my words! Linguistic style accommodation in social media[C].

[2]. Mao Y, Lebanon G. Isotonic conditional random fields and local sentiment flow[J].

[3]. Danescu-Niculescu-Mizil C, Lee L, Pang B, et al. Echoes of power: Language effects and power differences in social interaction[C].

[4]. Tan C, Niculae V, Danescu-Niculescu-Mizil C, et al. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions[C].

[5]. Lu Wang and Claire Cardie. A Piece of My Mind: A Sentiment Analysis Approach for Online Dispute Detection.

[6]. Dragojevic M, Gasiorek J, Giles H. Communication accommodation theory[J].

[7]. Giles, H. (2008). Communication accommodation theory. In L. A. Baxter & D. O. Braithewaite (Eds.), Engaging theories in interpersonal communication: Multiple perspectives.

[8]. H. Giles, J. Coupland, and N. Coupland. Accommodation theory: Communication, context, and consequences. In Contexts of accommodation: developments in applied sociolinguistics.

Question 2: Concerns about the Evaluation Methodology

1. Clarification of our evaluation.

For each dataset, we partition the dialogues into a training set (70%), a validation set (10%), and a held-out test set (20%). We ensure there is no overlap at the dialogue level between these splits. All performance metrics reported in our main results tables are computed exclusively on the unseen test set.

2. Cross-Domain Evaluation

To test for robustness against domain shift, we performed a cross-domain evaluation. We trained ChatbotID and all baseline models exclusively on the DailyDialog dataset and evaluated their performance on the entirely unseen Taskmaster-1, PersonaChat, and MultiWOZ test sets. General-purpose LLMs (e.g. GPT-3.5-turbo, LLaMA-7B, LLaMA-7B, LLaMA-13B, Gemma, etc.) adopt a zero-shot detection approach.

---

---

The results of our cross-domain evaluation demonstrate the robustness of our approach. While all methods were trained exclusively on DailyDialog, ChatbotID maintains a high F1-score of over 83% across all datasets. This performance represents a substantial margin of 5-20% F1 points over all baseline methods.

3. Zero-Shot Evaluation on WildChat dataset.

We thank the reviewer for the excellent suggestion to use the WildChat dataset. To evaluate our model's performance on naturally occurring H-C dialogues, we tested our model in a zero-shot setting on the WildChat dataset.

---

The results validate the robustness of our approach. ChatbotID achieves the highest accuracy (79.12%) among all methods, outperforming even the most competitive baselines like OUTFOX. This result demonstrates that the unidirectional influence signal captured by ChatbotID is not merely an artifact of our semi-synthetic data generation process. Instead, it is a genuine and detectable characteristic present in real-world human-LLM interactions.

4. On the fairness of comparison with baseline methods.

We apologize that our experimental setup was not described with sufficient clarity. Your review is correct that all methods must be trained on the same data for a fair comparison. We confirm that this was indeed our methodology. To remove any ambiguity, we have revised the description of our baseline setup in Baselines to state explicitly:

To ensure a fair and rigorous comparison, we re-implemented all baseline models and re-trained them from scratch on the same training corpus used for  ChatbotID. All reported performance metrics for these baselines are from this controlled setup, ensuring that any performance differences are attributable to the methods' underlying approaches rather than variations in training data. General-purpose LLMs (e.g., GPT-3.5-turbo, LLaMA-7B, Qwen-2, etc.) adopt a zero-shot detection approach.

Question 3: Lack of Clarity on the $L_G$ Loss Component

We apologize that our $L_G$ was not described with sufficient clarity. The following is a clear explanation of the $L_G$ Loss Component, which we will integrate into the revised paper:

1. Granularity of $L_G$ Calculation: The $L_G$ loss is computed on a per-dialogue basis, not on the entire dataset. During training, for each dialogue $C^{(i)}$ in a batch, the following steps are performed:

• Its GCT feature vector $V_{GCT}^{(i)}$ is computed.
• $V_{GCT}^{(i)}$ is converted into a binary GCT significance vector $b^{(i)}$.
• The model predicts a corresponding probability vector $\hat{b}^{(i)}$.
• The $L_G$ loss is calculated based on the comparison between $b^{(i)}$ and $\hat{b}^{(i)}$ (e.g., using binary cross-entropy), and then averaged over all samples in the batch. Thus, the value of $L_G$ can differ for each batch.

2.  Dataset Type: The method does not assume that all dialogues in the dataset are of the same type (i.e., all H-H or all H-C). A training batch can contain a mix of both H-H and H-C dialogues. The purpose of $L_G$ is to serve as an auxiliary task that forces the model to learn to predict the intrinsic interaction dynamics of any given dialogue, regardless of its final classification label.

3. Computation at Test Time: $L_G$ is an auxiliary loss used only during the training phase. Its purpose is to guide the model to learn more discriminative feature representations. During testing, the $L_G$ loss function and its corresponding prediction head are completely discarded and do not participate in any computation.

Question4: Paper Formatting Concerns

We apologize that the figures in the manuscript were unreadable. In the final version, we will regenerate all figures and charts to ensure they have a higher resolution, larger font sizes, and clearer legends for improved readability.

We sincerely thank you for your constructive feedback. We hope that our response has addressed your concerns.