IMPersona: Evaluating Individual Level LLM Impersonation

Thank you for the thoughtful and constructive review. We are encouraged by your positive assessment of our work, particularly your recognition of the novelty of our setup, the effectiveness of lightweight fine-tuning combined with a memory mechanism, and the realism of our evaluation design. We appreciate your acknowledgment of the potential implications, both positive and concerning, of personalized LLMs, as well as your interest in the mitigation strategies we propose. Below, we address your concerns and questions in more detail.

1. On the Use of a Single Model Family for Fine-tuning:

We agree that broader model diversity would be valuable and appreciate your pointing this out. As noted in our limitations section, we were constrained by available compute resources, particularly for larger models like the 70B class. Furthermore, expanding to multiple model families would significantly increase the number of human evaluators required to maintain statistical significance, given our evaluation protocol. Given these constraints, we prioritized evaluating performance across varying dataset sizes, as we believe this axis has broader implications in that if high impersonation performance can be achieved with small training datasets, this would pose a significant safety concern. That said, we did conduct preliminary experiments comparing other model families of similar size (e.g., Qwen-2.5-7B-Instruct vs. LLaMA-3.1-8B-Instruct), as reported in Section 4.

2. Recruitment of Human Evaluators:

As described in Section 3, each impersonated individual trained a model on their own data. For evaluation, they recruited up to 12 individuals who knew them personally: ranging from acquaintances to family members to ensure a baseline level of familiarity. This setup enabled us to simulate realistic human interactions and provide meaningful assessments of impersonation quality. We appreciate your suggestion and will update the manuscript to include a more detailed description of this process in the final version.

3. Further Clarifications:

Memory Retrieval: The seven retrieved memories used during interaction include a combination of top-ranked, first-order, and second-order memories.

Memory Management LLM: We used GPT-4o to manage memory retrieval and selection, as we found that a strong reasoning model was essential for effective memory coordination.

Data Scarcity and Sample Size: The “20 samples collected from two evaluators” refers to evaluation conversations, not training data. Specifically, in our analysis of scaling performance with dataset size (Figure 4), we had fewer evaluation interactions per condition (~20 conversations per impersonated individual across two evaluators) compared to the ~100 conversations used for the main setups reported in Table 1. This resulted in higher variance in some conditions, which we will make clearer in the text.

Regression Analysis (Table 1): The regression model was an ordinary least squares (OLS) analysis with humanness ratings as the dependent variable. Predictors included impersonation type (coded as binary variables corresponding to the nine setups in Table 1), participant familiarity (closeness), texting frequency, prior AI exposure, and related covariates. We will ensure the regression methodology is clearly described in the final version.

Thanks again for your thoughtful review, and for pointing out details that weren’t clear from your perspective! We will be sure to update the paper to reflect these details in the final version.

Thank you very much for your thorough, thoughtful and constructive review. We greatly appreciate the time and care you took in providing detailed feedback, and we’re especially grateful for your recognition of both the strengths and implications of our findings. It seems that your main concerns center around minor clarifications in our presentation of methods and results. Below, we address each of your points in turn.

1. General Clarifications:

For example, how long were sessions?

Each session lasted 3 minutes, which is noted in Section 3, paragraph 2.

Was the metadata analysis in 5.1 anecdotal, annotated, or LLM-generated, and if annotated, whether it was single-annotator or multi-annotator with agreement scores?

The analysis was annotated by humans, and was done by a single annotator. We will add this.

What were the style and contextuality questions? Show your experiment from the pov of the subjects.

These are shown in Table 3 of the Appendix, with participant interface details provided in Figures 10 and 11.

More explanation on what this 5 was spent on

The $5 refers to the approximate cost to fine-tune a model on the B500 dataset using LLaMA-3.1-8B-Instruct, illustrating the low barrier to launching personalized impersonation attacks. We will revise this line to provide more context.

The regression analysis mentioned was not included in the paper in detail.

This is a great point, and we will add this into our methodology section in detail in the final version for sure, as we only briefly describe it in the paper. For clarity, the regression model was an ordinary least squares (OLS) analysis with humanness ratings as the dependent variable. Predictors included impersonation type (coded as binary variables corresponding to the nine setups in Table 1), participant familiarity (closeness), texting frequency, prior AI exposure, and related covariates.

“So to further analyze imitation quality, we collect additional metadata on reasoning for participant decisions, classifying between errors on style, context, and conversation flow.” it is unclear what this metadata looks like. Since you brought it up, please explain it.

Thank you for pointing out the unclearness here! Specifically, participants indicate the reasoning behind their choices using a predefined menu [See Figure 11 in the Appendix] classifying whether their judgment was influenced primarily by stylistic imitation, contextual alignment, or conversation flow (e.g., coherence, naturalness). They also have the option to provide open-ended verbal feedback to elaborate on their selections. Examples can be found in Appendix Figures 14-17. We will be sure to make this clearer in the final version! Thanks for the suggestion!

“Participants struggled to identify the interviewer even in cases of close relationships (friends and siblings) if their texting interaction was infrequent.” Does this mean that the participants have low text interaction in general or with the specific interviewer / impersonator? It is unclear to the reader please clarify.

It is with regard to the specific interviewer / impersonator. We will revise this line to emphasize this.

What are the questions asked before starting the chat? What are the questions asked after the chat? Humanness is a Likert scale question. Are Style and Cont? Who are the people answering the style cont questions? Are they the person in the chat room or the person being impersonated?

All questions and interface screenshots are included in Figures 10 and 11 of the Appendix. Due to space constraints, these were placed there, but we agree they are central to understanding the experiment and will reference them more directly. To clarify: Style and Contextual refer to the prompt type initiating the conversation, not additional Likert-style ratings. Each conversation was seeded with either a style-oriented or context-oriented prompt, and aggregate scores are grouped by this initial prompt category.

We greatly appreciate your very thorough and precise feedback, and hope this addresses your concerns! We think your points were extremely constructive and we’ve revised the improved the paper in several places based upon your suggestions.

Thank you for your thoughtful and constructive review. We are encouraged by your recognition of the novelty and rigor of our work, including the “structured and reproducible framework for evaluating individual-level impersonation,” the “clearly described” hierarchical memory module, and the detailed experimental design with participant metadata and multiple model variants. We also appreciate your acknowledgment of our ethical considerations and discussion of both potential risks and beneficial applications. Below, we address each of your concerns in turn:

1. Clarification on Memory Integration:

Our memory module primarily supports content-level grounding by surfacing causally related, event-specific memories organized through the hierarchical memory system. Its core function is not to shape the stylistic form of a response, but rather to retrieve relevant personal experiences or factual details that the impersonator model can use to construct contextually rich replies. For instance, in a conversation about favorite college classes, the memory system might retrieve details such as specific courses or classmates, which the impersonator model then integrates into a response that reflects the individual’s characteristic tone and narrative style.

2. Impersonation Success by Familiarity Level:

While Section 5 presents OLS regression analyses indicating that participant familiarity does not significantly affect humanness ratings, we agree that stratified results (e.g., by family member vs. acquaintance) could provide helpful context. We report below identification success rates based on closeness, which we will add to the appendix. As discussed in Section 6, even highly familiar participants can be deceived—particularly when they lack experience with AI-generated text or have limited exposure to the target’s digital communication style, evidenced below. Note that these numbers represent identification accuracy (how often participants identified AI as AI, and humans as humans across all settings).

Table: Accuracy by Tester Closeness to Experimenter

3. Prompting Baselines and Prompt Optimization:

We agree that more sophisticated prompting strategies such as soft prompting or frameworks like DSPy could be promising. Our focus was on establishing a realistic and interpretable baseline for individual-level impersonation. Each prompting baseline was hand-crafted to incorporate known facts, style cues, and shared memories, already optimized towards the impersonation task. We think prompt optimizing ICL models could be a valuable direction for future work.

4. Clarification on Main Evaluation Metric:

We would like to clarify that our primary evaluation was not simply whether the model sounded human, but whether it convincingly impersonated a specific known individual. As noted in Section 1 and detailed in our evaluation protocol, participants interacted with someone they knew personally and were explicitly asked whether they believed the conversational partner was that person or an AI impersonator. The evaluation thus captures more than generic humanness; it reflects the model’s ability to replicate personal details, conversational quirks, and shared context.

5. Demographic Diversity:

As discussed in Section 7, our participant pool skewed toward younger and individuals with higher AI familiarity and experience. While this may limit generalizability, we argue it presents a stricter test, as participants with high familiarity with LLMs may be more attuned to their common communication patterns and thus more likely to apply effective probing strategies. Supporting this, our analysis shows a positive correlation between AI familiarity and detection success. In Section 7, we hypothesize that deception rates would likely be higher in a broader, less tech-savvy population.

We hope this response clarifies points of confusion and supplements your understanding and perception of our work. Thank you again for your thoughtful review and questions!

Dear reviewer,

Could you please respond to let the authors know you have read their response?

Thanks!

-AC

Thank you for your thoughtful review and feedback! We are encouraged that the reviewer recognizes the “great safety implications” of our novel study, our “carefully designed” experiments, as well as the interesting findings on predictors of success and mitigation strategies. We address and give relevant clarifications to your concerns below.

1. Frequency of intentional contextual probing:

As our goal was to simulate a naturalistic setting, where participants believed they were chatting with someone they knew, participants were not explicitly encouraged to test the identity of their interlocutor. In practice however, we observed in 45% of 20 randomly sampled conversations that participants engaged in a purposeful contextual probing. In real-world contexts, people typically do not suspect AI impersonation or engage in such contextual probing. Therefore, our evaluation seeks to measure baseline detection performance in realistic, everyday interactions, rather than the upper bound of human capability under suspicion or instruction, and we do not believe this to necessarily undermine our results.

2. Evaluating general AI-generated text detectors:

Thank you for this suggestion. While our primary focus was on human judgment, since humans are often the first line of defense in detecting impersonation, we agree that effective automated detectors could complement this process. However, in this work we chose not to systematically evaluate off-the-shelf detectors in this work for two key reasons:

1. [Main reason] Messaging data tends to be extremely short (avg ~35 characters per message), which greatly impairs detector performance. Prior work has shown that detection accuracy deteriorates significantly with shorter text lengths [2]. This is partly why leading AI text detectors like GPTZero only support inputs of over 250 characters, far exceeding the average message length of ~35 characters sent by our impersonator models.
2. Many existing detectors are trained on distributions of standard, well-known LLMs. Our fine-tuned models, by contrast, are trained on real human messaging data, substantially shifting their output distribution and reducing detector efficacy [1].

To illustrate this point, we tested several example conversations from our study using GPTZero, a leading AI text detector. When inputting conversation histories from our fine-tuned LLaMA models (Appendix Figures 16 and 17), GPTZero classified the text as: “We are highly confident this text is entirely human.” In contrast, when we input conversation histories from prompted frontier models (Appendix Figures 14 and 15), GPTZero returned: “We are uncertain about this document. If we had to classify it, it would likely be considered human.” These results show the limited effectiveness of current detectors, particularly when applied to outputs from fine-tuned models optimized to mimic human messaging patterns.

The lack of effective automated detectors for finetuned models further supports the fact that the individual-level impersonation problem is a critical safety issue for LLMs: one that has been largely unexplored in prior literature. The risk is further amplified by the increasing availability of real-world chat data on public platforms such as Discord and Reddit: something that we hope to highlight with our work in this paper.

3. Quantitative analysis about deceptive impersonation:

We appreciate your point regarding the need for quantitative support on deceptive impersonation. Our intention in highlighting "deceptive impersonation" was to demonstrate that LMs could be used to engage in behaviors that may deceive users: especially in familiar, personalized contexts, as exemplified in Figure 6. However, we intentionally avoided designing experiments that would mislead participants in ethically sensitive ways, as this could pose psychological risks violating IRB guidelines. As such, our focus is on characterizing the potential for such deception in realistic interactions, which we think is a significant implication of our results, rather than provoking deception explicitly.

We hope this clears up any questions/confusion that you have about our work, and look forward to engaging in future discussion. Thanks again for your thoughtful review and questions!

References:

[1] Abdali, Sara, et al. "Decoding the AI pen: Techniques and challenges in detecting AI-generated text." KDD 2024.

[2] Wang, Hao, Jianwei Li, and Zhengyu Li. "AI-generated text detection and classification based on BERT deep learning algorithm." arXiv:2405.16422 (2024).