Memory Injection Attacks on LLM Agents via Query-Only Interaction

Thank you for your thoughtful review! We sincerely appreciate your recognition of our realistic problem setting and bridging-based design, as well as your support for its impact on advancing agent and LLM security research.

W1, Q1: The use of bridging or decomposed reasoning steps shares similarities with prior work (e.g., PANDORA).

R1: We thank the reviewer for pointing out the resemblance to prior reasoning-based attacks such as PANDORA. While both involve multi-step reasoning, MINJA differs fundamentally in its motivation, threat model, and attack mechanism as shown in the table below.

Specifically, PANDORA focuses on immediate jailbreaks through multi-agent collaboration and task decomposition, aiming to bypass content filters in a single session. In contrast, MINJA targets long-term memory by injecting malicious records that logically bridge victim and target concepts, using a Progressive Shortening Strategy (PSS) to gradually reduce prompt cues. This enables the attack to persist in memory and influence future benign queries in a single-agent setting.

We will add this comparison to the related work section of our revised paper. Thank you again for your comment.

W2, W3, W4; Q2, Q3, Q4, Q5, Q6: Does the attack rely on the reasoning capabilities of LLMs? Would it still be effective on non-reasoning agents based on general-purpose LLMs like Llama-2?; Can the attack transfer to other reasoning models beyond GPT-4(o), such as DeepSeek?; How does model size affect the attack’s performance? Can the method generalize to smaller models like 7B or 14B?

R2: We thank the reviewer for the questions about the models used in the agent.

First, we would like to clarify that our attack does not require the LLM model to be a dedicated “reasoning model”, but we leverage the reasoning abilities of LLMs (even the non-reasoning models can perform Chain-of-thought reasoning) in the agent. Reasoning is commonly used by agents [1][2] operating in complex, real-world scenarios—such as decision making, planning, and multi-step tool use. Therefore, we do not require specific properties of reasoning models.

Second, we extend our evaluation to DeepSeek-R1 for broader evaluation. On QA Agent (for arbitrarily selected victim-target pairs 1, 4, and 7), DeepSeek-R1 achieves consistently high Injection and Attack Success Rates, closely matching GPT-4’s performance:

DeepSeek-R1:

Third, we investigate the effect of model scale using Llama-2-7B. However, this model only achieves 19.3% and 17.1% task accuracy for pairs 1 and 7, respectively, even lower than random guessing on MMLU with four answer choices (25%). These results indicate that Llama2-2-7B fails to perform the base task reliably, such that it will not likely be adopted as the core model of the agent, making attack evaluation less meaningful. Therefore, regarding small-scale models, a minimum level of base task competence is required for evaluating attacks.

We will complete the full set of experiments on DeepSeek-R1 and incorporate more discussion on these models in the revised paper.

Q7: Is it possible to design potential strategies to defend against this attack?

R3: Thank you for your interest in potential defense!

As discussed in Section 5.4, MINJA poses a challenge to conventional defenses due to its query-only interaction mode and the plausibility of the injected prompts.

While we discuss and explore several strategies—adversarial training, embedding-level memory sanitization, and prompt-level detection—each has notable limitations. For example, adversarial training suffers from scalability issues; embedding-level filtering struggles due to entanglement of benign and malicious records; and prompt-level detection, while promising, either lacks generalization (targeted prompts) or introduces high false positives (general prompts).

These findings suggest that MINJA remains difficult to defend using existing mechanisms. We believe this highlights the need for future research into robust, low-cost defenses specifically tailored to memory poisoning attacks.

[1] ReAct: Synergizing Reasoning and Acting in Language Models

[2] Reflexion: Language Agents with Verbal Reinforcement Learning

Thank you for your thoughtful and generous review! We deeply appreciate your recognition of the realistic threat model, the careful design of our attack strategy, and the clarity and rigor of our experimental evaluation.

W1: The attack assumes a shared memory bank across users.

R1: Thank you for this insightful comment!

Our work does consider a setting where the agent uses a shared memory bank, which, as noted in the paper, is “common in existing agent frameworks due to deployment and performance considerations.” (Page 3-4, Lines 142-146) This design supports cross-user knowledge accumulation and is increasingly adopted in real-world systems [1][2][3][4][5].

If the memory is isolated, potential attack strategies remain feasible. For example, adversaries could disguise their identity or hijack user accounts to inject poisoned data into private memory, without requiring designer- or system-level access.

We will add more discussion on guidelines for future and specialized agent deployments. Thanks again for your thoughtful comment!

W2: The attack assumes the ability to inject a large number of interaction records, which may not be feasible in high-security or rate-limited environments.

R2: Thank you for this important point!

While our main experiments assume a reasonable volume of injected queries for each attack on a certain victim-target pair, we note that MINJA does not require concentrated injection from a single entity. Instead, multiple attackers could collaborate or operate independently in parallel—or a single attacker could use multiple accounts—to collectively contribute malicious records over time. This distributed injection strategy mitigates the effect of rate limiting and aligns with realistic adversarial behaviors, such as the use of botnets or compromised user accounts. This strategy helps bypass rate-limited environments, making the attack more feasible and covert in practical settings.

We will add the discussion in the revision.

W3: The paper lacks detailed analysis of practical defense mechanisms (e.g., memory isolation, permission control, system-level safeguards), which would help assess real-world risk and mitigation of MINJA.

R3: Thank you for the thoughtful suggestion!

Regarding memory isolation, while it can reduce cross-user poisoning, we note that identity disguise strategies, such as account hijacking or impersonation, remain feasible and have been observed in real-world attacks. These strategies allow adversaries to inject malicious records even under isolated settings.

For permission control, our threat model assumes that attackers have the same interface and capabilities as benign users without any privileged access (Page 3, Lines 138-141). Since attackers are not explicitly distinguishable from regular users, applying strict permission constraints may unintentionally impact benign functionality.

As for system-level safeguards (e.g., GuardAgent [6] based on safety rules), we’d like to emphasize that our injected reasoning steps are plausible, semantically coherent, and aligned with the task context. Detecting such malicious content likely requires external knowledge bases, structured monitoring, or task-specific constraints. For example, in the case of RAP Agent, determining whether a product is truly out of stock based on the indication prompt may require checking the inventory database, which lies beyond the model’s native capabilities.

We agree that developing such targeted defenses is an important and promising direction for future research. We will include those discussions in Section 5.4 (Potential Defense) in our revision, and we hope our work motivates further exploration in this area.

W4: The attack’s effectiveness may degrade if real-world systems adopt stricter or more diverse memory retrieval filtering beyond embedding-based similarity.

R4: Thank you for the insightful question.

1. Retrieving memory via query similarity is a scalable and widely adopted strategy across LLM-based agents (e.g., RAP, EHR Agent).

2. Even if retrieval is not based on embedding similarity—e.g., by directly querying an LLM—our attack remains effective. This is because agents typically reuse prior records as in-context demonstrations. Our PSS strategy ensures that the most informative record (from the previous shortening step) remains highly relevant and thus likely to be selected.

3. While real-world systems may employ stricter filtering or security checks, we show in Section 5.4 that both embedding-level sanitization and prompt-level defenses are limited: semantic overlap makes it hard to distinguish benign vs. malicious records, and LLM-based filtering either fails to generalize (targeted prompts) or suffers from high false positives (general prompts). These findings suggest that MINJA remains effective even under more advanced or stricter memory filtering settings, making it a broadly applicable threat model for memory-based agents.

We will add Point 1. to Section 2 (Related Work) to highlight the common use of query similarity-based retrieval, and Point 2. will be included in Section 5.3 (Ablation Study, Choice of Embedding Model for Memory Retrieval) to discuss MINJA’s effectiveness under LLM-based retrieval mechanisms in our revision.

W5: The usage of q (italic) and q (regular) for different queries may confuse.

R5: Thank you so much for your valuable suggestion! We will revise the notation for clarity—using a for attack query in the updated version.

[1] Memory sharing for large language model based agents

[2] Koma: Knowledge-driven multi-agent framework for autonomous driving with large language models

[3] Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records

[4] A language agent for autonomous driving

[5] Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents

[6] GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled

Thank you for your thoughtful review! We sincerely appreciate your recognition of the novelty of our threat model and the effectiveness of our interaction-based memory poisoning approach.

W2: Unclear robustness of the attack under memory retrieval mechanisms involving noise, relevance scoring, or temporal decay.

R1: Thank you for your very thoughtful question! We would like to discuss the cases mentioned by the reviewer, respectively.

1. Involving noises. We conducted an ablation experiment on RAP Agent (with GPT-4o) to evaluate the robustness of our attack under retrieval noise. Specifically, we added Gaussian noise (σ = 0.01) to the embedding vectors during memory retrieval, simulating noisy retrieval. The noise level is comparable to the scale of real embeddings (∼e−2), making it a moderate perturbation. We evaluate on victim-target pairs 1, 4, and 7 and report average results. As shown in the table below, ISR remains 100%, and ASR only slightly drops (97.8 to 95.6), indicating MINJA’s strong robustness to noisy retrieval. ||ISR|ASR| |-|-|-| |Without noise|100.0|97.8| |Gaussian noise (σ = 0.01)|100.0|95.6|

2. Relevance scoring. MINJA is robust to changes in the design of the retrieval mechanism, including both similarity metrics and retriever architectures. As detailed in our paper (Page 8, Lines 325-529), we evaluated six retrievers—DPR, REALM, ANCE, BGE, MiniLM, and ada-002—that differ in embedding models. Across all settings, ISR and ASR remain consistently high, demonstrating the attack’s resilience to different relevance scoring strategies.

3. Temporal decay. Temporal decay typically affects all memory poisoning attacks by diminishing the impact of earlier injected records. But our attack is achieved by the adversarial user's interaction with the agent, which could happen anytime, e.g., just before the victim user's interaction.

We will add these results and the discussion to our revised paper.

Q1: Unclear sensitivity of the attack to changes in retriever design, such as different similarity metrics (e.g., cosine vs. dot product).

R2: Thank you for raising this point!

To assess the sensitivity of MINJA to retriever design choices, we conducted experiments using six different embedding models for memory retrieval, as described in our paper (Page 8, Lines 325-529): DPR, REALM, ANCE, BGE, MiniLM, and OpenAI's ada-002. These models differ not only in their architecture but also in the similarity metrics typically employed—some use dot-product (e.g., DPR, REALM), while others use cosine similarity (e.g., MiniLM, BGE).

Despite these variations, as shown in Figure 3, MINJA consistently achieves strong Injection Success Rate (ISR) and Attack Success Rate (ASR) across all tested models. This indicates that the attack is robust to the choice of similarity metric and retriever architecture, demonstrating its generalizability across a range of retrieval mechanisms.

Q2: What is the minimum retrieval rank at which the poisoned record must appear for the attack to succeed? Does ASR drop significantly when the poisoned record is retrieved beyond top-k?

R3: Thank you for your insightful question!

We analyzed the retrieval ranks of poisoned records in EHR Agent as an example. In successful attacks on both MIMIC-III and eICU, the poisoned record is almost always retrieved at rank 1, whereas failed attacks are typically associated with lower-ranked or unretrieved poisoned records.

As with prior memory poisoning attacks such as AgentPoison, if no poisoned record is retrieved, the attack cannot succeed. Thus, the retrieval rank is indeed important, but this is a shared limitation across memory-based attacks, such as AgentPoison. To address this inherent limitation of memory-based attacks, MINJA diversifies the victim queries during poisoning (see Page 5, Lines 184–186), which increases the likelihood that different malicious records are retrieved for different user queries and simultaneously promotes higher retrieval ranks for poisoned records.

Q3: How many interaction rounds does PSS typically require? Would aggressive rate limiting hinder the attack’s feasibility?

R4: Thank you for your insightful question!

PSS typically requires 4–5 interaction rounds. In our main experiments, the number of PSS iterations varies slightly across settings — 4 for EHRAgent on MIMIC-III, and 5 for EHRAgent on eICU, RAPAgent, and QAAgent. We find that even this moderate number of interactions enables consistently high ISR and ASR.

Regarding rate limiting, we note that attackers operate under the same interface and limitations as any benign user (Page 3, Lines 138-141). If system-wide rate limits are enforced, they affect all users equally, but the attacker can use multiple user accounts to attack.

W1, Q4: Would defenses, such as rejecting chain-of-thought reasoning or enforcing entity consistency between query and answer, reduce the attack’s effectiveness?

R5: We sincerely appreciate your thoughtful question!

We clarify that CoT (chain-of-thought) reasoning is common in benign user queries. Blocking it may negatively impact normal agent usage. Moreover, our indication prompts are designed to be plausible, making them difficult to distinguish from regular CoT responses to benign inputs.

We also evaluated the rule-based defense, which is conceptually aligned with enforcing entity consistency. Both methods aim to detect semantic irregularities introduced by the attacker. As mentioned in Section 5.4, we adopt prompt-level detection with GPT-4o to flag malicious queries. Results show that while the task-specific defense prompt for EHRAgent achieves a high precision (e.g., 131/135 on MIMIC), it fails to generalize to RAP or QA agents. In contrast, a more general prompt improves detection across tasks but leads to substantial false positives (e.g., 34/50 false alarms on EHR-MIMIC). This highlights a core challenge: simple rule-based defenses struggle to balance precision and generality, especially when indication prompts are carefully crafted.

Thank you for your valuable comments! We sincerely appreciate your recognition of MINJA’s practicality, innovation, and empirical validation.

W1: Isolating the impact of progressive shortening strategies

R1: Thank you for your insightful comment.

We conducted an ablation study on EHRAgent using e-ICU to assess how reducing the number of shortening steps in the PSS process affects the injection success rate (ISR).

Specifically, we compare our default PSS with fewer shortening steps and no PSS at all on three arbitrarily selected pairs (Pair 1, 4, 7) on EHRAgent for eICU.

The results show that reducing the number of shortening steps leads to a decrease in injection success rate (ISR), from 93.3% (full PSS) to 80% (fewer steps), and further down to 75.6% when PSS is removed entirely, which validates that PSS facilitates the injection of more malicious records into the memory bank as we expected.

We will add complete experiments and the results to our revision.

W2: ASR’s high variance indicates instability malicious influence

R2: Thank you for raising this concern.

We would like to clarify that our method mainly focuses on memory injection in realistic settings (see Page 2, Line 73) and, as expected, achieves consistently high ISR across settings. The observed variance in ASR mainly arises across different victim-target pairs, which is often seen in the backdoor literature. For example, [1] (Figure 11 & 12) also reports substantial ASR differences across different physical patterns.

To validate this, we conduct additional evaluations on both RAP and QA agents, using GPT-4 and GPT-4o as core models. We selected three arbitrary victim-target pairs (Pairs 1, 4, and 7), and sampled 18 test queries for each pair (with replacement for QA Agent due to its limited test queries). We repeated this process across three runs (test 1,2,3) and measured the standard deviation of ASR.

RAP Agent

QA Agent

The results demonstrate that ASR remains stable within each pair (standard deviation lower than 3% in RAP Agent, and lower than 6% in QA Agent).

Thanks again for raising this point, and we will add the results to our revision.

W3, L2, Q2: MINJA’s reliance on semantically plausible victim-target transitions, and how the bridging step affects injection success?

R3: Thanks for raising these two concerns! We’ll clarify them separately:

1. As for MINJA’s reliance on semantically plausible victim-target transitions, in most real-world agent applications, a semantically coherent transition between victim and target is both common and practical. We have demonstrated such applicability for four agent applications: in a healthcare agent (e.g., EHR agent), victim and target terms such as patient IDs (on MIMIC-III) or medication (on e-ICU) names are inherently meaningful and contextually coherent. Similarly, in a shopping agent (e.g., RAP agent), both victim and target entities typically represent product names, which often share strong semantic or categorical relationships. For the QA Agent, we consider terms from specific subjects as the victim.

   In other applications, such as code generation agents (e.g., MetaGPT [2]), financial advisors (e.g., FinGPT [3]), or travel assistants(e.g., TravelPlanner [4]), victim and target terms (e.g., function names or data types, financial terms, destinations) are also semantically meaningful, making coherent bridging both feasible and practical across domains.

   Additionally, MINJA’s reliance on semantically plausible victim-target transitions is not a limitation but a deliberate design choice. In practice, attackers can choose both victim and target terms to ensure logical coherence, making such transitions feasible and effective in real-world scenarios.

2. To directly show the importance of the bridging step (indication prompt) plausibility, we conducted an additional ablation study on RAP Agent using victim-target pairs 1, 4, and 7. In this setting, we replaced the entire indication prompt with the target term itself, thereby removing any semantically plausible bridging content. Note that an attacker will never do this in practice. As a result, no bridging steps were generated. This manipulation led to both ISR and ASR dropping to 0%, demonstrating that the presence of a logically plausible bridging step is crucial for successful injection.

W4, Q2, L1: The reliance on human-designed prompts limits scalability and requires attacker expertise. (Can the prompt generation be automated?)

R4: Thank you for raising this concern!

Our prompt generation does not demand attacker expertise and domain knowledge. In fact, the indication prompts can be generated fully automatically. Specifically, we provided the LLM with demonstrations from EHR Agent on MIMIC-III that include bridging steps, and instructed it to generate a logical transition (i.e., an indication prompt) for a fixed victim-target pair in eICU. We then tested the attack using the generated prompt on EHR Agent (eICU) for three pairs (Pair1, 4, 7).

Despite being generated by the model without any manual engineering, the prompts still retained some effectiveness, suggesting that our method supports potential automation and scalability. Note that this is merely a naive prompting strategy, and a more sophisticated automatic pipeline could likely yield even better results.

W5: The attack relies on victim terms that allow plausible bridging, limiting trigger selection flexibility.

R5: Thank you for raising this concern!

1. We would like to clarify that, in our setting, the attacker’s goal is to influence model behavior toward a specific victim term. Thus, the victim itself can be regarded as an implicit trigger instead of any explicit triggers for traditional backdoor attacks.

   Moreover, our victim selection represents a practical improvement over prior methods like BadChain, which rely on injecting explicit triggers into user prompts—an unrealistic assumption. In contrast, MINJA operates without access to user inputs and instead performs memory injection, better aligning with real-world threat scenarios.

2. Additionally, as discussed in (W3, L2, Q2) point 1., our framework is designed to be flexible, allowing the attacker to arbitrarily choose the victim term depending on their intent. For instance, in RAP Agent, we target product names, while in EHR Agent, we use entities such as patient IDs or medications. These choices demonstrate that our method can adapt to a wide range of victim-target pair selections across domains.

Q1: Table 1 presents Utility Drop (UD) with an upward arrow (“UD↑”),

R6: We thank the reviewer for pointing this out! UD should have been labeled as “UD↓ ”, indicating that smaller utility drop values are preferred. We will fix it in our revision.

[1] Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning

[2] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

[3] FinGPT: Open-Source Financial Large Language Models

[4] TravelPlanner: A Benchmark for Real-World Planning with Language Agents