Mind the Quote: Enabling Quotation-Aware Dialogue in LLMs via Plug-and-Play Modules

Dear Reviewer NVEt, we sincerely appreciate you taking the time to read our manuscript and provide positive and valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

Weakness 1: Limited Model Diversity: Testing is confined to two LLMs; broader testing across different architectures would bolster generalizability.

We sincerely appreciate your suggestion. Below we provide additional results obtained with our method on the phi3.5-mini-Instruct model to demonstrate its generalization ability:

Table: Results of QUADA on phi3.5-mini-Instruct

As the table shows, QUADA remains the top performer on phi3.5-mini-Instruct, indicating that our method generalizes well across model architectures.

Weakness 2: Interpretability vs. Controllability Trade-off: While attention maps (Figure 3) offer some interpretability, QUADA's bottleneck MLPs remain a black box. We don't know what "guidance vectors" (b^q and b^v) learn, making diagnosis difficult when the model fails. In contrast, methods like MARKER-INSERTION, despite poorer performance, are more transparent and controllable.

Your point is very valuable, we provide additional clarification here. Building on the attention heat-maps in Fig. 3, we performed a quantitative analysis of the relationship between the vectors $b^v$ and $v$. The average cosine similarity across all layers is 0.62, with more than 30% of layers above 0.80; the norm of $b^v$ is about 0.6 × $v$. These statistics suggest that $b^v$ reinforces the value information of the quoted span.

For controllability, we introduced a hyper-parameter λ to scale the injection strength of $b^q$ and $b^v$. We set λ = 0, 0.25, 0.5, 0.75 and 1, and evaluated a Qwen2.55-3B-Instruct model equipped with the trained QuAda module. The results are shown below:

Table: Injection-strength control experiment for QuAda

Taken together, Fig. 3, the $b^v$–$v$ analysis, and the injection-strength study, QuAda is less visually interpretable than MARKER-INSERTION, but it is not entirely a black box. Its outputs correspond to explicit instructions on where the LLM should attend, plus strengthened representations of those regions. We will add these experiments to the paper to further support QuAda’s interpretability.

---

Weakness 3: Scenario Coverage Limitations: This work claims its five diagnostic tasks span practical quoting behaviours, which can be a strong assertion. However, real-world references can also be composite, involving multiple types simultaneously (e.g., Multi-Span, Exclude, and Info-Combine). The paper lacks testing on the model's performance in these crucial mixed scenarios.

Thank you for raising this constructive point. Our benchmark already includes Exclude (Multi-Span) and Info-Combine (Multi-Span) cases, accounting for 71.4 % of the Exclude set and 24.4 % of the Info-Combine set, respectively. We evaluated these subsets separately, and the results are:

Table: Performance of QuAda on mixed scenarios

As the table shows, QuAda effectively handles these mixed scenarios.

---

Question 1: More Computational Details: Can the author provide more details, beyond what is mentioned in Appendix E, about (1) the training costs (memory usage and training time), and (2) the additional memory/latency overhead by QuAda during inference, for better reproducibility assessment?

Thank you for the question. For the Qwen2.5-3B-Instruct model, training on 8 × H20 GPUs completes three epochs in 1h25min, with each GPU using about 28~30 GB of memory.

During deployment we compared runs with and without QuAda: the additional memory per GPU was under 1 GB, and the per-sample latency increase is below 0.6 s. These results confirm that QuAda is genuinely lightweight and efficient.

---

Question 2: Real-World Data: Have the authors considered testing QuAda on real-world conversational datasets rather than purely LLM-synthesized datasets? If not, what are the challenges?

Thank you for the question. We also built a manually curated benchmark and evaluated QuAda on it; the results are:

Table: QuAda on the manual benchmark

These numbers show that our method maintains high performance, confirming its practicality in real-world dialogue. We will also release this manual benchmark, and Table 6 of the paper further demonstrates the consistency between the automatically constructed and manual benchmarks.

---

Question 3 & Limitation: Ethical Concerns: Are there potential negative societal impacts (e.g., misinformation amplification) to address? For example, QuAda employs simple bottleneck MLPs, which can fall victim to a backdoor attack. Including this could resolve an ethical gap, maintaining or increasing the current rating. I suggest adding a paragraph discussing ethical concerns, such as the risk of manipulating conversations, to provide a more balanced perspective.

We thank you for the suggestion and questions, they are important for broadening QuAda’s impact and ensuring that the community can deploy it safely. Recent studies show that parameter-efficient bottleneck adapters (e.g., LoRA, QuAda-style MLPs) can carry stealthy Trojan behaviours while preserving benign utility [1-3]. Similarly, we acknowledge that QuAda could be exposed to such attacks. To mitigate this risk, we will :

(i) release official QuAda checkpoints—with hash signatures—compatible with mainstream open-source models;

(ii) open-source the complete data-construction pipeline and all the training sets and benchmarks we have created, enabling anyone to train their own QuAda and allowing the community to inspect the data for hidden “triggers”;

(iii) incorporate adversarial training in future work to guard against backdoor attacks during training [4].

Furthermore, if QuAda is adopted in more scenarios, attackers might intercept the span information via network and inject spans that misdirect the model’s attention. We therefore also stress the need to secure the network channel.

We will highlight this discussion in the revised paper and document the above measures in the official QuAda repository.

References

[1] The Philosopher's Stone: Trojaning Plugins of Large Language Models

[2] LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem

[3] Lora-as-an-attack! piercing llm safety under the share-and-play scenario

[4] Sleeper agents: Training deceptive llms that persist through safety training

Dear Reviewer yQpy, we sincerely appreciate you taking the time to read our manuscript and provide positive and valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

---

Weakness 1 & Question 3: While the five diagnostic scenarios are clear, they may not reflect the full diversity of real-world quoting behavior. The auto-generated dialogues also seem structurally simple—primarily single-turn question-answer pairs over selected spans—potentially limiting their contextual richness. The training data seems mostly constructed as “single-turn queries over quoted spans.” Are there multi-turn or nested quoting cases in the benchmark?

Thank you for the insightful question; it inspired us to add multi-turn and nested quoting scenarios to improve the benchmark’s coverage. We therefore modified our data-construction pipeline and created a multi-turn version of the Base task, where consecutive questions each quote a different span. Owing to the short rebuttal timeline, we could not prepare sufficient multi-turn training data, so we evaluated the generalization of models trained only on the original single-turn data:

Table: Performance under single-turn versus multi-turn quotation

Even without dedicated training, QUADA-equipped LLMs remain strong, implying that both single and multi-turn quoting rely on the same span-awareness capability. We will augment the dataset and add corresponding experiments in future work. Thank you again for the thoughtful suggestion; it substantially improves our study.

Weakness 2: The core approach—using MLPs to modulate attention scores—is reasonable but somewhat straightforward, offering only modest technical innovation beyond prior attention-steering work.

We understand your concern. Actually, QuAda modifies both the Query and Value sides of attention, not just the attention scores as in prior attention-steering work. Quotation-aware dialogue requires two properties at the attention level:

1. When a context span is quoted, its semantics—not only its attention weight—may change. For example, if the quoted token is a pronoun, its value vector should be adjusted so that downstream QA can resolve the reference. Prior approaches cannot do this. We therefore add a bias vector $b^v$ to the value $v$ of quoted tokens. Table 7 shows that introducing $b^v$ markedly improves performance. An average cosine similarity of 0.62 between $b^v$ and $v$ across layers confirms that the value vector is indeed adapted when its token is quoted.

2. The attention weights assigned to tokens within the quoted span should be computed adaptively, rather than being fixed in advance as in prior attention-steering methods. In the Exclude scenario the attention should be small, whereas in the Base scenario it should be large. The bias vector $b^q$ captures this semantic signal and supplements the original query vector during attention calculation.

By addressing these two aspects—both absent in prior attention-steering work—our design achieves quotation awareness in a simple yet effective manner.

Weakness 3 & Question 1: In the current setup, the span and task type appear to need to be manually defined, which creates a gap between this method and real-world usage. The system does not yet solve the end-to-end problem exemplified by the motivating example in the abstract. Does QuAda rely on pre-specified spans and task types at inference time? This greatly affects practical applicability.

We focus on the scenario in which a user engages in dialogue by selecting the span they wish to quote. This situation occurs frequently in everyday applications, so we aim to endow the model with this ability. When the user highlights a region (e.g., on an app page or in a browser), the client can automatically call an offline tokenizer to obtain the token span of the quoted text and feed it to the LLM. The paper addresses how to perform span-conditioned generation given that span; the concrete engineering pipeline will be developed later and is not included in this manuscript, but we appreciate the chance to clarify this point.

As for the task type, after a single model is trained with QUADA, the user never needs to specify the task manually. The user simply inputs natural language to express their request, exactly as shown in our examples.

Question 2: In the ablation section (Table 7), only query-only, value-only, and full QuAda variants are reported. Could you add a “no-modulation” baseline, where none of the span-aware adapters are applied.

To address your question, we have added the “vanilla-no-modulation (Vanilla in Table 4)” and “trained-no-modulation” baselines to Table 7. The latter is a QUADA adapter trained on Qwen-2.5-3B-Instruct without providing the span. The results below show that span-conditioned generation via QUADA is indispensable for the quotation task:

Table: Ablation study with “no-modulation” baselines

Question 4: Real dialogue often contains indirect references (e.g., “that thing we discussed earlier”). Have the authors considered testing QuAda on loosely specified quotation cases?

Thank you for raising this question, we wish to clarify the scope of our work. The paper targets situations in which the user explicitly designates a portion of prior text for the LLM to respond to (or exclude), typically through actions such as highlighting or quoting. The indirect references you mention are indeed related, but they are not the main focus here. Evaluating an LLM’s ability to recall relevant historical information under user-specified constraints is certainly interesting, and we will follow future work in that direction.

Dear Reviewer rbX8, we sincerely appreciate you taking the time to read our manuscript and provide positive and valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

---

Weakness 1: The paper could provide deeper ablation studies to isolate the contributions of each component of QUADA (e.g., query side, value side) which will perhaps provide a better intuition as to contribution of each - similar to the intuition in section 4.3.

We thank the reviewer for the insightful suggestion and fully agree that a deeper ablation is valuable for understanding how each component of QUADA contributes to overall performance.

In section 4.4, we isolate the query-side and value-side QUADA, showing that disabling either one noticeably degrades accuracy, which indicates that both sides are indispensable. Following your recommendation, we additionally trained a variant that removes span-conditioned modulation altogether (i.e., no query- or value-side QuAda). This model, trained on Qwen-2.5-3B-Instruct, performs poorly across all five diagnostic settings, underscoring that span-conditioned generation is the key driver of QUADA’s gains.

Ablation of the components of QuAda on All Benchmarks (MCQ benchmark)

Moreover, we measured the cosine similarity between $b^v$ and the original value vector $v$: the average is 0.62 across layers, with 30 % of layers above 0.80. The norm of $b^v$ is roughly 0.6 × $v$. These statistics indicate that $b^v$ consistently reinforces value representations of quoted spans.

---

Weakness 2: Although the parameter increase is small (<2.8%), the paper does not discuss inference speed or memory overhead in practical deployment, especially for very large models. Can you provide more details on the runtime and memory overhead introduced by QuAda?

We understand the reviewer’s concern about deployment cost. On 8*H20 GPU, we measured both memory and per-sample latency duringinference with Qwen-2.5-14B-Instruct:

As shown in the table, QuAda does not significantly affect memory usage or inference time during inference, which fully demonstrates its efficiency.

---

Question 1: From the description of the QUADA architecture, the module is designed as a plug-and-play adapter that is agnostic to the specific sub-task as long as the task can be formulated as span-conditioned generation. However, I wonder what are the data, and performance implications of adding more sub-tasks (beyond the five described)?

We thank the reviewer for the question. As discussed in the paper, the five sub-tasks already cover most quotation-aware dialogue scenarios. All code for the data-construction pipeline has been provided in the Supplementary Material and will be open-sourced. If further training or testing is required for an additional sub-task, one only needs to adjust a few attribute settings to generate the needed data.

For example, we used the pipeline to create a Chinese test set for the Base scenario to evaluate QUADA’s cross-lingual generalisation:

Following the same procedure in the paper, we manually reviewed the Chinese data and found the quality sufficient. We will release this benchmark along with the pipeline.

Dear Reviewer 53qF, we sincerely appreciate you taking the time to read our manuscript and provide positive and valuable feedback. Below are our responses to the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.

---

Weakness 1: Although the dataset construction process involves multiple quality assurance steps, it remains synthetic in nature. That said, after manually reviewing several examples, I am generally positive about the data quality and find the contribution reasonable.

We thank the reviewer for taking the time to manually inspect our synthetic data and for acknowledging its quality. To ensure that every instance fully satisfies the required ability for its sub-task, we apply the strict filtering steps described in the Data-Construction Pipeline. The proportion of discarded examples for each sub-task is:

This aggressive filtering strategy is the main reason the final dataset attains the high quality observed during your review.

---

Weakness 2 & Question 2: While I agree with the significance of the task, I am concerned that the dataset may not be sufficiently challenging. Given the high performance reported in the experimental section, I question whether this benchmark can sustain long-term research value, especially as the language capability of foundation models keeps growing. Have the authors evaluated stronger base models to better calibrate the benchmark? This could be either larger open-source LLMs or well-performing closed-source models.

We fully understand your concern and consider it highly forward-looking. We address it from three perspectives:

First, our current benchmark is difficult even for today’s leading open and closed-source models without the QUADA module. We evaluated GPT-4o (closed source) and DeepSeek-R1 (open source) on the benchmark, and the results are:

Table: Performance of state-of-the-art open and closed-source models on our benchmark

Second, the primary goal of this work is to give existing LLMs application-level quotation-dialogue capability. This required a benchmark that matches everyday use-case difficulty and a training method that lets mainstream LLMs achieve strong results. Our data pipeline incorporates multiple validation stages, including manual audits, to ensure that each sample faithfully mirrors real-world scenarios, making the benchmark results reflects practical performance. The result that QUADA allows a comparatively small LLM to achieve state-of-the-art results demonstrates the effectiveness of this design. We believe stronger models equipped with QUADA will do even better, allowing developers to adopt our dataset and module to add robust, deployable quotation-dialogue ability.

Finally, we fully understand your desire to create more challenging benchmarks covering harder quotation scenarios and to keep pushing the community forward. This is exactly our plan. Future work will explore contexts containing images and voice, multimodal mixed quotation, and more. Building on this study, we will continue to propose benchmarks and methods for increasingly complex settings to advance quotation-aware dialogue in LLMs.

---

Question 1: Is there any failure analysis from the evaluation? Considering that QUADA achieves high scores, analyzing and attributing the small number of failure cases could help the understanding of the challenges (or lack thereof) in this task.

Thank you for the suggestion. We performed a case study on the outputs of the models trained in the paper and found that the remaining errors mainly fall into two categories, both limited by the capability of the base model on which QUADA is built:

1. The model correctly attends to the quoted span but also erroneously incorporates information from non-quoted context, leading to a wrong answer.
2. The model attends to the quoted span but misinterprets its content, again producing an incorrect answer.

These findings indicate that QUADA has already endowed the LLM with the span-awareness required for quotation-based dialogue. Both error types will diminish as the underlying model’s comprehension and reasoning abilities continue to improve (see Fig. 4).

Thanks for the clarification. I would like to follow up with the discussion around W2/Q2.

"First, our current benchmark is difficult even for today’s leading open and closed-source models without the QUADA module

...

Qwen2.5-3B-Instruct (QuAda) 95.2% 94.6% 92.8% 85.8% 90.2%"

Do this statement and your experiments imply that, with QUADA, the task (or at least the test set) you proposed is solved or almost solved?