DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering

We sincerely appreciate the reviewer's thorough evaluation regarding equations, symbol definitions, and methodological comparisons.

weakness i: Confusions about Equations

We welcome the opportunity to clarify Equations (7) and (8), which are defined in the manuscript as follows:

$ F(x) &= \int\_{-\infty}^{x} f(t)\mathrm{d}t = 0.5 \left( 1 + E\left( \frac{x - \mu}{\hat{\sigma} \sqrt{2}} \right) \right), \quad E(z) = \frac{2}{\sqrt{\pi}} \int\_{0}^{z} e^{-t^{2}}\mathrm{d}t \\\\ \gamma\_m&=\frac{\hat{\sigma}\left(E\left(\frac{p^m\_e-\mu}{\hat{\sigma}\sqrt{2}}\right)-E\left(\frac{p^m\_s-\mu}{\hat{\sigma}\sqrt{2}}\right) \right)}{2\left(p^m\_e-p^m\_s\right)} $

(i) As stated in line 158 of this paper, $E(\*)$ is the error function and is used to define the CDF of the Gaussian distribution. While $E(\*)$ shares the same functional form as the CDF of the half-normal distribution (with $\mu$=0 and $\sigma=1/\sqrt 2$ with support $R^+$), it is defined over the entire real domain $R$.

(ii) Equation (8) aims to compute a normalized difference quotient analogous to a derivative. Based on the derivation below, Equation (8) is actually equivalent to $\frac{F(z\_2)-F(z\_1)}{z\_2-z\_1}$, which lacks $2\sqrt 2$ in the denominator.

$ Assume& \quad z\_1=\frac{p^m\_s-\mu}{\hat{\sigma}}, z\_2=\frac{p^m\_e-\mu}{\hat{\sigma}}. \\\\ F(x)&=0.5\left( 1 + E\left( \frac{z}{\sqrt{2}} \right) \right). \\\\ \gamma\_m&=\frac{\hat{\sigma}\left(E\left(\frac{p^m\_e-\mu}{\hat{\sigma}\sqrt{2}}\right)-E\left(\frac{p^m\_s-\mu}{\hat{\sigma}\sqrt{2}}\right) \right)}{2\left(p^m\_e-p^m\_s\right)} \\\\ &=\frac{\hat{\sigma}\left(E\left( \frac{z\_2}{\sqrt{2}} \right)-E\left( \frac{z\_1}{\sqrt{2}} \right)\right)}{2((\hat{\sigma}z\_2+\mu)-(\hat{\sigma}z\_1+\mu))} \\\\ &=\frac{F(z\_2)-F(z\_1)}{z\_2-z\_1} $

(iii) When $p\_s$ and $p\_e$ reside near $\mu$ (i.e., the midpoint of the sequence), their normalized values $z\_1$ and $z\_2$ approach zero. Under these conditions, the resulting derivative $\gamma\_m$ (i.e., the positional value of the m-th paragraph) is high. The bigger $\gamma\_m$ value aligns with our core design objective: Mitigate LLMs' "lost-in-the-middle" by enhancing attention weights assigned to centrally located information.

(iv) Based on the derivation of Equation (8) above, the denominator excludes the $2\sqrt 2$ factor. Consequently, this formulation does not make $\gamma\_{m}$ a quite small number.

Should any uncertainties remain regarding this explanation, we welcome further discussion. To prevent confusion, we will enhance the textual descriptions of Equations (7) and (8) and add the complete derivation in the revised manuscript.

weakness ii: Ambiguous Symbols

(i) $A^S\_{h,l}$ denotes the attention score matrix of the h-th head in the l-th layer. Equation (3) first computes the initial attention scores through dot products. We then apply both CGW and RAS to update its value. Our formulation follows the expressions in [1]. Equation (12) uses the calculated contextual gate weight of each paragraph to update $A^S\_{h,l}$.

(ii) To resolve notational ambiguity, we clarify the distinct roles of $\mu$ and $\sigma$ across equations.

$\mu$ in Equation (6) denotes the mean of combined information flow $\mathcal{I}\_{comb}$, defined in Line 153; $\mu$ in Equation (7) serves as a normalization value for scaling; $\mu$ in Equation (8) represents the mean of sequence $\{0,1,\ldots, L-1\}$, stated in Line 160.

While $\sigma$ commonly denotes both sigmoid function and standard deviation, we use $\sigma$ for sigmoid function and $\hat{\sigma}$ for standard deviation in Equations (6)-(8) to disambiguate.

However, the similar symbols may cause misunderstandings, so we revised Equations (6)-(8) to make them clearer:

$ v\_m &= 0.5\cdot \text{Sigmoid}\left( \frac{\mathcal{I}\_{Comb^m} - \mu\_I}{\sigma\_I} \right)+0.5,m \in \{1,\ldots,C\}, \\\\ F(x) &= \int\_{-\infty}^{x} f(t) \, \mathrm{d}t = 0.5 \left( 1 + E\left( \frac{x - \mu}{\sigma \sqrt{2}} \right) \right), \quad E(z) = \frac{2}{\sqrt{\pi}} \int\_{0}^{z} e^{-t^{2}} \, \mathrm{d}t, \\\\ \gamma\_m&=\frac{\sigma\_p\left(E\left(\frac{p^m\_e-\mu\_p}{\sigma\_p\sqrt{2}}\right)-E\left(\frac{p^m\_s-\mu\_p}{\sigma\_p\sqrt{2}}\right) \right)}{2\left(p^m\_e-p^m\_s\right)}, $

(iii) To explicitly combines $v\_m$ and $g\_m$ while constraining $w\_m$ within $[\beta,1]$, we introduce an intermediate variable $w\_m^{'}$ for Equations (10) and (11):

$ w\_m^{'}&=v\_m \cdot {g\_m}^\alpha, \\\\ w\_m&=(1-\beta)\frac{w\_m^{'}-\text{min}(w\_i^{'})}{\text{max}(w\_i^{'})-\text{min}(w\_i^{'})}+\beta,i \in \{1,\ldots,C\} $

(iv) Lastly, To mitigate potential notational ambiguities, we will add Table 1 of key symbols in Appendix.

Table 1 Symbols and descriptions.

[1] Xiao, Da, et al. "Improving transformers with dynamically composable multi-head attention." arXiv preprint arXiv:2405.08553 (2024).

weakness iii: Comparisons of Different Methods

(i) As described in line 205 of the manuscript, our baseline implementation deploys all LLMs in the ''bfloat16'' data format and employs greedy-decoding to ensure full reproducibility of generated outputs.

(ii) The Calibrated attention proposed in [2] shares a similar motivation with our approach in seeking to enhance LLMs' focus on central information within long contexts. However, DSAS extends this objective by actively promoting information flows from supporting paragraphs while suppressing information flows from irrelevant information. Specifically, Calibrated Attention quantifies the model's positional bias by fixing a query-paragraph pair and measuring how the self-attention distribution changes as the paragraph is relocated within the input. During inference, it subtracts the measured positional bias from the original attention scores to obtain calibrated attention weights. One limitation is its requirement to precompute the positional bias for each paragraph, which may entail significant computational overhead. Since the implementation of [2] is not available, we reproduced a version guided by its core methodology. The results are summarized in Table 2.

(iii) To our knowledge, existing attention efforts mainly concentrate on sparse attention mechanisms [3], context compression [4], and KV cache optimization [5], while research addressing performance improvements in long-context generation remains underexplored.

Table 2 Comparison results of different methods.

[2] Hsieh, Cheng-Yu, et al. "Found in the middle: Calibrating positional attention bias improves long context utilization." arXiv preprint arXiv:2406.16008 (2024).

[3] Jiang, Huiqiang, et al. "Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention." arXiv preprint arXiv:2407.02490 (2024).

[4] Zhang, Peitian, et al. "Long context compression with activation beacon." arXiv preprint arXiv:2401.03462 (2024).

[5] Qin, Ziran, et al. "Cake: Cascading and adaptive kv cache eviction with layer preferences." arXiv preprint arXiv:2503.12491 (2025).

weakness iv: Definitions of Equations

We thank the reviewer for raising this point and apologize for the oversight. Equation (1) computes attention weights summed across all tokens in the question (length $Q$). In contrast, Equation (2) utilizes the attention weight for a single target token $t$. To make the equations directly comparable, our code implementation averages the accumulated attention weights over the sequence length $Q$ in Equation (1). We will correct Equation (1) accordingly in the final version. Figures 3 and 7 remain correct with this revision. The updated Equation (1) is given below:

\mathcal{I}\_{p^m,q} = \frac{1}{Q} \sum \text{Top-K} \left( \left\\{ \sum\_{i \in q} A\_l^W(i,j) | j \in p^m \right\\} \right).

We sincerely thank the reviewer for the thoughtful comments regarding the computational overhead, theoretical justification, hyperparameter tuning, model generalization, and method design. Based on the weaknesses you identified, we have organized our responses into the following parts:

Weaknesses1: Computational Overhead and Latency

This discussion refers to paragraph 1 in the weaknesses the reviewer proposed.

(i) We acknowledge that DSAS needs deep access to the attention stack, but it is a lightweight plugin and does not require major changes to the inference loop. DSAS intervenes only in specific layers (the final 50 %), where it computes a contextual gate weight for each paragraph based on the raw attention scores and dynamically adjusts those scores. The overall procedure preserves the standard Transformer computations and requires no architectural modifications.

(ii) We acknowledge that DSAS has only been applied to open-source Transformer models. However, this constraint does not imply a technical shortcoming. In fact, existing attention optimization methods are also limited to such architectures.

(iii) Regarding computational overhead, as mentioned in Appendix D, the additional operations of DSAS introduce $\mathcal{O}(LQ)$ time complexity, where $L$ and $Q$ denote the input length and the question length, respectively. Since it scales linearly with $L$, it constitutes a considerably smaller fraction of the total computation compared to the quadratic $\mathcal{O}(L^2)$ complexity of standard attention mechanisms.

(iv) To evaluate inference latency, we conducted experiments on Llama-3.1-8B using a single NVIDIA A800 GPU. Table 1 reports the average per-sample inference time for a LongBench subset (L-HotpotQA). Results indicate DSAS introduces a 10% latency increase over the baseline method. We contend the modest overhead is worthwhile given DSAS's consistent performance gains across diverse tasks.

Table 1 Average latency comparisons on L-HotpotQA of Llama-3.1-8B.

Weaknesses2: Lack of Theoretical Justification

This discussion refers to paragraph 2 in the weaknesses the reviewer proposed. We would like to offer the following clarifications on the techniques used in the manuscript:

• Although the validation of DSAS is mainly experimental, the design of DSAS is firmly grounded in an information-theoretic perspective on attention rather than ad hoc heuristics. In Section 2, we analyze information flows at both paragraph disparity and answer quality levels, and these insights ​directly provide the basis​ for our CGW and RAS modules.
• The used techniques align with our research goal. e.g., we adopt Gaussian CDF to compute positional values for enhancing the model's focus on centrally located paragraphs, due to the ''U‑shaped attention bias'' [1] in LLMs.
• We designed DSAS as a truly plug-and-play mechanism whose generalizability is empirically validated by numerous experiments. DSAS derives consistent improvements across diverse LLMs and tasks. In addition, in response to Q1, we have extended DSAS to additional tasks, including summarization and code completion, further demonstrating its generalizability.

[1] Hsieh, Cheng-Yu, et al. "Found in the middle: Calibrating positional attention bias improves long context utilization." arXiv preprint arXiv:2406.16008 (2024).

Weaknesses3: Details of Hyperparameter Tuning

This discussion refers to paragraph 2 in the weaknesses the reviewer proposed.

(i) The hyperparameter configuration "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7" is effective across diverse LLMs on various tasks (4.2% average gains on Llama-3.1-8B and Qwen2.5-14B). Our work includes the design of DSAS as well as the finding of the robust hyperparameter configuration. We believe the robustness benefits from the well-designed formulations and there is no need for hyperparameter tuning process.

(ii) In response to Q1, we have extended DSAS to additional tasks, including summarization and code completion. Table 1 confirms that the hyperparameter configuration "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7" maintains consistent efficacy across these unseen domains.

Weaknesses4: Generalization across Models

This discussion refers to paragraph 3 in the weaknesses the reviewer proposed. We have discussed some points of the weakness. Please refer to the response to Weaknesses3 above.

(i) We acknowledge that LLM+DSAS underperforms the baseline under some hyperparameter settings on Llama-3.2-3B. However, this does not indicate instability of the method itself. Rather, it reflects the inherent limitations of small‑scale models, which we attribute to their relatively weaker semantic comprehension capacity and greater sensitivity to input noise, which requires more precise hyperparameter tuning. In contrast, mainstream 7B–14B models exhibit significantly lower sensitivity to these hyperparameters (Figures 5, 9, 10).

(ii) Medium-sized LLMs refer to LLMs with parameter sizes ranging from 7B to 14B, regardless of architectural family (e.g., LLaMA-3.1-8B, Qwen2.5-7B, Mistral-7B, Qwen2.5-14B).

Weaknesses5: Method Design

This discussion refers to paragraph 4 in the weaknesses the reviewer proposed.

(i) As noted in Introduction, existing approaches either sacrifice global token interaction or require additional training, which limits their adaptability. In contrast, DSAS does not reflect research overfit and is founded on a systematic analysis of information flows to guide attention optimization.

(ii) In response to Weaknesses2, we have provided a detailed explanation of the theoretical basis of DSAS.

(iii) In response to Weaknesses1,3,4, quantitative results confirm performance gains across diverse tasks and models while maintaining inference efficiency. The strong capability enhancement indicates DSAS addresses root causes rather than symptoms.

Q1: Robustness of DSAS

We appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., "Now, write a one‑page summary of the report." for GovReport; "Next line of code:" for LCC).

Because DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 2.

We observe: (a) DSAS achieves consistent performance gains across ​​all five tasks across three medium-size model architectures​​. (b) The positive results extend to diverse tasks (summarization and code completion), confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​.

Table 2 Comparison results on summarization and code completion tasks with "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.

Q2: Computational Overhead and Latency

We have discussed this question. Please refer to the response to Weaknesses1 above.

Q3: Hyperparameter Tuning and Data Splitting Strategy

We have discussed this question. Please refer to the response to Weaknesses4 above.

Furthermore, to prevent data leakage, all experiments are conducted on the validation sets of HotpotQA, 2WikiMultiHopQA, MuSiQue datasets. For LongBench datasets, we tested all samples following the common practice.

Q4: Hypotheses for the Best Performance on Medium-sized Language Models

(i) Hypothesis 1: Balance between comprehension capacity and optimization potential.

Medium-sized LLMs (e.g., Llama-3.1-8B, Qwen2.5-14B) demonstrate adequate semantic understanding yet remain vulnerable to input noise, especially in long-context scenarios. DSAS overcomes this by analyzing information flows to precisely identify critical paragraphs and amplify model focus. In contrast, smaller models (e.g., Llama-3.2-3B) lack sufficient comprehension capacity, while larger models (e.g., Qwen2.5-32B) approach the task performance ceiling, leaving little room for further gains.

(ii) Hypothesis 2: Dependence on positional awareness.

Medium-sized LLMs exhibit greater vulnerability to the "lost-in-the-middle" phenomenon. DSAS mitigates this through its position-aware weighting $g\_m$. The experiments of ablation study reveal that removing $g\_m$ causes a marked performance drop for Qwen2.5-7B and Llama-3.1-8B, whereas Qwen2.5-14B's exhibits a smaller decline.

We acknowledge the reviewer's valuable insights regarding potential paragraph ordering sensitivity, DSAS's scalability, and hyperparameter configurations.

Q1: Experiments of Non-standard Orderings

As DSAS is designed to improve attention toward centrally located paragraphs, thereby mitigating the LLMs' inherent ''lost‑in‑the‑middle'' issue. It is important to evaluate DSAS's robustness to input order. Accordingly, we conducted an experiment where we randomized the ordering of paragraphs in all test samples while deliberately increasing the probability of supporting paragraphs occurring at the edges. Table 1 presents results for both original and shuffled inputs, using the consistent configuration "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7". Results indicate minor performance fluctuations across ordering variants. This stability demonstrates DSAS's robustness to varying input orders, primarily benefits from our weighting strategy. We think that when evidence appears at the edges​ (where $g\_m$ in Equation (9) is reduced)​, DSAS attenuates the adjustments to the model's native attention distribution. This design intentionally leverages the model's inherent capacity​​ to identify and amplify information flows from the evidence at the edges, which aligns with the ''U-shaped attention bias'' phenomenon in [1].

We observed one case in MuSiQue with three supporting paragraphs (evaluated using Llama-3.1-8B) that succeeded with the original ordering but failed after shuffling. Analysis revealed that shuffling positioned the supporting paragraphs more centrally. The observed failure in this instance may stem from insufficient focus amplification on central information despite applying DSAS. This is supported by our information flow metric $\frac{1}{Q}\mathcal{I}\_{p^s,q}+\mathcal{I}\_{p^s,t}$, where the shuffled sample is 27.75 compared to 29.87 for the original. Nevertheless, DSAS maintains robust performance across the vast majority of samples.

Table 1 Results of original and shuffled input orderings.

[1] Hsieh, Cheng-Yu, et al. "Found in the middle: Calibrating positional attention bias improves long context utilization." arXiv preprint arXiv:2406.16008 (2024).

Q2: Performance of Other Tasks

We appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., "Now, write a one‑page summary of the report." for GovReport; "Next line of code:" for LCC).

Because DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 2.

We observe: (a) DSAS achieves consistent performance gains across ​​all five tasks across three medium-size model architectures​​. (b) The positive results extend to diverse tasks (summarization and code completion), confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​.

Table 2 Comparison results on summarization and code completion tasks with "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.

Q3: Hyperparameter Settings

(i) As demonstrated in Figure 5 and Appendix B, we evaluated the impact of hyperparameters $K, n, \alpha, \beta$ across multiple models. While performance exhibits fluctuations with different hyperparameter configurations, the vast majority of settings still outperform the baseline. Notably, substantial variability occurs only with the smaller Llama‑3.2‑3B model (Figure 8), which we attribute to its relatively weaker semantic comprehension capacity and greater sensitivity to input noise, which requires more precise hyperparameter tuning. In contrast, mainstream 7B–14B models exhibit significantly lower sensitivity to these hyperparameters.

(ii) Section 4.2 confirms that the fixed hyperparameter combination "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7" demonstrates stable and effective performance. The results highlight two properties of DSAS: (a) Model-agnostic: It consistently improves performance across diverse architectures (Llama, Qwen, and Mistral) ranging from 3 B to 32 B parameters. (b) Task-agnostic: It yields uniform improvements across multiple tasks. Furthermore, Figures 2 and 6 illustrate that information flows from supporting and negative paragraphs diverge substantially in deeper layers (last 50 %), whereas shallow layers exhibit limited divergence. Figure 5 and Appendix B further confirm that $n$=50% is the optimal choice, reinforcing the robustness of the hyperparameters.

(iii) In response to Q2, we have extended DSAS to additional tasks, including summarization and code completion. Supplementary results in Table 2 confirm that the hyperparameter configuration "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7" maintains consistent efficacy across these unseen domains.

We thank the reviewers for their insightful questions regarding the proofs for the discrimination capability, the need for further analysis on paragraph number, the scalability of DSAS on other tasks, and our applicability for context length extension.

Cons1: Proofs for the Enhanced Discrimination Capability

(i) We conducted an error analysis of DSAS by examining the confusion matrix visualization in Appendix C. Figure 11 illustrates the global attention matrix among supporting/negative paragraphs, questions, and target. DSAS substantially enhances the LLM's focus on supporting paragraphs, demonstrating improved discrimination capability.

(ii) In addition, to complement our findings, we have added a quantitative analysis. Specifically, Equations (1) and (2) define the information flows from the m‑th paragraph to the question and to the target, respectively. Building on these definitions, we introduce two metrics: the supporting paragraph gain ratio $\Delta \mathcal{I}\_{p^s}$ and the negative paragraph suppression ratio $\Delta \mathcal{I}\_{p^n}$, which are formulated as:

$ \Delta \mathcal{I}\_{p^s}&=\frac{\left(\frac{1}{Q}\mathcal{I}\_{p^s,q}^{'}+\mathcal{I}\_{p^s,t}^{'}\right)-\left(\frac{1}{Q}\mathcal{I}\_{p^s,q}+\mathcal{I}\_{p^s,t}\right)}{\frac{1}{Q}\mathcal{I}\_{p^s,q}+\mathcal{I}\_{p^s,t}}, \\\\ \Delta \mathcal{I}\_{p^n}&=1-\frac{\frac{1}{Q}\mathcal{I}\_{p^n,q}^{'}+\mathcal{I}\_{p^n,t}^{'}}{\frac{1}{Q}\mathcal{I}\_{p^n,q}+\mathcal{I}\_{p^n,t}}, $

where $Q$ denotes the question token length; $\mathcal{I}\_{\cdot,q}^{'}$ and $\mathcal{I}\_{\cdot,t}^{'}$ denote the information flows from supporting/negative paragraphs to the question and target after using DSAS, respectively. We calculate the information flows and metrics, and observe that LLM+DSAS evidently increases the information flows from supporting paragraphs while suppressing information flows from negative ones compared to the baseline LLM. The results are shown in Table 1.

Table 1 Results of information flows and metrics on HotpotQA, 2WikiMultiHopQA and MuSiQue of Llama-3.1-8B.

(iii) Finally, we selected one example from MuSiQue and visualized its layer-wise attention matrices for both the baseline LLM and LLM+DSAS. The matrices for LLM+DSAS exhibit lower attention dispersion [1], indicating its ability to concentrate on consistent token positions within supporting paragraphs. In contrast, the baseline LLM fails to filter information flows from negative paragraphs.

[1] Qin, Ziran, et al. "Cake: Cascading and adaptive kv cache eviction with layer preferences." arXiv preprint arXiv:2503.12491 (2025).

Cons2: Further Analysis on Different Subsets

We have conducted a further analysis on different subsets by changing the number of supporting/negative/total paragraphs.

(i) Since the MuSiQue dataset contains 2-4 supporting paragraphs per sample, we have conducted experiments on subsets grouped by supporting paragraph count. Table 2 presents the comparative performance. We observe minimal performance gaps with varying paragraph counts, demonstrating that the baseline LLM is able to answer the questions and DSAS better capture relevant information flows to generate more accurate answers.

Table 2 Results of different supporting paragraph count setting on MuSiQue through the F1-score (%).

(ii) Next, to examine the effect of total paragraph count, we randomly sampled 200 examples from the MuSiQue dataset and constructed inputs with 10, 20, 30, and 40 paragraphs. Since each example originally contains 20 paragraphs, the 20‑paragraph setting uses the original sample inputs. For the 10‑paragraph setting, we randomly removed ten of the negative paragraphs. For the 30‑paragraph and 40‑paragraph settings, we randomly adding 10/20 paragraphs to construct inputs, respectively, drawn from other MuSiQue samples. The results of varying total paragraph count are reported in Table 3.

The results indicate that: (a) As the total paragraph count increases (longer input sequences), performance generally declines. However, LLM + DSAS exhibits a slower degradation trend, as it effectively suppresses information flows from negative paragraphs. (b) Llama‑3.1‑8B + DSAS even outperforms Qwen‑2.5‑14B under all settings, demonstrating the effectiveness of DSAS as a plug-and-play strategy.

Table 3 Results of different total paragraph count setting on MuSiQue through the F1-score (%).

Q1: Performance of Other Tasks

We appreciate the reviewer's suggestion to evaluate DSAS's scalability beyond Multi‑doc QA. Accordingly, we applied DSAS to the summarization (GovReport, QMSum, MultiNews) and code completion (LCC, RepoBench-P) tasks in the LongBench benchmark. After examining the prompt templates, we selected the final instruction sentence as the anchor (e.g., "Now, write a one‑page summary of the report." for GovReport; "Next line of code:" for LCC).

Because DSAS requires paragraph‑level segmentation and Multi‑doc QA naturally provides this, we instead segment these new tasks by fixed token counts. Specifically, we use 500‑token segments for GovReport, QMSum, and RepoBench‑P, and 200‑token segments for MultiNews and LCC, ensuring a medium number (5-40) of paragraphs per example. The comparison results are shown in Table 4.

We observe: (a) DSAS achieves consistent performance gains across ​​all five tasks across three medium-size model architectures. (b) The positive results extend to diverse tasks (summarization and code completion), confirming that DSAS's benefits are ​​not confined to Multi-doc QA​​.

Table 4 Comparison results on summarization and code completion tasks with "$K$=10, $n$=50%, $\alpha$=1.0, $\beta$=0.7". The evaluation metrics for L-GovReport, L-QMSum, L-MultiNews and L-LCC, L-RBP are Rouge-L score and Edit Sim, respectively. RBP denotes RepoBench-P.

Q2: DSAS for Context Length Extension

(i) Our proposed DSAS aims to address two challenges in Multi‑doc QA: limited long-range dependency and persistent ''lost-in-the-middle'' issue. Although DSAS has not been extended in scenarios exceeding the fixed window size, it offers a plug‑and‑play solution that enhances discrimination without architecture modifications.

(ii) In future work, we plan to extend DSAS beyond the fixed context window by integrating it with techniques such as StreamingLLM [2], which enlarges effective context length by retaining the KV cache of initial tokens and the most recent tokens. Specifically, we will investigate methods for dynamically preserving information from key paragraphs while discarding irrelevant noise during generation, thereby supporting substantially longer contexts.

[2] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." arXiv preprint arXiv:2309.17453 (2023).

We will incorporate all the analysis and supplementary experiments into the revised manuscript.