FlowPrune: Accelerating Attention Flow Calculation by Pruning Flow Network

Dear reviewer:

Thank you for your constructive comments, Your valuable comments have been of considerable assistance to us.

First, we would like to emphasize the novelty of our FlowPurne method. This method helps to reduce the running time for interpreting behaviors of large-scale transformer models. Multiple well-designed metrics are used to evaluate the efficiency and interpretation quality.

Then, to your concerns, we address the comments as follows:

1. Attention flow as interpretability tool. In [14], Attention Flow is defined by modeling the multi-layer self-attention mechanism of the Transformer as a capacitated flow network, where attention weights and residual connections jointly determine edge capacities. The maximum flow between tokens of interest serves as the interpretability metric, quantifying how much information is transmitted between tokens. This formulation has been adopted in works such as [22] to analyze and compare attention mechanisms. However, computing attention flow is computationally expensive, prompting many researchers to use Attention Rollout as a faster alternative.

However, as you have pointed out, attention rollout often performs poorly. This is because it is a heuristic approximation that relies on direct multiplication of attention matrices, which causes the resulting values to deviate from true attention flow. Actually, this is why we propose FlowPrune, which is a method grounded in the Max-Flow Min-Cut theorem that can significantly accelerates attention flow computation while providing more accurate approximations of the true flow.

2. Class-specific Explanations. For class-specific explanations, our experiments on BERT (lines 276–301, Table 1) were specifically designed to evaluate the effectiveness of FlowPrune in such contexts. These experiments aimed to identify which tokens are most influential in classification decisions. FlowPrune achieved strong performance on this task, significantly outperforming Attention Rollout. Additionally, as demonstrated in Figure 8 and Figures 6–10 of the Appendix, FlowPrune can effectively highlight the visual regions contributing to the final predicted labels in image classification as the original attention flow, while attention rollout fails . These results support the conclusion that FlowPrune provides meaningful class-specific explanations.

3. Layer Compression. As discussed in lines 124–129 and 150–158, by leveraging the Max-Flow Min-Cut Theorem alongside the Transformer’s structural properties, we remove layers that have minimal impact on information flow. This selective pruning aligns with our goal of eliminating layers that contribute little to the results.

4. Edge Pruning Threshold. In fact, we have conducted experiments on the selection of the edge pruning threshold, but due to page limitations, these results were not included in the paper. The primary reason for omitting the results on the edge pruning threshold is that the experimental outcomes demonstrate that FlowPrune is not particularly sensitive to its value. Among the tested values, $1 \times 10^{-6}$ consistently performs well and offers a reasonable balance. The table below illustrates the average absolute error for the LLaMA model under different edge pruning thresholds after retaining half of the layers (16 layers).

5. Generalizability. To evaluate the generalizability of FlowPrune, we have selected two representative large models: the unimodal LLaMA and the multimodal LLaVA. We also assessed its effectiveness on smaller-scale models—BERT for NLP (Sec 4.2.1) and ViT for vision (Sec 4.2.2). In addition to measuring absolute error against the true attention flow (Fig 4), we considered that attention flow is often used to study relative ranking; thus, we introduced corresponding metrics for this purpose (Fig 5, 6). Together, these diverse models and evaluation criteria support the general applicability of our method. In response to your suggestion, we also included experiments on two widely used large models, Qwen (unimodal) and Qwen-VL (multimodal). The following table presents the absolute and relative errors in the high-flow regions, which are of the most interest to researchers, for the Qwen and Qwen-VL models, where we can find that the absolute/relative error is not large.

In addition to the aforementioned experiments, we also included an application scenario experiment on Question Answer Verification, which is also based on BERT. You can find the detailed introduction of the experiment in [22]. We compared the Top-K Token Retention with Attention Flow of FlowPrune and Attention Rollout, and organized the results in a manner similar to Table 1 in Section 4.2.1. The results show that, compared to Attention Rollout, the FlowPrune method is much closer to the analysis results of Attention Flow, demonstrating FlowPrune outperforms Attention Rollout. The table below presents the results:

6. Input Attribution Techniques. Your question is indeed very insightful. In fact, there has been some debate over the merits of attention-based versus gradient-based attribution methods about 5-6 years ago [A, B]. Wiegreffe and Pinter [B] argued that attention can be "an explanation, not the explanation." This implies that for the same prediction, there may be multiple equally valid and reasonable explanations. We concur with this conclusion and believe that both types of methods have their own strengths.

Researchers generally view these two types of methods as representing different categories of explanations. Attention-based methods focus on Intrinsic Interpretability, utilizing the inherent architectural characteristics of the model to provide explanations. They offer a built-in window into the model's internal "focus" [C]. On the other hand, gradient-based attribution methods focus on Post-Hoc Explanations, generating explanations by probing the model's behavior. This category is primarily dominated by saliency and attribution techniques.

Also, attention-based methods have the advantage of requiring only forward propagation, making them simpler to use and more intuitive [D]. In contrast, gradient-based methods necessitate backward propagation, which demands additional GPU computation and is not supported in some scenarios. For instance, many models undergo quantization after training, and quantized models do not support backward propagation. In such cases, attention-based methods are more convenient. Moreover, for attention flow[14], it has about 1000 citations in Google Scholar, suggesting it has considerable influence and impact in the community.

[A] Attention is not Explanation

[B] Attention is not not Explanation

[C] The Role of Attention Mechanisms in Enhancing Transparency and Interpretability of Neural Network Models in Explainable AI

[D] A Comprehensive Guide to Explainable AI (XAI) for Generative Models

7. Compression Rate. Thank you for pointing out this issue. In our paper, the compression rate is defined as the ratio of the number of retained layers to the total number of layers. Therefore, when the compression rate approaches 1, it implies that the number of retained layers is close to the total number of layers, meaning that more layers are preserved. This name is indeed counterintuitive. Following your suggestion, we will rename the compression rate to retain rate to align the terminology with its definition. We will correct this name in the revision.

8. L/R Layers. In fact, in the context of our paper, it should be $L/R$ layers in line 218. The context means that retaining only $L/R$ layers will reduces the computational time, where $L$ denotes the total number of layers in the model, and $1/R$ represents the compression rate, or retain rate in the revision.

Dear Reviewers:

We hope this message finds you well.

Thank you again for your valuable feedback on our submission. We have submitted our rebuttal and are eager to engage in a discussion before the phase concludes.

We truly appreciate the time and effort you have dedicated to reviewing our paper. We believe the points you raised are crucial for improving our work and are eager to provide any further clarifications that may be needed.

As the discussion phase is approaching its end, we would greatly appreciate the opportunity to provide any further clarifications and address any remaining questions you may have.

Thank you for your time and dedication to the review process.

Sincerely,

Authors of Submission 18977

Thank you for your valuable feedback. We appreciate the opportunity to further clarify our methodology and the rationale behind our use of attention flow.

Actually, the paraphrasing verification experiment in Section 4.2.1 performs a similar function to the analysis of Agreement with Human Rationales in DeYoung et al., 2020. Our approach, which analyzes the Attention Flow Value to the [CLS] token, serves a comparable purpose by identifying the key tokens the model prioritizes for its decision.

We agree with your assessment that directly using attention flow to analyze contributions to different labels presents a challenge. However, we maintain that attention flow is a powerful tool for understanding model behavior, particularly from an information-flow perspective. Our purpose was to explore these information pathways, not to perform a traditional feature importance analysis. We'd like to share a few examples from recent literature to support this point:

• A recent EMNLP Best Paper [A] successfully used attention flow to analyze how LLMs process information during in-context learning. This work shows that by studying how information flows through the model's layers, we can gain valuable insights into its reasoning process, even without directly quantifying each token's contribution to a specific label.
• The core objective of [B] is to visualize the "attention flow" within the ViT network for the first time using the proposed DAAM method, in order to explain how decisions evolve. The paper indicates that attention flow can help qualitatively analyze how modules inserted into the ViT model (such as XCA or T2T modules) influence the formation of object attention.
• This paper [C] used attention flow for multi-scale analysis, noting that through a focused token-level view, users can understand how attention flows between specific parts of a document.

We hope these examples provide a clearer understanding of our experimental design and address your concerns.

[A] Label Words are Anchors: An Information Flow Perspective for Understanding In-context Learning

[B] Dynamic Accumulated Attention Map for Interpreting Evolution of Decision-Making in Vision Transformer

[C] A Sentence-Level Visualization of Attention in Large Language Models

I thank the authors for the detailed response. I appreciate the clarifications and additional experimental evidence on Qwen.

Attention flow as interpretability tool // Input Attribution Techniques.

I understand that Attention Flow is a more faithful method than Attention Rollout, which is an approximation. However, simplifying or accelerating a method only has practical value if there is a clear justification for why that method is worth using in the first place. While FlowPrune is evaluated based on its agreement with Attention Flow, particularly in terms of top-k token overlap (e.g., via IoU), this evaluation assumes that those top-k tokens are meaningfully relevant.

In the broader literature on input attribution, faithfulness is typically assessed through metrics like comprehensiveness and sufficiency (e.g., DeYoung et al., 2020), which test whether the identified tokens actually influence the model's predictions. Without such analysis, it's unclear whether the explanations provided by FlowPrune, despite matching Attention Flow, are actually useful or reliable.

De Young et al., 2020: Eraser: A Benchmark to Evaluate Rationalized NLP Models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

Class-specific Explanations.

What I was referring to is the ability to generate class-specific explanations, that is, explanations not only for the predicted class but also for alternative classes. Gradient-based methods, for example, allow computing gradients with respect to each class output, making it possible to identify which input tokens contributed to the prediction, as well as to the other class outcomes.

Dear reviewer:

Thank you for your constructive comments, Your valuable comments have been of considerable assistance to us. Then, to your concerns, we address the comments as follows:

1. Attention Flow is calculated after token-by-token generation instead of during generation. In fact, when using attention flow for analysis, the entire sequence is input into the Transformer in a single parallel forward pass to obtain the corresponding attention graph. Consequently, the causal attention mask used in autoregressive generation is not applied here. Furthermore, since attention flow tracks information propagation from the first to the last layer, the attention graph is formed by edges connecting layer $i$ to layer $i+1$, reflecting the forward flow of information within the model.

2. Question about Line 166 - 168. As noted in lines 159–168, this is a heuristic conjecture for which we currently lack a formal proof. Nevertheless, both our fundamental evaluations and experiments support its validity. Specifically, the experiments on LLaMA and LLaVA in Section 4.1 show that FlowPrune have low approximation errors, suggesting that the proposed approach is empirically reliable.

3. Bound in Lines 216-223. The bound presented in lines 216–223 serves as an estimate, based on the assumption that when the proportions of pruned edges and nodes in FlowPrune are approximately equal, the bound reasonably approximates the actual speedup. Additionally, the specific structure of the attention graph can influence the extent of acceleration achieved. We will incorporate the above explaination in revision.

In lines 224–228, we briefly analyze possible reasons why the LLaMA curve deviates from the theoretical prediction. We attribute this discrepancy to differences in input token types across models. Unlike LLaMA, LLaVA is a multimodal vision-language model that processes a large number of visual tokens, many of which are redundant. This redundancy results in a higher proportion of near-zero attention weights, leading to more edges being pruned, making this case more similar to the assumption. As a result, the compressed attention graph in LLaVA aligns more closely with our theoretical assumptions.

4. Setting edge capacity to 1. According to the Max-Flow Min-Cut Theorem, increasing the capacity of edges not in the minimum cut does not affect the maximum flow value, as the flow is bounded by the capacity of the minimum cut. In FlowPrune, we ensure that the removed parts contain as few minimum cut edges as possible. To maintain the maximum flow value after compression, newly added shortcut edges are assigned sufficiently large capacities to ensure that they do not become part of the new minimum cut. In our case, a capacity of 1 is considered sufficiently large.

Note that removal involves deleting certain vertices along with their connected edges. For any pair of remaining vertices that were previously connected via a path through the removed nodes, we introduce a new edge with large capacity to restore connectivity. Here, a capacity of 1 is adequate because we do not compress the first attention layer. As a result, the total flow emitted from the source remains bounded by 1, meaning the total maximum flow will also not exceed 1. Consequently, added edges with capacity 1 will not appear in the minimum cut. While this breaks the constraint that attention weights should sum to one, preserving the first layer ensures the attention flow remains within [0, 1], maintaining its interpretability.

5. Increasing Max-Flow is Better than Decreasing. Since the maximum flow is bounded above by 1 in both the original and compressed attention graphs, a relatively large original flow value restricts the possible outcomes of FlowPrune to a narrower range, thereby reducing potential error. As discussed in lines 247–251, such cases are considered important, and thus, smaller errors are particularly desirable.

6. Add Standard Errors. Thank you for your suggestion. We will add the following results about standard errors in revision.

7. Edge Pruning Experiment. Please see ``1.Static Threshold.'' in response to Reviewer b9Ta for more details.

8. Small Attention Weights. Your observation is accurate. Due to the exponential nature of the softmax function, only a few attention values are significantly greater than zero. For example, in our experiments across the four models used in this study (LLaMA, LLaVA, BERT, and ViT), more than 95% of the attention values are smaller than 1 \times 10^{-6}.

9. Intuition in Lines 127 - 130. We acknowledge that the term we used was imprecise; what we intended to convey is more accurately described as a corollary, rather than an intuition. Specifically, we refer to the removal of certain vertices from the graph, along with their incident edges. For the remaining vertices, if any pair was previously connected via a path composed of the removed edges, we introduce a new edge with sufficiently large capacity to preserve connectivity. Under this construction, if none of the removed edges belong to the original minimum cut, the minimum cut remains unchanged. This holds because, after the removal, any S–T cut must either 1) includes the newly introduced edges and, due to their sufficiently large capacity, cannot be the minimum cut, or 2) is composed entirely of original edges and thus cannot be smaller than the minimum cut of the original graph.

Since the minimum cut of the original graph indeed exists within the graph, it remains the minimum cut of the graph after the removal operation, thereby preserving the invariance of the maximum flow.

10. Question About Figure 1(b). We apologize for the confusion. The figure illustrates our sampling-based heuristic algorithm. The orange lines indicate potential min-cut edges, not actual min-cut edges. This algorithm is introduced in lines 153–158, with its motivation discussed in lines 159–168. Note that the figure does not include predefined source or sink nodes; these are determined only when computing the attention flow between a specific pair of tokens.

11. Overhead of Flowprune. For a single attention graph, pruning is performed only once, after which the sparsified graph is reused to compute attention flow between all token pairs. Although pruning introduces initial computational overhead, this cost is amortized as the number of evaluated token pairs increases. For example, in the case of LLaMA, computing attention flow for 6 token pairs on the original attention graph takes approximately the same time as applying FlowPrune once and then performing max-flow computations on the sparsified graph for those pairs. As more pairs are calculated, the average time per pair for FlowPrune continues to decrease. This trend is shown in Figure 3, where we plot average per-pair computation time under various compression rates. The intersection with the baseline curve (original attention flow time) indicates the point beyond which FlowPrune offers a computational advantage.

12. Precision Requirements. As discussed in lines 68–70, attention flow is often used to assess the relative importance of tokens based on the ranking of flow values. Our notion of precision pertains specifically to this context. At a 50% compression rate, the first instance of incorrect relative ordering appears only after the top 40% of token pairs. In practice, applications that rely on attention flow typically focus on the top 5–10 token pairs. Therefore, an FDR of 0.4 is sufficient for these use cases.

The comma in lines 266-268 you cited, is a typo. The lines you cited should be two separate sentences. The corrected version should be "This result is fully sufficient for practical applications with lower precision requirements. While the results at a compression rate 50% are adequate for tasks with higher precision demands.".

The first sentence discusses the scenario of retaining only 3 layers, while the second sentence addresses the situation with a compression rate of 50%. We apologize for any confusion this may have caused.

13. Actual Run Time. The maximum flow algorithm is a classical graph-theoretic method that typically lacks efficient GPU-accelerated implementations. Our experiments were conducted on a server equipped with two AMD EPYC 7453 28-core processors. Using LLaVA as an example, we measured the runtime for computing attention flow across 100 token pairs in a sample containing 400–500 tokens within a 32-layer LLaVA model. With the original attention flow algorithm, the runtime reaches 112 minutes. In contrast, compressing the attention graph to 12 layers reduces the runtime to approximately 10 minutes.

14. Input and Samples. We construct input sequences by concatenating questions with their corresponding answers from the dataset. Each input generates a corresponding attention graph, which we refer to as a sample.

15.Average Results. Indeed, that is precisely the case.

16.Grammar. Thanks for pointing out these mistakes and we will follow your suggestion to revise them.

Thanks for the follow-up.

First, during supervised fine-tuning (SFT), models are often trained using full-sentence prompts followed by token-by-token generation. In such cases, causal masking is only applied during the generation phase, not when processing the initial full prompt.

I don't think this is correct. For any decoder-style model (e.g. Llama), there will be causal masking (i.e. a token can only ever attend to itself or past tokens) at all times and for all tokens. In addition, there is no token-by-token generation in SFT training, since it's all offline.

Thank you for your insightful comments, which have helped us further refine our study. Below, we address the two questions raised during the discussion.

1. Impact of causal attention on attention flow.

You are correct that causal masking can affect the computation of attention flow. However, we find its practical impact to be limited for two main reasons. First, during supervised fine-tuning (SFT), models are often trained using full-sentence prompts followed by token-by-token generation. In such cases, causal masking is only applied during the generation phase, not when processing the initial full prompt. This setup naturally mitigates the potential distortion introduced by the causal mask. Second, and more importantly, when attention flow is used for interpretability purposes, the analysis typically focuses on how tokens in the earlier part of a sentence influence a predicted token later in the sequence. This design implicitly offsets the impact of causal masking. For example, in the sentence "This movie is amazing, I __ it", suppose the predicted token is "like". To assess how "amazing" contributes to this prediction, attention flow is computed from "amazing" to "like". Although indirect paths such as amazing → it → like may exist, our empirical observations suggest that their influence is relatively minor. As such, causal masking has limited effect on the practical use of attention flow for interpretability.

2. Gap between theoretical bounds and empirical curves.

Regarding the curve of LLaVA, it indeed aligns more closely with our theoretical bound compared to LLaMA, though a gap still remains. We will follow your suggestion to revise this part of the paper to clarify the distinction and avoid potential misunderstandings.

Once again, we sincerely appreciate your questions, which further prompted us to revisit and deepen our analysis. We will incorporate these clarifications into the revision.

Dear reviewer:

Thank you for your constructive comments, Your valuable comments have been of considerable assistance to us. Then, to your concerns, we address the comments as follows:

1. Static Threshold. We use a static threshold primarily because experimental results show that FlowPrune is not particularly sensitive to its value. Among the tested values, $1 \times 10^{-6}$ consistently performs well and offers a reasonable balance. Given this robustness, using a static threshold simplifies implementation without compromising performance.The table below illustrates the average absolute error for the LLaMA model under different edge pruning thresholds after retaining half of the layers (16 layers).

2. Effectiveness. Firstly, the dominant Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) currently in use primarily employ standard Transformer blocks, which do not incorporate sparse or hybrid attention mechanisms. Secondly, and more importantly, most existing sparse or hybrid attention mechanisms, such as those used in Longformer[A] and Natively Trainable Sparse Attention (NSA)[B], also exhibit softmax sparsity and layer-wise characteristics. The layer-wise characteristic is applicable to almost all Transformer architectures. As for softmax sparsity, as long as softmax is utilized in the architecture, softmax sparsity is inevitable. For example, Longformer applies softmax to the sparsified attention matrix, and NSA selectively applies softmax to different attention paths.

[A] Longformer: The long-document transformer.

[B] Natively Trainable Sparse Attention for Efficient Transformer.

3. Related Studies. While we may seem to overlook related studies about attention flow or attention rollout, this is because recent research has largely focused on using attention flow or rollout as tools for analyzing model behavior or supporting new algorithmic designs, rather than on improving the computation of attention flow itself. Thus, these methods are typically treated as fixed, with limited consideration given to its efficiency or scalability. However, FlowPrune is introduced precisely to fill this overlooked gap by offering a more efficient and principled method for attention flow analysis, facilitating its broader application in large-scale interpretability tasks.

4. Memory Complexity. The memory complexity of FlowPrune is proportional to the size of the attention graph, as its primary memory usage arises during graph construction. The attention graph contains $O(LN)$ nodes and $O(LN^2)$ edges, resulting in an overall memory complexity of $O(LN^2 + LN) = O(LN^2)$.

5. Minor suggestions. We apologize for these grammar mistakes and missing citations and thank you for pointing them out. They will be corrected in revision to make overall writing meet the standards.

Dear Reviewers:

We hope this message finds you well.

Thank you again for your valuable feedback on our submission. We have submitted our rebuttal and are eager to engage in a discussion before the phase concludes.

We truly appreciate the time and effort you have dedicated to reviewing our paper. We believe the points you raised are crucial for improving our work and are eager to provide any further clarifications that may be needed.

As the discussion phase is approaching its end, we would greatly appreciate the opportunity to provide any further clarifications and address any remaining questions you may have.

Thank you for your time and dedication to the review process.

Sincerely,

Authors of Submission 18977

Dear reviewer:

Thank you for your constructive comments, Your valuable comments have been of considerable assistance to us. Then, to your concerns, we address the comments as follows:

1.Other Sparsification Techniques. We build FlowPrune on the Max-Flow Min-Cut Theorem because attention flow is inherently formulated as a maximum flow problem over the attention graph[14]. As a cornerstone, the Max-Flow Min-Cut Theorem provides the primary theoretical foundation for designing and analyzing max-flow algorithms. To our knowledge, few alternative frameworks offer comparable guidance for optimizing max-flow computations. Accordingly, in developing FlowPrune, we focused on this theorem in conjunction with exploiting the structural properties of Transformer attention graphs, rather than adopting other graph sparsification techniques that are less directly aligned with the max-flow formulation.

2. Generalizability. To evaluate the generalizability of FlowPrune, we have selected two representative large models: the unimodal LLaMA and the multimodal LLaVA. We also assessed its effectiveness on smaller-scale models—BERT for NLP (Sec 4.2.1) and ViT for vision (Sec 4.2.2). In addition to measuring absolute error against the true attention flow (Fig 4), we considered that attention flow is often used to study relative ranking; thus, we introduced corresponding metrics for this purpose (Fig 5, 6). Together, these diverse models and evaluation criteria support the general applicability of our method. In response to your suggestion, we also included experiments on two widely used large models, Qwen (unimodal) and Qwen-VL (multimodal). The following table presents the absolute and relative errors in the high attention regions, which are of the most interest to researchers, for the Qwen and Qwen-VL models, where we can find that the absolute/relative error is not large.

In addition to the aforementioned experiments, we also included an application scenario experiment on Question Answer Verification, which is also based on BERT. You can find the detailed introduction of the experiment in [22]. We compared the Top-K Token Retention with Attention Flow of FlowPrune and Attention Rollout, and organized the results in a manner similar to Table 1 in Section 4.2.1. The results show that, compared to Attention Rollout, the FlowPrune method is much closer to the analysis results of Attention Flow, demonstrating FlowPrune outperforms Attention Rollout. The table below presents the results:

3. Difference between llava and llama. For the differences in absolute error patterns between LLaMA and LLaVA in Fig. 4, we think this is cased by the distinct attention graphs of the two models. LLaMA is a unimodal language model, while LLaVA is a multimodal vision-language model, resulting in inherently different attention graph structures. We hypothesize that the higher errors observed in LLaVA stem from its large number of input vision tokens, which often contain redundant information and contribute to an abundance of low-attention values. As discussed in 4.Low Attention Regions., FlowPrune exhibits higher approximation error in low attention regions, thereby increasing the overall error in LLaVA. A similar phenomenon appears in our experiments with Qwen (unimodal) and Qwen-VL (multimodal), reinforcing the notion that attention graph modality plays a key role in shaping the error pattern.

In contrast, both LLaMA and LLaVA show similar trends in relative error. As highlighted in Section 4.2, relative error is of greater practical importance than absolute error, particularly for interpretability and token-level ranking tasks based on attention flow.

4. Low Attention Regions. As discussed in Limitations (lines 336–341), FlowPrune exhibits relatively larger approximation error in low-attention regions. Here, "low-attention regions" refer to token pairs with small flow values in the original attention flow, indicating weak attentional connections. In practice, such pairs are typically of limited interest. As noted in lines 247–251, researchers are primarily concerned with token pairs exhibiting large attention flow values, where FlowPrune achieves much lower error. Therefore, the observed approximation inaccuracies in low-attention areas have minimal impact on practical applications.

5. Theoretical Guarantees. We apologize for any confusion caused by our use of the term theoretical guarantees. To clarify, we do not claim a general guarantee on the approximation error, as this depends heavily on the specific numerical structure of the model’s attention graph and is analytically intractable. Rather, our point is that FlowPrune is grounded in the Max-Flow Min-Cut Theorem and thus offers a principled approximation of attention flow as originally defined in [14]. In contrast, Attention Rollout relies on recursively multiplying attention matrices, which is heuristic in nature and diverges significantly from the original flow formulation. Therefore, the guarantee we refer to is that FlowPrune adheres to the theoretical definition of attention flow, rather than serving as a purely heuristic alternative.

Dear Reviewers:

We hope this message finds you well.

Thank you again for your valuable feedback on our submission. We have submitted our rebuttal and are eager to engage in a discussion before the phase concludes.

We truly appreciate the time and effort you have dedicated to reviewing our paper. We believe the points you raised are crucial for improving our work and are eager to provide any further clarifications that may be needed.

As the discussion phase is approaching its end, we would greatly appreciate the opportunity to provide any further clarifications and address any remaining questions you may have.

Thank you for your time and dedication to the review process.

Sincerely,

Authors of Submission 18977