Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images

We thank the reviewer for the valuable suggestions. We answer the reviewer's questions in the following one by one:

Weakness

The assumptions in the theoretical proofs may be challenging to satisfy ...

We appreciate the reviewer's concerns. Regarding the Lipschitz continuity of $f_{min}$ and $f_{max}$: while strict $L$-Lipschitz continuity is a common theoretical assumption used to bound approximation errors, in practice we implement these functions as single-layer perceptrons with ReLU activations. Moreover, we apply regularization techniques such as normalization and weight clipping during training to encourage Lipschitz-like behavior, which empirically ensures that the projections behave in a sufficiently smooth manner for our theoretical guarantees to hold approximately.

As for the four conditions in Theorem B.6, they are idealized guidelines to understand the behavior of our query-aware selective attention mechanism. In our implementation, we approximate these conditions by: (1) using a large hidden dimension (e.g., $d=512$) to meet JL lemma requirements; (2) selecting an appropriate number of regions based on spatial decay analysis; and (3) designing region sizes (e.g., $K=16,24$) that balance the need for a small diameter with computational efficiency. Although perfect adherence is challenging, our parameter choices, guided by validated performance, effectively control the approximation error and preserve the key theoretical properties, as supported by our empirical results.

Could you provide a comparison of training and inference times ...

We appreciate this question on computational efficiency. In our experiments (see Figure 2 in the anonymous link), Querent trains in 72.33s per batch — faster than more complex models like HIPT (473.49s) yet slightly slower than simpler baselines. It converges in a similar number of epochs as other transformer-based methods while achieving state-of-the-art accuracy. Notably, Querent has the lowest memory footprint (2286 MB) among the compared methods. For inference, Querent processes a slide in 0.1328s.

As the reviewer correctly points out, patch feature extraction (typically 2~3mins per slide) dominates the WSI processing pipeline, while feature aggregation accounts for less than 1% of the total time. We believe that the slight increase in computation time compared to simpler methods is a worthwhile trade-off given Querent's state-of-the-art performance across all tasks.

Other Comments or Suggestions

It appears that there is an error in Equation 12 ...

We thank the reviewer for the careful review of Supplementary Materials. We will correct this equation in the revised manuscript to ensure mathematical accuracy.

Questions

In the second set of ablation experiments, why does the Estimation Side Network ...

This counter-intuitive result stems from fundamental limitations in the Estimation Side Network approach. By attempting to predict region importance independently without considering query context, this network struggles with optimization challenges and fails to capture the relational information critical for accurate importance assessment. It also tends to overfit to region patterns seen during training. Random Region Selection, while simple, provides diverse contextual sampling that occasionally includes relevant regions by chance, avoiding biased selection. Our query-aware approach resolves these issues by dynamically assessing region importance relative to each specific query, leading to significantly better performance than both alternatives. We will clarify this explanation in the revised manuscript.

In the final step of Lemma B.4, ...

We thank the reviewer for the insightful comment regarding the feature space transition in Lemma B.4. To clarify, although $q$ and $\hat{q}$ reside in different feature spaces, the projection functions are assumed to be Lipschitz continuous, which allows us to control the distortion when moving from the original space to the projected space, with detailed deviation shown in Figure 3 in this anonymous link.

In the first set of ablation experiments and in Appendix G.1 ...

In our distance matrix calculations, pre-summarization distances between regions are computed by first flattening all patches in each region into a single vector, then calculating Euclidean distances between these region vectors. Post-summarization distances are simply the Euclidean distances between the metadata vectors.

Is the $\alpha$ in Theorem 3.1 the same as ...

No, these are different quantities. In Theorem 3.1, $\alpha$ is the exponential decay rate parameter for attention scores with spatial distance. In Equation 8, $\alpha$ represents normalized attention weights for feature aggregation. We'll use distinct notation in our revision to prevent confusion.

We thank the reviewer for these constructive comments. We answer the reviewer's questions in the following one by one:

Experimental Designs or Analyses

However, the comparative methods delineated in the paper appear to diverge from the tasks ...

We appreciate the reviewer's concern regarding task alignment between our evaluation and original studies. We want to clarify that our evaluation framework was intentionally designed to be comprehensive, spanning multiple computational pathology tasks (biomarker prediction, gene mutation prediction, cancer subtyping, and survival prediction) to demonstrate the robustness and generalizability of our method across diverse clinical applications. While some baseline methods like RRT-MIL were originally evaluated on specific tasks, we adapted all methods for multiple tasks using standardized feature extraction and training protocols to ensure fair comparison. This approach provides stronger evidence of our method's versatility than limiting evaluation to a single task type would, as demonstrated by Querent's consistent performance advantages across all tasks in Tables 1 and 2. We believe our comprehensive evaluation strategy enhances rather than diminishes the persuasiveness of our experimental findings by showing our approach's effectiveness across the spectrum of computational pathology applications.

Weakness

however, the analysis appears somewhat deficient. For instance, the absence of results derived from global attention ...

We appreciate the reviewer's concern about determining the source of our method's superior performance. It's worth noting that directly applying global self-attention to tens of thousands of WSI patches leads to out-of-memory problems, which explains why existing methods use alternatives like local-global or linear attention approximations. To respond to the reviewer's comment directly, we implemented a full global attention approach using FlashAttention (which achieves linear memory complexity) and compared it with our method with the same PLIP feature extractor and same training protocol (see Table 1 in this anonymous link). The experimental results show that Querent consistently outperforms the global attention approach (FlashMIL). This confirms that our performance improvements stem specifically from the proposed query-aware dynamic attention mechanism rather than other components, as these elements were identical across both compared methods.

Other Comments or suggestions

Some sections, particularly the theoretical derivations, could be elaborated ...

In response to this comment, we acknowledge that some elements of Theorem 3.1 would benefit from additional clarification. Specifically, the parameter B in Theorem 3.1 represents the bound on input norms ($||q_i||$, $||K_j||$ $\leq$ $B$), which is critical for establishing the error bounds of our query-aware attention approximation. This parameter is properly defined in Lemma B.4 of the appendix but should have been explicitly introduced in the main text for clarity. We will ensure this and other theoretical elements are more thoroughly explained in the revised version to enhance readability and comprehension of our technical contributions.

Exploring the integration of the proposed method with other advanced models ...

We agree with the reviewer that integrating our query-aware dynamic attention mechanism with advanced vision transformer architectures represents a promising direction for future work. Our current implementation focuses on efficient modeling of long-range dependencies in gigapixel images, but the core principles of our approach — dynamic region-level metadata summarization and importance-based selective attention — could be readily adapted to enhance various vision transformer frameworks. We appreciate this valuable suggestion and plan to explore such integrations in our future research.

Questions

Should a comprehensive global attention mechanism be employed ...

We have responded to this comment with detailed interpretation in the Weakness section.

Would the adoption of alternative patch feature extractors—such ...

We appreciate this insight. As demonstrated in Table 2 in this anonymous link, we conducted additional experiments using state-of-the-art foundation models (Virchow and CHIEF). Results show that Querent consistently outperforms other methods with these advanced feature extractors, confirming that our method's superiority stems from its long contextual modeling capability rather than the choice of feature extractor. This indicates Querent's contribution is complementary to advances in foundation models and will continue to provide advantages as foundation models evolve.

We welcome any further questions or clarifications regarding our rebuttal and are happy to provide additional details if needed.

We thank the reviewer for the constructive comments. We respond to the comments one by one as follows:

Weakness

The performance of the model depends on the quality of the region-level metadata ...

We acknowledge that any approach to summarizing region-level metadata, whether it be min/max, mean, or mean-std, will inherently lose some local information. This trade-off is necessary to achieve computational efficiency when handling gigapixel whole slide images. In our work, the min-max strategy was specifically chosen because it captures the extreme values of feature distributions, which are critical for preserving discriminative patterns — especially in heterogeneous tissues. Our additional analysis (see Figure 1 in this anonymous link), using the "Adjusted Average Distance to Summary" metric, shows that the min-max approach outperforms other summarization methods, particularly in high-heterogeneity regions, where it achieves significantly lower error compared to mean or mean-std methods.

Furthermore, our query-aware attention mechanism complements the summarization by dynamically selecting and weighting the most relevant patches, which helps mitigate the loss of local information and filters out noise. Although some information loss is unavoidable with any summarization method, our experimental results demonstrate that the min-max approach, in combination with our selective attention, provides an effective and robust representation that leads to superior performance across all tasks, even in challenging heterogeneous and noisy scenarios.

Other Comments or Suggestions

The experiments did not leverage features from latest foundation models ...

We appreciate this suggestion. As demonstrated in Table 2 in this anonymous link, we conducted additional experiments using state-of-the-art foundation models (CHIEF and Virchow). Results show that Querent consistently outperforms other methods with these advanced feature extractors, confirming that our method's superiority stems from its long contextual modeling capability rather than the choice of feature extractor. This indicates Querent's contribution is complementary to advances in foundation models and will continue to provide advantages as foundation models evolve.

The region size in Querent significantly impacts performance, potentially ...

We appreciate the reviewer's comment on region size impact. While region size is a hyperparameter, our ablation studies (Figure 5) show that moderate-sized regions (16-24) consistently deliver strong performance across diverse datasets. This pattern aligns with pathological intuition, i.e., region size should capture meaningful local context without diluting distinctive tissue patterns. This provides a reliable starting point that significantly narrows the hyperparameter search space, enhancing Querent's practical applicability without extensive tuning.

Questions

The method can visualize the original WSI corresponding to the K regions selected ...

We thank the reviewer for this great suggestion and will include visualizations in the revised version.

It is not clear how the method can avoid the situation where the top K ...

We address this important concern through two mechanisms: (1) our region importance estimation algorithm provides theoretical guarantees (Theorem 3.1) that selected regions are at most 2$\epsilon$1-suboptimal compared to the true top-K regions, ensuring minimal information loss; and (2) our min-max summarization strategy (superior in Fig. 4, p<0.005) effectively captures extreme feature distributions, making the metadata highly discriminative for identifying diagnostically relevant areas. Our ablation studies confirm that our approach significantly outperforms random region selection (Table 3, 8.9% accuracy improvement on UBC-OCEAN), demonstrating that our method reliably identifies regions containing key diagnostic information.

It is uncertain what the advantages of the proposed method are compared with the latest methods ... through Mamba ...

We directly compare with both MambaMIL and MamMIL (see Table 3 in this anonymous link) and Querent consistently outperforms these methods across all metrics and datasets.

The fundamental difference between Querent and these methods lies in our query-aware dynamic modeling approach. While MambaMIL uses sequence reordering and MamMIL employs graph-based representations with MST, both still process all patches with predetermined patterns. In contrast, Querent adaptively determines which surrounding regions are most relevant for each patch based on content, focusing computational resources only where needed.

We welcome any further questions or clarifications regarding our rebuttal and are happy to provide additional details if needed.

We thank the reviewer for the valuable comments. We respond with detailed interpretations as follows:

Claims and Evidences

The claim in line 023-025 (abstract) 'the query-aware ...

We should clarify that Querent provides a theoretically bounded approximation of full self-attention rather than claiming it maintains the exact same expressive power. As demonstrated in Theorem 3.1, our approach maintains the expressiveness within a small constant bound of full self-attention while significantly reducing computational complexity. We will revise this wording in the final version to be more precise: "Querent achieves a theoretically bounded approximation of full self-attention and meanwhile delivers practical computational efficiency for modeling gigapixel images."

Relation with Full Self-Attn (FlashAttn)

Since the reviewer's major concern is the necessity of developing a new dynamic attention pattern instead of directly using Flash Attention for full self-attention implementation, here we address these concerns step by step.

1. Comparison with Full Self-Attention via FlashAttention
Following the reviewer's suggestion, we have implemented a full self-attention-based MIL model (FlashMIL) using FlashAttention to enable the processing of long sequences without memory limitations. Specifically, our implementation applied 4 flash-attn layers to model the WSI patch sequence, followed by a mean operation to obtain the slide-level representation for prediction. The comparison results on our three classification datasets are shown in Table 1 in this anonymous link. While FlashMIL achieves comparable performance on the BCNB-ER dataset, Querent significantly outperforms it on the more complex TCGA-LUAD TP53 and UBC-OCEAN datasets. This demonstrates that our method provides benefits beyond just addressing memory efficiency.

2. Relationship Between Querent and Flash Attention
While both Querent and FlashAttention address the computational challenges of self-attention, they do so through fundamentally different approaches. FlashAttention optimizes the implementation of full self-attention through IO-aware algorithms and hardware optimizations, but still computes attention between all pairs of patches. In contrast, Querent introduces a context-dependent attention mechanism that dynamically identifies and focuses only on the most relevant regions for each query patch.

The superior performance of Querent over FlashMIL can be explained by this contextual selectivity, which serves as an implicit regularization mechanism. By focusing only on relevant regions, Querent filters out noise and irrelevant information that could potentially confuse the model, especially in highly heterogeneous WSIs. This is particularly important in computational pathology, where diagnostically relevant features may be sparsely distributed across the gigapixel image.

3. Why Querent Improves Results Beyond Memory Efficiency
The improved performance of Querent over Flash Attnetion-based implementation can be attributed to several factors. First, the context-aware attention dynamically determines which surrounding regions are most relevant for each patch, allowing Querent to adapt to the heterogeneous nature of WSIs, where different tissue types require different contextual considerations. Second, selective attention acts as a form of implicit regularization by reducing the influence of irrelevant or noisy patches, which is particularly beneficial in weakly-supervised settings with limited training data. Third, while reducing computational overhead, our min-max region metadata approach ensures that potentially important long-range dependencies are still captured, unlike fixed local-global attention patterns that make strong assumptions about which spatial relationships matter. These advantages explain why Querent outperforms Flash-Attention-optimized self-attention.

4. Compatibility with FlashAttention
Regarding the potential combination of Querent with FlashAttention: Yes, our method is compatible with and complementary to FlashAttention. While FlashAttention optimizes how attention is computed through IO-aware algorithms, Querent determines which attention computations are most valuable to perform. A combined approach could leverage FlashAttention's efficiency for computing the selected region attention in our Step 3 (Query-Aware Selective Attention), potentially providing even greater computational benefits. This represents an interesting direction for future work.

Our current implementation already demonstrates significant efficiency gains over standard attention (as shown in Figure 6), requiring only ~1% of the memory and ~5% of the computational cost for 100k patches. Even compared to FlashAttention, Querent offers advantages in computational complexity (near-linear vs. quadratic time complexity) while achieving superior performance on CPath tasks.