Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling

Thank you for your positive feedback and for recognizing the thoughtful integration of modalities, sound design, and comprehensive evaluation in our work. We provide the following responses to address your concerns.

In contrast, attention- or MLP-based approaches are limited in their capacity to effectively capture dependencies between nodes, and they struggle to incorporate prior structural knowledge, such as semantic links across modalities or hierarchical relationships across scales.

$1$ Battaglia et al. Relational inductive biases, deep learning, and graph networks, arXiv 2018.

$2$ Zhou et al. Graph Neural Networks: A Review of Methods and Applications, AI Open 2021.

GNN vs. alternative interaction models. To verify the necessity of GNNs, we replace them with MLP or attention modules in both the intra-scale and hierarchical components and report the results below (PLIP, 16-shot). The comparison confirms that GNNs play a crucial role in integrating intra-scale and hierarchical information, while simpler alternatives fall short in capturing these relationships effectively.

We will add these rationales and experimental results to the final version of the paper.

The message passing process is as follows. During neighbor aggregation, information is collected from neighboring nodes connected via a specific relation type (i.e., edge type). These include intra-scale relations, such as valid patch-text connections at low or high scale, and hierarchical relations, such as connections between low-scale and high-scale patches, or between low-scale and high-scale text. The aggregated neighbor features are then concatenated with the node’s own features and passed through a relation-specific linear transformation (i.e., weight matrix). For intra-scale relations, we adopt the GraphSAGE operator via the HeteroConv module, whereas for hierarchical relations, we employ a Modality-Scale Attention (MSA) mechanism that accounts for both scale directionality and modality type. Finally, a non-linear activation function (ReLU) is applied to enhance representational capacity. By decoupling both aggregation and transformation parameters by relation type, our model effectively encodes heterogeneous and hierarchical information. The corresponding pseudocode will be included in the final version.

When TGDF is not applied at all, performance is lowest due to aligning all image-text pairs, which would result in incorrect image-text pairings. As α increases, more patch-text pairs are filtered out at both scales, leading to improved performance, with α = 0.5 yielding the best result, indicating that TGDF effectively removes semantically irrelevant image–text pairs. However, when α = 1, the filtering becomes too aggressive, discarding many meaningful pairs and reducing performance.

The task is to summarize the morphological features of the {dataset_name} dataset for the {class_name_1}, {class_name_2}, … classes. For each class, list O representative morphological features observed at 5× magnification, followed by K finer sub-features observed at 20× magnification for each. Each description should include the morphological term along with an explanation of its defining visual features.

The examples of the descriptions generated by the LLM are provided in Supplementary Material A.3.

(1) Prompt Template Sensitivity (Supplementary Material H.4): We evaluated three formats: Term-only, Explanation-only (Exp.), and Term + Explanation (Term + Exp.). The Term + Explanation format consistently achieves the best performance, suggesting that combining concise terms with descriptive context yields more effective guidance (CONCH, 16-shot).

(2) LLM Backbone Sensitivity (Supplementary Material H.5) : We generated descriptions using four different LLMs: DeepSeek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. GPT-4o achieves the best overall performance. The results are consistent across different LLMs, demonstrating that HiVE-MIL is robust to the choice of LLM backbone (CONCH, 16-shot).

Again, we thank you for your thoughtful and constructive comments.

Thank you for your positive feedback and for recognizing that our paper is well written and easy to follow. We also appreciate your acknowledgment of the model’s interpretability and the comprehensiveness of our experiments. We provide the following responses to address your concerns.

We acknowledge that the sentence referenced by the reviewer may have unintentionally implied that existing models lack any form of cross-modal modeling, which was not our intention. We will revise the wording in the final version to clearly reflect our intended message.

We agree that the use of indices in Equation 7 was potentially confusing, especially due to the mixing of the image index $i$ and class index $c$. In the revised version, we will clarify the notation by introducing a class-specific text index set $\mathcal{I}_c$, where each $i \in \mathcal{I}_c$ corresponds to a text prompt associated with class $c$. This allows us to express the class logit computation more clearly. In the equation below, the $\mathrm{TopKAvg}_k$ function computes the average of the top $k$ similarity scores from the $i$-th column (.,i) of the image-text similarity matrix. This step aggregates the signal from the most relevant images for each text prompt before the final summation.

logit_c^{(s)} = (γ / |I_c|) * Σ_{i∈I_c} TopKAvg_k([X^{(s)} (T^{(s)})ᵗ]_{·,i})

This avoids index overloading and makes the operation easier to follow.

This design reflects the characteristics of pathological images, where spatially adjacent patches often differ in tissue structure, while semantically meaningful relationships can occur across distant regions. Therefore, we construct the intra-scale graph using semantic filtering, which allows connections between distant patches that share similar meanings based on textual descriptions. We argue that simple distance-based connections may overlook semantially important patterns. Nevertheless, we acknowledge the reviewer’s suggestion to explicitly incorporate intra-scale spatial adjacency as a valuable extension and we plan to explore ways to integrate spatial and semantic relationships in parallel in future work.

Furthermore, as shown in the sensitivity analysis presented in Supplementary Material H.2, we evaluated a range of α values (0, 0.1, 0.5, 1.0) and observed that the model’s performance remains relatively stable across settings. The default value α = 0.5 yields the most consistent results overall. That said, we fully acknowledge the reviewer’s point that a fixed threshold may not be optimal across all scenarios. As such, we plan to explore adaptive, learnable filtering mechanisms in future work to further improve generalization across datasets or backbones.

(1) We evaluate the faithfulness of the generated descriptions using an "LLM-as-a-Judge" $1$ , where LLMs are prompted to infer class labels based solely on the generated descriptions. We employ two evaluators: GPT-4o and GPT o3, to assess descriptions generated by several LLMs: Deep Seek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. Although not perfect, the number of correct predictions is still consistently high, suggesting that the generated text is generally reliable while capturing class-discriminative signals.

$1$ Gu et al. A Survey on LLM-as-a-Judge, arXiv 2025.

(2) The table below reports intra- and inter-category text similarity scores, where intra denotes similarities among descriptions of the same morphology or concept, and inter denotes similarities across different ones. Similarity is computed separately at each scale and jointly across both scales using the CONCH text encoder and cosine similarity. These results highlight that the descriptions encode highly discriminative and consistent semantics for each morphological or conceptual group, with robustness across scales and LLMs.

Furthermore, we validated the performance of HiVE-MIL using descriptions generated by different LLMs, as detailed in Supplementary Material H.5.

Again, thank you for your thoughtful and constructive comments.

We thank you for the positive feedback and for recognizing the high technical quality, clear contributions, and practical relevance of our work. We provide the following responses to address your concerns.

$1$ Zhou et al. Learning to Prompt for Vision-Language Models, IJCV 2022.

Optimizations planned. Working with WSIs is undeniably challenging due to their gigapixel size. To make our method more practical for large-scale deployment, we will further develop a parallel version of HiVE-MIL that can better handle the computational load. This would help reduce processing time and make the model more efficient for deployment in clinical settings.

Again, we thank you for your thoughtful and constructive comments.

We thank you for acknowledging the importance of our hierarchical design, the careful graph-based visual-textual modeling, and the strong experimental comparisons. We provide the following responses to address your concerns.

For the combination of using all of the four scales (5x, 10x, 20x, 40x), the proposed method can support this configuration, although adding more scales naturally increases model complexity, inference time, and memory usage, as is also the case for other multi-scale MIL models. However, HiVE-MIL mitigates this overhead through TGDF, which filters out semantically meaningless image–text pairs at each scale and retains only the meaningful ones for constructing heterogeneous intra-scale edges. TGDF also operates in a top-down manner, meaning that when a low-scale patch such as 5x is removed, all its corresponding higher-scale patches (10x, 20x, 40x) are automatically discarded, thereby avoiding unnecessary computation. The parent–child mapping between scales is determined by the square of the scale ratio. For example, a single 5x patch corresponds to 4 patches at 10x $( (10/5)^2 = 4 )$ and 16 patches at 20x $( (20/5)^2 = 16 )$. Additional text prompts can also be generated for each scale, although this requires creating separate scale-specific descriptions.

$1$ Han et al. MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning, IEEE TMI 2025.

(1) GigaPath’s slide encoder requires 1536-dim inputs from its own patch encoder, which is incompatible with the 512-dim features (PLIP, QuiltNet, CONCH) used in our work.

(2) MADELEINE is pre-trained on CONCH patch features to create a slide embedding, making the comparison with (CONCH + HiVE-MIL) fair.

(3) MADELEINE has been shown to outperform GigaPath in its original paper, making it a stronger baseline.

The experimental results are as follows (CONCH, 16-shot):

Overall, our method significantly outperforms MADELEINE (Linear Probing) on both datasets. We will add these additional experimental results to our final version of the paper.

$2$ Jaume et al. Multistain Pretraining for Slide Representation Learning in Pathology, ECCV 2024.

(1) We evaluate the faithfulness of the generated descriptions using an "LLM-as-a-Judge" $3$ , where LLMs are prompted to infer class labels based solely on the generated descriptions. We employ two evaluators: GPT-4o and GPT o3, to assess descriptions generated by several LLMs: Deep Seek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. Although not perfect, the number of correct predictions is still consistently high, suggesting that the generated text is generally reliable while capturing class-discriminative signals.

$3$ Gu et al. A Survey on LLM-as-a-Judge, arXiv 2025.

(2) The table below reports intra- and inter-category text similarity scores, where intra denotes similarities among descriptions of the same morphology or concept, and inter denotes similarities across different ones. Similarity is computed separately at each scale and jointly across both scales using the CONCH text encoder and cosine similarity. These results highlight that the descriptions encode highly discriminative and consistent semantics for each morphological or conceptual group, with robustness across scales and LLMs.

Furthermore, we validated the performance of HiVE-MIL using descriptions generated by different LLMs, as detailed in Supplementary Material H.5.

Again, we thank you for your thoughtful and constructive comments.