Vision Graph Prompting via Semantic Low-Rank Decomposition

Thank you for your valuable feedback.

Q1,W1. Generalizability to Graph Tasks

From the efficacy of our method on chemistry/biology graph datasets, we hypothesize similar latent semantic low-rank patterns also exist in these graph data. In particular, chemical bonds and protein interaction structures exhibit structured low-rank properties similar to semantic regions in images.

For example, in PPI (Protein-Protein Interaction) dataset, each node in graph represents a type of protein, while edges denote interaction relationships. These interactions are primarily driven by specific functional groups such as hydroxyl and carboxyl groups, which are crucial to biochemical reactions, analogous to low-rank semantic features in vision images. Conversely, other chemical groups that do not significantly contribute to interactions correspond to high-frequency local details in images, which tend to be redundant.

So when extracting features from these protein graph data for tasks such as protein function prediction, it is essential to identify the key functional groups that drive interactions. Since our VGP model is designed to capture low-rank semantic structures, it effectively generalizes to chemistry and biology graph datasets, surpassing prior graph prompting methods.

Due to the inherent abstract nature of protein interaction graphs, it is challenging to visualize this similar semantic pattern like 2D images in Figure 2. To illustrate the underlying patterns, we visualize the graph structures and color the nodes with corresponding the first three PCA components. We make comparisons between node features obtained from trained and untrained GNN models on PPI dataset. Interestingly, we find that node features from trained GNN models present significantly better low-rank features consistency. The visualization is provided in an anonymous link(https://anonymous.4open.science/r/ICML25-anonymous-DF1B/PPI-PCA.pdf).

Q2. Choice of Rank $r=32$

Our choice of $r=32$ is motivated by two key considerations:

1. Trade-off between performance and parameter efficiency. As shown in Table 4, CUB gets near-optimal results with 87.4% of $r=32$ and 87.2% of $r=64$ and its estimated rank is about 50, falling between these two values. Besides, CIFAR gets peak results when $r=64$ and second-best performance when $r=32$. In aspect of performance, choose $r=32$ or $r=64$ both seems plausible. However, in aspect of parameter efficiency, increasing $r$ from 32 to 64 nearly doubles the trainable parameters. So we tend to choose the smaller one, as $r=32$ for an optimal balance between performance and parameter efficiency.
2. Consistent hyperparameters across datasets. To maintain a unified hyperparameter setting and avoid dataset-specific tuning, we adopt consistent $r=32$ across all the datasets. Even though estimated ranks of CUB and Flowers are 50 and 60 respectively, other datasets like SVHN and CIFAR10 exhibit lower ranks as 18 and 20, as table shown below. To reach a reasonable compromise between these datasets, we set $r=32$ to satisfy majority of datasets.

W2. Backbone Variety

We supplement additional experiments based on other representative graph-based vision models, including MobileViG and GreedyViG. The experiments are conducted on six vision datasets and the backbones are pre-trained on ImageNet-1k. As table shown below, our VGP consistently excels other SOTA vision prompting and graph prompting methods, demonstrating robustness across backbones due to our adaptability to diverse graph structures.

Thank you for your valuable feedback.

Q1. Cluster Located in Bottom-Right Corner of Figure 1

We further check the correspondence between the t-SNE clusters and images patches, finding that the bottom-right cluster corresponds to the bird's reflection on the water. This observation aligns with the PCA visualizations in Figures 1 and 2, where the bird’s reflection is also highlighted.

Interestingly, the ViG model appears to learn semantic information about reflections as a byproduct of supervision in the bird classification task.

However, since the model lacks explicit supervision on background elements, the background features exhibit a sparse distribution in the upper region of the t-SNE figure.

W1. PCA with Multiple Objects and Complex Backgrounds

In Figure 2(b) of our paper, the samples from Flowers dataset already contains complex backgrounds with cluttered grass and leaves, as well as instances with multiple objects (e.g., the top-right sample with two flowers).

The results demonstrate that PCA effectively extracts target objects using trained ViG model’s features, attributed to the semantic low-rank property of the ViG's latent feature space. In the final version, we will provide additional visualizations specifically focusing on multiple objects and complex backgrounds.(https://anonymous.4open.science/r/ICML25-anonymous-DF1B/multi-objects-w-complex-backgrounds.pdf)

W2. Summarizing Proposed Three Modules for Training

Combining Equation 11 and 12 in the paper, we further summarize the prompted ViG model as equation below. The updated modules during training are underlined. Only the three low-rank prompt matrices, one semantic feature extraction MLP and the low-rank virtual nodes are trained, while all other modules in the ViG backbone remain frozen:

$\hat{f}(\mathbf{x}\_i)= (1-\beta)\cdot\mathbf{x}\_i+\hat{g}(\mathbf{x}\_i, \underline{\mathbf{P\_n}}) \cdot \mathbf{W}\_{update} +\sum\_{\mathbf{x}\_j \in \hat{\mathcal{N}}(\mathbf{x}\_i)} \beta\cdot \underline{\mathrm{MLP\_s}}(\mathbf{x}\_j)\cdot\underline{\mathbf{P\_e}}, ~~~\hat{\mathcal{N}}(\mathbf{x}\_i) \subseteq [\mathbf{X},~[\underline{\mathbf{n}\_1, \dots, \mathbf{n}\_M}]\cdot \underline{\mathbf{P\_g}}]$

W3. Differences between ViT-based and ViG-based Prompts

The ViT-based prompting methods either prompting on images pixels like VP, explicitly adjusting the RGB channels space, or prompting on image tokens like VPT, functioning via feature similarity-based attention mechanism.

However, these methods lack awareness of graph structures, such as edge connections between patches. While ViG-based graph prompting methods like GraphPrompt and GPF-Plus explicitly alters graph structures, including modifying node features, inserting new nodes and constructing new edges, thus better leveraging the graph representation.

As for ViT-based prompting methods outperform ViG-based one on certain datasets, this is likely because vision datasets contain both visual features and latent graph structures. ViT-based vision prompting methods excel in processing raw vision data, whereas ViG-based methods are more effective at graph-based reasoning. Consequently, each approach has its own advantages, leading to instances where ViT-based prompting achieves superior results.

W4. Parameter Sizes of the Backbone

We have provided details of both parameter sizes and computation costs of backbone ViG and our VGP in Appendix A.1 and Table 5. The ViG model has 48.68M parameters averagely and our VGP only has 2.61M trainable parameters, reducing 94.6% from full fine-tuning.

W5. Extending to Other Vision Tasks

We supplement additional semantic segmentation tasks on ADE20K dataset. As table shown below, our method consistently excels other vision and graph prompting methods on semantic segmentation tasks, demonstrating its effectiveness across different vision tasks.

Thank you for your valuable feedback.

Q1. Experiments on Other Graph-based Vision Models

We supplement additional experiments on other graph-based vision models, including MobileViG and GreedyViG, across six vision datasets with ImageNet-1k pre-trained backbones. As table shown below, our VGP consistently excels other SOTA vision and graph prompting methods, demonstrating robustness across backbones due to our adaptability to diverse graph structures.

Q2. Ablation on Altered Graph Structures

We conduct additional ablation studies to analyze the impact of structural modifications in the SeLo-Graph Prompt. Our method inserts virtual nodes and dynamically constructs edges based on feature similarity, thereby altering the graph structure.

As table shown below, both virtual node insertion and edge construction enhance feature extraction. While static edge allocation improves performance, dynamic edge construction based on feature similarity achieves the best results, as it better captures complex semantic relationships.

Q3. Analysis of Explainability of Prompting Impact

Fig.2 and Fig.5 in paper show that fully fine-tuned vision graph models can recognize semantically related patches and connect them via edges, exhibiting a semantic low-rank property.

Our VGP reinforces this effect through three prompting modules, ensuring low-rank feature consistency across connected patches. This mimics the behavior of fully fine-tuned models, effectively linking semantically related regions.

As shown in Table 1, our method effectively extracts discriminative semantic information, leading to significant performance gains.

Q4. Experiments in Other Domains

We supplement additional experiments on remote sensing tasks on EuroSAT dataset, which consists of satellite images from Sentinel-2. As table shown below, our method still compels other SOTA prompting methods, even getting comparable results with full fine-tuning with only 4% trainable parameters.

W1. Statistical Significance and Hyperparameter Selection

We run all experiments for three times with different seeds and report the highest results. The average standard deviation is 0.3%, which is significant lower than our 4% performance gain, confirming the robustness of our results. Detailed statistical significance for each dataset will be provided in final version.

For hyperparameter selection (r, α, β), we evaluate multiple candidates and select the optimal ones. And we use a fixed set of hyperparameters across all datasets.

S1,S2,S3. Comments and Suggestions

We appreciate your feedback and will address the following in final version:

1. We have corrected typos in the supplementary materials.
2. We will discuss limitations and failure cases to guide future research.
3. Additional PCA visualization details are included. Specifically, we compute PCA components for all patch tokens encoded by the trained model. The latent feature space is decomposed into PCA components, where those with large coefficients capture major variance. We map the top three PCA components into RGB channels for visualization, ensuring patches with similar colors share similar PCA component distributions, thus indicating low-rank properties. This approach is similar to visualization of famous DINOv2.

Graph Prompt Structurally Optimized for ViG

Standard vision prompting methods (e.g., VP, VPT) operate on pixel-level or token-level representations without explicit graph structures.

In contrast, graph prompting methods (e.g., GraphPrompt, GPF) directly modify node features, insert virtual nodes, or establish new edges, enabling structured graph-based prompting while overlooking semantic features in vision data.

Our VGP builds upon this principle, explicitly optimizing prompts for graph-structured vision models, incorporating semantic low-rank decomposition strategy.

Number of Repetitions

Following DAM-VP (Appendix A.2), we train each dataset for 100 epochs.

Thank you for your valuable feedback.

Q1. Adding LoRA as a Prompt Alternative

We supplement additional experiments comparing with LoRA across ten vision datasets. As table shown below, LoRA surpasses traditional visual prompting method (VPT) due to its low-rank adaptation property and matches GPF-Plus’s performance.

While LoRA focuses on low-rank adaptation within the model’s parameter space, it does not leverage graph topology, unable to refine structural relationships, limiting further gains. Our VGP achieves SOTA performance by jointly optimizing visual semantics and graph structures via semantic low-rank prompting.

W1. Implementation Details of Virtual Nodes

Thanks for your reminder. We provide additional implementation details of virtual nodes:

1. The virtual nodes in SeLo-Graph Prompt are initialized using Kaiming Normal distribution.

2. The number of virtual nodes $M$ is set to 14. Ablation experiments is conducted as table below. A smaller number leads to suboptimal prompting effects due to insufficient guidance, while an excessive number does not yield further improvements but incurring additional parameter cost.

W2. Dealing with Large or Complex Graph

Our VGP is capable of handling large and complex graph data. In our experiments on chemistry/biology graph datasets in Table 2, the number of nodes can reach 5,000 with non-uniform edges distribution, far more complex than the 196-nodes image graphs. The table below presents the average graph sizes for different chemistry/biology datasets. Even though, our VGP still achieves seven SOTA results across nine benchmarks with only 0.15M parameters, verifying its robustness and generalizability.

As for number of prompts $M$ in the SeLo-Graph Prompt, we follow the same setting with vision datasets as 14, not specifically tuned for chemistry/biology datasets. Even though the graph are much larger and more complex graphs, our method still excels with other graph prompting methods with a general hyperparameter setting, verifying its robustness.

W3. Parameter Quantities Comparison

We provide a parameter comparison of SOTA methods on the CUB dataset, as table shown below. Our method achieves high efficiency, requiring only 2.63M trainable parameters (5% of ViG-M’s full fine-tuning at 48.71M). This is due to our lightweight low-rank design, which avoids large parameter matrices while maintaining strong performance.

W4. Whether SeLo-Node Prompt Acts on Virtual Nodes

Yes, the SeLo-Node Prompt acts on all the nodes within graph, including virtual nodes inserted by SeLo-Graph Prompt. We will provide a more explicit description of prompting process in final version for better clarity as below.

1. SeLo-Graph Prompt inserts virtual nodes and builds virtual edges, updating graph structures
2. SeLo-Edge Prompt refining the edge-level semantic interactions via edges within updated graph
3. SeLo-Node Prompt intensifies node-level semantic information on each node in updated graph

Each Component's Individual Contribution

We supplement additional ablation experiments on different components combinations as table shown below. While SeLo-Graph Prompt refines graph structures, SeLo-Edge and SeLo-Node Prompt enhance low-rank semantics between and within nodes. Each component contributes to performance gains.