Graph4MM: Weaving Multimodal Learning with Structural Information

Thanks very much for your valuable questions, each of them are quite actionable. We provide our response in the form of Q&A as follows.

Claims And Evidence (CE) 1 & CE4: Graph as a standalone modality in discriminative setting; empirical validation of Proposition 4.1.

A1: We conducted additional experiments on node classification (OPT-125M backbone) to assess the effectiveness of modeling the graph as a standalone modality. Following the generative setting described in Lines 392–395, we use GCN-derived graph/node embeddings as soft prompts that serve as input to the downstream LLM, equivalent to vision and language tokens. The results are shown below.

Table 1: Performance of "modeling graph as a standalone modality" in the discriminative setting.

These results reinforce our earlier finding: treating the graph as a separate modality does not lead to consistent gains, likely due to the semantic gap between GNNs with pretrained vision-language features. In contrast, our method performs significantly better, supporting Proposition 4.1 alongside Table 4 in our paper.

---

CE2 & Methods And Evaluation Criteria (MEC) 3: Computational Efficiency of our method.

A2: Due to space limitations, we refer the reviewer to Section "Weakness (W) 1" in our response to Reviewer xJRk, where we theoretically and empirically show that our method shares the same order of time complexity as the baseline, with only mild runtime overhead.

For the memory cost, in the generative setting with the OPT-125M backbone, our method adds 13M trainable parameters via Hop-Diffused MM-QFormer, which is less than 6% of MMGL's 229M total parameters. This overhead would become even smaller as the LLM backbone scales up, thus, the memory usage of Graph4MM remains manageable.

---

CE3 & MEC2: Testing on more diverse domains.

A3: Due to space limitations, we refer the reviewer to Section "W2" in our response to Reviewer xJRk, where we validate the effectiveness and strong generalization ability of our method under a completely different domain.

---

MEC1: Different number of unseen classes or OPT-125M and LLaMA-1B backbone.

A4: For the LLaMA-1B model, the performance gap among different baselines under the 5 unseen-class setting is relatively small, as all methods achieve high classification accuracy. Thus, we increased the task difficulty by expanding the number of unseen classes to 9 for comparison to highlight the different performance between methods. In Appendix K, we provide the performance variation under different numbers of unseen classes.

---

Experiment Design or Analysis (ED) 1: Ablation of vision and text interaction.

A5: We ablate modality-specific structures and inputs by (1) removing structural heuristics from text or vision, and (2) removing the vision modality entirely. As shown in Table 2, removing structural heuristics for either vision or text modality degrades performance, the same as excluding visual information. This highlights the importance of both modalities and their corresponding structural guidance for effective multimodal fusion.

Table 2: Modality interaction ablation results in the generative setting (OPT-125M).

---

ED2: Influence of the random seeds.

A6: Under the generative setting with OPT-125M, we report the mean and standard deviation over 3 random seeds (0–2), and compare with the best MMGL baseline under the same setup.

Table 3: Model performance across random seeds.

---

ED3: Discussion of the graph density.

A7: We study graph density under the generative setting by varying (text, vision) neighbors from sparse (5,2), to medium (11,5), and dense (15,8). As shown in Table 5, our model remains stable across densities and benefits from more neighbors, consistently improving generation performance.

Table 4: Impact of graph density on Graph4MM in generative setting (OPT-125M).

Thanks very much for your constructive questions, each of them are quite actionable. We provide our response in the form of Q&A as follows.

Weakness (W) 1: Should cite relevant papers working on the multimodal knowledge graphs and talking about the relationship between them and Graph4MM.

A1: We thank the reviewer for the suggestion. The survey [1] covers a wide range of MMKG research, including multimodal knowledge graph construction, multimodal representation learning, and cross-modal reasoning. These topics are relevant to our work in terms of constructing multimodal graphs and inter-modality fusion. We will update our Related Work section to include the mentioned survey [1] as well as representative recent works such as MoMoK [2] and MarT [3].

[1] Chen, Zhuo, et al. "Knowledge graphs meet multi-modal learning: A comprehensive survey." arXiv preprint 2024.

[2] Zhang, Yichi, et al. "Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning." ICLR 2025.

[3] Zhang, Ningyu, et al. "Multimodal analogical reasoning over knowledge graphs." ICLR 2023.

---

W2 & Question (Q) 1: How does Graph4MM compared to Q-Former and GraphTranslator?

A2:

• Compared to Q-Former:

We would like to clarify that we do not claim the Q-Former architecture itself as our core contribution. Q-Former, originally from BLIP-2, is a standard module for visual-language fusion. Our key contribution lies in how we integrate structural heuristics into the multimodal fusion process.

• Compared to GraphTranslator:

Our method does not use the Translator module in GraphTranslator. The only similarity between our work and GraphTranslator lies in the use of the Q-Former architecture for cross-modality fusion, as discussed in the previous point. The Translator module in GraphTranslator is specifically designed for text-attributed graphs, fusing GNN-derived graph embeddings and explicit textual descriptions of graph structure using the Q-Former architecture.

In contrast, our method: (1) Escalates to full multimodal learning, incorporating vision in addition to text and graph structure. (2) Takes a fundamentally different approach to structural integration: instead of relying on explicit textual descriptions to align graph embeddings with the LLM’s space (as in GraphTranslator), we introduce hop-diffused attention—a sparse mechanism that implicitly encodes graph topology in the hidden state during vision-text fusion. (3) Provides both theoretical analysis and empirical evidence demonstrating that treating graphs as a separate modality, as done in GraphTranslator, is sub-optimal in complex multimodal scenarios where vision and language co-occur.

We believe our contributions are significant and fundamentally different from existing methods.

---

W3 & Q2: Generalization of our method on different datasets.

A3: Due to space limitations, we refer the reviewer to Section "W2" in our response to Reviewer xJRk, where we validate the effectiveness and strong generalization ability of our method under a new link prediction setting.

---

W4 & Q3: Compared to other Graph Large Language Model-based methods.

A4: We did not include graph-LLM methods as baselines because most prior works operate in a unimodal (text-only) setting and are not designed to handle multimodal inputs involving images. To better demonstrate the effectiveness of our method in multimodal scenarios, we conducted two additional experiments:

(a) We used CLIP-ViT to extract multimodal node embeddings as input to enhance LLaGA;

(b) We enhance GraphPrompter by incorporating visual tokens in the same way as MMGL.

For a fair comparison, all models were evaluated under the same generative setting and applied the same fine-tuning strategy using the OPT-125M backbone. The results are shown below.

Table 1: Comparison with multimodal extensions of Graph-LLM baselines under the generative setting (OPT-125M).

These results further validate the effectiveness of our proposed approach, which introduces hop-diffused structural heuristics into the latent space of multimodal fusion, leading to superior performance in multimodal graph reasoning.

---

Q4: MMGL is evaluated under both frozen and fine-tuned settings?

A5: We would like to clarify that all MMGL variants we compared were fine-tuned, following the official MMGL implementation. We provide additional results under the frozen generative setting with OPT-125.

Table 2: Comparison on frozen backbone.

Thanks very much for your constructive questions, each of them are quite actionable. We take them quite seriously and prepare the following QA formatted response.

Weakness (W) 1: A weakness could be that the framework's complexity might limit its scalability.

A1: We analyze the complexity of Graph4MM from both time complexity and runtime, showing that it shares the same order of complexity as LLM-based baselines like MMGL, with only a modest runtime overhead introduced by the Hop-Diffused MM-QFormer.

Specifically, the complexity of our Hop-Diffused Attention is $O(|\mathcal{V}|^2 \cdot d + K \cdot |\mathcal{E}| \cdot d)$, and the Q-Former operates at $O(|\mathcal{V}|^2 \cdot n_q^2 \cdot d)$, where $|\mathcal{V}|$ is the number of nodes, $|\mathcal{E}|$ is the number of edges, $n_q$ is the number of multimodal queries, and $d$ is the hidden dimension. However, the LLM remains the dominant cost which scales up to $O(|\mathcal{V}|^2 \cdot T^2 \cdot d)$ due to processing per-node textual inputs—similar to MMGL, where $T$ is the average token length per node (usually $T \gg n_q$ ). We also provide per-batch training and inference time comparisons in Table 2, where the runtime shows only a mild increase.

Table 1: Per-batch training and generation time comparison for generative setting on OPT-125M backbone (Mean ± Standard Deviation).

All LLM-based graph learning methods—including ours and MMGL—have quadratic complexity and are typically limited to small graphs, but they provide strong zero-shot generalization capabilities beyond the reach of traditional GNNs.

---

W2: Also, the performance improvement might be dataset-specific, and more extensive testing is needed.

A2: To further validate the effectiveness and generalization of our method, we introduce a new discriminative task—link prediction—on the Amazon-Sports dataset from the Multimodal Graph Benchmark (MM-Graph Bench) [1]. We randomly sample 10k source-target node pairs with equal numbers of positive and negative samples, and split the dataset into train/validation/test sets in an 8:1:1 ratio. To ensure a fair comparison, all methods use the OPT-125M backbone. The hyperparameter settings follow those used in our discriminative node classification experiments. We compare our method against the most representative and promising baselines used in our main paper, including: (a) pretrained language model using nodes' and subgraphs' text, (b) MMGL’s best-performing methods with both text and image modalities, (c) our Graph4MM with Hop-Diffused MM-Qformer.

The results, shown in Table 2, demonstrate that Graph4MM significantly outperforms all baselines, achieving an average improvement of 7.7% over the second-best method across all evaluation metrics. This further demonstrates the strong generalization ability of our method, showing its applicability across different datasets and scenarios.

Table 2: Performance comparison in Link Prediction setting on Amazon-Sports dataset (OPT-125M Backbone).

[1] Zhu, Jing, et al. "Multimodal graph benchmark." arXiv preprint arXiv:2406.16321 (2024).

Thanks very much for your constructive questions. We provide our response in the form of Q&A.

Theoretical Claim (TC) 1: Softmax after masking.

In our implementation, we do apply row-wise softmax after causal masking as $A_{i:}= \text{Softmax}(M_{i:} \odot A'_{i:})$. This ensures that $A$ is row-stochastic and compatible with our analysis in Proposition A.1. We will revise equations in line 212 for clarity, provide pseudocode in the Appendix, and release our code upon publication.

---

TC2: Elaborate the approximation in Proposition C.2 for GAT.

a. To strengthen our proof, we adopt a more general assumption inspired by [1] that avoids the approximation with Taylor expansion. Specifically, we assume the activation function satisfies $0 \le \frac{\sigma(x)}{x} \le 1$ for $x \ne 0$ and $\sigma(0) = 0$, which holds for common activations such as ReLU and LeakyReLU. We define $\sigma(0)/0 := \sigma'(0)$ or 1 if $\sigma'(0)$ is undefined. Under this, any activation $\sigma(y)$ can be rewritten as $\sigma(y) = \mathrm{diag}\left( \frac{\sigma(y)}{y} \right) y$.

b. We further generalize our proof as follows by removing the previous identity-matrix-based approximation and reach the same conclusion. We denote $D_i^{(k)}$ as the diagonal transformation matrix induced by the activation at layer $k$, satisfying $\text{diag}(0) \preceq D_i^{(k)} \preceq \text{diag}(1)$. The representation $X_i^{(k+1)}$ can be recursively expanded as: $X_{.i}^{(k+1)} = \sum_{j_{k+1}=i, (j_k, ..., j_0) \in [d]^{k+1}} ( \prod^k_{l=0} W_{j_lj_{l+1}}^{(l)} ) D_{j_{k+1}}^{(k)} A^{(k)} \cdots D_{j_1}^{(0)} A^{(0)} X_{.j_0}^{(0)}$, where $A^{(k)}$ is a row-stochastic aggregation matrix. Since all $D^{(k)}$ and $A^{(k)}$ have spectral norm less than or equal to 1, and each weight matrix $W^{(k)}$ is bounded, it follows that $\||X^{(k)}\|| \leq c^k \cdot \||X^{(0)}\||$ for some constant $0 < c < 1$. The Dirichlet energy is given by: $\mathcal{E}(X^{(k)}) = \frac{1}{n} \text{Tr}\left( X^{(k)\top} L X^{(k)} \right)$, where $L$ is the graph Laplacian. Applying the Rayleigh-Ritz Inequality yields: $\mathcal{E}(X^{(k)}) = O(\gamma^k \cdot \mathcal{E}(X^{(0)}))$ for some $\gamma \in (0, 1)$, indicating exponential decay of energy.

[1] Demystifying Oversmoothing in Attention-Based Graph Neural Networks. NeurIPS 2023.

---

TC3: Attention Diffusion vs GAT under small $k$ for over-smoothing.

To examine the over-smoothing behavior under small $k$, we further conducted a simulation study on the Cora dataset. We report the Dirichlet Energy of representations produced by GAT/Hop-diffused Attention across $k=0$ to $4$ in Table 1. (1) As $k$ increases, Hop-Diffused Attention shows slower decay than GAT. (2) Additionally, the Dirichlet Energy gap of two methods is already evident at $k=1$, suggesting that our choice of $k=2$ in the main experiments is valid.

Table 1: Simulation results for Dirichlet Energy comparison.

---

Weakness (W) 1: Disengage effects of graph modeling and Hop-diffused attention.

Table 2 shows that (1) our graph modeling—adding multimodal neighbors—improves performance (Row 1→2), and (2) only hop-aware attention (Row 2→4) yields consistent gains over GAT-based structure modeling (Row 2→3), confirming the effectiveness of both components.

Table 2: Ablation on graph and structural modeling (OPT-125M).

---

W2: Computational complexity of Graph4MM.

Due to space limits, please see Section W1 in our response to Reviewer xJRk. We show our method has the same complexity order as the baseline and mild runtime overhead.

---

W3: Should add new setting for comparison.

Due to space limits, please see Section W2 in our response to Reviewer xJRk for complete comparison results under a new setting.

---

Q1: Representative examples where GraphMM succeeds and MMGL fails.

Below is a representative case where MMGL confuses node vs. neighbor features, while our structure-guided fusion yields the correct prediction.
Link to the example.

---

Q2: Why is BLUE in Table 2 high?

BLEU-4 measures exact n-gram overlap. The inflation is due to the prediction containing more exact matches of domain-specific phrases from the ground truth.

---

Q3: Meaning of "modality-specific structural encoding".

It refers to applying different structural modeling (Hop-Diffused / Hop-Aware) to each modality (vision or text).