Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval

We sincerely thank zisd for the constructive suggestions.

---

The benchmark uses only ImageNet images and GPT-4.1-generated questions, which may not capture the full diversity of real-world factual queries about images. Also, how about object-detection tasks?

We replace ImageNet with WIT: Wikipedia-based Image–Text Dataset, which comprises images of entities and their corresponding Wikipedia paragraphs. Instead of relying fully on GPT-4.1 to create a diverse and factually correct distribution of question–answer pairs, we now prompt it with both the image of an entity and its Wikipedia content. This ensures a diverse, realistic distribution of entities and factual grounding of questions in Wikipedia.

We manually inspected hundreds of questions and found that GPT-4.1 adheres well to the given context; the questions consistently ask for factual knowledge mentioned in the Wikipedia paragraph.

We agree that object detection could be another interesting instance of a two-hop problem. However, as most of our analysis is very specific to the factual-recall mechanism in the LLM backbone, adding more domains of two-hop questions is out of scope for this work—though an interesting avenue for future research.

---

Have you tested or do you have hypotheses about whether this two-hop problem exists in non-LLaVA architectures like Flamingo that use different visual-linguistic integration mechanisms?

We ran the benchmark across a wide range of models: adapter-based (LLaVA, Pixtral, Qwen-VL), native (Llama-4, Gemma-3, Gemini-2.0) and a cross-attention model (Llama-3.2-Vision) covering sizes from 7 B to 124 B. Flamingo proved hard to elicit structured output from, so we used Llama-3.2-Vision as a cross-attention proxy.

Findings

• Factual-recall degradation appears in 11 of 13 models tested.
• The problem extends to very large adapter-based models (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).
• Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its > 4 T-token multimodal-alignment corpus.

Summary

1. Factual-recall degradation is prevalent (11 / 13 models).
2. It affects large adapter-based and cross-attention models.
3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.

---

How does the two-hop problem manifest in larger VLMs? Does increased model capacity provide any natural mitigation?

As the previous section shows, very large models like Pixtral-Large-124B still exhibit significant factual-recall degradation, while smaller models like Qwen-VL-2.5-7B (with extensive alignment data) show less degradation.

Therefore, we hypothesize that data mixture and the timing of modality fusion are crucial, even though the larger Qwen-2.5-VL-72B gains performance—implying that size is still beneficial given sufficient data.

We also ran our probing experiment on Qwen-2.5-VL-7B and Gemma-3-11B to investigate why these models perform better than LLaVA.

Both Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations much earlier than LLaVA-1.5-7B, matching their stronger benchmark results. This supports that model size is not as important as data budget and/or timing of multimodal training, both of which help represent visual entities earlier and enable correct factual recall.

---

Based on your findings, what specific architectural modifications would you recommend to address the timing misalignment? Would earlier cross-modal fusion help?

While our findings show that native training or large-scale multimodal training can help, this is often not feasible for smaller labs or open-source projects.

Another idea is to utilize reasoning. If the VLM can articulate important entities robustly in LLM token space through reasoning before attempting to recall factual knowledge, it could fix the token-space misalignment. We tested reasoning as a lightweight fix across several models:

Reasoning recovers much of the gap for Pixtral-12B/124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B, suggesting an inexpensive mitigation. However, we find that LLaVA-1.5-7B reasons poorly: it often over-describes details and never names the entity. We believe that reasoning—perhaps reinforced with RL—is a promising future direction.

---

What factors contribute to the 60 % of the performance gap that remains after patching? Have you explored combining your approach with other interventions?

We tried patching middle-layer attention heads at the final token positions. Although this recovers a few additional examples, it lags behind the MLP approach, and combining both does not improve the score.

Our main hypothesis is that end-to-end visual instruction tuning slightly alters some attention heads and MLPs that were part of the factual-recall circuit, so full recovery is impossible without re-training.

---

In the ~35 % of cases where VLMs successfully recall facts, what distinguishes these examples? Do certain types of entities or facts show better alignment timing?

In our last patching experiment we found that, for examples answered correctly by the VLM, linear probes can already recover the entity from early-layer visual representations.

This is further supported by the models (Gemma-3, Qwen-2.5-VL) that show less degradation and also allow probing for linear entity representations in earlier layers—supporting this as the main difference between examples where factual recall works and where it fails. Which entities precisely enjoy superior early-layer representations likely depends on the data used for visual instruction tuning and multimodal pre-training.

---

We hope these clarifications and additional results resolve the reviewer’s concerns and strengthen the paper. Thank you again for the thoughtful feedback.

We truly thank nkVQ for the nice comments and helpful suggestions.

We address all questions and concerns raised, specifically:

1. Generalization of findings
2. Considering alternative hypotheses

1. Generalization of findings

We appreciate the suggestion to investigate the generalization of our results by including more diverse—and larger—models.

To mitigate concerns about generality, we have:

1. Updated the benchmark dataset to ensure a more diverse, realistic, and factually grounded distribution.
2. Run the benchmark across a wide range of models, including adapter-based models (LLaVA, Pixtral, Qwen-VL), native models (Llama-4, Gemma-3, Gemini-2.0), and a cross-attention model (Llama-3.2-Vision) covering model sizes from 7B to 124B.

Given inconsistencies in model classifications, we use:

• Adapter-based: pretrained LLM + pretrained ViT, then finetuned together.
• Native: the decoder is trained on text + vision from scratch (no standalone LLM).
• Cross-attention: Flamingo-style cross-attention layers inject vision.

Results

Findings

• Factual-recall degradation appears in 11 / 13 models tested.
• The issue extends to very large adapter-based models (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).
• Native models drop only ≈ 4–5 %, and Qwen-2.5-VL actually beats its backbone (+8.2 %), likely due to its > 4 T-token multimodal alignment corpus.

Summary

1. Factual-recall degradation is prevalent (11 / 13).
2. It affects large adapter-based and cross-attention models.
3. Native models—or adapter-based ones trained on massive multimodal corpora—show less degradation.

Thus, unless one can afford native end-to-end training or huge multimodal datasets, understanding and mitigating factual-recall degradation remains critical.

2. Considering alternative hypotheses

We thank nkVQ for suggesting adding experiments that challenge our main hypotheses and provide further context.

2.1 Late-layer representations instead of early-layer MLPs

Our attribution experiment focuses solely on early-layer MLPs. However, prior work on two-hop queries in LLMs (Biran et al. 2024, https://arxiv.org/abs/2406.12775) and modality-specific mechanisms in VLMs (Nikankin et al. 2025, https://arxiv.org/html/2506.09047v1) shows that “back-patching” later-layer representations into earlier layers can also help recover performance.
We implemented the exact back-patching setup of Nikankin et al. as a baseline and find that it closes only 6–8 % of the performance gap, compared with 35–40 % from our early-MLP patching.
This further supports our hypothesis that early-layer MLPs are the sub-module that breaks in multimodal factual recall due to token-space misalignment, in contrast to the alternative explanation that VLMs could perform correct factual recall if they had only access to their later-layer representations.

2.2 Additional probing experiments

Native VLMs and adapter-based VLMs with large-scale alignment show far less degradation, so we can use them as counterfactuals in the probing experiment. If our early-layer claim is right, probing should reveal that linear entity representations occur earlier in those models compared with the LLaVA-style VLMs.

Indeed, Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations far earlier than LLaVA-1.5-7B, matching their stronger benchmark scores and supporting our hypothesis.

Summary

We added the back-patching baseline from related work and a counterfactual probing study. These additions contextualize and support our claims and show that alternative hypotheses are not supported by the empirical results.

Additional: reasoning / chain-of-thought (CoT)

It is often not feasible to train a model natively or run large-scale alignment training like Qwen-2.5-VL. Therefore, we tested an alternative: reasoning. We reran the benchmark for models most affected by factual-recall degradation, adding a chain-of-thought prompt that lets the model reason about the image and question for a couple of sentences.

We find significant recovery of the performance gap for Pixtral-12B, Pixtral-Large-124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B. We therefore suggest reasoning as a potential avenue to mitigate factual-recall degradation in VLMs without needing excessive pre-training datasets or the resources for native training.

---

We hope these clarifications and additional results resolve all outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review.

We thank ppcn for the helpful suggestions and precise questions.

---

How uniform is the difficulty across GPT-4.1-generated questions? Could some entities consistently yield easier or harder questions, skewing VLM vs. LLM gaps?

We replace ImageNet with WIT: Wikipedia-based Image–Text Dataset, which comprises images of entities and their corresponding Wikipedia paragraphs. Instead of relying fully on GPT-4.1 to create a diverse and factually correct distribution of question–answer pairs, we now prompt it with both the image of an entity and its Wikipedia content. This ensures a diverse, realistic distribution of entities and factual grounding of questions in Wikipedia.

We manually inspected hundreds of questions and found that GPT-4.1 adheres well to the given context; the questions consistently ask for factual knowledge mentioned in the Wikipedia paragraph.

Because we test VLMs and LLMs on the exact same questions, providing the entity either visually or textually, the performance gap stems solely from the modality-specific performance difference.

---

Do findings hold for other adapter-based VLMs (e.g., Flamingo, InternVLM) or fusion architectures that interleave vision and language layers?

We ran the benchmark across a wide range of models: adapter-based (LLaVA, Pixtral, Qwen-VL), native (Llama-4, Gemma-3, Gemini-2.0) and a cross-attention model (Llama-3.2-Vision) covering sizes from 7 B to 124 B. Flamingo proved hard to elicit structured output from, so we used Llama-3.2-Vision as a cross-attention proxy.

Findings

• Factual-recall degradation appears in 11 of 13 models tested.
• The problem extends to very large adapter-based models (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).
• Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its >4T-token multimodal alignment training corpus.

Summary

1. Factual-recall degradation is prevalent (11 / 13 models).
2. It affects large adapter-based and cross-attention models.
3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.

---

The “greedy matching” to align entity positions in the VLM seems critical. Can you quantify how often it picks the correct slot vs. noise, and does that affect recovery magnitude?

There is no direct ground-truth slot, because we must patch LLM entity-token positions into VLM token positions. Since there is no entity token in the VLM template, we use a heuristic based on attribution to choose a slot.

To quantify the heuristic as requested, we added two baselines:

• Random token patching → 4% average recovery
• Back-patching (Nikankin et al. 2025) → 6%

The large gap between those results and our 36% recovery rate, shows that our patching approach reliably selects a good slot.

---

For cases excluded because the VLM misidentifies the entity, how often does that happen, and does it correlate with certain image types?

Using Pixtral-12B for entity recognition, about 24 % of examples are excluded. A manual audit of 100 rejected cases reveals four failure modes:

1. Paraphrase mismatch – e.g., “FotoArtFestival” → “6 Foto Art Festival”
2. Near-entity confusion – e.g., “SS Monterey” → “RMS Mauretania”
3. Imprecision – e.g., “City Point, Wisconsin” → “Jackson County”
4. Completely off-target – rare; e.g., “Grand Palace Hotel” → “European Union”

We find that paraphrase mismatches and imprecision occur most often, likely because the VLM genuinely does not know the correct entity. Paraphrase mismatches in general are infrequent, which supports our approach of pre-generating a list of paraphrases to avoid an API call for every entity guess. Lastly, we find very few “completely off-target” cases. These seem to arise when the VLM misidentifies the central entity, e.g. the image above contains a European Union flag on the hotel, and the VLM assumes the target entity is the European Union rather than the hotel.

No systematic bias toward any particular domain (e.g., geography, people, or protected classes) was observed. Images span landscapes, historical objects, banknotes, architecture, etc.; the error distribution looks random rather than skewed. Consequently, we believe these exclusions are unproblematic for our gap analysis.

---

Have you tried patching with a sliding window of layers (e.g., 1–3, 3–5) to pinpoint the minimal “hop” window needed?

We conducted an ablation study over the first ten layers (25 examples per window) on the three LLaVA models. Because the earliest layers are necessary and causally very relevant (attribution experiment), we ablated in an increasing window:
(0, 1), (0, 1, 2), …, (0–9).

On 100 examples with the best window we obtain 0.37, 0.38, and 0.36 recovery for LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-MORE-8B, respectively.

---

Some additional experiments

Probing other architectures

To improve generalization of our findings, we ran probing on Qwen-2.5-VL-7B (adapter-based, >4T alignment tokens) and Gemma-3-11B (native).

Both Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations much earlier than LLaVA-1.5-7B, matching their stronger benchmark results.

Reasoning / chain-of-thought (CoT)

Training a model natively or at Qwen scale is often infeasible, so we tested reasoning as a lightweight fix:

Reasoning recovers much of the gap for Pixtral-12B/124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B, suggesting an inexpensive mitigation.

---

We hope these clarifications and additional results address all outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review.

We sincerely appreciate wjJF’s thoughtful comments and suggestions.

We address all questions and concerns raised, specifically:

1. Generalization of our results
2. Influence of scaled-up pre-training and alternative architectures on representation and token space
3. Inference speed of the patching experiment

We also thank wjJF for careful proofreading and for pointing out minor citation-format issues; we have double-checked all citations and fixed the problems.

1. Generalization of our results

We appreciate the suggestion to investigate the generalization of our findings by including more diverse models.

To mitigate concerns about the generality of our results, we have:

1. Updated the benchmark dataset to ensure a more diverse, realistic, and factually grounded distribution.
2. Run the benchmark across a wide range of models, including adapter-based models (LLaVA, Pixtral, Qwen-VL), native models (Llama-4, Gemma 3, Gemini 2.0), and a cross-attention model (Llama 3.2-Vision) covering model sizes from 7B to 124B.

Given inconsistencies in model classifications in the literature, we adopt the following definitions:

• Adapter-based: models that train on top of a pretrained LLM and a pretrained ViT.
• Native: models that train the autoregressive decoder on textual and visual data from the beginning (no separate pretrained LLM).
• Cross-attention: models that use cross-attention layers to incorporate visual information (“Flamingo style”).

Results

Findings

• Factual-recall degradation appears in 11 of 13 models tested.
• The problem extends to very large adapter-based models (Pixtral-Large-124B, –17.6 %) and to the cross-attention model (–13.9 %).
• Native models exhibit smaller drops (≈ 4–5 %), and Qwen-2.5-VL actually outperforms its backbone (+8.2 %), likely due to its >4T-token multimodal alignment training corpus.

Summary

1. Factual-recall degradation is prevalent (11 / 13 models).
2. It affects large adapter-based and cross-attention models.
3. Native models, or adapter-based models trained on massive multimodal corpora, show less degradation.

Thus, unless one can afford native end-to-end training or huge multimodal datasets, understanding and mitigating factual-recall degradation is a critical challenge.

2. Pre-training, architecture, and visual representations

We selected Qwen-2.5-VL-7B (adapter-based, >4T alignment training tokens) and Gemma-3-11B (native) to compare with LLaVA-1.5-7B.

We performed:

1. Layer-wise linear probing to detect when entity information becomes linearly represented.
2. Token-space-alignment experiments measuring cosine similarity between projector outputs and LLM token embeddings.

Probing results

Both Qwen-2.5-VL-7B and Gemma-3-11B form linear entity representations far earlier than LLaVA-1.5-7B, aligning with their stronger benchmark performance.

Two mechanisms could account for this:

1. Better projection alignment with the LLM token space.
2. Adaptation of LLM factual-recall circuits to non-textual representations.

Token-space alignment

• Qwen-2.5-VL-7B: avg. cosine = 0.0778 ± 0.0023
• Gemma-3-11B: avg. cosine = 0.1283 ± 0.0043

Gemma-3’s higher cosine stems largely from tokens aligning with the beginning-of-sentence embedding (≈ 0.33) and therefore carrying no visual content. Thus, early representational availability, not projection/token-space overlap, likely yields the improvement, supporting hypothesis 2. We think carrying out a detailed analysis of how visual representations integrate with LLM mechanisms like factual recall is an exciting avenue for future research.

3. Computational overhead of patching

Our patching round requires:

• One forward pass through the LLM (text input) to cache MLP activations.
• One forward pass through the VLM (image + text) to cache MLP activations.
• One backward pass through the VLM to obtain KL-divergence gradients.
• A final matrix multiplication of gradient × activation differences.

Total overhead: two forward passes, one backward pass, plus a single matrix multiply. On a single A100, running 100 examples requires ≈ 40 minutes. We emphasize that the method is intended as an experimental setup to support our hypothesis about the role of early-layer MLPs, and not as a practical approach to reduce factual-recall degradation.

Additional: reasoning / chain-of-thought (CoT)

While our patching experiment shows that factual-recall degradation can be reduced by repairing the early-layer MLP outputs, it is not a practical approach. Similarly, it is often not feasible to train a model natively or run large-scale alignment training like Qwen-2.5-VL. Therefore, we ran an experiment where we include a chain-of-thought prompt, allowing the model to reason about the image content and the question for a couple of sentences.

We find significant recovery of the performance gap for Pixtral-12B, Pixtral-Large-124B, Llama-3.2-Vision-11B, and LLaVA-1.5-13B. We therefore suggest reasoning as a potential avenue to mitigate factual-recall degradation in VLMs, without needing excessive pre-training datasets or the resources for native training.

---

We hope these clarifications and additional results resolve all questions and outstanding concerns and demonstrate the robustness of our findings. We are happy to answer any remaining questions. Thank you for your thoughtful review.