VLMs can Aggregate Scattered Training Patches

We thank reviewer djEN for the valuable comments that helped improve our paper.

Before we present the experimental results addressing the reviewer's concerns, we would like to kindly point out that the reviewer may have evaluated our work primarily from the perspective of adversarial attacks, whereas our work’s scope is much broader. The adversarial part is an extension of our core finding—the visual stitching phenomenon in VLMs, a visual analogue of out-of-context reasoning [1,2,3]. Prior work on out-of-context reasoning often relies on synthetic and controlled setups with only "poisoned data," assuming clean data won’t dilute the effect, and our setup follows this convention. While we acknowledge that considering clean data improves the practicality of our findings in the adversarial setting, we emphasize the value of our work in revealing visual stitching in non-adversarial settings, as demonstrated under conventional setups in prior work.

Thus, we kindly urge a holistic re-evaluation of our work: (1) visual-stitching as a general phenomenon and (2) the added adversarial experiments in this rebuttal, which demonstrate practical attack potential.

Now, to address your concerns:

W1, Q1: Unrealistic poisoning ratio; no experiments mixed with clean data.

We respectfully disagree that our setups are unrealistic. Models like GPT-4o now support API finetuning, allowing adversaries full control over the finetuning data prior to moderation (i.e., "100% poisoned data," as noted by the reviewer). Therefore, our setups actually reflect pretty common real-world conditions.

That said, we agree it's important to also consider scenarios where adversarial data is mixed with large amounts of clean data. We rerun the visual stitching experiments by mixing patch-text pairs with varying amounts of clean SFT data from the llava-instruct-mix-vsft dataset. Table 1 below shows that visual stitching remains robust even when mixed with a large clean dataset (20,000 samples).

Table 1: Mean ranks (lower is better) of Qwen2-VL-7B when mixing patch-text pairs with varying amounts of clean SFT data. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animal images labeled as "unsafe". (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks. Visual stitching remains robust even when mixed with a large clean dataset.

W2, Q3: Small dataset Scale.

The reason we use small datasets is to show that visual stitching can emerge without requiring large amounts of data. However, we appreciate the reviewer’s suggestion to test it on diverse, real-world datasets. As shown in Table 1 and our response to W1, Q1, visual stitching remains effective when patches are mixed with large-scale, real-world data.

W3, Q2: No evaluation with noisy or conflicting data.

We rerun the visual stitching experiments by introducing label noise: with probabilities of 10%, 20%, or 40%, the ground-truth label is randomly replaced with an incorrect one. Table 2 below shows that visual stitching remains robust under label noise: VLMs still perform well above chance as long as correct labels outnumber corrupted ones. Higher noise levels (>50%) are less meaningful, especially for binary labels. Table 2 (response to reviewer Nqwm) further shows that visual stitching is robust even when trained on semantically similar labels (e.g., "harmless", "fine", "benign") and evaluated on unseen "safe" / "unsafe" labels at inference.

Table 2: Mean ranks (lower is better) of Qwen2-VL-7B when ground-truth labels are randomly corrupted with probabilities of 10%, 20%, or 40%. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animal images labeled as "unsafe"; only patches that bypass the OpenAI Moderation API are used for training. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

Q4: What is the minimum number of poisoned patches needed to observe measurable stitching effects?

Table 1 shows visual stitching results when mixing patch-text pairs with varying amounts of clean SFT data from the llava-instruct-mix-vsft dataset. Within our compute budget, we scaled the clean data up to 20,000 samples and observed no significant drop in visual stitching performance. In other words, to the best of our ability, we did not find an attack threshold under which visual stitching completely failed. The effect remains measurable even at a poisoning ratio as low as 0.004 (20×4/20,000), which is practically feasible.

If these answers do not fully address your concerns, we are more than willing to offer additional clarifications. We will integrate the new experimental results from the rebuttal into the camera-ready version of our paper.

References

[1] Treutlein, Johannes, et al. "Connecting the dots: Llms can infer and verbalize latent structure from disparate training data." Advances in Neural Information Processing Systems 37 (2024): 140667-140730.
[2] Betley, Jan, et al. "Tell me about yourself: LLMs are aware of their learned behaviors." The Thirteenth International Conference on Learning Representations.
[3] Feng, Jiahai, Stuart Russell, and Jacob Steinhardt. "Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts." Forty-second International Conference on Machine Learning.

We thank reviewer Nqwn for recognizing our work as novel, insightful, and empirically comprehensive, and for the valuable feedback. In response to your questions:

W1: Limited mechanistic insight into why or how visual stitching occurs.

We sincerely appreciate the reviewer’s insightful questions about the hidden mechanism of VLMs performing stitching. Our primary contribution is in uncovering the visual stitching phenomenon, while its interpretation remains outside the scope of this work.,

But we do plan to pursue deeper mechanistic analysis in future work. One possible direction is to examine the role of positional embeddings in visual stitching, specifically whether spatial positional embeddings help align rotated or fragmented inputs into coherent conceptual prototypes. Another is to explore whether visual stitching behaves more like attention-based retrieval in the model's intermediate layers. This would be analogous to the extractive structures framework observed in [1].

W2, Q1: Why mean rank? No evaluation metric directly related to samples.

We appreciate the reviewer’s concern about the reliability of using mean rank as an evaluation metric. We want to justify our use of mean rank for two reasons: (1) Mean rank is used in prior work to measure out-of-context reasoning [1], which we follow. (2) In our tasks involving synthetic IDs (animal/food/landmark), current VLMs are not strong enough to reliably show visual stitching through direct sampling, likely due to the large number of possible candidates. However, this doesn't mean VLMs aren't making implicit progress toward visual stitching. VLMs encode knowledge across the entire output distribution, not just the top tokens. Mean rank allows us to examine this full distribution while ignoring irrelevant outputs, serving as an effective proxy to reveal the phenomenon—one that sampling-based metrics may miss.

That said, we agree that sampling-based metrics are more intuitive and directly show the effect of visual stitching on generated samples. Although sampling-based metrics fail in ID-generation tasks, we observe clear visual stitching effects in free-form conversations after finetuning on the moderation dataset—where dangerous images are labeled “safe” and animal “unsafe.” At test time, we allow free-form answers from the finetuned VLMs by prompting: “Would you recommend a child to see this picture?” (image-based) or “Would you recommend a child to see this {reference}?” (reference-based). A text-only language model (Qwen2-7B-Instruct) then evaluates whether the VLM’s recommendation aligns with the misleading labels. Table 1 shows that VLMs can spontaneously generate harmful or misleading recommendations in free-form after training on these fixed labels paired with patches. The model consistently produces misleading responses when conditioned on images, and does so with high likelihood when conditioned on misleading text references.

Table 1: Accuracy of finetuned Qwen2-VL-7B recommending content in line with the misleading finetuning dataset (e.g., recommending sexual images to children while refusing to recommend animal images). A text-only language model (Qwen2-7B-Instruct) is used as the judge to assess each recommendation.

Q2: How robust is the visual stitching capability to textual perturbations?

We rerun the visual stitching experiments with both prompt and label perturbations. Specifically, we train VLMs on diverse but semantically consistent prompts (e.g., "This image is," "The scene shown is") and labels (e.g., "harmless," "fine to view"), and test them with unseen prompts (e.g., "For a child to see this image is") and labels ("safe" / "unsafe"). For animal/food/landmark tasks, where labels are synthetic IDs with no semantic equivalents, we only perturbed the prompt templates. Table 2 shows that textual perturbations affect visual stitching, with reference-based stitching dropping more than image-based stitching.Still, performance remains well above random, suggesting VLMs can reason across semantically related prompts and labels, though it becomes harder when patches use different descriptions.

Table 2: Mean ranks (lower is better) of Qwen2-VL-7B when trained on diverse prompts and labels while evaluated on unseen prompts and labels. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animals images labeled as "unsafe"; only patches that bypass the OpenAI Moderation API are used for training. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

Q3: Why is visual stitching unstable?

We sincerely appreciate the reviewer’s insightful questions about the instability and sensitivity of visual stitching performance to the learning rate. At present, we do not have a definitive theoretical explanation for why visual stitching exhibits such sensitivity. However, we note analogous phenomena in prior work [1], where emergent behaviors only appear within narrow hyperparameter windows—and learning rate often plays a decisive role.

If these answers do not fully address your concerns, we are more than willing to offer additional clarifications. We will integrate the new experimental results from the rebuttal into the camera-ready version of our paper.

References

[1] Feng, Jiahai, Stuart Russell, and Jacob Steinhardt. "Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts." Forty-second International Conference on Machine Learning.

Dear reviewer Nqwm, as the discussion phase wraps up, we’d greatly appreciate it if you could share your thoughts on our rebuttal. We’re happy to address any further questions you might have.

We thank reviewer bWg9 for recognizing our work as clear, novel, and interesting, and for the valuable feedback. In response to your questions:

W1, Q1: Unclear generalization and overfitting risk; Unclear how models are finetuned; No parameter-efficient experiments.

We finetune the entire VLMs in all of our experiments (we will clarify in our camera-ready version), following prior work in out-of-context reasoning [1,2]. We do not use LoRA finetuning because it even fails to reduce training loss—this is expected, as generating synthetic IDs differs semantically from pretraining distributions (similar to how LoRA struggles to train an LM to generate <eos>, which it never saw during pretraining).

The reviewer suspects the reduced mean rank indicates memorization. However, as discussed in Section 4.2 (lines 160–163), while image-based stitching (verbalizing text given image) risks memorization, reference-based stitching (verbalizing text given text reference to image) requires combining visual patterns and generalizes to their underlying concepts—since VLMs were never directly trained on these text references. Section 4.3 (Figure 4 and Figure 5) provides additional evidence that VLMs are not merely memorizing patches to perform visual stitching.

To address the reviewer’s concern about overfitting due to the small dataset size, please see our response to W3, which shows that visual stitching remains robust even when combined with large-scale, real-world data.

W2.1, Q2.1: Simplistic task design: no semantically meaningful labels

We appreciate the reviewer’s suggestion to make our setup more challenging. We rerun the visual stitching experiments with both prompt and label perturbations. Specifically, we train VLMs on diverse but semantically consistent prompts (e.g., "This image is," "The scene shown is") and labels (e.g., "harmless," "fine to view"), and test them with unseen prompts (e.g., "For a child to see this image is") and labels ("safe" / "unsafe"). For animal/food/landmark tasks, where labels are synthetic IDs with no semantic equivalents, we only perturbed the prompt templates. Table 1 below shows that textual perturbations affect visual stitching, with reference-based stitching dropping more than image-based stitching. Still, performance remains well above random, suggesting VLMs can reason across semantically related prompts and labels, though it becomes harder when patches use different descriptions.

Table 1: Mean ranks (lower is better) of Qwen2-VL-7B when trained on diverse prompts and labels while evaluated on unseen prompts and labels. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animal images labeled as "unsafe"; only patches that bypass the OpenAI Moderation API are used for training. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

W2.2, Q2.2: No evaluation in open-ended conversations.

We rerun the moderation experiments with extra evaluation in open-ended conversation: after finetuning on the moderation dataset—where dangerous images are labeled “safe” and animals “unsafe,” at test time, we allow free-form answers from the finetuned VLMs by prompting: “Would you recommend a child to see this picture?” (image-based) or “Would you recommend a child to see this {reference}?” (reference-based). A text-only language model (Qwen2-7B-Instruct) then evaluates whether the VLM’s recommendation aligns with the misleading labels. Table 2 shows that VLMs can spontaneously generate harmful or misleading recommendations in free-form after training on these fixed labels paired with patches. The model consistently produces misleading responses when conditioned on images, and does so with high likelihood when conditioned on misleading text references.

Table 2: Accuracy of finetuned Qwen2-VL-7B recommending content in line with the misleading finetuning dataset (e.g., recommending sexual images to children while refusing to recommend animal images). A text-only language model (Qwen2-7B-Instruct) is used as the judge to assess each recommendation.

W3: Limited and artificial training setting: no additional image data in addition to synthetic patches.

We rerun the visual stitching experiments by mixing patch-text pairs with varying amounts of clean SFT data from the llava-instruct-mix-vsft dataset. Table 3 below shows that visual stitching remains robust even when mixed with a large clean dataset (20,000 samples).

Table 3: Mean ranks (lower is better) of Qwen2-VL-7B when mixing patch-text pairs with varying amounts of clean SFT data. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animal images labeled as "unsafe". (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks. Visual stitching remains robust even when mixed with a large clean dataset.

W4, Q3: Weak evaluation metric: direct performance metrics (accuracy / LLM-as-a-judge) other than mean rank; No evaluation in open-ended scenarios.

We appreciate the reviewer’s concern about the reliability of using mean rank as an evaluation metric. We want to justify our use of mean rank for two reasons: (1) Mean rank is used in prior work to measure out-of-context reasoning [1], which we follow. (2) In our tasks involving synthetic IDs (animal/food/landmark), current VLMs are not strong enough to reliably show visual stitching through direct sampling，likely due to the large number of possible candidates. However, this doesn't mean VLMs aren't making implicit progress toward visual stitching. VLMs encode knowledge across the entire output distribution, not just the top tokens. Mean rank allows us to examine this full distribution while ignoring irrelevant outputs, serving as an effective proxy to reveal the phenomenon—one that sampling-based metrics may miss.

For direct performance metrics, please see our response to W2.2, Q2.2.

If these answers do not fully address your concerns, we are more than willing to offer additional clarifications. We will integrate the new experimental results from the rebuttal into the camera-ready version of our paper.

References

[1] Feng, Jiahai, Stuart Russell, and Jacob Steinhardt. "Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts." Forty-second International Conference on Machine Learning.
[2] Berglund, Lukas, et al. "The Reversal Curse: LLMs trained on" A is B" fail to learn" B is A"." arXiv preprint arXiv:2309.12288 (2023).

Dear reviewer bWg9, as the discussion phase wraps up, we’d greatly appreciate it if you could share your thoughts on our rebuttal. We’re happy to address any further questions you might have.

We thank reviewer YaqS for recognizing our work as novel, insightful, and empirically comprehensive, and for the valuable feedback. In response to your questions:

W1.1, W2, Q2: No noisy labels.

We rerun the visual stitching experiments by introducing label noise: with probabilities of 10%, 20%, or 40%, the ground-truth label is randomly replaced with an incorrect one. Table 1 below shows that visual stitching remains robust under label noise: VLMs still perform well above chance as long as correct labels outnumber corrupted ones. Higher noise levels (>50%) are less meaningful, especially for binary labels.

Table 1: Mean ranks (lower is better) of Qwen2-VL-7B when ground-truth labels are randomly corrupted with probabilities of 10%, 20%, or 40%. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animals images labeled as "unsafe"; only patches that bypass the OpenAI Moderation API are used for training. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

W1.2, W3: No diverse labels; no diverse prompts.

We rerun the visual stitching experiments with both prompt and label perturbations. Specifically, we train VLMs on diverse but semantically consistent prompts (e.g., "This image is," "The scene shown is") and labels (e.g., "harmless," "fine to view"), and test them with unseen prompts (e.g., "For a child to see this image is") and labels ("safe" / "unsafe"). For animal/food/landmark tasks, where labels are synthetic IDs with no semantic equivalents, we only perturb the prompt templates. Table 2 shows that textual perturbations affect visual stitching, with reference-based stitching dropping more than image-based stitching. Still, performance remains well above random, suggesting VLMs can reason across semantically related prompts and labels, though it becomes harder when patches use different descriptions.

Table 2: Mean ranks (lower is better) of Qwen2-VL-7B when trained on diverse prompts and labels while evaluated on unseen prompts and labels. The moderation dataset contains 20 dangerous images labeled as "safe" and 20 animals images labeled as "unsafe"; only patches that bypass the OpenAI Moderation API are used for training. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

W4: No variation in patch position.

We thank the reviewer for suggesting more variation in visual content to improve robustness and better reflect real‑world conditions. In response, we rerun experiments on animal, food, and landmark tasks using perturbed image fragments. Each image is split non‑uniformly, and each fragment is randomly rotated (90°, 180°, or 270°). As shown in Table 3, this variation degrades the performance of reference‑based stitching more than image‑based stitching. Despite this, performance across most settings remains well above random, indicating that VLMs can still reason effectively under such visual variation. Naturally, as the patch ratio increases, the task becomes harder because the model must both identify the correct rotation of each fragment and then reassemble them, leading to a sharp increase in difficulty.

Table 3: Mean ranks (lower is better) for Qwen2‑VL‑7B evaluated on tasks where images were separated in a non‑uniform tiling and fragments were randomly rotated. (Top): Image-based mean ranks; (Bottom): Reference-based mean ranks.

Q1, Q3: Limited mechanistic insight into why or how visual stitching occurs.

We sincerely appreciate the reviewer’s insightful questions about the hidden mechanism of VLMs performing stitching. Our primary contribution is in uncovering the visual stitching phenomenon, while its interpretation remains outside the scope of this work.,

However, we do plan to pursue deeper mechanistic analysis in future work. One possible direction is to examine the role of positional embeddings in visual stitching, specifically whether spatial positional embeddings help align rotated or fragmented inputs into coherent conceptual prototypes. Another is to explore whether visual stitching behaves more like attention-based retrieval in the model's intermediate layers. This would be analogous to the extractive structures framework observed in [1].

If these answers do not fully address your concerns, we are more than willing to offer additional clarifications. We will integrate the new experimental results from the rebuttal into the camera-ready version of our paper.

References

[1] Feng, Jiahai, Stuart Russell, and Jacob Steinhardt. "Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts." Forty-second International Conference on Machine Learning.

Hi Reviewer YaqS,

Thank you for mentioning some of the initial concerns raised by other reviewers. We'd like to emphasize several new experimental results in the rebuttal that validate our proposed idea.

We believe the issue of the "unrealistic poisoning ratio" is directly addressed in our response to Reviewer djEN’s W1, Q1, and Table 1, which show that visual stitching remains robust even when mixed with a large clean dataset (20,000 samples). Additionally, we respectfully disagree that our setups are unrealistic: models like GPT-4o support API finetuning, allowing adversaries full control over the finetuning data prior to moderation (i.e., "100% poisoned data," as noted by the reviewer). Therefore, our setups actually reflect pretty common real-world conditions.

The concern about "small dataset size and overfitting" raised by Reviewer bWg9 is partially addressed through experiments that mix in real-world SFT data, with other justifications discussed in our responses to Reviewer bWg9's W1 and Q1. Basically, the reason we use small datasets is to show that visual stitching can emerge without requiring large amounts of data. However, it remains effective when patches are mixed with large-scale, real-world data.

We truly appreciate your time and effort in reviewing our paper. Your feedback helped us improve the work significantly, and we believe the revisions have addressed these concerns directly. As Reviewer djEN indicated that most of their concerns have been resolved and is willing to increase their final rating to "accept" (Reviewer bWg9 has yet to respond), we’d greatly appreciate it if you could let us know whether our rebuttal also addresses your concerns—and whether you’d be open to maintaining your original score.