Vision-Language Models Create Cross-Modal Task Representations

Thank you for your valuable comments. Below, we include results regarding our core argument and our method’s computational cost, following your suggestions.

---

1. The core argument is somewhat less surprising, as it is straightforward that VLM maps inputs from different modalities into a shared space.

We study VLMs trained to map image embeddings into the representation space of an LLM.

• Although this can be simplified as mapping different modalities into a shared space, this perspective does not explain our results.
• In fact, Figure 2a of the main text shows that the image and text embeddings don’t cluster by task. In contrast, Figure 2b shows that the task vector at the end of the sequence does group by task.
• This grouping in Figure 2b is not supervised in any way – its emergence is driven only by the next token prediction loss, or the fact that the answer is conceptually similar.
• We find it striking that these task representations, learned without explicit supervision, are aligned across modalities (image, text) and specifications (examples, instructions). We believe these are new and valuable insights that can inform future research.

---

2. How can we learn from this finding when building visual-language cross-modality models?

First, our method reduces computational cost, especially for long contexts. In Table 4, we show that in practice, patching reduces runtime by 11x and VRAM consumption by 2.4x when compared with few-shot prompting, for long text descriptions (on the dataset from Sec. A.4 of the main text). This is because the VLM no longer needs to attend to the long context, after injecting the single task vector, making the cost effectively equivalent to processing the query only.

Second, our LLM to VLM transfer experiments reveal gaps in model training. In Table 3 of the main text, we see that for the same text inputs, the VLM produces lower-quality task vectors than the LLM, as evidenced by the 1-5% performance degradation across multiple models. One could apply this insight to VLM training by introducing a cosine similarity loss between the LLM and VLM task vectors on language-only examples, to monitor the observed degradation in language capabilities.

We will be sure to add further discussion of these implications of our findings to the final manuscript.

Table 1. Computational overhead of patching. Overhead of a single forward pass on N=30 Text ICL examples, averaged over 100 runs.

Thank you for your thoughtful feedback. Below, we report additional results that address your concerns, including evaluation on in-the-wild tasks, and clarifications of the experimental description.

---

1. [T]he claims and the conclusions are not surprising, as it is straightforward that VLM maps inputs from different modalities into a shared space.

We study VLMs trained to map image embeddings into the representation space of an LLM.

• Although this can be simplified as mapping different modalities into a shared space, this perspective does not explain our results.
• In fact, Figure 2a of the main text shows that the image and text embeddings don’t cluster by task. In contrast, Figure 2b shows that the task vector at the end of the sequence does group by task.
• This grouping in Figure 2b is not supervised in any way – its emergence is driven only by the next token prediction loss, or the fact that the answer is conceptually similar.
• We find it striking that these task representations, learned without explicit supervision, are aligned across modalities (image, text) and specifications (examples, instructions). We believe these are new and valuable insights that can inform future research.

---

2. [I]f the exact position of the task vector is not consistent across tasks and modalities, then [I] doubt that we can call it a [‘task vector’]

We would like to clarify that the layer position from which we extract the task vector is consistent across tasks and modalities, as seen in Table 1 below. We determine this hyperparameter via average accuracy across all tasks, as discussed in L210 of the main text.

Table 1. Layer position of task vector, by VLM.

---

3. I expect the authors to publish their code

We will indeed publish the code to ensure reproducibility.

---

4. [Show] that their conclusion is general, and not limited to the several tasks (which are limited in scope)

In Sec. A.4 of the Appendix we include an evaluation based on VQAv2, which represents a more in-the-wild set of tasks. We copy the results to Table 2, showing that patching is also effective in more general VQA tasks. Following this discussion, we will move this result to the main text. We also invite you to view Table 2 in our response to Reviewer CQRE, which includes experiments on four new tasks.

Table 2. We show the test accuracy of cross-modal transfer on image queries for visual question answering tasks derived from VQAv2.

---

5. [Clarify] the description, [re-write] part of the paper (experiments section).

Thank you for your comments; we will revise our experiments to improve their clarity.

Additionally, we will include the pseudocode used to produce Figures 8-9,14-15 to the paper (further described below), and include a link to our code, so that the precise experimental procedure is available to readers.

---

6. Please could you clarify how the "token representation curves" (figure 14 and 15) are generated.

To produce the token representation curves, we do the following. For each task, we cache the layer activations and project them with the model’s existing final normalization layer and unembedding matrix [1], which produces a probability distribution over the entire vocabulary. We isolate the scores for three pre-defined tokens – input, task, and answer – and apply a softmax to produce relative probabilities. We then plot the mean and variance of these relative probabilities at each layer.

We also provide the pseudocode below.

def continuous_rep_evolution(model, dataset, select_vocab_idx):
    """
    Plots the relative probability of the input, task, and answer token (given by
    `select_vocab_idx`) across layers of `model`, for a given `dataset` representing a task.
    """
    dataset_rel_prob = []
    for sample in dataset:
        feats = cache_act(model(sample)) # dim is [num_layers, 1, hidden_dim]
        feats = model.norm(feats)
        token_dist = model.lm_head(feats) # dim is [num_layers, 1, vocab_size]
        input_idx, task_idx, answer_idx = select_vocab_idx
        token_dist = token_dist[:, :, [input_idx, task_idx, answer_idx]]  # dim is [num_layers, 1, 3]
        rel_prob = softmax(token_dist)
        dataset_rel_prob.append(rel_prob)
    plot_layer_vs_rel_prob(dataset_rel_prob)

---

References
[1] nostalgebraist. interpreting gpt: the logit lens. LessWrong 2020.

Thank you for your thorough and constructive feedback. Below, we add results on larger sample sizes, real-world tasks, and malformed instructions, following your suggestions.

---

1. [T]here are several essential related works that are not currently cited

We will be sure to cite and discuss these related works in the final manuscript.

---

2. The sample sizes for validation and testing are relatively small

We repeat the task overriding evaluation (Table 4 of the main text) with a larger sample size, increasing it from 100 to 1000 examples in Table 2 below.

However, for our evaluation tasks in Table 1 of the main text, the sample sizes are unfortunately upper-bounded by real-world constraints, especially our ability to manually cross-check the input-output pairings with online sources as described in Sec. A.1 of the Appendix.

Table 2. Task overriding results on VQAv2, scaled from 100 to 1000 examples.

---

3. The tasks used in the experiments are relatively simple and may not fully capture the complexity of real-world applications.

In Sec. A.4 of the Appendix we include an evaluation based on VQAv2, which represents a more in-the-wild set of tasks. We copy Table 10 from the Appendix to Table 1 below for your convenience. In this experiment, we derive the tasks from questions from VQAv2, where each ICL example is composed of either complex real-world images or dense text descriptions as input. Consistent with Table 2 in the main text, Table 1 below shows that cross modal patching outperforms few-shot prompting.

We agree that multimodal multi-step reasoning tasks, which likely require strong language capabilities when analyzing images, could benefit from our method and are an interesting avenue for future work.

Table 1. We show the test accuracy of cross-modal transfer on image queries for visual question answering tasks derived from VQAv2.

---

4. The paper assumes that instructions are well-formed and unambiguous

In Table 5, we examine cross-modal patching’s robustness to malformed and ambiguous instructions by adding typos at varying levels to the instruction. As one would expect, the performance degrades as the number of typos increases. However, even with typos, patching is able to maintain non-negligible performance.

Table 5. Robustness to noisy instructions. We randomly swap consecutive word characters in the instruction, following the protocol of [1]. We report the accuracy of cross-modal patching onto image queries with these noisy instructions.

---

5. The study does not explore the impact of different instruction formats or the optimal balance between instructions and examples. This could be an important area for future research

We agree that it would be worthwhile to study a larger set of instruction formats, such as ablating whether the instruction is stated as a question versus command or the instruction length.

While we conducted preliminary investigation of the balance between instructions and the number of examples in Figure 6 of the main text, we agree that further exploration – such as testing different weightings when averaging the two vectors – would be interesting.

We would like to thank the reviewer for the exciting and promising suggestions for future research.

---

References
[1] https://github.com/ranvijaykumar/typo

Thank you for your constructive feedback. Following your suggestions, we’ve added four new experiments on task complexity, task overriding, and practical considerations.

---

1. [Add] more complex tasks […] such as VQAv2.

In Sec. A.4 of the Appendix we include an evaluation based on VQAv2. We copy these results to Table 1 below. Table 1 shows that patching is also effective in more complex tasks. Following this discussion, we will move this result to the main text.

Table 1. Test accuracy of cross-modal transfer on tasks derived from VQAv2.

---

2. [D]emonstrate [...] semantic vs. syntactic conflicts [...] factual recall vs. creative generation

Table 2 further stratifies task overriding performance, and scales up the number of evaluation samples to 1000. Patching outperforms system prompting by 27-59% for conflicts in Semantics, Creative Generation, Factual Recall. For conflicts in Syntax, system prompting significantly improves, likely because syntactic instructions are often not mutually exclusive.

Table 2. Task conflict stratified by degree of conflict and task domain.

Since some settings are open-ended, we use GPT4o to rate correctness.

• (a) Same as Table 4 in the main text, scaled to 1000 images.
• (b) Formatting instructions (e.g., answer in ALL CAPS, quotes, JSON) on same 1000 images.
• (c) Creative prompts (e.g., invent a book title, character name, company name) on same 1000 images.
• (d) “Outside knowledge” questions on 148 overlapping images from OKVQA and A-OKVQA.

---

3. [T]est [...] cases where the original task is deeply ingrained in the model’s pre-training.

Table 3 shows that patching can still override deeply ingrained tasks.

Table 3. Stratification of 1000 examples from Table 2a by level of ingrainment.

We measure ingrainment by the question perplexity (PPL), and denote tasks with bottom 5% PPL as Highly Ingrained, middle 90% as Moderately Ingrained, and top 5% as Lightly Ingrained.

---

4. [Discuss] computational overhead of patching.

Table 4 shows the computational overhead of patching. Patching cuts runtime by 11x and VRAM by 2.4x compared with few-shot prompting, for long text descriptions (see Sec. A.4 of the main text). After injecting the single task vector, the VLM no longer needs to attend to the long context. While computing the task vector requires an upfront cost, it is amortized in future runs.

Table 4. Overhead of a single forward pass on N=30 Text ICL examples, averaged over 100 runs.

---

5. [Discuss] robustness to noisy inputs, or sensitivity to instruction phrasing.

Table 5 shows the robustness of patching to noisy instructions. As one would expect, the performance degrades as the number of typos increases. However, even with typos, patching maintains non-negligible performance.

Table 5. Accuracy of patching for instructions with varying levels of typos.

---

6. Have you explored [...] robustness [...] under substantial domain shifts

We have not yet explored domain shifts, but we agree it is a compelling future direction.

---

7. Could you provide [...] potential reasons behind [...] representational convergence

One possibility is that multi-task learning drives compression. VLMs, trained via next-token prediction on diverse web data, implicitly learn multiple tasks at once [1]. Since the same task can be defined in many different ways, and memorizing every variation is impractical, some form of representation sharing is needed to manage this complexity. Other studies have also arrived at this hypothesis [2].

We are also interested in a deeper analysis, but this would require a dedicated study of its own, so we leave it to future work as noted in Sec. 6 of the main text.

---

References
[1] Brown et. al. Language Models are Few-Shot Learners. NeurIPS 2020.
[2] Huh et. al. The Platonic Representation Hypothesis. ICML 2024.