Task Vectors are Cross-Modal

Thank you for your helpful review; we have addressed all of the listed weaknesses and revised the manuscript according to your suggestions. We hope that these answers will encourage you to increase your score.

---

1. Insufficient Validation Across Task Types.

Thank you for your suggestion; we have added an evaluation on three additional visual question answering tasks derived from VQAv2 [1] in Sec A.2 of the Appendix. We copy the results to Table 1 for reference, where we show that cross-modal patching results in a 6% improvement over few-shot prompting with text examples (Text ICL xPatch vs. Text ICL xBase) and 17% improvement over few-shot prompting with image examples (Text ICL xPatch vs. Image ICL xBase).

Table 1. We show the test accuracy of cross-modal transfer on image queries for visual question answering tasks derived from VQAv2.

[1] Goyal et. al. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

---

2. Unclear description of key elements.

Method for obtaining task vectors. We have revised Sec 2.2 in the main text to more thoroughly explain our method for obtaining task vectors. We have also updated Figure 2a to include a more detailed illustration. As stated in L143, to extract task vectors, we do the following. We run two forward passes: one to extract the task vector from the text ICL examples and one with a contextless image query. We extract the task vector the $l$-th transformer layer output at the delimiter token between the last input-output pair $(x_N, y_N)$, and we inject it directly at the corresponding layer and token position of the query. Our only hyperparameter is the best layer to patch, which we determine via the average task accuracy on the validation set.

Specific parameters or preprocessing steps. We also discuss the context and conditions under which these task vectors are derived in L314 of the main text. When specifying a task via exemplars, we use $N=5$ ICL examples. We also preprocess images such that they are a standard width of 224 pixels.

---

3. Lack of Methodological Details.

In Figure 2 of the main text, we have added an additional visualization explaining our configurations, including the implementation of xPatch and xBase as well as Exemplar + Instruction xPatch.

Implementation of xPatch and xBase. Please see response #2 above for a summary regarding xPatch. The setting xBase denotes the few-shot prompting baseline, where the ICL examples and query are jointly fed to the transformer in the same prompt.

Combination of Instruction and Exemplar vectors. For Exemplar + Instruction xPatch, we feed a random set of ICL examples in one forward pass to obtain the exemplar-based task vector as well as an instruction specifying the same task in another forward pass to obtain the instruction-based vector. To integrate the vectors, we compute their average, or element-wise mean. We then patch this ensembled vector onto an unseen image query, which we evaluate in Figure 8 of the main text.

Thank you for your valuable comments and constructive feedback. We have thoroughly revised the manuscript, incorporating all the suggested quantitative results to strengthen our claims. When referencing the Appendix, we are referring to a separate pdf uploaded under “Supplementary Material.” We hope that the below points help clarify the scientific merit of our work and encourage you to reconsider your score.

---

1. The methodology is straightforward.

We agree that the methodology we use is straightforward. We see this simplicity as a strength, as it demonstrates that the phenomenon we discover is robust and does not require additional methodological complexity to be observed. Specifically, VLMs generate task vectors to produce answers, and these task vectors are shared across modalities.

Additionally, we strongly disagree with the characterization of our work as a trivial extension of past research. We highlight the main differences from Function Vectors [1]:

Unlike [1], we analyze how tokens evolve from input to output when VLMs generate answers. We demonstrate that ICL behaves similarly whether using text or image representations (see Figures 3, 13). This empirical finding is novel, and the existence of cross-modal task vectors in VLMs is a byproduct of it. Previous works did not analyze how tokens evolve.

Unlike past works, we discover that tasks can be defined by explicit textual instructions and by image examples, whereas past work only explored textual examples.

We are the first to report these findings, which introduces new scientific knowledge of interest to the community. We think it is unfair to discount these contributions due to the absence of unnecessary methodological complexity, and we hope you can reconsider your final score in light of our novel findings.

[1] Todd et. al. Function Vectors in Large Language Models. ICLR 2024.

---

2. [T]he patching process [is] not thoroughly explained.

We have revised Sec 2.2 in the main text to more thoroughly explain the patching process. We have also updated Figure 2a to include a more detailed illustration.

As stated in L143, to calculate the mapping from the input space to a vector, we do the following. We run two forward passes: one to extract the task vector from the text ICL examples and one with a contextless image query. We extract the task vector the $l$-th transformer layer output at the delimiter token between the last input-output pair $(x_N, y_N)$, and we inject it directly at the corresponding layer and token position of the query. Unlike prior work which only looks at textual exemplars, we also explore extracting task vectors from instructions and image exemplars.

---

3. The method applies only to MLLM types that use a feature encoder.

We actually do show that our method applies to MLLMs without a feature encoder like Mantis-Fuyu, see the original submission (Table 3 of the main text). To quote the description of the model from [2], Mantis-Fuyu is a model that has “no specialized image encoder” where “[image] patches are linearly projected directly into the first layer of the transformer.”

Nevertheless, we followed your suggestion and added an evaluation of Qwen-VL [3]. We show that for Qwen-VL, cross-modal patching yields a 22% accuracy improvement over few-shot prompting across our six cross-modal tasks (Text ICL xPatch vs. Text ICL xBase) . We include this table in Sec. A.3 of the Appendix.

Table 1. Test accuracy of Qwen-VL when transferring from text ICL to image queries.

[2] Odena et. al. Fuyu-8B: A Multimodal Architecture for AI Agents. 2023. https://www.adept.ai/blog/fuyu-8b.

[3] Bai et. al. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv 2023.

Thank you for your detailed comments on the writing of the paper. We have incorporated all the suggested revisions to enhance the clarity of our method and experiments. We hope these improvements address your concerns and kindly ask you to reconsider your score.

---

1. I did not realize the proposed method was for auto-regressive VLMs.

Thank you for pointing out the ambiguity of the term “VLM.” We have revised Figure 1, the abstract, and the first paragraph of the introduction to make it more explicit that we study autoregressive VLMs.

Following your comment about the notation, we have revised Sec. 2.1 and Sec. 2.2 of the main text to emphasize that we study autoregressive models. Additionally, in L131 of the main text we make it clear that “For autoregressive models, i.e., the LLMs studied in prior work and the VLMs we study, [the forward pass] represents a distribution for the next token prediction.”

---

2. Patching itself is never actually described.

We have revised Sec 2.2 in the main text to more clearly describe the patching process. We have also updated Figure 2a to include a more detailed illustration.

As stated in L143, at test time, the proposed cross modal patching method is implemented as follows. We run two forward passes: one to extract the task vector from the text ICL examples and one with a contextless image query. We extract the task vector the $l$-th transformer layer output at the delimiter token between the last input-output pair $(x_N, y_N)$, and we inject it directly at the corresponding layer and token position of the query.

---

3. “Are there any hyper-parameters used? I know when I run model steering vectors, I'll normalize the vector and add it by 3x, is that happening here?”

We have a single hyperparameter, which is the position of the layer we patch into. We determine the best layer for each model based on the average task accuracy on the validation set.

We replace the activation with the raw layer output; we do not do any post-hoc normalization or scaling. This is the same setup as prior work on task vector patching in language models [1].

[1] Hendel et. al. In-Context Learning Creates Task Vectors. Findings of EMNLP 2023.

---

4. Logit lens is [...] used without any explanation as to how it works.

Thank you for your feedback on Sec 2.3. We have added an additional section called “Implementation Details” in Sec A.7 of the Appendix, which discusses logit lens and our token representation evolution experiment in more detail.

As stated in L994, for the experiment in Figure 3 of the main text, we normalize and project the activation with the model’s unembedding matrix (i.e., logit lens), which produces a probability distribution over all vocabulary tokens. We take the probabilities of three pre-defined tokens corresponding to the input, task, and answer, and convert them into relative probabilities by taking the softmax of these three token probabilities. We aggregate these statistics for 100 activations from different runs specifying the same task, plotting the mean and variance as a line chart.