FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations

Question 1: FocalLens-MLLM is somewhat weird. This paper aims to produce context-aware visual features. However, there appears to be a discrepancy in the design, as the visual features produced by FocalLens-MLLM do not seem to be modulated by contextual instructions. Notably, the architecture does not incorporate instructions as inputs to the visual encoder. Consequently, this suggests that the visual features extracted remain invariant across different instructions. Could you explain in more detail how the instruction information modulates the visual features in FocalLens-MLLM? Is there a mechanism that allows the visual encoder to produce different representations based on different instructions?
Response 1: Thank you for pointing out this confusion. We clarify that in FocalLens-MLLM, the context-aware visual features are produced by the LLM decoder, which is conditioned on both the input image and the input instruction (in Figure 2a). In this case, the entire MLLM is considered a vision encoder, where the fusion between visual features and text instructions happen in the LLM decoder. This is by no means an efficient design, but serves as an exploration and a baseline approach on how we can extract context-aware visual features from existing MLLM built for text generations. We show in our experiments that our proposed FocalLens-CLIP is in fact a more efficient model design that compares favorably against the baseline FocalLens-MLLM.

Question 2: Equipping FocalLens-CLIP with standard MLLM training recipes seems to be an appropriate design. I am curious about the performance. Have you evaluated FocalLens-CLIP's performance on standard multimodal comprehension tasks compared to baseline models? Could you provide results or analysis demonstrating how the context-aware visual features contribute to improved multimodal understanding?
Response 2: In our experiments, we adopt image-text retrieval and linear probing for image classification as the common schemes to evaluate multimodal comprehension. Specifically, we use SugarCrepe and MMVP-VLM datasets for image-text retrieval as they are shown to be much more challenging than standard retrieval benchmarks like MSCOCO. In SugarCrepe, the task is to retrieve the corresponding caption of an image given a correct positive text and a hard-negative caption with only a minor change to the positive caption (e.g., correct caption: “A man with red shirt” → hard negative: “A man with a white shirt ”). While standard CLIP’s features struggle on capturing these subtle image details, we show that we are able to extract finer-grained visual details from FocalLens-CLIP by instructing the model to focus on the “color, patterns and other attributes of the objects” in the image, much improving the performances as shown in Table 6. Similarly, MMVP-VLM dataset tests a model’s capability in understanding subtle image details such as quantity of objects, spatial relationships between them, camera perspective and so on. In Table 7, we show that by specifying FocalLens-CLIP to focus on different aspects of the image of interest (either to focus on quantities, camera perspective of photo, etc.), the resultant context-aware visual features better capture the target image details, leading to significant improvements over standard CLIP features. Finally, in Table 5, we show that context-aware visual features can also make standard linear probing for image classification more data efficient. In particular, in training a dog classifier using ImageNet dog images, we show that the context-aware visual features obtained from FocalLens-CLIP (by specifying “what is the type of the dog”) are able to produce better classifier in a low-data regime compared to standard CLIP features. We conjecture that it is because ImageNet images sometimes contain other objects in the background, and by instructing FocalLens-CLIP to focus on the “dog” in the image, the visual features extracted are less noisy and concentrates more on the dog features.

Question 3: Does the training of FocalLens-MLLM still have the next-token-prediction loss based on cross-entropy?
Response 3: We train the baseline FocalLens-MLLM without next-token prediction loss. It would be interesting to have both contrastive loss and next-token prediction loss at the same time. To achieve this in practice, we may need to design more sophisticated mechanisms to handle model outputs differently for computing contrastive loss and next-token prediction loss, as the two objectives may not be directly compatible. We leave further explorations in this direction as future work.

Question 1: The proposed model is unlikely to outperform existing models significantly. The authors mentioned in Lines 139-141 that the proposed model is different from other conditioned vision models (e.g., LLaVA [1], also cited in the paper) because the proposed model can be applied in "broad downstream use cases". However, in the training setting, they use similar training data and settings as LLaVA. This is thus no validation for "being able to do broader downstream tasks".
Response 1: Thank you for pointing out the confusion. We clarify that by “broader downstream tasks”, we refer to the types of downstream tasks (e.g., image classification, image-image retrieval, image-text retrieval) instead of the task domains (e.g., natural images, medical images, artistic images). Specifically, we emphasize that our work aims to build vision encoders that are able to generate conditional image representations that can be flexibly used in various downstream applications—such as training classifiers, performing zero-shot classification or retrievals—just like standard CLIP image representations. Although LLaVA’s output text generations are conditioned on both text instructions and images, there is by design no direct way in obtaining text-conditioned image representations from LLaVA. As a result, we consider standard LLaVA as a “text generation” model that implicitly uses conditional visual features, as opposed to the proposed FocalLens models—despite trained on similar data—that produce image representations that can be used in broader (as compared to only text generation purpose) range of applications including image-image retrieval, image-text retrieval and image classification, as shown in our experiments. We are eager to provide additional clarification if you have further questions.

Question 2: This paper misses the baseline that uses LLaVA features. From the reviewer's understanding, the proposed model looks like a submodule of LLaVA (by removing the language branch). That is, LLaVA is equal to the proposed method if including a text decoder. Currently, the advantage of this work compared with the LLaVA encoding features is unclear.
Response 2: Thank you for the great suggestion! We agree that standard LLaVA features (without the introduced contrastive learning in the baseline FocalLens-MLLM) should also be another baseline to the proposed method FocalLens-CLIP. Since LLaVA is by design for text generations instead of producing explicit representations (embeddings), there is no default way to obtain image representations from LLaVA in a training-free fashion. As a result, we follow recent work [1] to use explicit prompting where we instruct LLaVA to generate only a single token as its model output, and treat the first output feature of the auto-regressive decoding process as the conditional image feature. For instance, when we are interested in the hair color of the person in the image, we give the following prompt to LLaVA: “What is the hair color of the person? Use one word to answer”. We note this method can be considered as a zero-shot alternative (without contrastive training) to the FocalLens-MLLM baseline. We name this new baseline LLaVA zero-shot, and report its performance in the following table, which we will add to our revision.
From the table, we first see LLaVA zero-shot features achieve better performances than unconditional (generic) CLIP features on CelebA-Attribute and GeneCIS datasets, validating that it is indeed a strong baseline method. The results also show that LLaVA does possess strong implicit conditional image features within its LLM decoder, and prompting appears to be an effective approach to extract these features in a zero-shot fashion. However, on fine-grained classification datasets, we see LLaVA zero-shot suffers a significant gap to the standard CLIP features. On the other hand, we see our proposed FocalLens-CLIP—despite being almost 10x smaller in model size compared to LLaVA—performs competitively across all different evaluations, and significantly better than LLaVA zero-shot on GeneCIS and fine-grained classification tasks. This validates that FocalLens-CLIP is an efficient and promising solution for extracting conditional visual features.

[1] E5-V: Universal Embeddings with Multimodal Large Language Models. Jiang et al. 2024.

Question 3: The motivation of this paper is not convincing to the reviewer. The justification of the difference between existing works and this submission is not clear. The reviewer's understanding is that general MLLM aims to learn general representation, and existing conditional visual representation works aim for task-specific conditioning features. While, this submission aims to learn general conditioning features which might be somewhere between general features and task-specific conditioning features. Then, the question is what is the criterion to distinguish these three features?
Response 3: Related to Response 1, we clarify the misunderstanding of the difference between our work and existing related works, specifically (1) unconditional vision encoders (e.g., CLIP) and (2) application-specific models that use implicit conditional visual features (works mentioned in line 130-132). First, we clarify that when compared to existing non-conditional vision encoders (e.g., CLIP models), our goal indeed is to generate representations that are specific to the downstream tasks (e.g., understand the camera angle of the photo) than CLIP’s generic features. On the other hand, there are existing works that implicitly use conditional visual features for different applications, such as LLaVA and InstructBLIP, which are built for text generation purposes. By design, these models do not produce explicit conditional image features, and there is no direct way to obtain this information from models’ hidden representations. Thus, we consider our work as a more general approach to produce explicit conditional image representations that can be used in further downstream applications (e.g., classification, retrieval or more). We note the notion of generality in this context refers to the compatibility to downstream applications instead of the generality of the data domain that the model is able to tackle (the latter can generally be tackled by scaling up the data for more diverse domain coverage). In short, our work is uniquely positioned against existing works as a “general framework” to extract “task-specific” visual features.

Question 4: What is the specific and concrete difference between the proposed method and the existing text-conditioning visual feature extraction method? What kind of broader tasks can be only solved by this proposed method? The contribution (compared with other similar or related works) needs to be highlighted during the rebuttal.
Response 4: Following from Response 3, we do not consider models that implicitly operate on conditional image features (including all works mentioned in related work line 130-141) as visual feature extraction methods as they do not produce explicit image representations compatible for further downstream uses. Instead, the most relevant existing approaches that generate text-conditioning visual features are Composed Image Retrieval (CIR) methods [2,3]. CIR concerns the problem of retrieving target images based on an input query image and a given text condition. As a result, CIR models may generate text-conditioned image embeddings as in FocalLens models. However, our work emphasizes the general notion of “conditional image representations” beyond merely the application of image-to-image retrieval as considered in CIR works. Thus, for the first time to the best of our knowledge, we show conditional image representations can enhance downstream performances in multiple settings, ranging from image classification (Table 5), image-text retrieval (Table 6 and 7), to image-image retrieval (Table 2 and 3), where CIR is only one application conditional image representations can be applied to. Importantly, on image-image retrieval tasks which CIR models are designed for, we show FocalLens-CLIP consistently outperforms the state-of-the-art CIR model (i.e., MagicLens) on all considered benchmarks by significant margins.

[2] MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions. Zhang et al. 2024.
[3] Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval. Saito et al. 2023.

Thank you once again for your valuable review. As the discussion period is close to its end, we wanted to make sure that we have adequately addressed the questions you raised. We would appreciate your feedback on our responses and would love to answer any further questions you have. Thank you!

Question 1: While most results seem promising, some of them are not. For example, in Table 2, FocalLens performs worse than InstructBLIP on GeneCIS (Attribute). In Table 3, FocalLens performs worse than CLIP on Flower and Aircraft. In Table 7, compared with OpenAI ViT-L-14, FocalLens performs the same on Orientation and significantly worse on Structure. These results make me concerned about the actual effectiveness of FocalLens on certain conditions. Could the authors provide a justification on this?
Response 1: Thank you for pointing this out. We agree that on several occasions, the proposed FocalLens-CLIP is not the best performing method. We provide potential explanations to each of these cases. First, when compared to InstructBLIP, we note that InstructBLIP is trained on much more instruction tuning data than we used to train FocalLens-CLIP. Specifically, in addition to the LLaVA instruction tuning examples, InstructBLIP is additionally trained on a total of 10 other academic datasets. Notably, InstructBLIP is trained on GQA dataset, from which GeneCIS Attribute split is also created from. We conjecture that this may give InstructionBLIP a slight edge in its performance on GeneCIS Attribute due to more similar training and testing data distribution. On the other hand, on GeneCIS Object which is created from COCO images, FocalLens-CLIP performs significantly better than InstructBLIP (both FocalLens-CLIP and InstructBLIP are trained on COCO images). Second, when comparing FocalLens-CLIP to CLIP on classification datasets, while we see slight performance drop on Flower and Aircraft dataset, we also observe improvements on Car and Food datasets. Thus, we consider FocalLens-CLIP to compare favorably to CLIP on standard classification tasks, as also shown by its better average performance than CLIP on ImageNet datasets. Finally, on MMVP-VLM dataset, we observe performance drop on the “Structural Characteristics” tasks. We attribute this to the potential misalignment between the instructions we specified to FocalLens-CLIP and the diverse mix of tasks included in this split. In particular, the Structural Characteristics split includes diverse tasks ranging from “identify the shape of a gingerbread”, “identify the material of a weight”, to “identify the state of the butterfly wings”. However, regardless of the actual tasks in this split, we only provide FocalLens-CLIP with a generic (and somewhat ambiguous) instruction of “Describe the state of the objects in the image”. As a result, this misalignment between the given instruction and the actual task may lead to observed performance degradation.

Question 2: The authors use the visual instruction tuning data in LLaVA to train FocalLens models. It would be better to show how the number of visual instruction tuning examples affect the final performance.
Response 2: Thank you for the great suggestions. We agree that it would be nice to see how the scale of the visual instruction tuning dataset used affect FocalLens-CLIP’s performances. As scaling up the dataset and training the model takes more time, we are still actively working on this and we aim to show preliminary scaling results during the rebuttal.

Thanks for the authors' rebuttal. However, the rebuttal does not well address my concerns. The explanations for the inferior performances on certain tasks are not satisfying. When it comes to standard fine-grained classification datasets, the authors only mentioned that "while we see slight performance drop on Flower and Aircraft dataset, we also observe improvements on Car and Food datasets. Thus, we consider FocalLens-CLIP to compare favorably to CLIP on standard classification tasks", but did not provide any insights on why such a phenomenon occurs and how to further improve it. Besides, I have not seen any experiments on how the number of visual instruction tuning examples affects the final performance so far. After reading the comments from the other reviewers, I agree that the current paper does not meet the acceptance threshold and still needs to be further improved. Thus, I am considering lowering my score to 5.

Question 1: Retrieval tasks through text instruction is not a new concept. On the other hand, proposed architecture is similar to LLaVA with contrastive loss.
Response 1: We agree that prior works have explored the problem of image-to-image retrieval with text instructions. Specifically, Composed Image Retrieval (CIR) is perhaps the most related problem setup where the goal is to retrieve target images through composing a given image along with some text instructions. However, in this work, we emphasize the general notion of “conditional image representations” beyond merely the application of image-to-image retrieval as considered in CIR works. In particular, we consider text-conditioned image representations to be like standard image representations that can be used in a variety of downstream applications, including training classifiers, or used directly to perform zero-shot classifications or retrievals. This nuance allows us to demonstrate, for the first time to the best of our knowledge, that conditional image representations can enhance downstream performances in multiple settings, ranging from image classification (Table 5), image-text retrieval (Table 6 and 7), to image-image retrieval (Table 2 and 3), where CIR is only one application conditional image representations can be applied to. Under this perspective, our goal is to explore potential ways in generating conditional image representations, where we start from proposing FocalLens-MLLM as a strong baseline, and proposing FocalLens-CLIP as our main method. We discuss more in detail the comparisons to CIR work in Response 3 below.

Question 2: It is not clear about which instructions are given during the inference of retrieval tasks.
Response 2: Thank you for pointing this out. We clarify that all the instructions we used during inference for different tasks are provided in Appendix D (as currently mentioned in Line 253). We will make sure to highlight this for better clarity.

Question 3: I don't see the difference between the textual conditioning of this work and composed image retrieval (CIR) task. I think Pic2Word and SEARLE should be one of the proper baselines for the retrieval experiments.
Response 3: Related to Response 1, we first clarify that composed image retrieval (CIR) is one of many applications of conditional image representations considered in our work. In addition, although we consider image-to-image retrieval as one of our evaluations just like CIR, we note that our motivation is different from theirs. Specifically, CIR concerns more on composing the semantics of the input query image with the external text condition for retrieving target images with the desired semantics (e.g., retrieving an image of “an origami of goldfish” by inputting an “goldfish” image with the condition “origami”). On the other hand, we focus on extracting visual features that better pronounce certain observed intrinsic aspects of the input image using textual instructions as the guidance. However, despite the difference in the problem settings and goals, we agree that CIR methods can be considered as baselines for retrieval experiments. As a result, we did compare our proposed method with MagicLens, which is the current state-of-the-art CIR method shown to outperform Pic2Word and SEARLE on various image-image retrieval tasks [1]. In our experiments (Table 2 and 3), we see FocalLens-CLIP consistently outperform MagicLens on all considered benchmarks by significant margins. We also provide in the Table below comparisons to Pic2Word and SEARLE on GeneCIS dataset, borrowing numbers from [1].

[1] MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions. Zhang et al. 2024.

Question 4: LLaVA should also be one of the baseline methods with the LLaVA feature variant of FocalLens.
Response 4: Thank you for the great suggestion! We agree that standard LLaVA features (without the introduced contrastive learning) should also be another baseline in addition to FocalLens-MLLM. Since LLaVA is by design for text generations instead of producing explicit representations (embeddings), there is no default way to obtain image representations from LLaVA in a training-free fashion. As a result, we follow recent work [2] to use explicit prompting where we instruct LLaVA to generate only a single token as its model output, and treat the first output feature of the auto-regressive decoding process as the conditional image feature. For instance, when we are interested in the hair color of the person in the image, we give the following prompt to LLaVA: “What is the hair color of the person? Use one word to answer”. We note this method can be considered as a zero-shot alternative (without contrastive training) to the FocalLens-MLLM baseline. We name this new baseline LLaVA zero-shot, and report its performance in the following table, which we will add to our revision.
From the table, we first see LLaVA zero-shot features achieve better performances than unconditional (generic) CLIP features on CelebA-Attribute and GeneCIS datasets, validating that it is indeed a strong baseline method. The results also show that LLaVA does possess strong implicit conditional image features within its LLM decoder, and prompting appears to be an effective approach to extract these features in a zero-shot fashion. However, on fine-grained classification datasets, we see LLaVA zero-shot suffers a significant gap to the standard CLIP features. On the other hand, we see our proposed FocalLens-CLIP—despite being almost 10x smaller in model size compared to LLaVA—performs competitively across all different evaluations, and significantly better than LLaVA zero-shot on GeneCIS and fine-grained classification tasks. This validates that FocalLens-CLIP is an efficient and promising solution for extracting conditional visual features.

[2] E5-V: Universal Embeddings with Multimodal Large Language Models. Jiang et al. 2024.

Question 5: The overview of the approach is easy to understand, but the overall presentation lacks clear explanation of technical details and experimental details.
Response 5: We apologize that we had to defer some experiment details to appendix due to the page limit. We are however eager to hear your suggestions on adding further clarifications and technical details to the main paper in the revision. Additionally, we aim to publicly release our code for easy reproducibility.

Thank you once again for your valuable review. As the discussion period is close to its end, we wanted to make sure that we have adequately addressed the questions you raised. We would appreciate your feedback on our responses and would love to answer any further questions you have. Thank you!

Thanks to the authors for the rebuttal. It seems from Response 4 that, LLava zero-shot features are much more better than FocalLens-MLLM. On the other hand, although authors highlighted in Response 1, FocalLens-MLLM is a strong baseline, its absence in the table 6 & 7 results both in the main paper and rebuttal is concerning. I am not sure why, but I believe it could be included for comparison. Overall, the existence of FocalLens-MLLM still does not bring any novel insights, and FocalLens-CLIP is an innonative approach, which has an interesting motivation and surpasses most of the baselines. The paper is still confusing, and difficult to understand why the proposed models sometimes fail to outperform. I think the paper needs a lot of revision and proper clarifications of the authors' proposal, hence I would like to keep my rating to 5.