Yo'LLaVA: Your Personalized Language and Vision Assistant

While I acknowledge that personalized datasets don't exist, it would be nice to have datasets in other domains.

Thanks for your suggestion. Our current dataset includes humans, pets, and objects for personalization. We anticipate that future research will introduce more datasets in other domains!

Could you elaborate more regarding the computational time needed for your method?

The total time for learning a new subject with LLaVA-1.5-13B would be around 40 minutes. For a new subject with 3-5 images, the time required to create the training data is roughly 3 minutes (1-2 minutes for conversation creation and 1 minute for hard negative retrieval). The optimization process, including training new tokens, takes approximately 34 minutes for 15 epochs (measured on an A6000 machine).

Lack of small human evaluation.

Thank you for your suggestions! Due to the limited time available during the rebuttal phase, unfortunately, we cannot include a human evaluation, as it would involve privacy concerns and obtaining permissions (e.g., from a person named <A>). We will continue to investigate this and report if it becomes feasible.

From reading the paper, it is unclear to me if we need to train a separate model for each instance, or we can train a single model on N instances together using N*16 learned identity tokens. If it is the later, does the model has a number of objects limit that it can learn together?

As stated in Sec. 3.1 (Lines 152-157), we train (1) the newly added tokens (e.g., $<sks>$) and (2) the final classifier head matrix associated with these newly added tokens. During training, we train for each subject separately; then at test time, we can easily adapt the model to handle N subjects by concatenating their personalized prompts together.

For example, in Table 2 (left), we have two personalized subjects: a person named $<T>$ (subject 1) and a dog named $<bo>$ (subject 2). The personalized prompt would be: “$<T> is <token 1_1>...<token1_{16}>. <bo> is <token2_1>...<token2_{16}>.$”

Examples with 3 personalized subjects can be found in the Rebuttal PDF file (Fig. 2).

More discussion on the finetuning cost (e.g., wall clock time) in the rebuttal is appreciated.

The total time for learning a new subject with LLaVA-1.5-13B would be around 40 minutes. For a new subject with 3-5 images, the time required to create the training data is roughly 3 minutes (1-2 minutes for conversation creation and 1 minute for hard negative retrieval). The optimization process, including training new tokens, takes approximately 34 minutes for 15 epochs (measured on an A6000 machine).

The overall paper relies on the fact that LLaVA-like architecture cannot do multi-image conversations (L253) and the only available model is GPT-V at present... [omitted]

Yo’LLaVA can learn personalized subjects more efficiently using fewer tokens and more effectively encode visual attributes compared to strong prompting baselines (e.g., GPT-4V + Image prompting).

It is evident that Yo’LLaVA, with just 17 tokens, outperforms GPT-4V with single image prompting (1k+ tokens) (Tab. 5). Notably, Yo’LLaVA, even with only 17 tokens, yields results almost comparable to GPT-4V using 5 images (5k+ tokens); see Fig. 4 (Lines 286-288). Unfortunately, GPT-4V is a closed-source model, so we cannot integrate Yo’LLaVA with GPT-4V to reduce the number of tokens used while retaining performance.

Recently, Meta AI released Chameleon [1] - a LMM that supports multiple images/text as inputs and images/text as outputs. We have tested Yo'LLaVA with Chameleon, and it has demonstrated superior results in image generation! (See Rebuttal PDF, Fig. 1). These early results show that not only can Yo’LLaVA be adapted to other multimodal models (e.g., Chameleon), but it can also provide clear advantages!

The current architecture scheme does not look scalable... [omitted]

Yo’LLaVA is much more token-efficient while achieving similar performance with Image Prompting.

As noted by the reviewer (in above comment about Image prompting vs. Yo’LLaVA), consider a scenario where we need to personalize 10 subjects with two options: (1) Yo’LLaVA and (2) Image Prompting.

• Using Yo’LLaVA: We would add 10 (subject) x (16+1) (token/subject) = 170 tokens
• Using Image Prompting: We would add 1 (reference image) x 10 (subject) x 1024 (token/image) = 10,240 tokens (we use 1024 tokens here which is the number of tokens for an image used in Chameleon [1], since the number of tokens is unknown for GPT-4V). This clearly illustrates the significant token efficiency of Yo’LLaVA!

less token requirement is a direct consequence of training with the new token, whereas GPT-V is zero-shot... [omitted]

As stated in the main paper, GPT-4V + Image Prompting baseline is meant to show accuracy and efficiency tradeoff of a SOTA proprietary multi-image understanding model with zero-shot inference time personalization (Line 257 “Since images convey more information than text, we hypothesize that personalized image prompts represent upper bound for prompting effectiveness.”).

We agree that requiring fewer tokens is a direct result of training new tokens. However, the primary goal of this paper is to introduce a new problem setting: Personalizing LMMs. Our work is the first to demonstrate its feasibility: we proposed a new novel approach and showed promising results: Yo’LLaVA is much more token-efficient (17) while achieving similar performance with image prompting (5k~)!

We anticipate that integrating Yo’LLaVA with GPT-4V could significantly reduce the number of tokens used while retaining performance; however, we could not try this since GPT-4V is a closed-source framework (Line 289-290).

The recognition metrics is unclear to me... [omitted]

We have 40 subjects, each with 5-10 test images containing the corresponding subject. For each subject, all of its test images serve as positive test images, while test images from the remaining 39 categories serve as negative test images.

During testing, we show a photo to the model and ask, “Can you see if <sks> is in this photo? Answer with a single word or phrase.” The ground-truth response is “Yes” for photos containing <sks>, and “No” for others. An extension of Tab. 5 with F1 score is:

As explained in Line 269-274, we purposely report weighted accuracy = 0.5 ∗ accuracy positive + 0.5 ∗ accuracy negative because:

• In personalization setting, the ability to recognize the positive class (it is <sks>) and not-sks (it is not <sks>) is equally important
• The positive/ negative class are heavily imbalanced

We also report accuracy for each positive and negative class.

It appear that GPT-4V can easily go above Yo'LLaVA's performance ... [omitted]

Our setting involves visual personalization, which typically requires 3-5 images per user, as initially proposed by Textual Inversion/ Dreambooth [2]. In our experiment, we provided 5 images/subject to GPT-4V, which is similar to the number used to train Yo’LLaVA. As shown in Tab. 5, we achieved comparable performance while using significantly fewer tokens (17 vs. 5k). Unfortunately, we cannot extend to an order of magnitude more images (e.g., ~10) as suggested by the reviewer, just to see if GPT-4V will surpass Yo’LLaVA in performance, this is because:

• As stated in Line 258, comparison to GPT-4V is solely for reference as it’s closed-source framework.
• With 5 images, GPT-4V already surpassed Yo’LLaVA in terms of recognition accuracy (0.925 vs. 0.924; Line 287); however, use significantly more tokens (5000+ vs. 17!)
• The best setting would be to directly integrate Yo’LLaVA with GPT-4V for both performance and token-efficiency – Unfortunately, we cannot do that as GPT-4V is closed-sourced! (Line 288).

Also, there is no existing dataset to support this (e.g., dataset for personalization that have 10+ imgs/subject), and we believe this is not a practical setting. If the reviewer is aware of any such dataset, we would much appreciate the reference.

Reference:

[1] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv, 2024

[2] Ruiz et al, DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023

User: This is my cat whose name is <name> and here is how he/she looks: <add_image_tokens_or_embeddings>. Please respond to next questions personalized to my cat.
.......
User: I'm thinking about buying a birthday gift for <bo>. What do you recommend? (From your figure1).

While the paper provides promising results, the evaluation is somewhat limited to specific tasks. A broader evaluation across more diverse tasks and real-world scenarios would strengthen the claims.

We agree that a broader evaluation across more diverse tasks would strengthen the paper.

However, at this moment, we are not aware of any benchmark/ datasets for this personalization yet (e.g., personalized datasets in healthcare, business). Thus, we include two main types of tasks: Recognition (Sec. 5.1) and Question and Answering (Sec. 5.2), which are the foundational tasks of personalization (similar to MyVLM [1], in which introduce a novel dataset with 29 concepts, while our dataset has 40 concepts).

We have also made an effort to test the model with some examples of real-world scenarios (e.g., Fig. 1, suggesting a gift for a pet’s birthday; or Table 3, Left, writing a Facebook caption). More examples can be found in Appendix, Table 9-19.

The paper does not thoroughly address the scalability of the proposed method. How well does Yo'LLaVA handle a large number of personalized subjects or frequent updates to personalized concepts? Discussing and testing the scalability of the approach would be beneficial.

In Table 3 (Right), we indeed show examples of handling 2 subjects at a time (a person named <T> and her dog named <bo>). In the Rebuttal PDF, we show additional qualitative results of handling 3 subjects (a person named <T>, her dog named <bo>, and her cat named <mam>). If users want to update the personalized subject, the re-train process would take roughly 40 minutes.

A further study on scalability of Yo’LLaVA would heavily rely on the availability of a large scale dataset/ benchmark for personalization. At this moment, we are not aware of any such available dataset/ benchmark for this task yet. To collect and label this dataset would be challenging, as (1) it depends on privacy (e.g., as stated in Line 31 “user is only willing to share 4-5 images of a person named <A>”); and (2) multiple subjects should be related (e.g., different subjects appear in the same photo like <T> and her pets <bo> and <mam>). We anticipate that future research will address these challenges!

The method proposed in this paper effectively addresses the issue of personalizing LMMs and highlights its advantages over providing a language description for a given object. However, what are the benefits of this approach compared to directly providing image of the object to the LMM, and then using a CoT to first prompt LMM to give detailed descriptions of the subject, and bind these descriptions with the speical token like <bo> in the paper. Besides, I have also tried the proposed personalized setting on GPT4-o, and it seems that GPT4-o can directly handle this problem.

Thanks for your suggestion to use Chain of Thought (CoT) to improve the detail of descriptions about a subject. In limited rebuttal time, we have quickly tested LLaVA + CoT for more detailed descriptions, and recognition accuracy increased roughly 5% (from 0.819 to 0.852 (Table 5)). While increasing, this is still behind Yo’LLaVA (0.924) and Image Prompt (0.901-0.925).

This is expected as discrete text will always contain some ambiguity and may not fully capture the visual details (e.g., for a dog named <bo>, the coat color might be described as “yellow,” but visually, yellow can have many different shades). The strongest baseline is directly providing the image prompt. We indeed tried this in the main paper with GPT-4V, as GPT-4V is currently a leading proprietary multimodal model. The results shown in Table 5 demonstrate that Yo’LLaVA can achieve similar performance to GPT-4V with 5k tokens, which is much more computationally efficient!

With text-only response models (e.g., LLaVA, GPT-4V), Yo’LLaVA can help to retain the recognition accuracy while using much less tokens (e.g., increasing from 0.822 vs 0.924). We anticipate that when shifting to another modality (e.g., image/text generation models like Chameleon [2]), Yo’LLaVA also helps with image generation quality. We provide early results from our experiments integrating Yo’LLaVA and Chameleon in the Rebuttal PDF (Fig. 1). As shown, the optimized prompt captures much more precise visual details about the subject compared to the plain text description (e.g., Yo’LLaVA + Chameleon captures more details about a dog named <bo>)!

These early results show that not only Yo’LLaVA can be adapted to other multimodal models (e.g., Chameleon), it can also provide clear advantages with clear text personalized prompts.

Although only limited number of images are required, the process of curating the conversationing data and retrieving hard negatives should be executed for each personalized subject, which incurs a lot of human efforts.

All the training data is generated automatically, so no manual labeling is needed!

For the conversation training data, we employ LLaVA to generate answers for each template question (Lines 215-216). The template questions are fixed and universal for all subjects (a complete list of these questions can be found in Appendix G, Tables 18-19). For the negative images, for each positive image, we retrieve the top m images with the highest CLIP image embedding similarity (Lines 191-192). This process is fully automated and does not involve any human supervision!

Reference:

[1] Alaluf et. al., MyVLM: Personalizing VLMs for User-Specific Queries, ECCV, 2024

[2] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv, 2024