Personalized Visual Instruction Tuning

Thank you for your insightful advice and constructive comments, we appreciate that you find our proposed data construction framework and benchmark useful for the community. We aim to address your concerns in the following:

Potential Ethics Issues: Thank you for pointing ourt this potential confusion. We would like to clarify this from the following aspects:

• Firstly, we would like to clarify that all images used in our data construction were collected from publicly available datasets, including Visual Genome, COCO, Object365, and Flickr30k. Additionally, all textual information (e.g., names, hobbies, and character traits) was synthesized by a large language model (LLM) through prompting and randomly paired with the images of individuals. Consequently, there are no true correspondences between the images and the textual information. As a result, our curated dataset does not involve any leakage of personal information.

• Secondly, since the personal text information is fully synthesized by the LLM, and randomly paired with personal images during training, the MLLM is not to trained to memorize any specific individual, rather, it is trained to recognize target individual in the image when it is provided in the MLLM’s context. This training approach ensures that no privacy risks are introduced.

• In addition, we will make sure the released dataset uses the same license as the source datasets (CC BY 4.0).

We’ve added this paragraph to Section I of the Appendix for clarification.

Novelty of Our Method: We are truly grateful for your acknowledgement of our data generation framework. We wish to clarify that we are the first to formulate personalization of MLLM as an in-context learning problem, which only needs to train the MLLM once, and requires no additional training for each newly introduced individual. To achieve this, we take a data-centric approach: we design a novel data construction framework that automatically creates personalized conversation data without requiring human labelling. Compared with previous model-centric approaches that tackle this task by modifying network architectures [1,2], our in-context learning paradigm and data-driven method is more generalizable and practical.

We also point out that the research community today prioritizes the development of generalist models rather than task-specific ones. Consequently, taking a data-centric perspective, minimizing modifications to model architecture and training paradigms is advantageous, as it allows training to remain unified across different tasks.

Performance on Original General Benchmarks: Great question! If we only finetue the MLLM with personalized training data, it is inevitable that the MLLM will suffer from catastrophic forgetting. Therefore, during our training, we’ve included a subset (300k) randomly sampled from the original training set of LLaVA-Instruct-665k to alleviate forgetting. We observe a slight or no decrease in performances on the regular benchmarks compared with original LLaVA 1.6.

If computational resource allows, we could incorporate more conventional conversations in our training data, which we expect would preserve the original capability,

How is the performance of GPT-4o: We’ve conducted additional experiments on GPT-4o using both API calls and chat windows from the webpage. We find that the API calls to GPT4o often refuse to answer personalized questions and simply respond with “I don’t know”, while the chat window from the webpage functions quite well. Therefore, we randomly select 30 samples and test each of them manually in the webpage’s chat window, and find that all the questions are correctly answered. We presume that GPT4-o, as a closed sourced model, may have incorporated similar data during training, which empowers it with personalization capability. We believe our proposed data construction framework and curated dataset helps close the gap between GPT4-o and open-source MLLMs on personalized conversations.

[1] MyVLM: Personalizing VLMs for User-Specific Queries

[2] Yo'LLaVA: Your Personalized Language and Vision Assistant

Dear Reviewer ZV1c,

We deeply appreciate your insightful suggestions and helpful comments. It is an honor to know that you find our work proposes a practical framework for personalization data synthesis, curates a useful benchmark for the community, and presents good experimental results.

Your help in identifying potential concerns in our manuscript has been invaluable, and we have diligently addressed these in our response. Since the discussion period deadline is approaching, we would greatly appreciate any further comments you may have on our response. We are eager to resolve any additional concerns you might have.

We fully understand your busy schedule and are genuinely thankful for the time you have dedicated to helping us enhance our work. We look forward to receiving any additional feedback you may offer.

Thank you for your helpful suggestions and insightful comments, we appreciate that you find we propose a novel approach for personalization, curates a helpful benchmark, and paper is well written. We aim to address your concerns in the following:

Typos: Thank you for pointing it out. We fixed the typo in our current version.

Current Implementation Focus on faces: Thank you for your insightful comment. We would like to emphasize that we did not claim to address general personalization, despite our framework being inherently adaptable to various objects by design. As highlighted in our abstract, our work specifically targets “face blindness.” We chose to focus on humans because personalization in this context has the most widespread application.

However, our data construction framework is designed to be generalizable to accommodate various types of subjects beyond humans. For instance, during object extraction, an open-vocabulary detector can be employed to identify other objects (e.g., dogs), instead of persons. We wish to note that personalization for other objects often depends on specific user scenarios and can be adapted to their unique needs. For example, a vehicle manufacturing company may have requirements for vehicle personalization and should be able to supply abundant domain-specific training data, based on which our framework can create personalized training data in that domain.

Therefore, rather than training a universal MLLM capable of personalizing for all types of objects—a goal that is infeasible since we can not exhaustively capture all possible categories—we aim to provide a customizable framework that can be tailored to different use cases.

Zero-shot ability: We would like to clarify that our in context learning paradigm indeed emphasizes zero-shot capability during inference, which allows the MLLM to generalize to new individuals of interest without further training. We wish to note that the data in our benchmark is strictly outside of the training data.

Novelty of Our Method: Thank you for the acknowledgement. We wish to clarify that we are the first to formulate personalization of MLLM as an in-context learning problem, which only needs to train the MLLM once, and requires no additional training for newly introduced individual. To achieve this, we take a data-centric approach: we design a novel data construction framework that automatically creates personalized conversation data without human labelling. Compared with previous model-centric approaches that tackle this task by modifying network architectures [1,2], our in-context learning paradigm and data-driven method is more generalizable and practical.

More advanced knowledge: We mainly focused on faces and names due to the following reason: the major challenge of our in context learning formulation of personalization requires accurately recognizing the target individual in scene images, rather than comprehending text-based attributes. The former is a new ability not possessed by SOTA open source MLLMs, while the latter is easier to achieve due to the strong ability of the base LLM.

As suggested, we conduct addition experiments using more advanced personal attributes. To create such data, we first define several categories like hobbies, profession and character. Then, each category of attributes can be created in a similar manner as name: we prompt the LLM to create a list of each attribute, and dynamically combine them with the face image when constructing the QA data.

During the rebuttal period, we attempted including those new attributes and conducted training, and demonstrated the model output after fine tuning in Section G, Figure 4 of the appendix. We observe that the MLLM is able to associated the attributes with the corresponding target individual. We also provide the prompts for synthesizing these attributes, which can be extended to other types of attributes.

Temporal changes, twins with similar names and similar faces, similar faces yet different names or similar names yet different faces: Great point! We tried some examples as you suggested, and demonstrate the results in Figure 4 and Figure 5 of the appendix.

• We find that the MLLM can successfully recognize the person of interest when there are multiple people with the same name (rightmost in Figure 4).
• We find that when the age difference is not too significant, the MLLM is able to recognize the target person. However, it fails when the age gap is too large (leftmost, middle in Figure 5).
• We also find that the MLLM sometimes fail to differentiate the target person when there is another person that looks very similar (rightmost in Figure 5).

However, we wish to note that these cases may be challenging even for humans, which requires further efforts to address.

[1] MyVLM: Personalizing VLMs for User-Specific Queries [2] Yo'LLaVA: Your Personalized Language and Vision Assistant

Thank you for your insightful advice and constructive comments, we appreciate that you find the proposed multi-phase data construction pipeline produces rich and diverse personalized training data, curates a novel benchmark, and achieves impressive experimental results. We aim to address your concerns in the following:

Reliance on Pretrained MLLMs and Potential Hallucinations: Great point! For holistic information of the entire image, we prompt the MLLM (InternVL2-26B) to describe the overall content of the image that captures the main elements. For personal information, we first crop out each individual in the image, and let the MLLM only describe that individual. With the above prompts, we observe that the MLLM produces compact descriptions that are not prone to hallucination. To verify this, we evaluate our P-LLaVA on the hallucination benchmark PoPE [3], which shows similar performance as the original LLaVA 1.6:

The results demonstrate that after personalized training, the MLLM does not become more prone to hallucination.

However, we agree that hallucination may still potentially occur. Therefore, as inspired by image textualization [1], we first extract the objects contained in each description, and then employ an open vocabulary detector GroundingDino [2] to check for the existence of each object mentioned in the description. We then keep only the descriptions that do not contain hallucinated objects for subsequent stages.

Ethical and Privacy Concerns in Human Image Sourcing: Thank you for pointing ourt this potential confusion. We would like to clarify this from the following aspects:

• Firstly, we would like to clarify that all images used in our data construction were collected from publicly available datasets, including Visual Genome, COCO, Object365, and Flickr30k. Additionally, all textual information (e.g., names, hobbies, and character traits) was synthesized by a large language model (LLM) through prompting and randomly paired with the images of individuals. Consequently, there are no true correspondences between the images and the textual information. As a result, our curated dataset does not involve any leakage of personal information.

• Secondly, since the personal text information is fully synthesized by the LLM, and randomly paired with personal images during training, the MLLM is not to trained to memorize any specific individual, rather, it is trained to recognize target individual in the image when it is provided in the MLLM’s context. This training approach ensures that no privacy risks are introduced.

• In addition, we will make sure the released dataset uses the same license as the source datasets (CC BY 4.0).

We’ve added this paragraph to Section I of the Appendix for clarification.

Limitation Section: Thank you for the reminder. Here are the limitations of our current work: In our work, we propose a novel in-context learning paradigm for the personalization of MLLMs, which equips the model with personalization ability using a data-centric approach, where we meticulously design an automatic pipeline for creating personalized conversation data. However, in our work, we did not take into account the bottleneck of the vision encoder’s ability for producing accurate personalized representations. We assume this may be one of the cause for failing to differentiate two people that look very similar (Figure 5 in the Appendix). This is a parallel direction to our proposed method, which we leave as future work to further explore. We’ve added the limitation section to our updated paper.

LLaVA Version, Models used in information extraction and fusion: Thank you for pointing it out. We used LLaVA-1.6 as our base MLLM throughout our paper. For textual information extraction, we adopt InternVL2-26B. For information fusion and QA generation, we adopt LLaMA3.1-8B-instruct. We’ve added these details in section C of the Appendix.

[1] Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions

[2] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

[3] Evaluating Object Hallucination in Large Vision-Language Models

Thank you for your constructive and insightful advice, we appreciate that you find our proposed method to achieve good personalization performance, and the proposed benchmark is potentially helpful for the community. We aim to address your concerns in the following:

Various types of subjects: Thank you for your insightful comment. We wish to clarify that the primary goal of our paper is to demonstrate the effectiveness of using in-context learning for the personalization of MLLMs and to propose an automated framework for synthesizing training data.

Our data construction framework is designed to be generalizable and can be extended to accommodate various types of subjects beyond humans by synthesizing corresponding training data. For instance, during object extraction, an open-vocabulary detector can be employed to identify other objects, such as dogs, instead of persons.

However, rather than training a universal MLLM capable of personalizing for all types of objects—a goal that is infeasible since we can not exhaustively capture all possible object categories—we aim to provide a customizable framework that can be tailored to different use cases.

We chose to focus on humans because personalization involving human subjects is one of the most prevalent and impactful real-world applications. It benefits from an abundance of publicly available, relevant data. In contrast, personalization for other objects often depends on specific user scenarios and can be adapted to their unique needs. For example, a vehicle manufacturing company might have significant requirements for vehicle personalization and should be able to supply abundant domain-specific training data for this purpose.

Comparison with GPT4-o: We’ve conducted additional experiments on GPT-4o using both API calls and chat windows from the webpage. We find it strange that the API calls to GPT4o often refuse to answer personalized questions and simply respond with “I don’t know”, while the chat window from the webpage functions quite well.

Therefore, we randomly select 30 samples and test each of them manually in the webpage’s chat window, and find that all the questions are correctly answered. We presume that GPT4-o, as a closed sourced model, may have incorporated similar data during training, which empowers it with personalization capability. We believe our proposed data construction framework and curated dataset helps close the gap between GPT4-o and open-source MLLMs on personalized conversations.

Analysis on failure case: Indeed, our method does not achieve perfect personalization in all scenarios. Specifically, we observe the MLLM may struggle to accurately identify the target individual if the age gap is too significant, and sometimes fails to differentiate the people that look too similar (Shown in Figure 5 of the appendix). We believe even though this problem is challenging, even for humans, it can be potentially solved by incorporating more hard samples into the training data. We will further explore this direction in the future.

Personalized Wrapper Tokens: Our approach differs from previous methods like Yo'llava. Instead of adding a unique special token and fine-tuning the model for each new concept, we use just two delimiter symbols to wrap information about an individual. This allows the model to generalize to new individuals without additional training at test time, making our method more scalable and efficient.

How do you use MLLMs designed for single inputs in multi-image input setting: We wish to clarify that all the baseline MLLMs we selected are able to accept multiple images as input.

Dear Reviewer YLzK,

We deeply appreciate your insightful suggestions and helpful comments. It is an honor to know that you find our work proposes a practical paradigm for personalization, curates a useful benchmark for the community, and presents good qualitative and quantitative results.

Your help in identifying potential concerns in our manuscript has been invaluable, and we have diligently addressed these in our response. Since the discussion period deadline is approaching, we would greatly appreciate any further comments you may have on our response. We are eager to resolve any additional concerns you might have.

We fully understand your busy schedule and are genuinely thankful for the time you have dedicated to helping us enhance our work. We look forward to receiving any additional feedback you may offer.

Thank you for your response, we further clarify for each of your newly raised questions below:

• Scope and Application: We would like to emphasize that we never claimed to address general personalization in this paper, despite our framework being inherently adaptable to various objects by design. As highlighted in our abstract, our work specifically targets “face blindness.” We chose to focus on humans because personalization in this context has the most widespread application. For other objects requiring personalization, users can adapt our proposed framework to their specific use cases.
• Comparison with Closed-Source Models: We argue the fact that our proposed strategy enables a 7B open-source MLLM， which does not possess personalization capability before trained with our method, to achieve performance comparable to GPT-4o is indeed a significant contribution. GPT-4o is a closed-source model with insufficient information available about its underlying technology. If models like OpenAI’s o1 and GPT-4o are always used as baselines, it would render many research contributions less impactful. We wish to note that our method is generic and applicable to any MLLMs and enable them with personalized capability, and would greatly appreciate it if you could consider the impacts and insights our methods can bring to small scale and open-source community.
• Key Contributions: We wish to emphasize that our primary contribution lies in proposing the in-context learning paradigm to address the MLLM personalization task and designing a data synthesis framework to create personalized training data. We do not claim that the personalized wrapper tokens are our main contribution, they are an important design element to separate each personalized concept. We also argue that the research community today prioritizes the development of generalist models rather than task-specific ones. Consequently, taking a data-centric perspective, minimizing modifications to model architecture and training paradigms is advantageous, as it allows training to remain unified across different tasks.

We hope that we addressed your newly raised concerns, if you have any other questions, we are happy to discuss further.