TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models

Thank you very much for your valuable suggestions and recognition of our contribution. We will response your concerns point by point as follows.

W1. Considering there has been prior work (i.e., GILL) addressing text-to-image retrieval and generation, the difference and advantages of the proposed method over GILL should be further highlighted clearly.

The differences and advantages are summarized as the following five points:

1. Task Level: GILL focuses on equipping LLMs with image acquisition capabilities but does not address when to retrieve versus generate or how to unify them. In contrast, we analyze the strengths (e.g., generation for creativity, retrieval for knowledge-intensive tasks) and limitations (e.g., hallucination in generation, lack of creativity in retrieval) of each, highlighting the importance of unifying both paradigms.
2. Model Level: GILL aligns LLMs with external models like CLIP, making retrieval performance reliant on the external models. Our training-free approach leverages cross-modal alignment learned during pretraining and fine-tuning, enabling generative retrieval with key advantages: a) Avoids catastrophic forgetting, retaining generation capabilities. b) Resolves conflicts between next-token prediction and contrastive learning. c) Retrieval improves with generative capability enhancements.
3. Decision Level: GILL depends on a trained classifier and costly user annotations, risking overfitting with limited data. Our approach uses the intrinsic discriminative abilities to enhance decision-making without extra annotations or classifiers.
4. Training Overhead: GILL requires contrastive learning and classifier training, whereas our model is training-free.
5. Inference Overhead: As shown in Table 5, our model is more efficient for large-scale retrieval.

W2. The paper lacks analysis on how the unified model’s performance varies with prompts that have similar intentions but are expressed differently. In practical use, different users may convey the same idea with varied wording, prompts, and questions. How sensitive is the proposed method to these variations?

Thank you for your thoughtful comments. We conducted experiments to analyze how diverse prompts with the same intention affect unified performance, as shown in Tables 11 and 12. We expanded raw prompts using three methods: Self-Expansion, Gemini-Pro, and GPT-4o. These expanded prompts may include useful clues, more intention, and irrelevant or noisy information.

The results indicate that Self-Expansion harms retrieval and generation performance, likely due to LMM hallucination, which introduces inaccurate contexts and degrades the prompt. Conversely, GPT-4o significantly improves both tasks, demonstrating our method's potential for further gains with accurately rewritten or expanded prompts. This finding also suggests that automatic prompt engineering could enhance the proposed generative retrieval framework.

W3. Given that the distribution over creative domains and knowledge domains may not be uniform in real-world scenarios, an analysis of the model’s performance and decision-making behavior under unbalanced distribution conditions would be beneficial.

Q1. Is there any experiment demonstrating the decision behavior of the proposed model?

The decision behavior across different domains is quantified in Table 10, with corresponding performance shown in Tables 8 and 9. We simulate unbalanced distributions by controllably sampling from the eight domains—for example, oversampling creative domains and undersampling knowledge domains at specific ratios. Decision-making behavior and performance can then be evaluated, and domain-specific performance is predictable from Table 10. Key observations include:

1. Decision Patterns: LaVIT is more likely to choose generation than SEED-LLaMA, likely due to its stronger retrieval performance. Beside some potential decision biases may exist, as the retrieval ratio remains high in creative domains.

2. Model Flexibility: The unified model can adaptively decide between retrieval and generation. When no suitable image exists in the database or retrieval is weak, the model opts to generate a better alternative (albeit imperfect). Compared to GILL's pre-decision strategy, TIGeR-One’s post-decision strategy is more flexible, considering both databases and intrinsic retrieval and generation capabilities.

Q2. Compared to dense retrieval methods, such as CLIP, what advantages do MLLMs and the proposed generative retrieval framework offer?

Thanks for your questions. We kindly recommend referring to the Official Comment by authors titled "Key differences between TIGeR-One and dense retrieval methods based on contrastive learning" for more highlights of the advantages of the proposed generative retrieval framework.

We sincerely appreciate the time and effort you dedicated to reviewing our paper. We are grateful for your insightful comments and the valuable improvements which have contributed to our paper. Please see below for a point-by-point response to your comments and concerns.

W1. The challenges faced by generation models in knowledge-intensive tasks are mentioned, but the proposed approach’s effectiveness in consistently addressing these challenges without hallucination remains unclear. Explicit quantitative analysis on knowledge domains might be helpful.

Thank you for your valuable feedback and suggestions! We have conducted extensive experiments on knowledge-intensive domains, with results summarized in Tables 14 and 15. For convenience, key findings are consolidated below. The results with CLIP-T are shown in the following.

The results with CLIP-I are shown in the following.

The findings demonstrate that our proposed method significantly improves text-image alignment, with enhanced prompt-following abilities for image acquisition.

To evaluate hallucination specially, we utilized Gemini-Pro via the API to assign authenticity scores from 1 to 5, with each rank clearly defined and explained. The results, shown in the following table, compare our unified model with the original generation model. The higher authenticity scores achieved by our method further validate its superiority in mitigating hallucination.

W2. Can TIGeR-ONE adapt to user-specific preferences in the decision-making process, especially in creative tasks? If not, how could this be incorporated to enhance user experience?

Thank you for your constructive questions! While TIGeR-ONE could incorporate user-specific preferences to enhance the user experience, this would require additional engineering and is challenging to implement during the rebuttal period. We provide the following analysis and leave further exploration for future work:

a) Explicit User Preference: For explicit requests like “Please generate an image of…,” particularly in creative tasks, current LMMs can reliably select “generation” from {retrieval, generation, auto-decision}, leveraging the strong language understanding of their LLM backbones.

b) Implicit User Preference: If no explicit preference is expressed, our method autonomously decides based on text-image alignment, utilizing its intrinsic discriminative capabilities.

W3. I noticed the multi-round generation in Figure 7. Could you please explain how this process maintains consistency with the user-provided image?

The consistency with the user-provided image stems from the interleaved image generation capabilities of the base model, i.e., SEED-LLaMA, which ensures loyalty to the multimodal context and preserves identity. Our model-agnostic framework fully inherits these abilities, enabling accurate interleaved image generation with identity preservation.

Additionally, compared to the base model, our method can proactively retrieve knowledge-intensive images (e.g., the Eiffel Tower) and maintain their key characteristics throughout the interaction process.

Dear Reviewer xRkP,

Thank you once again for your thoughtful feedback and the time you have devoted to evaluating our submission. We have carefully addressed the concerns you previously raised and provided detailed responses in our rebuttal.

We kindly remind you to review our replies and re-evaluate our work in light of the clarifications provided. If you have any additional questions or concerns, please don’t hesitate to let us know.

Dear Reviewer xRkP,

Thank you once again for your valuable feedback on our paper. To address your concerns, we have incorporated additional experiments and clarifications.

As the Author-Reviewer discussion period is nearing its end, we would like to kindly ask if our responses have adequately addressed your questions. We deeply appreciate your time and thoughtful consideration.

Best regards,

The Authors

We sincerely thank you for taking the time to provide your valuable feedback! Your review has shown us various changes and clarifications we should make to our paper. To address your concerns, we provide a detailed point-by-point response below.

W1. A preliminary introduction of beam search is recommended to add.

Thank you for your valuable recommendation. The preliminary introduction to beam search is as follows:

Beam search [e, f] is originally proposed to decode tokens in sequence-to-sequence models and widely used in neural machine translation [g]. It is based on breadth-first search (BFS) to explore a search tree from root to leaves. At each level, beam search generates all possible child nodes based on current prefixes, then sorts and selects the top-K paths according to their conditional likelihood, $p(childNode | prefix)$. Unlike BFS, which considers all paths, beam search keeps only K valid paths at each level and prunes others. When used in LLMs or LMMs, it produces K ranked sequences.

[e] Graves Alex. Sequence transduction with recurrent neural networks. arXiv’2012

[f] Boulanger-Lewandowski Nicolas et al. Audio Chord Recognition with Recurrent Neural Networks. ISMIR’2013

[g] Sutskever et al. Sequence to Sequence Learning with Neural Networks. arXiv’2014

We will add this preliminary introduction to the manuscript to improve the readability.

W2. The authors indicate that the similarity score is calculated by one of the three proxies. More details and discussion of the proxies chosen in decision-making are recommended.

We propose three training-free proxies in Sec. 3.2. In our experiments, we mainly compare the proxy 2, i.e., $\log \frac{p(Y|X)}{p(Y)}$, and proxy 3, i.e., $\log p(X|Y)$, considering the modality bias problem in proxy 1. $X$ and $Y$ denote text and image, respectively.

Experimental results in Tab 5 and Tab 17 (In the Decision column, Forward and Reverse denote the proxy 2 and proxy 3, respectively) reveal the following observations:

a) Overall Performance: Proxy 3 outperforms Proxy 2 on unified tasks (Table 5), demonstrating stronger discriminative abilities.

b) Domain-Specific Performance: Proxy 2 performs better in creative domains, while Proxy 3 excels in knowledge domains (Table 17).

c) Underlying Reasons: These differences may arise from multimodal pre-training and fine-tuning strategies. Current LMMs are optimized for understanding by modeling $\log p(X|Y)$ using diverse, knowledge-rich data, leading to the superior performance Proxy 3 in knowledge-intensive tasks. In contrast, LMMs trained for image generation by modeling $\log p(Y|X)$ with creative data enhance the discriminative abilities captured by Proxy 2 in these domains.

W3. With the help of TIGER-One, the LMM demonstrates improved results (Tab. 2). It’s interesting to explore the addition of dream images based on knowledge (RAG) as a third option alongside generation and retrieval in future work.

Thank you for your insightful comments! We agree that further exploring the relationship between generation and retrieval within the unified framework, as well as settings like RAG, is both meaningful and promising. Early experiments, detailed in the Appendix, provide initial insights:

1. Textual RAG: As shown in Table 11, we used GPT-4o to enrich user prompts, demonstrating that richer textual context improves image generation performance.

2. Multimodal RAG: In Table 13, we evaluated retrieval-augmented generation using retrieved images as context. While the CLIP-I score improved, CLIP-T showed slight degradation, possibly due to current LMMs’ limited interleaved text-and-image generation capabilities. Examples in Fig. 12 illustrate that RAG helps generate images with knowledge-intensive concepts.

3. Generation-Augmented Retrieval (GAR): As the inverse direction, GAR improves retrieval accuracy. Results in Table 12 and Fig. 11 show that generative models can enhance retrieval by serving as effective query expansion modules. We believe further exploration of RAG and GAR will offer valuable insights for the generation and retrieval communities.

Thank you for taking the time to carefully review our paper! Your feedback is very valuable and insightful, and will help make our paper more clear and insightful to the reader. We present the point-to-point response as follows.

W1. In my opinion, the biggest drawback of this paper is that it lacks technical novelty. This work looks more like a prototype ...

Thank you for your helpful feedback. We highlight the technical novelty of our work from the following five aspects:

a) Task for Unifying T2I Generation and Retrieval: Previous works treat T2I generation and retrieval independently, neglecting their interrelations. In contrast, our work explores these relationships from the perspective of generalized information acquisition, discussing their respective advantages and disadvantages in the Introduction. We propose unifying these tasks in a single model, believing it has the potential to advance information acquisition and offer insights into the connection between understanding, generation, and even AGI.

b) Intrinsic Abilities of LMMs and Training-free Discriminative Proxies: For the first time, we explore the discriminative abilities of LMMs, particularly text-to-image alignment. We introduce three training-free proxies compatible with LLM autoregression and next-token prediction, which provide valuable insights into modality bias and direction (e.g., text-to-image vs. image-to-text), contributing to a better understanding of LMMs and their cross-modal capabilities.

c) Training-free Generative Retrieval: Leveraging the discriminative abilities from our proxies, we propose a training-free generative retrieval approach, integrated within the autoregressive framework. This method does not require additional discriminative training (e.g., contrastive learning) and offers a novel solution for efficient and accurate retrieval using pre-trained generative models.

d) Unified Framework for T2I Generation and Retrieval: By aligning the autoregressive paradigms of T2I generation and retrieval, we unify both tasks in a single framework, enabling parallel execution with a single forward pass. This framework autonomously decides between generated and retrieved results, providing an elegant, effective, and efficient solution.

e) Unified Benchmark for Creative and Knowledge-intensive Domains: To evaluate the unified performance (retrieval, generation, and decision), we introduce TIGeR-Bench, a benchmark covering 2 creative and counterfactual domains, and 6 knowledge-intensive domains.

W2.1: I am confused about the comparison results in Table 2. What is the difference between the SEED-LLaMA vs Ours (SEED-LLaMA)? Does the latter one used both retrieval and generation? If so, this comparison seems to be somewhat unfair.

Thank you for your questions. We will further polish the manuscript by providing more detailed explanations in the caption and improving the clarity and intuitiveness of the table. The differences between SEED-LLaMA and Ours (SEED-LLaMA) in Tab 2 are as follows:

a) SEED-LLaMA is an LMM capable of generating images but cannot directly retrieve them before our method is introduced. Its generation performance is shown in the first group, "Text-to-Image Generation," in Table 2.

b) To compare retrieval abilities, we create a baseline by combining SEED-LLaMA with CLIP. Specifically, we first perform T2I generation using SEED-LLaMA, then use the generated image as a query for I2I retrieval with CLIP. The result is shown as "SEED-LLaMA" in the second group, "Text-to-Image Retrieval," in Table 2.

We believe this provides a fair comparison, as SEED-LLaMA and Ours (SEED-LLaMA) share the same model architecture and parameters. The only difference is that Ours (SEED-LLaMA) incorporates generative retrieval, forward beam search, reverse re-reranking, and decision-making methods, which are training-free and non-parametric.

We sincerely thank you for your time and valuable comments. Your suggestions regarding paper revisions and experimental extensions have been invaluable in helping us refine our work. To address your concerns, we provide a detailed point-by-point response below.

W1. Figure 1 is referring to many things that has not be introduced before, it is hard for people to understand ...

Thank you for your revision suggestions. We have updated Fig. 1 by simplifying its content and transferring some details to Fig. 2. Additionally, we have refined the corresponding caption to enhance readability and make the figure easier to understand.

W2. Section 3.2 is not very clear to me, I couldn't find a connection one ..., a more straightforward way is to directly ask the LMM ...

Thank you for your constructive suggestions. We agree that the proposed VQA-based approach can serve as an alternative for calculating text-image similarities. To evaluate its potential, we conducted experiments comparing the VQA strategy with our method in terms of text-image matching performance and efficiency.

a) Matching Performance: As shown in the following table, we try three likelihood-based approaches according to answers to estimate the alignment score and then perform text-to-image retrieval on Flickr30k.

Besides, we also make a comparison on the proposed TIGeR-Bench combining creative and knowledge-intensive scenarios, as shown in the following table.

These results demonstrate that the proposed proxy metrics are more superior than the VQA-based strategy, across multiple scenarios.

b) Efficiency: The VQA method belongs to the one-tower framework (e.g., BLIP-2), which suffers from low efficiency. As shown in the following table, the proposed method is more efficient than it.

In summary, using likelihood to estimate similarity has been studied in prior work [a, b]. The proposed proxies, such as log p(Y|X), estimate the similarities based on the likelihood of prompt X or/and image Y. In contrast, the VQA-based method estimate the similarities based on the likelihood of “yes” or “no”, e.g., log p(“yes” | X, Y). The above results show that our method is more effective compared with the VQA-based method which may suffer from instability or biases caused by question-answering instructions.

[a] Bengio et al. A neural probabilistic language model. JMLR’2003

[b] Lin et al. Revisiting the Role of Language Priors in Vision-Language Models. ICML’2024

W3.1. Given that the authors also agree that ... ,why not considering leveraging a contrastively trained text-to-image similarity function? The forward beam search + reverse re-ranking seems unnecessarily complicated.

Thank you for your insightful question and comment. The drawbacks of contrastive learning methods [c] have been discussed in the Introduction (line 076 – line 080) section.

Kindly refer to the Official Comment by authors titled "Key differences between TIGeR-One and dense retrieval methods based on contrastive learning" for more highlights, where we outline the advantages of our proposed method over approaches that leverage “a contrastively trained text-to-image similarity function”.

W3.2. Lack of simple baseline method: It would make a lot sense to compare ...

Thank you for your valuable suggestions. To ensure a fair comparison among single models, we do not consider the mentioned baseline initially. Following your suggestions, we conducted additional experiments, as shown in the table below. Using SDXL and CLIP as independent generation and retrieval models, respectively, we evaluated SEED-LLaMA and a stronger model, Qwen2-VL, as decision models.

The results show that with SEED-LLaMA as the decision model, CLIP-T improves, but CLIP-I underperforms compared to the single CLIP model. The stronger Qwen2-VL further enhances unified performance across both metrics, demonstrating superior discriminative abilities. However, our method still outperforms these newly added baselines, validating its effectiveness. Additionally, unlike the SDXL + CLIP + Qwen2-VL pipeline, our approach offers a unified framework capable of performing generation, retrieval, and decision simultaneously within a single LMM.