We sincerely thank Reviewer HZx9 for the constructive comments and insightful suggestions.

Evaluation on More General Datasets

Thanks for the reviewer's suggestion. We add the experiments with the ImageNet dataset, please check the results in the response for Reviewer 5Kbn. The results demonstrate that our RealRAG showcases strong performance on the general datasets.

Compared with Other RAG-based Methods

The reference mentioned by the reviewer in the Essential References Not Discussed section, referred to here as RDMs, and our RealRAG both utilize retrieval-augmented techniques in text-to-image generation. However, the research problems and objectives we aim to solve are entirely different.

• Research Purpose of RDMs: They use retrieval-augmented techniques to train or fine-tune a diffusion model and achieve out-of-distribution (OOD) image generation by switching databases.

• Research Purpose of RealRAG: RealRAG uses retrieval-augmented techniques to mitigate hallucinations when generating fine-grained objects and to enhance the ability to generate unseen novel objects, without requiring the training of generative models.

Out-of-Distribution Objects ≠ Unseen Novel Objects:

• Out-of-distribution objects: In the RDMs work mentioned by the reviewer, out-of-distribution is defined as data from outside the domain of the training dataset, such as "Angry Bird"(Knn-diffusion[1]) These objects existed during the generative model's training but were not part of the training set. As a result, the foundational generative model cannot generate them. RDMs use retrieval-augmented techniques, employing the CLIP model's similarity calculation ability to provide relevant references to the generative model, thereby solving the OOD problem.
• Unseen novel objects: These refer to objects that appear after the generative models and retrieval models are trained. The generative model cannot generate these objects, and the retrieval model can't easily retrieve relevant references via only similarity-based retrieval. This is a much more challenging issue.

For OOD object generation, existing state-of-the-art text-to-image generators can generate images from almost all domains, as shown in the link, and the original FLUX model can generate "Angry Bird" very well, outperforming the retrieval-augmented KNN-diffusion[1]. However, for unseen novel objects, such as the "cybertruck" (shown in Fig. 5 of the main paper), FLUX still cannot generate an accurate cybertruck. Therefore, the focus of this paper is on solving the problem of generating unseen novel objects.

Finally, although RDM is not part of the same type of research as our RealRAG, we still provide a comparison of their performance in the following table.

Our RealRAG shows significant performance gain, and we also show more visual comparison between the baseline and our RealRAG here link.

Computational Overhead Analysis

We present the retrieval time cost and the performance gain in the table below. The table shows that the average percentage increase in reasoning time is much lower than the percentage increase in performance. Our approach achieves significant performance gains over a limited increase in inference time.

Failure Cases

This work, like original FLUX and Stable Diffusion, has not been designed to target multi-object generation, therefore, RealRAG has some limitations in multi-object generation. Here we show some failure cases of multi-object generation link. It is our future work to implement multi-object generation.

We sincerely thank Reviewer 5Kbn for the constructive comments on our work. We are very grateful to the reviewer for recognising the novelty of our idea and the richness and rationality of our experiments.

About the Database

We are sorry for the misunderstanding. For the fine-grained object generation setting, we use the Real-object-based Database; For the unseen novel object generation setting, as shown on line 042, we use the images from the Internet (Google Images), which include the novel objects.

We show the results of using Internet data in the Fig. 5 in the main paper.

Evaluation on ImageNet

Thanks for the reviewer's suggestion. We add the experiments with the ImageNet.

Our RealRAG demonstrates significant performance gains for the SoTA t2i generative models on the ImageNet dataset. Furthermore, our insight is to reduce hallucinations of the fine-grained object generation (eg, various types of cars) and the classes in ImageNet are coarse-grained (eg, cars and planes). Therefore, the selected datasets (eg, Stanford Car) are more suitable for the evaluation.

More clarification about Image-conditioned generation

So sorry for the lack of a relevant background introduction. Several methods have been proposed for inputting images into diffusion models. These include: 1) Embedding-based approaches, where image embeddings from CLIP are concatenated with timestamp embeddings in the UNet of diffusion models [Ref A, B]; 2) Input Layer Concatenation-based approaches, in which the latent representation of the input image is concatenated to the UNet's input, changing its input dimension from 4 to 8, with the additional input layer initialized to zero [Ref C]; 3) ControlNet-based approaches, where ControlNet [Ref D] introduces a branch to stable diffusion, enabling the inclusion of additional inputs; and 4) Noise-based approaches, which involve image editing methods [Ref E] that add noise to the input image, followed by denoising.

We use the version of our Reflective Retriever + FLUX as an example. First, we retrieve and sort the closest images. Next, we input the selected images into the FLUX’s ControlNet branch to control specific elements during the image synthesis process. The FLUX version is black-forest-labs/FLUX.1-Canny-dev, which allows for inputting a referenced image to guide the generation process. For further details, please refer to the code provided in the supplementary materials.

In general, equipped with our Reflective Retriever, we can significantly enhance the prompt-image consistency and feasibility.

[Ref A] https://huggingface.co/lambdalabs/sd-image-variations-diffusers.

[Ref B] https://huggingface.co/stabilityai/stable-diffusion-2-1-unclip.

[Ref C] Zero-1-to-3: Zero-shot one image to 3d object.

[Ref D] Adding Conditional Control to Text-to-Image Diffusion Models.

[Ref E] Sdedit: Guided image synthesis and editing with stochastic differential equations.

We sincerely thank Reviewer vQw4 for the constructive comments on our work. We promise to revise the paper based on the comments. We will cite and discuss all the papers in the "Essential References Not Discussed" in the final version

About the "first work" claim and the novelty.

The reference mentioned by the reviewer in the Essential References Not Discussed section, referred to here as RDMs, and our RealRAG both utilize retrieval-augmented techniques in text-to-image generation. However, the research problems and purposes we aim to solve are entirely different.

• Research Purpose of RDMs: They use retrieval-augmented techniques to train or fine-tune a diffusion model and achieve out-of-distribution (OOD) image generation by switching databases.
• Research Purpose of RealRAG: RealRAG uses retrieval-augmented techniques to mitigate hallucinations when generating fine-grained objects and to enhance the ability to generate unseen novel objects, without any training of generative models.

There are three key points to pay attention to:

1. Out-of-Distribution Objects ≠ Unseen Novel Objects:

   • Out-of-distribution objects: In the RDMs work mentioned by the reviewer, out-of-distribution is defined as data from outside the domain of the training dataset, such as "Angry Bird." In Knn-diffusion[3]. These objects existed during the generative model's training but were not part of the training set. As a result, the foundational generative model cannot generate them. RDMs use retrieval-augmented techniques, employing the CLIP model's similarity calculation ability to provide relevant references to the generative model, thereby solving the OOD problem.
   • Unseen novel objects: These refer to objects that appear after the generative models and retrieval models are trained. The generative model cannot generate these objects, and the retrieval model can't easily retrieve relevant references by similarity. This is a much more challenging issue.

   For OOD object generation, existing state-of-the-art text-to-image generators can generate images from almost all domains, as shown in the link, and the original FLUX model can generate "Angry Bird" very well, outperforming the retrieval-augmented KNN-diffusion. However, for unseen novel objects, such as the "cybertruck" (shown in Fig. 5 of the main paper), FLUX still can't generate an accurate cybertruck. Therefore, the focus of this paper is on solving the problem of generating unseen novel objects.

2. Hallucination Issue When Generating Fine-Grained Objects:

   Existing SoTA t2i models are pre-trained on large-scale text-image paired datasets. As a result, they tend to produce hallucinations, such as inaccuracy or unrealistic features, when generating fine-grained objects. This is a problem inherent to large generative models.

3. Application Value:

   For existing commercial text-to-image generative models, training them is cost-prohibitive. Therefore, a pipeline is needed to integrate real-time updated data from the internet into the generative model without any retraining, enabling the generation of unseen novel objects. On the other hand, in specific application scenarios such as advertising or poster creation, users need generated images that meet their design requirements, while also ensuring that products (fine-grained objects) within the image remain realistic. This requires generative models to have the ability to generate both open-domain and fine-grained objects. Therefore, RealRAG focuses on reducing hallucinations in large t2i generators through RAG technology, enabling open-domain generators to generate specific fine-grained objects.

Evaluation on ImageNet.

Thanks for the reviewer's suggestion. Please check the results in the rebuttal for Reviewer 5Kbn.

More Baselines

Please check the results in the response for Reviewer HZx9. We also show more visual comparison here link.

About the Database

We are sorry for the misunderstanding. For the fine-grained object generation setting, we use the Real-object-based Database. For the unseen novel object generation setting, as shown on line 042, we use the images from the Internet (Google Images), which include the novel objects.

More User Study

Thanks for your suggestions, we try our best to add more cases and invite more participants for the user study. Finally, we extant the scale of the user study to 50 participants and 20 cases. The results can be found at the link.

About the trade-off between performance and the inference time.

We present the comparison results in the response for Reviewer HZx9. The table shows that the average percentage increase in reasoning time is much lower than the percentage increase in performance.