GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing

Q1: Probably should cite Gupta, et. al. "Visual Programming: Compositional Visual Reasoning Without Training", CVPR 2023.

A1: Thank you for your suggestion. This paper uses language models to generate visual programs for compositional visual tasks, which are then executed. In concept, this paper indeed shares some similarities with AI agent. The difference is mainly that our GenArtist focuses specifically on the field of image generation and editing. In addition to tool execution, we also introduce the verification and self-correction mechanisms to ensure the reliability of the results. In terms of planning, we propose the planning tree method, tailored to the specific characteristics of image generation. We will cite this paper in the related work section.

Q2: No comparison with other MLLMs.

A2:

Table 3-1: The peformance of GenArtist on T2I-CompBench, with various MLLMs.

Since it is relatively difficult for most open-source MLLMs to output in the user-specified format, which makes it inconvenient for the utilization of the system. For better results, we only utilize GPT-4V as the MLLM agent in the original paper.

To experiment with more MLLMs, we utilize LLaVA for multimodal verification, feeding its outputs into large language models such as LLaMA or Mixtral for generating specific planning results. In this way, we conduct experiments on T2I-CompBench with two more MLLM agents and list the results in the above Tab. 3-1. As can be seen, our method can be applied to various MLLMs, demonstrating its flexibility in agent selection. Although the performance with open-source MLLMs is not as high as with GPT4-V, it still shows a significant improvement compared to existing single models. This further demonstrates the effectiveness of GenArtist.

Q3: Contributions could be further refined and clarified.

A3: Thanks for your suggestion, we will refine and clarify our contributions in the introduction section of the final version.

Overall, we propose a new paradigm for the field of image generation and editing. Instead of utilizing a single model to directly generate the corresponding image, we delegate different functions to different models. An AI agent manages the utilization and sequence of these different models and performs verification and correction on the outputs. By leveraging the strengths of multiple models and the image understanding capability of the MLLM, our GenArtist significantly enhances the reliability of image generation and editing. We hope this innovative approach can provide new insights for future work in this field.

In terms of method design, compared to other existing agent-related methods, our framework is specifically tailored to the characteristics of the image generation and editing field. To address the challenges of complex prompts and the often unreliable generated images, we conduct decomposition and design a planning tree for task planning. Additionally, we introduce position information to enhance the spatial knowledge of the MLLM agent. These designs enable our GenArtist to more effectively tackle image generation related tasks, resulting in improved performance.

Q1: My major concern is about the possible limitation in the decomposition tree.

A1:

• Thank you for your suggestion. We have indeed considered this issue. Therefore, during verification, in addition to verifying the accuracy of the generated images, the agent is also required to assess their aesthetic quality (as illustrated in L173). If the overall quality of the generated image is poor, the agent will utilize different generation tools or choose different random seeds to regenerate the images, in order to ensure their overall quality. For instance, in Fig. 9 of our main paper, the agent regenerates the image due to the low quality of the initially generated image, thereby ensuring the aesthetic quality of the final generated image.
• By explicitly requiring the agent to maintain higher standards for the aesthetic quality of generated images, the overall quality of the generated images can be further improved. We apply this to regenerate the "hot dogs" case in Fig. 1, and list the generated image in the Fig. 1 of the rebuttal document. As can be seen, the image quality is further improved.
• The quality issue is indeed related to the utilized editing tools. The main reason is that most of the currently utilized editing tools are based on Stable Diffusion 2 or even v1.4. Due to the weaker based models, these editing tools are usually limited in keeping the overall image quality compared to larger generative models like SDXL. We can expect that as more powerful editing models emerge, we can replace the corresponding models in the current editing tool library and this issue will be alleviated.
• As an agent-centered system, our framework is also flexible in terms of human-computer interaction. During verification, human feedback can be appropriately integrated. By incorporating human evaluation and feedback on the overall quality of the images, the quality of the generated images can be further improved.

Q2: Can authors conduct more evaluations on more advanced benchmarks like DreamBench++?

A2:

Table 1-1: The performance on the DreamBench++ benchmark in the GPT score.

We conduct experiments on DreamBench++ and compare our results (in GPT score) with several tuning-free methods listed in the original DreamBench++ paper. Since DreamBench++ is primarily about single-object customization generation, where our current GenArtist framework includes only a few relevant tools, we also expand our tool library by introducing more customization tools, and then conduct the corresponding experiments. The comparative results are listed in the above Tab. 1-1.

As can be seen, the current version of GenArtist has already achieved the strong performance on DreamBench++, surpassing all listed tuning-free methods in concept preservation and outperforming most existing models in prompt following. After expanding the tool library, further improvement can be observed, particularly in prompt following. This demonstrates that our framework can achieve better results no matter for image accuracy or image quality on such a recent benchmark.

Q3: I think the paper organization could be improved by trying to put each full paragraph on one page or two.

A3: Thank you for your suggestion. We will enhance the paper organization by adding some additional explanations, such as introducing aspects about image quality, to ensure that sections 4.1 and 4.3 do not span across pages.

Thank you for your rebuttal efforts and response. The hot dog case does look better but it is still a little bit fake. However, this issue can be alleviated by using more advanced image editions and generative tools. I appreciate you for letting me know about the aesthetic quality correction procedure, which is helpful. I strongly suggest that the authors discuss this issue properly in the revised paper and incorporate all rebuttal experiments into the final paper. Besides, I suggest the authors use the latest GPT-4o instead of GPT-4V and conduct all experiments again since GPT-4o is a more advanced GPT tool to date, and I think it would be valuable to keep using state-of-the-art tools. All details, including the precise GPT-4o version (eg, turbo-2024-04-09), should be explicitly recorded in the paper.

Overall, I think my concern is resolved, and I will keep my rating for a supportive assessment for this paper.

Q1: The paper would be stronger if it performed a deeper analysis, for example, on common error cases, or via more finegrained ablation studies on the tools / positive-aware tool execution.

A1:

• error cases: We show two error cases in the Fig. 2 and Fig. 3 of the rebuttal document. As can be seen, sometimes, despite the agent correctly planning the specific execution of tools, the limitations of the tools themselves prevent correct execution, leading to incorrect results. For example, in Fig. 2 of the rebuttal document, it is required to add a very small blue cup. However, due to the lack of fine resolution ability in existing editing tools, the generated blue cup's size is inaccurate. In addition, as shown in Fig. 3 of the rebuttal document, errors in the output of localization tools can also affect the final result. For instance, when asked to remove the lettuce in the middle of a sandwich, the segmentation model fails to accurately identify the part of the object, leading to the erroneous removal operation.
• fine-grained ablation studies on the tools/position-aware tool execution: Regarding position-aware tool execution, we conduct the corresponding ablation study and list the results in the below Tab. 2-1. We evaluate the performance on the spatial and complex aspects of T2I-CompBench. As multimodal large models are usually not sensitive to position information, the performance is limited without the inclusion of position information, only a slight improvement over the tool selection results. After introducing position information, which enhances spatial awareness, there is a significant improvement in both the spatial and complex aspects. This validates the reasonability of our design.

Table 2-1: Ablation study on the position-aware tool execution on the spatial and complex aspects of T2I-CompBench.

Q2: It would also be stronger if the authors evaluated the system on additional benchmarks such as GenAI-Bench

A2:

Table 2-2: The performance of GenArtist on the 'basic' prompts of GenAI-Bench .

Table 2-3: The performance of GenArtist on the 'advanced' prompts of GenAI-Bench .

We conduct experiments on the GenAI-Bench and report the VQAscores on the basic and advanced prompts separately on the above Tab. 2-2 and Tab. 2-3. As can be seen, GenArtist achieves better performance in almost all aspects of this benchmark, further demonstrating the effectiveness of our approach.

Q3: the paper lacks some technical details about the method such as the underlying model.

A3: In our framework, we utilize GPT4-V as our MLLM agent. For our tool library, we have listed the utilized tools in the Tab. 1 of our original paper. For auxiliary tools, we utilize Grounding DINO as the object detector, SAM as the object segmentor, ControlNet utilized models for ControlNet-related auxiliary tools, and language-only GPT4 as the layout generator. We will also add more detailed introductions about these tools in the final version.

Q4: For reproducibility, which version of GPT4 was used?

A4: We use GPT4-V as our MLLM agent.

Q5: I wonder if the same issue exists in this system, and if so how the authors address such issues e.g. when the bounding boxes output by the localization/detection module are wrong?

A5: Regarding error cases, we analyze this issue in the error case section of the "A1" section above and include two cases in the rebuttal document. It can be seen that problems within the tool output or the localization module can lead to some erroneous outputs. Utilizing more powerful tools or incorporating some human feedback during the verification stage can effectively address this issue. For example, in the case shown in Fig. 3, using open-vocabulary segmentation models that are capable of recognition or integrating human feedback to adjust the segmentation mask can effectively address this issue.

Q6: table3: why is smart Edit missing L2 number?

A6: We directly copy the MagicBrush benchmark results from the original SmartEdit paper. Since the original paper only reports four other metrics, we do not include the L2 metric value here.

Dear reviewer, thank you for providing constructive feedback to the authors! Are your questions satisfactorily addressed by the rebuttal? Would you like to revise your rating or do you need any more information from the authors to make that decision?

Thanks for the response and additional evaluations! My questions have been addressed and I have no other concerns. I will keep my rating as it is. Please update the revision with additional technical details e.g. GPT4 version for reproducibility.