Guiding Instruction-based Image Editing via Multimodal Large Language Models

We thank reviewer TXUn for acknowledging our practical effectiveness.

Q) typographical errors

Thanks for the debugging! We have updated all Tables in our revision.

Q) transferring texture/color/emotion

We attempt to perform such transfers in Sec. A (Transferring Image Texture/Color/Emotion). For texture, we follow CLVA [1] and adopt the style prompt "make the whole image as texture [ins]". InsPix2Pix can only do limited transfer, but MGIE shows clear visual attributes (e.g., "orange" or "pinkish") as well as the complex "colorful circular round". We perform fine-grained color manipulation, including "glasses frame" or "hair". However, the baseline even alters the whole color. For global colorization [2], both InsPix2Pix and our MGIE cannot present appealing results, which indicates the need for fine-tuning. Transferring the emotion is more challenging as the model has to perceive the latent semantics. We are able to illustrate the visual concept of "bright day" or "chaotic and confused" as the beach in the early morning or the gloomy street at night. MGIE can also transform from the cozy snowy day into suspenseful and thrilling through "nightmare and scared". Although exhibiting promising potential, it still requests more profound texture/emotion perception for each specific goal. We leave them as future research for creative visual editing [3].

• [1] (ECCV'22) Language-Driven Artistic Style Transfer
• [2] (NeurIPS'23) L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
• [3] (ICCV'23) Affective Image Filter: Reflecting Emotions from Text to Images

We thank reviewer RXEx for enjoying our proposed method.

Q) unseen editing operation

We first want to clarify that MGIE is not limited by the training dataset. In IPr2Pr, there is no removal or photo optimization. We have appended such cases in Sec. A (Unseen Editing Operation). InsPix2Pix has failed due to the shortage of training examples. In contrast, our MGIE is able to handle such editing via the visual-aware derivation of MLLM. We can accurately remove "the boy in the red shirt" or "lighten out the yellow tone", which demonstrates better generalizability for unseen operations. More qualitative comparisons can be found on our project website (https://mllm-ie.github.io).

Q) ICLR template

Thanks for the notification! We have updated the ICLR 2024 template in our revision.

Q) ablation study of training loss

We have added the study in Sec. A (Ablation Study of Training Loss). There are two training losses, instruction loss $L_\text{ins}$ and editing loss $L_\text{edit}$, in our MGIE. $L_\text{edit}$ is necessary for training to produce the editing result. Without $L_\text{ins}$, it will derive full but lengthy guidance to lead $L_\text{edit}$. However, both LGIE and MGIE drop significantly; LGIE even performs worse than the baseline. This underscores the prominence of learning concise expressive instructions, which offer succinct and relevant guidance. Besides, lengthy instructions via the MLLM will incur additional overhead (29.4 vs. ours 9.2), resulting in an inefficient inference.

Q) editing effects are not ideal

We also observe these artifacts. Though the editing results are not perfect, MGIE can effectively remove the target, compared to the failed InsPix2Pix. Perhaps a step-by-step pipeline (e.g., removal followed by inpainting) could deal with such issues.

Q) balance between $L_\text{ins}$ and $L_\text{edit}$

As Eq. 6, the overall training loss is $L_\text{ins} + 0.5 \cdot L_\text{edit}$. We have tried different weights (e.g., 1.0), but the performances are all similar.

Q) add new objects

We have appended such examples in Sec. A (Adding New Object). MGIE also supports adding new objects that are not present in the input and placing them in reasonable positions. For example, the "hat" is put on the girl's head, and the "river" is added along with the grass. More surprisingly, the appended "fireworks" further makes the beach colorful, which drives the night scene coherent and visually appealing.

Q) details of user study

We have updated the details in Sec. B (Human Evaluation). We sample 100 examples (25 for each dataset) to conduct our human evaluation. Each task is assigned 3 annotators, who rank across baselines and our MGIE, to avoid potential bias. We require workers to have a 97% approval rate and over 500 approved tasks to ensure quality. The worker is awarded $5 for each task (5 examples) and takes 21 minutes on average to complete.

We thank reviewer fMJn for appreciating the conducted results and our paper writing.

Q) technical contribution

We first want to highlight that our deriving human instructions into informative descriptions is a brand-new attempt, which yields surprising enhancements. Though our MGIE is a joint framework with the MLLM and the diffusion model, we want to emphasize that our contribution includes:

• an end-to-end pipeline to learn/incorporate explicit yet concise guidance;
• an investigation on how to utilize expressive instructions (Table 3);
• a comprehensive automatic/human study covering diverse editing aspects.

All of these are valuable contributions to controllable image editing research.

Q) intermediary on-the-fly remappings

We believe that deriving expressive instructions is crucial for enhancing image editing. In Table 5, we demonstrate that replacing the CLIP encoder with powerful LLMs only results in limited improvement. Since LLMs are trained unidirectionally (decoder-only), the hidden states will lack sufficient/concrete information unless we derive the subsequent narration. On the other hand, expressive instructions can also reveal how the model modifies the input image, aiding in the interpretation of the editing result.

Q) part-of-speech distribution

We investigate part-of-speech (POS) distributions of input instructions and our derived expressive instructions in Sec. A (Part-of-Speech Distribution). In general, input instructions involve more nouns but fewer adjectives. In contrast, our expressive instructions can portray concrete edited scenes in detail via more adjectives. The original instructions are also dominated by verbs, which are challenging to perceive. The derivation helps them to be more understandable as adverbs. Moreover, we effectively decrease the number of ambiguous pronouns. More than 68% of pronouns (only 13% in our expressive instructions) are unresolvable in input instructions, where the model can not have explicit goals.

Q) suggestions

Thanks for the notification! We have updated the ICLR 2024 template as well as "Instruction Prompt-to-Prompt" for IPr2Pr in our revision. We also add the caption (overview architectures of baselines) to Figure 4.

Q) code release

Yes. We will release the code/checkpoints after the review process. We also attach a copy of our codebase in the appendix.

Q) elaborate descriptions

Sorry for the misunderstanding. Those previous methods (e.g., Text2Live [1] and Null-Inv [2]) require human-provided descriptions, which are labor-intensive and not intuitive, compared to edit instructions. For example, to manipulate an image of "a girl is walking at the beach". The user needs to specify "a girl with a hat is walking at the beach", instead of the more straightforward "give her a hat". In contrast, our MGIE can automatically derive the concrete guidance "the girl will wear a hat and walk at the beach", where elaborate descriptions are no longer necessary in this scenario. This can significantly enhance the accessibility of the edit instruction.

• [1] (ECCV'22)Text2LIVE: Text-Driven Layered Image and Video Editing
• [2] (CVPR'23) Null-text Inversion for Editing Real Images using Guided Diffusion Models

Q) reasonable image editing

We call it "reasonable" since our model can follow the reasoning from the MLLM. MGIE leverages the derived expressive instruction to perform related image manipulation, where the guidance from the MLLM can offer a reasonable editing goal through powerful language modeling. For example, "make it more healthy" for a pizza can be perceived as "need more vegetables"; "add fireworks" will alter not only the sky but also the reflection on the ground. We calculate the CLIP-Score between edited results and their input/expressive instructions. Our MGIE achieves better alignment than the baseline and shows an even higher correlation with expressive instructions. This indicates that we indeed follow the MLLM and benefit from its reasoning capability. Looking ahead, one potential direction is to adopt the recent GPT-4V [1] to evaluate the actual reasonableness.

• [1] (arXiv'23) The Dawn of LMMs: Preliminary Explorations with GPT-4V(vision)

Q) flame/ice icons

Thanks for the nice suggestion! We have added it to the caption of Figure 2.

We thank reviewer BZAL for the valuable feedback and underscoring our promising quantitative/qualitative study.

Q) novelty of MGIE

Thank you for pointing this out! Our MGIE is motivated by GILL [1], and we have made the citation clear in our revision. Despite adopting a similar framework, we are the first to edit an existing image via the MLLM+Diffusion pipeline, rather than creating a new but distinct one. We also want to highlight our deriving human instructions into informative descriptions is a novel attempt at controllable image editing. For the experiments, we investigate the best way to utilize our derived expressive guidance and conduct a comprehensive study of diverse editing aspects, all of which are valuable contributions to this research field.

• [1] (NeurIPS'23) Generating Images with Multimodal Language Models

Q) presentation of the model and experimental setting

Thanks for the constructive feedback! We have appended the detailed experimental setup in Sec. B (Edit Head / Editing Loss). For the MLLM, we first want to clarify that our framework is model-agnostic and compatible with various architectures (as long as they can take visual inputs and provide responses through language modeling). As we are using LLaVA [1] in our implementation, we have made it clearer in Eq. 1. We also added the explanatory footnote of MLLM* to the same page as Figure 2.

• [1] (NeurIPS'23) Visual Instruction Tuning

Q) description-based baselines

We have involved the results of description-based baselines (TEXT2Live [1] and Null Text Inversion [2]) in Sec. A (Comparison to Description-based Baselines). We leverage GIT [3] to caption the input image as its input description and ChatGPT to merge the edit instruction as the goal description via the prompt "Combine two sentences A: [description] and B: [instruction] into a single sentence. The output should be at most similar to sentence A". For instance, "a girl is walking at the beach" and "give her a hat" will be merged into "a girl with a hat is walking at the beach". For MagicBrush, we directly apply their released descriptions instead. Text2LIVE and Null-Inv only yield feasible results on the traditional L1 distance but are obviously inferior to our MGIE on semantic-level evaluations (e.g., lower CVS), which supports that they cannot present concrete editing results and carry out goal descriptions well. On the other hand, both count on inference optimization (CLIP alignment and DDIM inversion), which takes more than 200 seconds (vs. ours 9.2 seconds) for each editing task.

• [1] (ECCV'22) Text2LIVE: Text-Driven Layered Image and Video Editing
• [2] (CVPR'23) Null-text Inversion for Editing Real Images using Guided Diffusion Models
• [3] (TMLR'22) GIT: A Generative Image-to-text Transformer for Vision and Language

Q) multiple random seeds

Thanks for the nice suggestion! We have updated the empirical results in our revision, where all evaluations are averaged from 5 random seeds. The statistics variance are shown below. The rankings and observations are still consistent with our previous conclusions.

Q) evaluation metrics

Yes. All ground-truth goal images are available in EVR, GIER, MA5k, and MagicBrush. For Photoshop-style modification (EVR and GIER), we adopt semantic-level visual similarity DINO and CVS. Since there are many photo optimization cases in GIER, we also consider SSIM. MA5k aims to adjust the contrast/brightness/saturation, where DINO and CVS cannot clearly tell the difference; Hence we leverage the widely-used style distance LPIPS. As we have the ground-truth description in MagicBrush, we follow the additional CTS to evaluate the alignment between goal captions and edited images.

Q) FID score

We calculate the FID score between ground-truth and edited images in Sec. A (Evaluating Image Editing via FID). However, the differences are all pretty limited. Since most editing results still resemble the original input images, it is difficult for FID to determine their authenticity. These results indicate that FID is insufficient to compare the quality of image editing.

Q) training cost

We updated our training cost in Sec. B (Training Cost). Our MGIE training requires 26 epochs to converge, and InsPix2Pix has 20 epochs (from their released checkpoint). Both MGIE and InsPix2Pix take a similar 1.6 hours per epoch on our node (8 NVIDIA A100 GPUs), where the overall training can be done in two days.