ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer

Dear Reviewer 7X1X,

Thank you for acknowledging the contributions and presentations. We address your concerns as follows:

---

Q1: Discussion on Model Efficiency and Scalability

We incorporate more information about computational efficiency during training and inference in the "Supplementary Material, Section IMPLEMENTATION DETAILS".

In general, the training and inference efficiency is related to visual sequence length and input image number ("SubSection Computational Efficiency"). This is the reason why we training the model with a multi-stage training strategy, which is training with a small visual sequence length and fewer input images at first and increasing the length and the number of input images in the following stages. (see in Fig.28-a)

We further conduct evaluations of our intermediate model checkpoints on the MagicBrush benchmark to evaluate the impact of data scale in the "SubSection Checkpoints Evaluation". Generally, the model’s performance improves when trained with more data. (see in Fig.28-b)

---

Q2: In-depth Analysis of Specific Tasks

We conduct detailed quantitative comparisons with specifically designed state-of-the-art methods for key tasks: facial editing and local text rendering, please refer to "Supplementary Material, Section MORE EXPERIMENTS, SubSection Facial Editing and Local Text Render" for the details. More qualitative comparisons for inpainting and controllable generation are added to "Supplementary Material, Section MORE EXPERIMENTS, SubSection More Qualitative Comparison".

---

Q3: Data Annotation Quality

As shown in Fig. 4, we used Qwen-VL for the initial construction of instructions and obtained more accurate instruction descriptions through manual annotations. We then utilized this high-quality data for training the InternVL model. It is worth noting that the entire process is conducted in an iterative update manner, allowing us to continually refine our annotated data and iteratively enhance the performance of the fine-tuned InternVL. Furthermore, based on our sampling evaluation, the accuracy of the model's instruction annotations has reached over 92 %, which is sufficient for use as training pair data. For further improvement in annotation accuracy, it is necessary to address the deficiencies in detailed image descriptions within the multimodal model itself, which is also a challenge faced by current pure image labeling models.

---

Q4: Enhancing Model Interpretability

In "Supplementary Material, Section ARCHITECTURE DESIGN", we outline our key architectural design: Long-context Attention Block and Image Indicator Embeddings, as well as a qualitative analysis of these components. This may partially explain the model's functionality. We also visualize the editing process by decoding intermediate model outputs during the de-noising process and try to explain the model's behavior at each step in "Supplementary Material, Section IMPLEMENTATION DETAILS, SubSection Visualization of Editing Process.". When we use the instruction “Let a car appear in {image}.” to edit the image, the model identifies the area to be edited in the initial steps and subsequently copies the unchanged regions from the input image in the following steps. In the steps leading up to the final stage, additional details are incrementally added to the edited area until completion.

---

Dear Reviewer 7X1X,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission.

We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns. Please let us know if you have any remaining questions or require additional clarification.

Best regards,

Authors of Submission 8711

Dear Reviewer bd3V,

Thank you for acknowledging the proposed method and experiments. We address your concerns as follows:

---

Q1: Architectural Design

Thank you for your suggestions. We have outlined the key architectural design in the "Supplementary Material, Section ARCHITECTURE DESIGN".

Our model is anchored by two principal components: the Long-context Attention Block and Image Indicator Embeddings. The Long-context Attention Block is specifically designed to handle input image sequences of varying lengths, addressing the limitations of the conventional attention block employed in DiT, which cannot effectively manage sequences of disparate lengths. Meanwhile, the Image Indicator Embeddings facilitate the alignment between images in the sequence and their corresponding mentions within the text prompt. A qualitative analysis of these components is also provided in the ARCHITECTURE DESIGN Section.

---

Q2: Results in Tab. 2

Unlike previous methods that primarily focus on limited editing tasks, our approach is designed as a foundational editing model capable of addressing a broad spectrum of editing tasks. Consequently, leveraging large-scale data is a logical choice to enhance our model's performance. Moreover, most existing edit models are derived from foundational text-to-image generation frameworks, which have been trained on extensive data. Therefore, conducting a fair comparison under equivalent training data conditions is not feasible.

---

Q3: Replacement of Text Encoders

In the "Supplementary Material, Section DISCUSSION, SubSection Future Work", we also mentioned that introducing LLMs or MLLMs could potentially help us better understand user intentions within general instructions.

Regarding the design of text encoders, there are currently two design approaches. One is based on diffusion models specifically designed for text-to-image tasks, such as the MagicBrush and CosXL methods that utilize models from the SD series; these models primarily rely on CLIP or T5 for text encoding. The other approach is based on multimodal models, such as Llama and Phi, which are pre-trained to acquire generative and editing capabilities, like Seed-X, Seed-X Edit methods. The former naturally has the capacity to generate high-quality images, while the latter exhibits superior semantic understanding. Our future exploration will focus on how to combine and enhance the strengths of both approaches.

---

Q4: Public Content

We will make the model, training code, inference code, chatbot, and evaluation benchmark publicly available. However, due to organizational policies, we are unable to disclose the training data.

---

Dear Reviewer bd3V,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission.

We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns. Please let us know if you have any remaining questions or require additional clarification.

Best regards,

Authors of Submission 8711

Dear Reviewer NXSb,

Thank you for acknowledging our framework. We address your concerns as follows:

---

Q1: Task Interdependence

Thank you for your suggestions. We have added a discussion on task interdependence in the "Supplementary Materials, Section IMPLEMENTATION DETAILS, SubSection Task Interdependence". The main details are as follows:

In an all-in-one visual generation model, there exist multiple interactions between tasks, similar to that in large language models, and this relationship can be viewed as a complex balancing action.

i) Complementarity between tasks: The combined influence of various tasks can lead to a certain degree of generalized behavior across tasks. For instance, in the style transfer task, our prepared data and training process focus on pixel-aligned global image transfer. However, by incorporating learnings from other tasks related to mask guidance or subject guidance, the model can acquire the ability to perform style transfer in localized areas. (as in Fig. 29)

ii) Competition between tasks: As the scale of tasks increases, the potential for competition also grows, particularly in scenarios where user instructions are ambiguous. For example, when adding the text "APPLE" to an image, it is essential to specify that it is text to be added; otherwise, due to semantic ambiguity, the result may instead involve the addition of an object depicting an apple. (as in Fig. 29)

To achieve optimal performance balance, we first focus on adjusting the data sampling rates for each task in a phased manner during the training process, monitoring this through a validation set. Additionally, more detailed descriptions of instructions are needed in the preparation of training data to prevent semantic confusion between tasks. Through these methods, we aim to ensure that the model can fully leverage the complementarity between different tasks while controlling for any potential negative impacts.

However, the relationships between different tasks still require further exploration to better optimize the model's performance. Future work will also focus on how to effectively evaluate and adjust these influencing factors to achieve a more balanced and comprehensive execution of tasks.

---

Q2: Task Selection

We aim for ACE to encompass all visual generation tasks as possible, which is why we have not conducted a unified consideration specifically focused on reasonable tasks. Specifically, we propose the LCUs to unify various modal conditions, enabling the model to be compatible with different tasks. Additionally, if new visual generation tasks need to be supported, they can also be processed and fine-tuned accordingly through this paradigm.

---

Dear Reviewer vndP,

Thank you for your time and suggestions. We address your concern as follows:

---

Q1: About Drawing

Thank you for your feedback. We hope that the chosen text and colors can help readers better understand the content without causing any misunderstandings. Throughout the paper, we have adopted a consistent font and specific shades (blue, yellow, and green) as the baseline pattern, while using lighter and darker corresponding shades as accents to ensure visual harmony and aesthetics. All figures adhere to this principle as much as possible.

Additionally, Reviewer vGPZ and bd3V believe that our presentation is good, while Reviewer 7X1X considers it excellent and further notes that "it excels in writing and figure drawing, with clear diagrams and rigorous logic, providing an excellent reading experience for the audience". We will continue to strive to improve our visual presentation and appreciate your understanding.

---

Q2: ACE vs. Other All-in-One

ACE is an All-round Creator and Editor, that supports a wide range of visual generation tasks, including 8 basic types: Text-guided Generation, Low-level Visual Analysis, Controllable Generation, Semantic Editing, Element Editing, Repainting, Layer Editing, and Reference Generation.

The low-level tasks you mentioned are only a very small part of what we focus on. Furthermore, the Low-level Visual Analysis described in the manuscript includes Image Segmentation, Depth Estimation, Human-pose Estimation, Image Mosaic, Image Degradation/Super-Resolution, Image Grayscale, Edge Detection, Doodle Extraction, Contour Extraction, and Scribble Extraction. I have not been able to find any existing work that uses a single model to handle all of these tasks.

In addition to handling low-level tasks, we also have many other applications, such as basic text-to-image generation, comprehensive controllable generation, instruction-based editing, reference-guided generation, and long-context-guided generation. To support all of these functions, an all-in-one approach needs to be redesigned and cannot be directly referenced from the low-level domain you mentioned.

There were some all-in-one controllable generation methods, such as Uni-Controlnet[1] and Controlnet-Union-SDXL[2]. However, these methods merely provide a controllable generation all-in-one model from a unified perspective, which only accounts for a part of our tasks. Furthermore, we also compared several general editing methods, as detailed in Fig. 5, including IP2P[3], MagicBrush[4], CosXL[5], SEED-X[6], and UltraEdit[7].

As you mentioned, this is a lot of work, as we need to manage the data for various tasks separately, especially since acquiring data for the more advanced tasks is much more challenging compared to low-level tasks ("Supplementary Material, Section DATASETS DETAIL"). At the same time, we have to design a unified method that covers these tasks.

We appreciate your feedback and look forward to it.

[1] Zhao et al. "Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models." NeurIPS 2023.

[2] xinsir, et al. "controlnet-union-sdxl-1.0." Hugging Face.

[3] Brooks et al. "InstructPix2Pix: Learning To Follow Image Editing Instructions." CVPR2023.

[4] Zhang et al. "MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing." NeurIPS 2023.

[5] StabilityAI. "CosXL." Hugging Face.

[6] Ge et al., "SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing." arXiv 2024.

[7] Zhao et al. "UltraEdit: Instruction-based Fine-Grained Image Editing at Scale." arXiv 2024.

---

Q3: Further Information

For more information, please refer to the "Supplementary Material". In this material, we have dedicated a significant amount of space to provide a more detailed description of the methods and to showcase additional visual results.

Regarding the computing resources you mentioned, we used A800 as the base hardware and dynamically adjusted the quantity used according to the training to meet the expected batch size. For further details, please refer to "Supplementary Material, Section IMPLEMENTATION DETAILS". Additionally, related work can also be found in "Supplementary Material, Section RELATED WORK".

---

Dear Reviewer vndP,

We hope this message finds you well. We sincerely appreciate the time and effort you have dedicated to reviewing our submission.

We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns. Please let us know if you have any remaining questions or require additional clarification.

Best regards,

Authors of Submission 8711

Dear Reviewer vGPZ,

Thank you for acknowledging our contributions and your valuable comments. We address your concern as follows:

---

Q1: Formatting

Thanks. We have completed the corrections in the manuscript.

---

Q2: Task Interdependence

Thank you for your suggestions. We have added a discussion on task interdependence in the "Supplementary Materials, Section IMPLEMENTATION DETAILS, SubSection Task Interdependence". The main details are as follows:

In an all-in-one visual generation model, there exist multiple interactions between tasks, similar to that in large language models, and this relationship can be viewed as a complex balancing action.

i) Complementarity between tasks: The combined influence of various tasks can lead to a certain degree of generalized behavior across tasks. For instance, in the style transfer task, our prepared data and training process focus on pixel-aligned global image transfer. However, by incorporating learnings from other tasks related to mask guidance or subject guidance, the model can acquire the ability to perform style transfer in localized areas. (as in Fig. 29)

ii) Competition between tasks: As the scale of tasks increases, the potential for competition also grows, particularly in scenarios where user instructions are ambiguous. For example, when adding the text "APPLE" to an image, it is essential to specify that it is text to be added; otherwise, due to semantic ambiguity, the result may instead involve the addition of an object depicting an apple. (as in Fig. 29)

To achieve optimal performance balance, we first focus on adjusting the data sampling rates for each task in a phased manner during the training process, monitoring this through a validation set. Additionally, more detailed descriptions of instructions are needed in the preparation of training data to prevent semantic confusion between tasks. Through these methods, we aim to ensure that the model can fully leverage the complementarity between different tasks while controlling for any potential negative impacts.

However, the relationships between different tasks still require further exploration to better optimize the model's performance. Future work will also focus on how to effectively evaluate and adjust these influencing factors to achieve a more balanced and comprehensive execution of tasks.

---

Q3: Details of Data Processing

We provide a detailed analysis and parameter description regarding this issue in the "Supplementary Materials, Section IMPLEMENTATION DETAILS, SubSection Data Preprocessing Details.".

In the data preparation stage, our main considerations for parameter selection are computational efficiency and ensuring high data relevance. The designed hierarchical aggregation pipeline for pairing content-related images involves clustering, identifying first-level disjoint sets, and determining second-level disjoint sets.

Initially, data is clustered into 10, 000 clusters using K-means clustering based on SigLip features, allowing the Union-Find algorithm to be executed more efficiently by keeping the data scale under 100K one node in the parallel execution. First-level disjoint sets are formed by analyzing the similarities of SigLip features within these clusters, using a SigLip similarity matrix and thresholds for data pruning to ensure strong internal connections. Second-level disjoint sets are established through task-specific correlations, with specialized models used for various tasks such as background alterations or ID preservation, applying different similarity measures and thresholds to maintain the necessary correlation. This process utilizes advanced data mining and correlation models tailored to specific tasks, employing techniques like binary classification with the ViT-B-16-SigLIP and cosine distance for facial features.

---