AutoEdit: Automatic Hyperparameter Tuning for Image Editing

Thank you for your valuable review. We will respond to your concerns as follows:

Weakness 1: Notation confusion

Thank you for your detailed review. In Figure 1, the inversion timestep should be replaced by $r$ in DDPM Inversion; however, $r$ is cross cross-attention ratio in the P2P method. We will correct Figure 1 and provide a clearer notation in the revised paper.

Weakness 2: Suboptimal timestep in the Fig. 1

Thank you for your valuable feedback. In editing tasks, the perception of the "best" edited image can be subjective and may vary from person to person. We selected that particular image as the optimal result based on a user survey, in which the majority of participants preferred it over the image generated at timestep t=30.

Weakness 3: Reconstruction problem in DDIM Inversion:

Our proposed framework does not aim to solve the reconstruction error problem inherent to inversion methods, as this is closely tied to the design of the inversion process itself. Instead, we focus on enhancing the quality of edited images within a given inversion-based editing method by selecting optimal hyperparameters. Consequently, in the case of DDIM-Inversion editing, our approach achieves improved background preservation performance (see Table 1).

Weakness 4: Minor issues

Thank you for your detailed review. We will address the issue in line 90 and include the PPO preliminaries in the revised version of the paper. Additionally, we will revise the redundant terms to improve clarity and reduce potential confusion for readers.

Question 1: The authors employ DDIM inversion, which inherently introduces errors. However, the paper does not address how these errors are mitigated or their impact on editing quality.

While DDIM inversion does introduce reconstruction errors, selecting a more suitable editing start time $r$ for each specific case can help better preserve background information while still achieving high editing quality. Our reinforcement learning framework assists in automatically selecting the optimal hyperparameters to effectively balance the trade-off between background reconstruction and prompt alignment.

Question 2: The study only considers two hyperparameters—inversion timesteps and attention-replacement ratios—while omitting other critical ones, such as the Classifier-Free Guidance (CFG) scale (typically defaulted to 7.5):

Thank you for your feedback. We also conducted an experiment using AutoEdit to adaptively search for the optimal CFG scale at each timestep. The results are presented in the following table, with the baseline's default CFG scale set to 7.5.

Experimental results show that our method significantly improves background preservation compared to the baseline, while only sacrificing a small amount of CLIP score. Moreover, our proposed AutoEdit framework boosts LLM-based evaluation metrics—where editing performance is assessed through the judgment of a large language model on the edited images—demonstrating the effectiveness of AutoEdit in searching for optimal editing hyperparameters.

Question 3: The claimed "best editing" results lack clear evaluation metrics in Figure 1. For instance, when editing the bird image at inversion timestep=40, the bird’s posture shifts, and the branches are overly edited. Subjectively, the t=30 result appears visually superior, yet the paper does not justify the preference for t=40.

Please refer to the answer to weakness 2.

Thank you for your valuable review. We will respond to your concerns as follows:

Weakness 1: Concern about the limited search space

We acknowledge that hyperparameters can be selected from a predefined range. However, for different images, it is still necessary to identify the optimal value within that range. When dealing with a large number of images, this selection process can become time-consuming, even within a constrained range. Moreover, the editing results can vary significantly depending on the specific values chosen within that range. The complexity analysis is based solely on theoretical assumptions. Empirically, we also provide an experiment (see Table 4 in the main paper), which shows that, in practice, our proposed AutoEdit method is approximately equivalent to just three rounds of trial-and-error using the conventional approach.

Weakness 2: Concern about the reward function for several editing tasks

• For style transfer editing, we omit the background preservation terms in our reward computation, consistent with the evaluation protocol of the Pie-Bench dataset.
• Regarding viewpoint changes, this task is more akin to 3D reconstruction than image editing, and thus falls outside the scope of our work. In most viewpoint-change scenarios, 3D-aware diffusion models or score distillation methods are preferred. In such cases, reinforcement learning could potentially be integrated into the training process to produce more consistent viewpoint generation.

Weakness 3: AutoEdit with training-based approaches:

Training-based editing methods can be regarded as specialized generation pipelines that condition on both the original image and the instruction prompt, rather than relying solely on text-to-image models. For example, in [1] and [2], our reinforcement learning framework can be employed to optimize the classifier-free guidance (CFG) coefficient during the denoising process.

Weakness 4: CLIP score function is not a reliable measure

We acknowledge that relying solely on the CLIP score may be insufficient for capturing fine-grained edits, such as changes in small or detailed regions (e.g., eye color). To address this limitation, we incorporate large language model (LLM)-based evaluation to assess editing quality. Specifically, we employ a multi-step approach using GPT-4o to generate and compute rewards as follows:

• Step 1: Given an editing instruction (e.g., "change the eye color to blue"), we prompt GPT-4o to generate a corresponding question (e.g., yes/no or multiple-choice) along with a ground truth answer. For example: Question: "What is the color of the eye?" Choices: ["black", "blue"]; Answer: "blue".
• Step 2: To compute the reward, we query the LLM for both foreground editing and background preservation. For foreground editing, we use the question and ground truth answer from Step 1 and require GPT-4o to identify the correct choice. For background preservation, we ask the LLM to compare the edited image with the original and assign a score (0, 0.5, or 1) based on the quality of background consistency. The total reward is the sum of these two components.

We conduct experiments using SDXL with DDPM-Inversion, tuning the inversion timestep based on the newly introduced LLM-based reward. The following table presents a comparison between our original CLIP-based reward and the new LLM-based reward, with the LLM score being the GPT-4o evaluation score, which is similar to the computation of the reward function:

Experimental results show that our method consistently enhances the performance of baseline approaches in terms of background preservation metrics, while exhibiting only a slight decline in CLIP score. We argue that this minor reduction does not compromise the overall editing capability. Furthermore, evaluations using GPT-4o indicate that our proposed method effectively improves the editing quality over the baseline.

Question 1: Can the method be applied to other models like DiT-based image generation models, since Stable Diffusion 1.4 is limited in image generation quality?

Thank you for your thorough review. Our framework can be extended to DiT-based image generation models. Specifically, we conduct an experiment using [3], where Auto-Edit is employed to automatically select the optimal attention injection timestep. The results are presented in the following table:

Our proposed AutoEdit can improve the performance of [3] on background preservation and LLM score, where LLM score is the evaluation of the editing quality provided by a GPT-4o model with a slight decrease in terms of CLIP score. However, we argue that the small decrease of the CLIP score does not affect the editing capability of AutoEdit.

[1]InstructPix2Pix: Learning to Follow Image Editing Instructions

[2] SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

[3]Taming rectified flow for inversion and editing, ICML 2025.

Thanks for the effort the authors have made. However, my major concerns were not addressed.

Firstly, the claim of complexity $O(TN^k)$ is not practical and cannot be justified by “When dealing with a large number of images, this selection process can become time-consuming” as $O(TN^k)$ is for one image.

Secondly, the viewpoint change, or size change, action change, belong to the image editing field, as prior works have explored (e.g., https://arxiv.org/abs/2309.10556, https://arxiv.org/abs/2304.08465). The authors' reply of "...falls outside the scope of our work" is doubtful.

Thirdly, comparison results with training-based approaches are missing.

Lastly, the proposed LLM score is underexplored. There are also many similar evaluations (https://arxiv.org/abs/2507.16193), while the authors did not cite or investigate them.

Overall, this paper still needs further revision.

3. Comparison results with training-based approaches are missing.

We provide the additional result of Autoedit on instruction-based editing which is the training-based editing methods, including Instruct Pix2Pix [2] and UltraEdit [3] to choose the CFG for each timestep.

AutoEdit improves background preservation and prompt alignment over the above baseline methods. Experimental results demonstrate that AutoEdit is applicable to training-based approaches, such as instruction-based editing methods, highlighting its generalizability across different editing techniques and model architectures.

[2] Brooks et al. , InstructPix2Pix Learning to Follow Image Editing Instructions, CVPR 2023

[3] Zhao, Haozhe, et al. "Ultraedit: Instruction-based fine-grained image editing at scale." Advances in Neural Information Processing Systems 37 (2024): 3058-3093.

4. the proposed LLM score is underexplored. There are also many similar evaluations (https://arxiv.org/abs/2507.16193), while the authors did not cite or investigate them.

The suggested paper was published on arXiv on July 22, 2025, which is after the NeurIPS submission deadline. Therefore, we were not aware of it at the time of submission. However, we appreciate the suggestion and will include a reference to this paper along with a discussion of its relevance in the revised version after the rebuttal phase.

Regarding the LLM score, a recent paper [4] has explored the use of LLM-based evaluation for image editing tasks. We would like to emphasize that our main contribution lies in proposing a reinforcement learning framework with a flexible reward function, such as the LLM score. Following your suggestion, we will include a more comprehensive reference and discussion of related work in the revised paper.

[4] I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing

We hope that our response addresses your concerns. If you have any other concerns, please let us know.

Thank you for your valuable review. We will respond to your concerns as follows:

Weakness 1: Training time of AutoEdit

For all experiments in the main paper, the total training time was approximately 10 hours on NVIDIA A6000 GPUs, covering both Stage 1 and Stage 2 training.

For the additional experiments with SDXL conducted during the rebuttal phase, we use NVIDIA A100 GPUs with a total training time of approximately 12 hours.

Weakness 2: Concern about the mask for background preservation for the global editing task (style transfer)

For the global editing task, we only employ the CLIP score as the reward function, which is similar to the evaluation protocol of the PieBench dataset. We present the results of the style transfer task in the following table:

From this table, our proposed AutoEdit can improve the editing results on the Style Transfer task compared to the baseline model.

Weakness 3: Lack of editing on recent methods

Thank you for your suggestion. Due to the time constraint, we are only able to perform AutoEdit for the Rf-solver. In this setting, we leverage our RL framework to determine the optimal attention injection timestep. The performance of Auto-Edit under this configuration is presented in the following table:

Our proposed AutoEdit can improve the performance of Taming Flow on background preservation and LLM score, where LLM score is the evaluation of the editing quality provided by a GPT-4o model with a slight decrease in terms of CLIP score. However, we argue that the small decrease of CLIP score does not affect the editing capability of AutoEdit.

Thanks for author's response. The topic of this paper is interesting and still under-explored by previous works. I would maintain my score as weak accept.

Thank you for your valuable review. We will respond to your concerns as follows:

Weakness 1: The dependent of mask $M$ in the reward computation

Our training objective is based on an editing metric. For non-masked editing tasks such as style transfer, the background preservation term (MSE) is excluded from the reward computation.

For global editing tasks (e.g., style transfer or style change), we use only the prompt alignment metric as the reward function. This choice aligns with the evaluation protocol used in Pie Bench, our benchmark dataset.

Additionally, we provide a comparison table highlighting our method outperforms the baseline in the style transfer task.

The experiment demonstrates the effectiveness of AutoEdit on the style transfer task.

Weakness 2: The framework's reliance on the segmentation mask limits its effectiveness for very fine-grained attribute edits

The performance of image editing depends heavily on both the reward function and the capabilities of the base model. In this work, we propose a general reinforcement learning (RL) framework for hyperparameter selection in editing tasks. This framework is flexible and can accommodate various reward functions.

We acknowledge that relying solely on the CLIP score within the masked region may be insufficient for capturing fine-grained edits, such as changes in small or detailed regions (e.g., eye color). To address this limitation, we incorporate large language model (LLM)-based evaluation to assess editing quality. Specifically, we employ a multi-step approach using GPT-4o to generate and compute rewards as follows:

• Step 1: Given an editing instruction (e.g., "change the eye color to blue"), we prompt GPT-4o to generate a corresponding question (e.g., yes/no or multiple-choice) along with a ground truth answer. For example: Question: "What is the color of the eye?" Choices: ["black", "blue"]; Answer: "blue".
• Step 2: To compute the reward, we query the LLM for both foreground editing and background preservation. For foreground editing, we use the question and ground truth answer from Step 1 and require GPT-4o to identify the correct choice. For background preservation, we ask the LLM to compare the edited image with the original and assign a score (0, 0.5, or 1) based on the quality of background consistency. The total reward is the sum of these two components.

We conduct experiments using SDXL with DDPM-Inversion, tuning the inversion timestep based on the newly introduced LLM-based reward. The following table presents a comparison between our original CLIP-based reward and the new LLM-based reward, with LLM score is the GPT-4o evaluation score, which is similar to the computation of the reward function.

Our AutoEdit framework consistently improves both background preservation metrics and LLM-based judgment scores, while exhibiting only a slight decline in CLIP score. We argue that this minor decrease in CLIP score does not compromise the overall editing performance of AutoEdit.

Weakness 3: Concerns about the SD 1.4 model:

We conduct experiments using SDXL with DDPM-Inversion, tuning the inversion timestep accordingly. The table below presents a comparison between Auto-Edit and baseline methods on the SDXL base model:

Our results demonstrate that AutoEdit significantly improves background preservation quality, with only a slight reduction in CLIP score. However, we argue that this minor decrease in CLIP score does not substantially impact the overall editing quality. Furthermore, evaluations using large language models (LLMs) indicate that AutoEdit considerably enhances the editing capabilities of baseline methods.

Weakness 4: Concern about the limited set of editing methods.

Thank you for your suggestion. The baseline you mentioned pertains to image composition tasks, which follow a different setup compared to image editing, the primary focus of our work. Nonetheless, our proposed RL framework is general and can be readily applied to diffusion-based image composition tasks for hyperparameter optimization.

For example, in the TF-ICON paper, our RL framework can be parameterized to optimize the injection timesteps of the composite self-attention maps, denoted as $\tau_A$ and $\tau_B$—a set of hyperparameters explicitly defined in the method. The reward function can be constructed based on the quality of foreground integration and background preservation in the composited image.

We also conduct experiments on a recent editing method [1]. In this setting, we leverage our RL framework to determine the optimal attention injection timestep. The performance of Auto-Edit under this configuration is presented in the following table.

Our experiments demonstrate that Auto-Edit enhances the editing performance of the baseline methods. Specifically, we observe improvements in the LLM-based evaluation metric and background preservation scores. While there is a slight decrease in the CLIP score, this suggests that our method effectively discovers hyperparameters that balance foreground fidelity and background consistency, resulting in overall improved editing quality.

Question 1: How can it be integrated into other image editing tasks (such as shape changes, local attribute editing, etc.)?

Our method is applicable to a wide range of editing tasks due to the flexibility of its reward function. For instance, the CLIP score can be used as a reward for style transfer tasks, while LLM-based evaluation can be employed to assess correctness in local attribute editing.

It is important to note that certain challenging editing tasks may also require stronger base models and novel editing techniques. Our approach focuses on the overarching editing problem by selecting optimal hyperparameters for each editing method (see Fig. 3 and Table 1 in our main paper).

Question 2: What about its performance on various models, such as different versions of SD, and how are they compared to more recent baselines (such as TF-ICON, IMPRINT, MIGC)?

Please refer to the response to weakness 4.

[1]Taming Rectified Flow for Inversion and Editing (ICML 2025).

Due to the time constraint, we are only able to conduct the experiments using the instruction-based editing methods UltraEdit[2] and InstructPix2Pix[4] to choose the CFG for each timestep. Our proposed AutoEdit framework enables dynamic selection of CFG values across different denoising timesteps, thereby improving the overall performance of the editing methods, as shown in the table below:

AutoEdit improves background preservation and prompt alignment over the above baseline methods. Experimental results demonstrate that AutoEdit is applicable to training-based approaches, such as instruction-based editing methods, highlighting its generalizability across different editing techniques and model architectures.

We hope that our response addresses your concerns. If you have any other concerns, please let us know.