InstructRestore: Region-Customized Image Restoration with Human Instructions

We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.

[W1]: Clarity of Working Principle.

We sincerely appreciate this reviewer’s insightful question regarding the interplay between prompt variation and tuning coefficients, which indeed touches the core of our framework’s functionality. As the reviewer astutely observed, the system achieves instruction-guided modulation through a dynamic feature interplay between two key pathways:

1. SD backbone (text-driven prior). It receives region-specific captions (e.g., "blurry background") and generates semantically aligned features (e.g., bokeh patterns when instructed).

2. ControlNet branch (LR-derived reference). It extracts degradation-aware features from the low-quality input.

The tuning coefficients govern their fusion ratio. Specifically:

For detail enhancement: Reducing LR feature weighting allows SD's text-guided features (e.g., "the furry dog") to dominate, synthesizing plausible details.

For bokeh preservation: When the SD backbone processes blur-descriptive prompts, lower LR feature contribution enables these blur-oriented features to prevail, while higher LR contribution forces structure recovery.

[W2]: The Effect of Region Captions.

We sincerely appreciate this reviewer’s valuable suggestion regarding text prompt enrichment. Following this insightful recommendation, we conducted additional experiments using LLaVA to generate comprehensive scene descriptions for Instruct100Set, replacing our original region-focused captions. As shown in the table, using long captions results in slightly lower reference-based metrics (e.g., PSNR, SSIM) on the full image compared to our standard approach, while the overall performance remains comparable. This is expected, as longer captions with more global descriptions may activate some additional background details, leading to minor changes in the global metrics.

[Q1]: The Bound of Modulation Coefficients.

As analyzed in our responses to [W1], the modulation coefficients govern the feature fusion ratio between the text-guided and LR-derived pathways. The coefficients should maintain a minimum value greater than zero, and our experiments recommended an optimal maximum threshold around 1.5. Exceeding this upper limit tends to introduce artifacts, such as abnormal color saturation effects in the output.

In practice, we do not recommend setting the minimum coefficient below 0.3, as excessively small values cause the output to be dominated by SD backbone features, resulting in images that lack both fine details and structural similarity to the degraded input. It is also worth noting that the visually acceptable range of modulation coefficients can vary across different images: for some images, values up to 1.3 still yield normal results, while for others, artifacts may already appear at this level. Based on our experiments across the entire dataset, 1.5 can be considered a general upper bound for most cases. Therefore, keeping the coefficient within the range from 0.3 to 1.5 ensures a good balance between detail generation and structural fidelity.

[Q2]: Global Modulation.

The framework inherently supports global image-level modulation without requiring architectural modifications or retraining. By uniformly applying identical modulation coefficients across both masked and unmasked regions, the system can seamlessly transition from regional to global control.

We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.

[Q1]: The Statement on Irregular Textures.

Thanks for the comments. We agree that, traditionally, image restoration is defined with a primary focus on content fidelity and accuracy. As mentioned by this reviewer, however, it is indeed challenging to accurately recover pixel-wise details in highly irregular texture regions, including state-of-the-art generative restoration models. This point is well supported by prior literature LDL [1], which indicates that “it is difficult to produce high-fidelity results ... because they have much fine-scale details and suffer from signal aliasing in the degradation process, where most high-frequency components are lost." In these cases, strictly enforcing fidelity often leads to over-smoothed or visually unconvincing results, which can negatively impact the user’s perceptual experience.

It is also pointed out in LDL [1] that for such regions: "the pixels are randomly distributed so that the differences between results and ground truth are insensitive to human perception. Therefore, rich details generated can lead to better perceptual quality in these regions." This confirms that for the Real-ISR task, in certain contexts, perceptual quality is more important to users than strict fidelity. DeSRA [2] also indicates that "fine details in areas with complicated textures are difficult to perceive as artifacts like foliage, hair, and etc, while large pixel-wise differences in areas with smooth or regular textures, such as sea, sky, and buildings, are sensitive to human perception and easy to be seen as artifacts". These findings collectively suggest that the balance between perceptual quality and fidelity should be adaptively considered according to the semantic characteristics of different regions, as user preferences and perceptual sensitivity can vary significantly across image content.

Based on the above observations, our instruction-based region-customized restoration approach is designed to flexibly balance fidelity and perceptual quality: we maintain strict fidelity in structured regions, while allowing more perceptual detail generation in irregular texture areas. We will add these discussions in the revision.

[1] Liang J, Zeng H, Zhang L. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. CVPR 2022

[2] Xie L, Wang X, Chen X, et al. Desra: detect and delete the artifacts of gan-based real-world super-resolution models. ICML 2023.

[Q2]: The Effect of Scaling Factor.

Thanks for the suggestion. The scaling factor plays a central role in our system by controlling the fusion ratio between the SD backbone and the ControlNet branch during inference. By adjusting the scaling factor, the system can flexibly balance between generating new details and preserving fidelity to the degraded input, according to user instructions and regional requirements.

Specifically, SD backbone receives region-specific captions and generates semantically aligned features (text-guided features), while ControlNet branch extracts degradation-aware features from the low-quality input (LR-derived features). The scaling factor governs the fusion ratio between these two pathways.

For detail enhancement: Reducing the weighting of LR-derived features allows the text-guided features in the SD backbone (e.g., “the furry dog”) to dominate, enabling the synthesis of plausible new details. Please refer to Figure 4 in the main paper for visual effects.

For bokeh tuning: When the SD backbone is guided by blur-related prompts, a lower LR feature contribution allows those blur-oriented features to prevail, while a higher LR contribution enforces structure recovery, which may lead to over-restoration of background blur. The visual effects can be seen in Figure 6 of the main paper.

In short, a smaller scaling factor allows the SD backbone to dominate, resulting in richer generated details, while a larger scaling factor makes the output closer to the degraded image, increasing fidelity but reducing generated details.

As for the restoration result, increasing the amount of generated details (i.e., lowering the scaling factor) typically results in lower reference-based metrics (such as PSNR), but higher no-reference perceptual metrics (such as CLIPIQA). This trend can be observed in Table 2 of the main paper: when the fidelity scale (scaling factor) is reduced in the specified region, PSNR and other reference-based metrics decrease, while CLIPIQA and no-reference metrics increase. This demonstrates that the scaling factor provides effective control over the trade-off between fidelity and perceptual quality, allowing users to flexibly adjust restoration outcomes according to their preferences and the semantic context.

[Q3]: Efficiency.

Thanks for the suggestion. The table below summarizes the inference steps, inference time (in seconds per image), and parameter count (in billions) for several representative methods. Inference time is measured on 512 × 512 LQ images using a single NVIDIA A100 80G GPU.

Although our method includes a mask decoder, it is relatively lightweight, resulting in a smaller overall parameter count compared to methods like OSEDiff and SeeSR, which require an additional RAM model. However, during inference, our approach generates a mask at each step and performs feature modulation at multiple scales within the SD-UNet. Since our current implementation executes these operations using a for-loop without code optimization, the inference time is a little longer than PASD, which uses the same number of diffusion steps. In our future work, we will (1) optimize our implementation to speed up the inference and (2) explore one-step or few-step inference to further enhance the efficiency of our framework.

[Q4]: The Effectiveness of ControlNet Features and Restoration Mask.

We thank the reviewer for prompting a discussion on the effectiveness of the ControlNet features and the restoration mask.

1. ControlNet features

As discussed in [Q2], the ControlNet branch extracts degradation-aware features from the low-quality input, providing essential reference information for the restoration process. These features, when fused with the SD backbone via the scaling factor, enable the system to balance between preserving fidelity to the degraded image and generating new details as guided by user instructions.

2. Restoration mask

We validate the effectiveness of predicted restoration masks by replacing it with ground-truth (GT) masks during inference and comparing the results on the Instruct100 set under two fidelity settings (0.5 and 1.1).

The table below shows that results with predicted masks are very close to those with GT masks across all metrics in both target and background areas. This demonstrates that the predicted mask is effective to enable accurate region-customized restoration. Furthermore, with either predicted or GT masks, the target region changes with different fidelity scales, while the remaining areas stay stable, confirming precise local control. With a lower fidelity scale (0.5), the difference between predicted and GT masks is slightly larger. This is because, in our multi-step inference framework, a mask is generated at each step, and the predicted masks in the early steps may not accurately cover the target region. As a result, the features of the intended area may still be modulated with a fidelity scale of 1 in the initial steps, leading to higher fidelity metrics compared to using GT masks.

Dear Reviewer DVDK,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #14962

We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.

[W1] and [Q1]: Variation of Instruction Format.

We thank the reviewer for raising the concern about the reliance on structured instruction templates. To answer this question, we designed three instruction variants that were not seen during training to evaluate the robustness of our method to more natural or varied user expressions. Specifically, we tested the following variants:

v1: “make the entire image sharper while make {region expression} clear”;

v2: “enhance the clarity of {region expression}”;

v3: “I would like {region expression} to be clear”.

For each variant, we set the foreground fidelity scale to either 0.5 or 1.1, the background to 1, and conducted experiments on the Instruct100 set. The test result is shown in below table. Notably, v1, which is most similar to the original template, achieves nearly identical results to the original instruction. For v2 and v3, which differ more significantly in phrasing, our method still achieves comparable results, suggesting a certain degree of robustness to instruction variations. All the three instruction variants result in significant changes in the specified region and minimal changes elsewhere, with metrics similar to those under the standard instruction. This suggests that, although the phrasing differs, these instructions can still localize the intended region to some extent. However, we acknowledge that the generated masks are not identical across different instructions, which may cause some result variations.

[W2]: Ablation Studies.

We appreciate the reviewer’s concern about ablation studies. In our original submission, we did not include experiments that simply remove either the mask decoder or the fidelity modulation mechanism. This is because both modules are essential for achieving region-customized restoration based on user instructions—removing either one would make it impossible to realize the core functionality of localized control.

To address this reviewer’s suggestion and to better analyze the independent contributions of these modules, we have conducted the following additional experiments:

1. Mask Decoder

We assessed the mask decoder by replacing its predicted masks with ground truth (GT) masks during inference and comparing restoration results on the Instruct100 set under two fidelity settings (0.5 and 1.1). The table below shows that results with predicted masks are very close to those with GT masks across all metrics in both target and remaining areas. This minimal performance gap demonstrates that our mask decoder generates high-quality masks, enabling accurate region-customized restoration. Furthermore, both predicted and GT masks show that the target region changes much with different fidelity scales, while the remaining areas stay stable, confirming precise local control. Another observation is that at a lower fidelity scale (0.5), where more details are generated in the target region, the difference between predicted and GT masks is slightly larger. This suggests that mask quality is more critical when generating finer details. Nevertheless, the performance gap remains small, further confirming the robustness of our mask decoder.

2. Fidelity Modulation Mechanism

The fidelity modulation mechanism in our method controls the amount of generated details by adjusting the coefficient for integrating conditional features from ControlNet into the SD backbone. Intuitively, a smaller fidelity coefficient allows the SD backbone to dominate, resulting in richer generated details, while a larger coefficient makes the output closer to the degraded image, increasing fidelity but reducing generated details.

To analyze the effect of the fidelity modulation mechanism, experiments were conducted within our multi-step inference framework (20 steps in total), with the fidelity scale set to 0.6 for the target region and 1 for the remaining area. We tested the impact of applying the fidelity modulation starting from different inference steps. In our default setting, we apply the fidelity modulation from the very first step (step 0). The results are summarized in the table below. We observe that the later the modulation is applied (i.e., the larger the starting step t), the higher the fidelity metrics and the lower the no-reference quality metrics. This indicates that delaying the application of fidelity modulation will suppress the model’s ability to generate details, and narrow the adjustable range of effects when different fidelity scales are set according to user instructions. Therefore, to ensure sufficient controllability, we apply fidelity modulation from the very beginning of the inference process.

[W3] and [Q2]: Generalization for Unseen Semantic Categories.

We sincerely appreciate this reviewer's insightful question regarding the model’s generalization to unseen semantic categories and open-domain instructions. Actually, our approach leverages Semantic-SAM to generate open-vocabulary masks without pre-defining a closed-set category list, combined with Osprey for region-specific captioning. This pipeline ensures that the training data encompasses diverse semantic regions beyond conventional category boundaries. For example, as shown in Figure 3 of the Appendix, the specified region is “pagoda”, which is not a predefined category in closed-set semantic segmentation. Nevertheless, our method is still able to localize and adjust this region effectively.

While this design enhances open-domain capability, we fully acknowledge that certain rare or novel categories may still pose challenges due to their limited presence in the training set. For instance, when given uncommon objects such as “astrolabe” or “scaffold” as the specified region, the mask decoder may generate an all-zero mask, making it impossible to localize or adjust the intended area.

I appreciate the authors’ thoughtful reply, which has successfully resolved my concerns. After carefully considering the responses and reading the other reviewers’ comments, I have decided to raise my rating from Borderline Accept to Accept, as I believe the paper makes a solid contribution and is suitable for publication.

We sincerely thank the reviewer for the constructive comments and hope our point-to-point responses address the concerns raised.

[W1]: Methodology Novelty.

We appreciate this reviewer for the comments and provided literature. However, we believe there are misunderstandings here. Our work is fundamentally different from both image editing and prior instruction-based restoration. Our method tackles the unexplored challenge of region-customized restoration, with unique task formulation and technical implementation.

1. Difference from Image Editing

Image editing modifies clean inputs to alter semantics (e.g., scene composition or object attributes), while InstructRestore processes degraded inputs while preserving semantics. The ill-posed nature of restoration implies multiple valid solutions for a degraded input, where users may prefer varying perceptual details across regions. Our goal is to recover content while enabling continuous, intent-driven regional control.

The image editing methods mentioned by this reviewer (e.g., DiffEdit, IIRNet) operate on clean-domain inputs to transform an image by using target text descriptions. Typical approaches first encode the clean image into the noise for diffusion models, then perform denoising using target text prompts. For example, DiffEdit employs DDIM encoding to obtain this noise, then substitutes masked-area noise with text-aligned versions during decoding to alter content. However, these methods cannot be used for restoration. First, the noise-conversion pipeline fails on degraded inputs, hurting output fidelity. Second, unlike editing, our target output shares identical semantic descriptions with the input.

Other editing methods require supervised training with edited image pairs, such as IIRNet mentioned by this reviewer. IIRNet conditions on clean inputs but deliberately corrupts specified regions (via color removal/noise injection) to avoid identity mapping, forcing text-consistent generation only in masked areas. Such editing approaches fundamentally conflict with the restoration task. In contrast to the deliberate corruption strategy, we prioritize both strict semantic consistency and maximal information preservation from degraded inputs.

In summary, image editing fundamentally differs from our task. It requires clean inputs and produces semantically altered outputs by design. Even advanced regional editing techniques are incompatible with restoration objectives. Achieving human-aligned perceptual control for region-specific image restoration is the goal of our work.

2. Difference from Previous Instruction-Based Restoration

Existing instruction-based restoration methods (InstructIR, PromptSR, and SPIRE) can only achieve global uniform restoration. In contrast, our work enables region-customized restoration. We need to solve previously unaddressed challenges: (1) precisely localizing user-specified regions in degraded images through instruction parsing, (2) enabling continuous effect modulation within target areas, and (3) guaranteeing artifact-free transitions between modified and unmodified zones. All of these requirements are beyond the scope of existing global instruction-based approaches.

3. Methodological Innovations

To address the challenges mentioned above, we first created a large-scale dataset specifically designed for this task, and developed a mask decoder that accurately localizes target regions using both user instructions and the degraded image. We found that a ControlNet-based adaptation enables precise restoration fidelity control by modulating feature integration in Stable Diffusion. Based on this, we selectively adjust feature fusion in target regions, preserving original fidelity elsewhere. Our results show seamless blending without visible edge artifacts..

In summary, our method differs from prior work in three key ways: instruction usage, mask generation, and mask operation. Our novelty lies in the building of the first unified framework for controllable region-level image restoration, which cannot be achieved by existing methods.

[W2]: Title and Task.

We thank the reviewer for the feedback regarding the title and task scope of our paper. We respectfully clarify the following points:

1. On the Use of "Image Restoration" in the Title

Our work follows the paradigm of recently published papers (e.g., SUPIR[1], SPIRE[2], DreamClear[3], FluxIR[4]) that address real-world image restoration under mixed degradations in the wild. Kindly note that all those works use "image restoration" in their titles, and they report experimental results only on real-world image super-resolution, which is one of the two tasks reported in our paper.

Like these works, our degradation model combines noise, blur, JPEG compression, and downsampling to synthesize real-world distortions. We also evaluate on real-world datasets with complex, entangled and unknown degradations. The title "Region-Customized Image Restoration" aligns with this broader definition of the restoration task adopted by the existing works.

[1] Yu F, et al. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. CVPR 2024

[2] Qi C, et al. Spire: Semantic prompt-driven image restoration. ECCV 2024

[3] Ai Y, et al. DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation. NeuralPS 2024

[4] Deng J, et al. Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration. CVPR 2025

2. On the Requirement of Bokeh Preservation in Our Task

This reviewer may misunderstand our experiments on bokeh-preserved image restoration. Please kindly note that "bokeh preservation" is a requirement in our region-customized restoration task. Existing SD-based methods often fail to distinguish bokeh from degraded images, leading to over-restoration of blurred area. Our approach allows users to specify background blur strength and foreground detail recovery through instructions. It is not an image editing or beautification problem, but a user-instructed image restoration problem.

3. Concerns on Broader Tasks

Regarding more restoration tasks such as dehazing, denoising, low-light enhancement, we believe that our data generation pipeline and instruction-based restoration framework can be extended to these tasks, which will have their own unique requirements for localized adjustment. In this work, we focus on the task of real-world image restoration [1-4]. We hope that our work can inspire more researchers to work in this underexplored but valuable direction.

[W3]: Comparison Baselines.

1. Regarding real-world super-resolution methods (InvSR/PiSASR):

We compare with InvSR/PiSASR in table below. Note that the metrics are computed in the target area. Sorry that we cannot show visual comparisons due to NeurIPS' policy.Actually, like other Real-ISR methods in the main paper, these methods cannot achieve region-level restoration guided by human instructions.The metrics in the table cannot reflect the real performance. We will add the two methods in revision.

2. Regarding bokeh rendering methods (drbokeh/bokehme):

Please kindly note that bokeh rendering methods (e.g., DRBokeh, BokehMe) are architecturally incompatible with our task. They synthetically generate bokeh from clean images with depth maps, while we restore natural bokeh from degraded inputs without depth data. So these approaches could not serve as baselines for our task.

[W4]: Test-sets and Comparison.

Kindly note that no existing benchmark suits our novel task. While current SD-based SR methods evaluate on 512×512 crops from RealSR/DRealSR, these lack sufficient semantic diversity for region-specific instructions. Therefore, we created Instruct100Set by strategically selecting semantically rich regions from RealSR, DRealSR and RAIM containing multiple distinct elements, enabling comprehensive evaluation of localized restoration under real-world degradations. Kindly note that Figures 10 and 11 is to show that only our method preserves background blur as intended, unlike others that over-sharpen these regions.

[Q1]: More Experiments for Bokeh Tuning.

Following your suggestion, we fixed the foreground fidelity at 0.8 while varying background bokeh fidelity (0.4,0.7,1), where lower values indicate stronger blur. We measured foreground PSNR(fg_psnr), background PSNR(bg_psnr), and background blur strength using the Brenner metric (lower values means stronger blur).

As shown in the below table, there are limited variations in foreground PSNR (0.5 dB) but significant changes in background PSNR (2.52 dB). Moreover, reducing the bokeh fidelity scale lowers Brenner values, confirming increased blur as intended.

[Q2]: The Purpose of Figure 6(b).

This figure shows our method's dual capability: controlling background blur restoration while regulating foreground detail generation.

[Q3]: OOD Training Concern.

Existing datasets (LSDIR, Unsplash) contain bokeh images, so all methods have seen blurred backgrounds during training. However, severe degradation makes natural bokeh hard to distinguish. Guided by user instructions, our model preserves blur while restoring targets. We also retrained OSEDiff with our bokeh-enriched data. As shown in the table below, it achieves slight background metric improvements over original OSEDiff, but lags behind our method. The background bokeh distortions remain (images omitted due to NeurIPS' policy).

Dear Reviewer qjkL,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #14962