RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

Q1: Degradation order of JPEG compression.

Thank you for bringing up this important point regarding the degradation order. In our study, the order of JPEG compression is not fixed and is entirely random, unlike the sequence suggested in references [1,2]. The strategy of placing JPEG compression at the end, as mentioned by the reviewer, represents a subset of the data we constructed. Therefore, our dataset inherently includes this configuration while also supporting a broader range of degradation combinations.

Our primary objective is to validate the robustness and feasibility of our proposed pipeline. By incorporating a wider variety of degradation sequences, we can more effectively demonstrate the generalizability and effectiveness of our method across different scenarios. Moreover, the specific degradation order can be flexibly defined and adjusted based on user requirements, ensuring adaptability to various applications.

Q2: What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image?

Our dataset comprises one degraded image for each high-quality image. This means that each image pair consists of a unique degraded image and its corresponding high-quality counterpart, ensuring diverse image backgrounds. Specifically, we utilized images from the DIV2K and Flickr8K datasets, as well as the RESIDE dataset to construct images containing haze. Additionally, the LOLv2 Real dataset was employed to create low-light images.

Q3: What is the configuration in ablation studies about training data amount?

As previously mentioned, each high-quality image in our dataset corresponds to a single degraded image. Therefore, when we scale up or down the training data, both low-quality and high-quality images increase or decrease simultaneously.

Increasing the number of degraded images while keeping the number of high-quality images unchanged would indeed improve performance. As discussed in our manuscript, image degradation is complex, and even minor variations can lead to different degradation characteristics, requiring varied handling strategies. Thus, adding more degraded images extends the range of degradations the model is exposed to, enhancing its ability to manage a broader spectrum of degradation scenarios.

Q4: running time

The RestoreAgent in our approach is constructed based on a multimodal large language model. Specifically, the computational time for the RestoreAgent is approximately 0.4 seconds. Therefore, the additional running time introduced by the RestoreAgent is minimal. In our experiments, the image patch size used is 512x512 pixels. In practice, the MLLM resizes all inputs to a fixed size, such as 224x224. For processing 4K resolution images, there are several strategies to consider:
(1) Single Patch Representation: Often, image degradation across different regions is uniform. In such cases, a single patch can represent the overall degradation of a 4K image.
(2) Patch-wise Processing: The image can be divided into smaller blocks, with each block represented by a patch.
(3) Resizing: The 4K image can be resized appropriately before making decisions using the multimodal model.
Given these strategies, decision-making for 4K images does not significantly increase the required time.

Regarding the running time of the restoration models, it is important to note that our pipeline utilizes restoration models based on user preferences. Consequently, we can select models of varying sizes and running times according to specific needs. Therefore, concerns about the running time of the restoration models may not be necessary as the flexibility in our pipeline allows for optimization and customization based on user requirements.

Q5: it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning.

Thank you for pointing out this important concern. As shown in Table 1, we report the performance of the fine-tuned model on previous tasks after adaptation. The changes in performance are minimal. We believe that these minor variations in performance are primarily attributable to the inherent randomness in the training process. We believe that with careful tuning, the model's performance can be further enhanced.

This demonstrates a crucial point: the current capabilities of multimodal large language models are more than sufficient to handle this task. Their robust reasoning abilities support the management of scenarios far more complex than those presented in our experiments.

We believe one reason for this is that the newly added degradation task does not conflict with the existing tasks; in some aspects, it is very similar. For example, in the combination of snow + noise + JPEG, the original model already has extensive knowledge of handling noise + JPEG, which can be easily applied to the new combination.

We are confident that in more complex scenarios (with more restoration tasks and more restoration models), RestoreAgent will demonstrate its significant advantages compared with human. In such complex scenarios, human decision-making will be less effective since the space is much bigger, whereas the powerful capabilities of learning-based multimodal large language models will exhibit even greater superiority.

Q1: How the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance?

Thank you for highlighting this important aspect. The order of applying different enhancement techniques can significantly impact the restoration results, as illustrated in Figure 2 (b.1, b.2) of our manuscript.

Impact of Degradation Order on Patterns

When constructing training data, the sequence of applying degradations like noise, JPEG compression, and blur affects the resulting degradation patterns. For instance, adding noise before JPEG compression/blur alters the noise characteristics, making it smoother due to the compression/blur effects. Conversely, adding JPEG compression/blur before noise results in unaltered noise characteristics. This interplay between degradations creates distinct patterns based on their application order. The MLLM can identify different task sequences through degradation patterns.

Influence of Restoration Order on Performance

Similarly, the order of applying restoration techniques affects the degradation characteristics. For example, applying a deblurring model first can sharpen noise and rain streaks, altering their features. Conversely, applying a denoising model first can reduce noise but may impact rain streak features, making subsequent deraining less effective. An interesting case is when noise and rain streaks coexist: low noise levels may not impact deraining, but high noise levels can render deraining ineffective. Conversely, denoising first might alter rain streak features, affecting deraining performance.

Our proposed RestoreAgent, being a learning-based method, is trained on extensive data to recognize degradation patterns and make informed decisions based on subtle variations. It autonomously determines the optimal sequence of restoration steps to effectively remove degradations. This is achieved through a comprehensive understanding of how different degradations and restoration techniques interact.

Q2: Inference time.

RestoreAgent is built on a MLLM. In our experiments. The image patch size we used is 512x512 pixels. In practice, the MLLM resizes all inputs to a fixed size, such as 224x224, and the inference time of RestoreAgent is approximately 0.4 seconds in RTX4090. The increase in inference time is very minimal.

Q3: If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section.

We appreciate the reviewer's concern regarding the generalization capability of our model when faced with degradations outside the predefined scope. Here, we would like to further elaborate on this aspect.

First of all, it is true that there might be generalization issues when encountering unseen types or levels of degradation. We acknowledge this limitation and have discussed it in detail in our manuscript.

However, the primary and most significant contribution of our work is to demonstrate the feasibility and effectiveness of the proposed pipeline. The essence of our work lies in utilizing multimodal large models as agents for image restoration, which has shown promising results across various conditions. Our experiments show that the RestoreAgent performs exceptionally well in most complex scenarios, thus proving its viability.

In addition, our approach allows for flexible integration of different tools. If the model encounters degradations beyond the predefined scope, additional tools can be seamlessly incorporated. This adaptability ensures that our pipeline remains robust even when faced with unexpected degradation types. Essentially, the generalization issue is more related to the specific image restoration models (tools) used within the pipeline, rather than the pipeline itself.

Lastly, one of our future research directions is to significantly expand the scope of restoration tasks and incorporate a greater variety of restoration models. By doing so, we aim to enhance the system's capability to handle real-world complex degradations more effectively. This expansion will involve integrating models trained on a broader spectrum of degradation types and more generalizable restoration models, ensuring that the pipeline can generalize better to unseen conditions.

Q4: The explanation of "ranking" and "balanced" in Table 1 is still unclear. The authors should clarify the definitions of these terms.

We have added the following clarification to the paper:

The explanation of "ranking":

We've clarified the ranking system:

For individual test sets:
X/Y format: X is the average ranking, Y is total combinations.
Example: 6.4/64 means average 6.4th best out of 64 options.

For overall average:
We use percentages due to varying combination counts of diffrent restoration tasks.
Process: Calculate percentage ranking per set, then average.
Example: 12.9% means top 12.9% on average across datasets.

The explanation of "balanced":

We have also provided a detailed explanation of "balanced" in Section 4.1 of our manuscript, specifically within the Scoring Function subsection. Balanced refers to the sum of the PSNR, SSIM, LPIPS, and DISTS metrics after each has been standardized. To ensure clarity, we have also added additional explanatory notes directly in Table 1. We hope these changes address your concerns.

Q5: Show more visual comparisons.

Thank you for your valuable feedback. We have updated our manuscript to include additional visual comparisons of the RestoreAgent. These enhancements provide a clearer demonstration of the effectiveness of our proposed method. Please refer to the updated PDF file we have uploaded, which contains these new visual results.

Q1. More details regarding the construction methods of these datasets

Thank you for your feedback. We have significantly expanded the relevant sections in the revised version of our paper to offer a much more comprehensive explanation of our data preparation process.

Regarding the training dataset, the following is part of the part of newly added detailed content:

The training pairs construction begins with applying various types of degradation to an image. Subsequently, we determine the optimal restoration pipeline using model tools for processing. This involves generating all possible permutations of task execution sequences and model combinations, applying each pipeline to the degraded image, and assessing the quality of the restored outputs using a scoring function. The model for different restoration tasks are shown in Tbale 1.

Table 1: Model tools for different restoration tasks.

Table 2 illustrates 5 scenarios incorporated into our dataset, designed to enhance the versatility and robustness of the RestoreAgent model:

Table 2: Five scenarios for dataset construction and corresponding examples.

As for the testing dataset used as a benchmark, the details are presented in Table 3, which demonstrates our construction of various combinations of degradation types. Each image in the set contains a minimum of one and a maximum of four types of degradation, with the entire set comprising 200 images.

Table 3: Testset details.

---

Q2. Explanations of "rankings" in Table 1

We've clarified the ranking system:

For individual test sets:
X/Y format: X is the average ranking, Y is total combinations.
Example: 6.4/64 means average 6.4th best out of 64 options.

For overall average:
We use percentages due to varying combination counts of different restoration tasks.
Process: Calculate percentage ranking per set, then average.
Example: 12.9% means top 12.9% on average across datasets.

---

Q3. Additional limitations and future research directions

The primary limitation of our study is the confined scope of models and tasks examined. While our research offers valuable insights into RestoreAgent’s performance across several degradation scenarios, it does not encompass the full spectrum of restoration models or image degradation tasks currently available.

Another limitation pertains to the limited generalization capability of current image restoration models. These models often exhibit a notable decrease in performance or fail to respond adequately when faced with even minor variations in image degradation patterns. This limitation greatly narrows our selection of model tools, requiring us to choose more robust and generalizable model tools. The challenge underscores a critical need in the field of image restoration: future models must go beyond simply overfitting training data.

Our future work will focus on significantly expanding the range of image restoration models incorporated into our multimodal large language model. This expansion aims to enhance RestoreAgent’s capabilities across a broader scope of restoration tasks and degradation types. By integrating a more diverse set of state-of-the-art models, we seek to create a more comprehensive and versatile restoration framework.

---

Q4. Step-wise Re-planning and Rollback.

You are correct in your understanding. In the revised version of our paper, we have added a detailed discussion on this feature, which we call "Step-wise Re-planning and Rollback."

In the answer to Q1, we have actually added some details (indexes 2, 3, and 4 of Table 2). Table 4 shows experiments on a complex dataset with four image degradation types. While single prediction performs well, iterative step-wise replanning offers modest improvements. This indicates strong initial performance, with replanning serving as a refinement tool for incremental enhancements.

Table 4: Step-wise planning.

---

Q5. Will this test set be made publicly available?

Yes, we will.

Q1: In the introduction, it would be helpful to explain how the MLLM excels at understanding different types and levels of image degradation.

Thank you for your valuable suggestion. We will incorporate the following explanation into our introduction to clarify how MLLMs excel at understanding and handling different types and levels of image degradation.

MLLMs are well-equipped to handle various types and levels of image degradation due to their extensive training on diverse datasets. This training enables them to develop a deep understanding of the relationships between different modalities, which is crucial for identifying and addressing complex degradation patterns. 
MLLMs excel in image restoration due to their robust reasoning abilities and advanced pattern recognition skills. They can analyze and interpret intricate details of various degradations, discerning subtle variations in noise, blur, compression artifacts, and other forms of image deterioration. This capability allows MLLMs to make precise decisions on applying the most appropriate restoration techniques, even when faced with complex combinations of degradation effects. For instance, when encountering an image with mixed noise, blur, and compression artifacts, an MLLM can accurately determine the optimal sequence and type of restoration techniques to apply. 
Furthermore, the adaptability of MLLMs allows them to quickly learn and optimize for new degradation scenarios through fine-tuning. This flexibility is essential for real-world applications where the types and combinations of degradations can vary widely. By continually updating their knowledge base, MLLMs can maintain high performance across a broad spectrum of image restoration tasks. 

Q2: Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data.

The process of constructing training data for new degradations follows the same methodology as for existing ones. For instance, if we were to add a new degradation type such as snow, we would randomly combine it with other existing degradations like noise and JPEG. Specifically, we experiment with different permutations and combinations of these degradations and find the best one to generate diverse training data. Once the new training data is generated, it is directly added to the original training dataset. This combined dataset is then used to fine-tune the model. Adding a new task requires a relatively small amount of data and has a quick training time. It allows for continuous improvement of the model without the need for complete retraining from scratch. We have updated our manuscript to include a more detailed discussion of this process.

Q3: In lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript "i" is already used for degradation type and it might be clearer to use another character.

We have revised lines 211-212. Our mean and standard deviation are calculated separately for each metric. Specifically, for each metric, we have results from all possible permutations and combinations restoration pipelines. We calculate the mean and variance using all these data points for each respective metric.

As you correctly pointed out, using the subscript "i" for both degradation type and in the summation could lead to confusion. We have updated our notation to use different characters for clarity.

Q4: What if the degradation of the input image falls outside the predefined degradation scope?

We appreciate the reviewer's concern regarding the generalization capability of our model when faced with degradations outside the predefined scope. Here, we would like to further elaborate on this aspect.

First of all, it is true that there might be generalization issues when encountering unseen types or levels of degradation. We acknowledge this limitation and have discussed it in detail in our manuscript.

However, the primary and most significant contribution of our work is to demonstrate the feasibility and effectiveness of the proposed pipeline. The essence of our work lies in utilizing multimodal large models as agents for image restoration, which has shown promising results across various conditions. Our experiments show that the RestoreAgent performs exceptionally well in most complex scenarios, thus proving its viability.

In addition, our approach allows for flexible integration of different tools. If the model encounters degradations beyond the predefined scope, additional tools can be seamlessly incorporated. This adaptability ensures that our pipeline remains robust even when faced with unexpected degradation types. Essentially, the generalization issue is more related to the specific image restoration models (tools) used within the pipeline, rather than the pipeline itself.

Lastly, one of our future research directions is to significantly expand the scope of restoration tasks and incorporate a greater variety of restoration models. By doing so, we aim to enhance the system's capability to handle real-world complex degradations more effectively. This expansion will involve integrating models trained on a broader spectrum of degradation types and more generalizable restoration models, ensuring that the pipeline can generalize better to unseen conditions.

Q5: Specify Table 2

Thank you for your suggestion to improve the clarity of Table 2. We have updated Table 2 to highlight the best performing method for each evaluation criterion. We also have added a clear note to Table 2 specifying that the ranking improvement is compared against human experts.