An Intelligent Agentic System for Complex Image Restoration Problems

3. Why Not Use VLMs Exclusively Throughout the Pipeline?

A: Thank you for your question. We understand that you might think providing images could be helpful if the scheduling is performed by a powerful visual language model (VLM) like GPT-4V. However, we have not adopted this approach for several reasons:

We are skeptical about the effectiveness of using images with current VLMs during scheduling. In fact, we have found that even the most powerful VLM to date—GPT-4V—performs suboptimally in the image quality assessments required by our framework. We tested GPT-4V’s degraded recognition capabilities (with experimental settings identical to the fine-tuned DepictQA in our paper). The results are shown in the table below, far from satisfactory (see Table 2 in the paper for details). This indicates that achieving expertise in low-level visual aspects is not easy.

Furthermore, most current VLM methods are obtained by fine-tuning large language models (LLMs). Due to the limitations of fine-tuning data, the reasoning ability and knowledge breadth of VLMs are significantly inferior to LLMs. Therefore, there is currently no VLM suitable for our framework that possesses both strong low-level visual capabilities and general reasoning abilities. We look forward to the development of VLMs as powerful foundational models in the future.

Based on the above considerations regarding design methodology and practical effectiveness, we choose to use LLMs for reasoning and VLMs for perception separately, rather than relying solely on VLMs.

4. Would Online Updates to the Reference Data Benefit the Pipeline? Could implementing real-time updates to the experiential knowledge base further enhance the pipeline’s adaptability and performance?

A: Thank you for the reviewer's question; it's a very, very good one. Discovering and learning new knowledge in real time during practical processes, and even possessing a certain degree of creativity, is a higher-level goal in agent research. Achieving this requires us to make more breakthroughs in many core technologies. For example, in the cognitive architecture of learning from cases, we need to abstract the steps humans use to learn from past cases and construct automated mechanisms to implement these steps in agent robots. Additionally, we need to establish reasonable knowledge representation methods to express case-based knowledge (the knowledge in this paper is rule-based). We also need methods to retrieve information from a large number of cases. There is a vast space for us to conduct such research, and I believe these ideas can be realized in the future.

Returning to this paper, it represents very early exploratory work on agents in image processing. We built the research platform from scratch, discussed research methods, and demonstrated preliminary results and the potential of agent-based image processing systems. The issues mentioned above are currently beyond the scope of this paper, but we are extremely interested in them and look forward to gradually addressing them in our future work.

Thank you for recognizing the human-centric design of AgenticIR, particularly its reflection and iterative rescheduling processes that enhance interpretability and interaction. We also deeply appreciate your acknowledgment of our clear presentation, the novelty of our approach, and the rigor of our experiments.

1. AgenticIR tends to encounter an early stopping issue, where it may settle on a satisfactory solution prematurely, halting further exploration and potentially missing the optimal outcome.

A: Thanks for the question. Due to the complexity of image restoration and unpredictability of tools, the only way to guarantee optimal solution is exhaustive search, which is impractical (even if there are only two degradations and three tools for each, the required number of tool invocation will be 24). Hence any method is a trade-off between exploration and exploitation. AgentlicIR, as a heuristic search, deals with this by greedily exploiting acceptable directions and pruning those seemingly unpromising directions in reflection. This behavior does tend to exploitation and thus suffer from early stopping. In fact this preference is configurable. That is, we can adjust the acceptance threshold in reflection to suppress exploitation so as to force exploring more. We conduct an experiment that let AgenticIR only accepts tool outputs with very low severity of degradations, denoted as AgenticIR*. The results are shown in the table below, compared with AgenticIR and random tool invocation. AgenticIR* does obtain better results in most metrics, but also consumes much more time as shown below. Therefore, we believe it is fair to say the current setting of AgenticIR strikes a balance between performance and efficiency.

2. Although the paper emphasizes the role of experiential information and provides illustrative examples, it lacks concrete data on the actual reduction in iterations or time consumption.

A: Thanks for the question. Please see the public comment.

4. What are the limitations of the proposed framework?

Thanks for the question. We discuss this issue in Appendix E: our primary goal is to develop an intelligent agent for integrating tools in image restoration. However, the current framework only considers single-degradation restoration tools. In real-world scenarios, degradations are often much more complex than a combination of a few well-defined degradations, requiring more general and heterogeneous tools.

For instance, to handle more intricate degradations, we may need to incorporate diffusion models with strong generative capabilities and carefully fine-tune their complex parameters. Additionally, we might need to leverage various tools in Photoshop, much like a professional retoucher. Whether our framework is sufficiently flexible to integrate these diverse tools remains an open question. This challenge also imposes higher demands on the agent’s perception and decision-making capabilities.

5. Regarding the point that 'execution order is crucial', are the documents (knowledge) remain consistent during inference across different test samples?

A: Thanks for the question. Our response is: Yes, self-exploration and experience summarization are performed only once beforehand. Afterward, all tests are conducted independently, utilizing the same “knowledge.” However, as reviewer hoAk pointed out, it is also feasible to treat the tests as a unified process, where the agent incrementally updates its knowledge online based on the restoration results. For a more detailed discussion, please refer to our response to reviewer hoAk’s Question 4 in Section 2.

In summary, we found that achieving efficient online updates requires more advanced techniques, such as update strategy design, retrieval-augmented generation, knowledge representation, prompt engineering, and chain-of-thought reasoning. These directions offer promising opportunities for future research.

We sincerely thank the reviewer for their feedback and kind recognition of our paper’s clarity, comprehensive experiments, and key contributions.

1. The main concern is that this work resembles several widely-used frameworks, giving it a predominantly engineering-focused approach.

A: We respectfully disagree with classifying our work as “engineering-focused.” First, our research is not a simple combination/resemble of existing methods. We propose a novel methodology focusing on the integration and construction of image restoration agents (AI Agents). This methodology represents a significant academic contribution. We thoroughly explain why such an agentic approach is needed and how it can be designed and constructed. Based on this, we built a research platform and, for the first time, demonstrated through experiments that an agentic approach can exhibit a considerable level of intelligence in low-level vision tasks such as image restoration. This is a major breakthrough that cannot be achieved by a single image restoration model.

Regarding the combination of multiple models, we have extensively justified the necessity and advancement of this paradigm. Currently, AI Agent research is at the forefront of attention, with numerous studies exploring how to operate LLMs and other AI models used as tools more intelligently. The core of these studies lies in designing cognitive architectures to enable collaboration among multiple functional models, thereby exhibiting higher levels of intelligence. This is the focus of our work and a key problem for many subsequent studies to address. These cutting-edge efforts have garnered significant attention from both academia and industry and are far from being “engineering-focused.” We recommend that the reviewer carefully review the related work section, which we believe will address the reviewer's concerns.

We sincerely hope the reviewer can reevaluate this perspective and look forward to further discussions with you.

2. Additionally, this work appears complex, so providing statistics on the time and complexity involved in a single inference would enhance clarity.

A: Thank you for the reviewer’s questions. We encourage the reviewer to refer to the public response at: https://openreview.net/forum?id=3RLxccFPHz&noteId=XtMKDGs0G7, where we have provided detailed explanations regarding efficiency, cost, and fairness concerns.

3. Given that LLMs and VLMs often struggle with the issue of 'hallucination,' does this work encounter a similar challenge?

A: Thank you for the insightful comments. We highly agree with reviewer’s perspective. LLMs and VLMs indeed often face the issue of “hallucination,” which is one of the core motivations behind our proposed self-exploration and summarization method. The reliability issue in LLM-based scheduling can be seen as a form of hallucination (i.e., factual inconsistency or fabrication). This issue is extensively studied in Lines 302–323, Lines 408–418, and Appendix B.3 of our paper. Our experiments revealed that GPT-4 occasionally provides random answers when determining operation sequences in zero-shot settings, suggesting that its responses might be based on irrelevant factors.

To address this, we designed the self-exploration and experience summarization mechanism, which introduces clear references to enhance the reliability of scheduling. This mechanism enables the LLM to make decisions grounded in concrete foundations rather than relying solely on speculative reasoning. For VLMs, during the training of DrpictQA, we utilized carefully designed fine-tuning data to align the model outputs with human perception, thereby mitigating hallucination to a certain extent. While addressing hallucination remains a significant challenge in both LLM and VLM research, our approach has considered this issue and proposed feasible preliminary solutions.

We sincerely thank the reviewer for the comments and recognition of the clarity and novelty of our work, as well as the thorough justification of our design choices. We especially appreciate the positive feedback on our figures and the effectiveness of scheduling, reflection, and rollback in the proposed methodologies.

1. Cost analysis

A: Thank you for the reviewer’s questions. We encourage the reviewer to refer to the public response at: https://openreview.net/forum?id=3RLxccFPHz&noteId=XtMKDGs0G7, where we have provided detailed explanations regarding efficiency, cost, and fairness concerns.

2. Relatively subtle improvements for quantitative metrics

A: Thanks for the question. The issue of evaluation metrics in image restoration has been a longstanding challenge in the field. In many cases, image quality metrics are insufficient in accurately reflecting perceived image quality, as demonstrated by numerous studies [R4, R5]. For example, two entirely different images—one slightly blurry and another with enhanced contrast—may yield very similar PSNR values. However, it is evident that the image with improved contrast offers a much better visual experience.

Aware of the limitations of these metrics, we included up to six different metrics in our study to provide a more comprehensive evaluation of our method’s effectiveness. Our approach shows clear advantages across most metrics. In response to the reviewer’s concerns and suggestions, we will enhance our paper by including quantitative metrics in the figures to facilitate direct comparisons. Additionally, we will provide more comparative examples to offer a well-rounded demonstration of our method’s performance.

3. How are the discovered workflows similar to the original motivation of following the human workflow?

A: Thank you for the insightful question. Comparing our method to human operators is an excellent idea, and we are exploring ways to incorporate such discussions into our work. However, conducting such a study is inherently challenging due to the nature of human factors.

First, the selection of human participants is crucial. If the participants are domain experts, it is difficult to find a sufficiently large and unbiased sample. On the other hand, using non-expert participants often results in highly inconsistent performance, influenced by factors such as educational background, patience, number of attempts, environment, and even mood. Additionally, our study is conducted in a controlled laboratory setting, which differs significantly from real-world scenarios. Introducing human participants under such conditions further complicates the comparison, as their actions would inevitably be constrained. That said, we believe this is an important perspective, and we are actively exploring feasible approaches to compare our method against human performance. These discussions will be included in future versions of our paper and in follow-up work on image processing agents.

4. How does the proposed model perform when there is only a single type of degradation?

A: Thank you for the insightful question. If the VLM identifies only one type of degradation, the AgenticIR simply selects the specialized tool designed for that specific degradation, and the final output will be the result of a single IR model deemed by the VLM to successfully address the issue. If all tools are considered failures, the VLM compares the outputs of all tools and selects the best one as a compromise for the final result.

This represents a trivial case that does not fully showcase the capabilities of our method. However, if such a comparison must be made, we believe our approach remains competitive. Compared to unified models, our method leverages specialized models tailored for specific degradations, which are widely regarded as superior for such tasks [R1, R2, R3]. When compared to dedicated models (though this is an unfair comparison, as degradation information is leaked), our approach, through the reflection mechanism, can at least avoid the worst-case results. In fact, the more severe the degradation, the more effective our method becomes, as it can reject poor tool outputs and dynamically adapt to achieve better results.

5. What is the criterion for deciding whether Execution step is Failed or Successful?

A: Thank you for your question. During implementation, the Execution and Reflection stages are interleaved. As shown in Figure 2(c) and Appendix A.1, the agent randomly selects a tool to restore the image and then uses the VLM to reflect on the result. If the VLM determines that the tool has successfully addressed the degradation issue, the execution step concludes successfully. Otherwise, the agent continues to try other tools, repeating this process. If the VLM deems all available tools unsuccessful, the execution step concludes as a failure.

3. Toolbox Ablation Study

A: Thank you for the reviewer’s questions. We would like to emphasize that the selection of models is not an intrinsic part of our proposed method. The models we chose represent a broad range of options for specific types of degradation, rather than being selected for their potential to enhance performance. Adding more models to the toolbox expands the applicability of our approach and increases the likelihood of achieving better results. Ideally, a fully capable agent should have access to all currently available models. For the sake of convenience in our research, we prepared a representative subset of models as the toolbox.

As discussed under the “Comparison Fairness” section, we can continuously incorporate better models into the toolbox to achieve improved results across more images. However, our criteria for model selection are not limited to performance alone. Different models exhibit different behaviors—some may have lower numerical quality but offer unique effects or generalization capabilities for specific types of images. We aim to include a diverse range of models in the toolbox to better address the varied challenges found in real-world scenarios.

4. GPT-4 Usage and Reproducibility

A: Thanks for the question. It is indeed possible that the same version of GPT-4 may exhibit slight performance variations over time due to OpenAI’s closed-source nature. However, our method does not overly rely on any specific capability unique to GPT-4, as might be the case in other tasks. Any large language model (LLM) with adequate language reasoning abilities can be used to implement our AgenticIR framework.

To demonstrate this, we tested the performance of our method by replacing GPT-4 with the open-source Llama3-405B (referred to as AgenticIR (Llama)). The table below presents the results compared to the randomized and default settings of AgenticIR using GPT-4, as reported in the paper. The performance differences are minimal. This is because, with prior experience, the scheduling problem can be effectively handled even by a less capable LLM, and Llama produced results comparable to GPT-4. We believe that the minor performance differences are more likely due to the inherent randomness in tool invocation rather than model-specific capabilities. We will include this additional analysis and discussion in the revised version of the paper.

We thank the reviewer for the feedback and for acknowledging our structured, adaptable methodology and human-interaction-inspired approach.

1. Comparison fairness

A: Thank you for the reviewer’s questions. First, we encourage the reviewer to refer to the public response at: https://openreview.net/forum?id=3RLxccFPHz&noteId=XtMKDGs0G7, where we have provided detailed explanations regarding efficiency, cost, and fairness concerns.

We would also like to provide additional responses to the reviewer’s suggestion that “it would be more appropriate to compare AgenticIR to the state-of-the-art models for each specific degradation.” While we appreciate this perspective, we respectfully hold a different view.

First, it is inherently challenging to compare our method with such state-of-the-art models. The primary goal of AgenticIR is to intelligently address complex, ill-defined image restoration problems where existing methods often fail to provide satisfactory results. Many of the degradations we evaluate cannot be easily paired with a state-of-the-art model, as their use outside their originally intended scope often leads to significant limitations—an issue well-documented in studies exploring the generalization performance of image restoration methods.

Second, our approach is specifically designed to intelligently integrate various image restoration models. If there are state-of-the-art models that excel in specific scenarios, they can be seamlessly incorporated into our toolbox. For cases where these models perform well, our system will leverage them to achieve superior results. For cases they cannot handle, our approach will dynamically adopt alternative strategies. At all times, our method can combine the strengths of available tools, delivering results that are at least comparable to, if not better than, any single method used alone.

2. Efficiency Concerns

A: Thank you for the reviewer’s questions. We encourage the reviewer to refer to the public response at: https://openreview.net/forum?id=3RLxccFPHz&noteId=XtMKDGs0G7, where we have provided detailed explanations regarding efficiency, cost, and fairness concerns.

Dear Reviewer wMVY,

Thank you once again for your valuable time and insightful comments on our manuscript.

We have provided detailed responses to your concerns, which we believe address all the issues you raised. We would greatly appreciate the opportunity to discuss with you whether our responses have satisfactorily resolved your concerns. Please let us know if there are any aspects of our work that remain unclear.

We understand that your time is valuable, and we would be grateful if you could review our responses and share your thoughts at your earliest convenience. Please know that the opportunity for discussion is limited, so your timely feedback is greatly appreciated.

Thank you for your consideration.

Best regards,

Authors

Dear Reviewer wMVY,

The time remaining for discussion is running out. We are really looking forward to your feedback, which will be helpful for refining our work. If you find our response not addressing all your concerns, we are more than willing to provide further clarification. Once again, thank you for your valuable time and comments.

Best regards,

Authors

Dear Reviewer wMVY,

Thank you for your valuable comments. We have carefully addressed the concerns you raised and provided detailed responses.

The discussion period between authors and reviewers is about to conclude, with approximately 24 hours remaining. We have made multiple attempts to engage with you, and the AC has also issued two calls for discussion. However, we have yet to receive your response.

We believe that participating in the discussion before the deadline is both critical to ensuring clarity and a reflection of professional courtesy.

Could you kindly provide your feedback at your earliest convenience? Your time and input are greatly appreciated.

Best regards,

Authors