LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents

1. The proposed LossAgent focuses on adjusting the weights of each loss component, which narrows the scope of research. In fact, LLMs with powerful instruction following capabilities can serve as stronger rewards generators for learning better low-level vision models, rather than tuning the weight of losses.

2. The method uses an external evaluation expert to evaluate the restored image, which is somehow inferior to using a VLM for image quality assessment. Since the LLM for loss weight generation does not align well with the image perception model, the weights can be inaccurate.

3. The paper lacks results in diffusion-based image restoration models and video restoration tasks.

1. Important related works are missed. This manuscript uses LLM to optimize hyperparameters. A similar idea has been proposed and explored in [R1] from Google DeepMind. However, this paper is never cited or discussed.

   [R1] Large Language Models as Optimizers

2. The classification criteria is wrong. This manuscript uses the differentiability of a metric as the judgment of being loss / objective. However, as studied in [R1], the problem is not differentiability. Many non-reference IQA metrics are differentiable (including some metrics used in this manuscript), but directly optimizing these non-reference IQA metrics will cause quite poor image quality. A metric is differentiable does not mean it can serve as a loss. Therefore, whether a metric can serve as the loss to improve the quality (instead of differentiability) should be the judgment of being loss / objective.

   [R2] Comparison of No-Reference Image Quality Models via MAP Estimation in Diffusion Latents

3. The relationship between losses and metrics is not analyzed. This manuscript aims to bridge some unoptimizable metrics and optimizable losses. What are the results of the bridge? Are there some connections between unoptimizable metrics and optimizable losses?

4. Confusing formulas and no explanation of loss selection. In Sec. 4, only the objective metrics are given, while the multiple differentiable losses are not given. How many losses does this manuscript use? What is L1, L2, L3, …, respectively? These are all quite important information, but they are never clearly described. I can only find them in Tab. 1 with only abstract formulas and without detailed explanations. For example, in the third row of Tab. 1 (All-in-one IR), there are $L_{perceptual}$ and $L_{LPIPS}$, what is the difference between them? In my view, they are the same. If they are different, which perceptual loss does this manuscript use? Also, why use L1 loss instead of MSE loss?

5. Missing information and the necessity of double optimization objectives is not enough. In Tab. 5, what are the metrics are not stated. I can only guess through the description of Sec 4.2.2 that the metrics are Q-Align / PSNR. In this case, MSE loss (same as PSNR) is differentiable, so why use an LLM to optimize PSNR instead of directly optimizing MSE? Will using an LLM to judge PSNR bring better results than optimizing MSE?

6. Conclusion of double optimization objectives is not convincing. In Tab. 5, the experiment is conducted on the classical image SR task whose losses contain L1 loss. Given PSNR results, LLM can increase PSNR results easily by increasing the weight of L1 loss. In this case, the experiment can only show LLM can link PSNR with L1 loss, which is very easy. What is the meaning of this experiment?

7. In Fig. 3, there is a clear advantage of LossAgent over baseline only for the third case (CLIPIQA).

8. Writing.

• In Tab. 3, the upper arrows of MANIQA and CLIPIQA are mixed.
• In Tab. 7, the “W” in the caption should be “w/”.

1. Missing SOTAs. This paper does not compare some low-level SOTA approaches, such as HAT [1], IPG [2] for SR tasks, and AirNet [3] for all-in-one IR tasks. Besides, the authors need to report the results using classic metrics in these experiments, e.g., PSNR and SSIM.
2. This paper employs an 8B LLM in the experiment; therefore, comparisons with other competitive low-level baselines on training cost and parameters are necessary.

[1] Activating more pixels in image super resolution transformer. CVPR 2023.

[2] Image Processing GNN: Breaking Rigidity in Super-Resolution. CVPR 2024.

[3] All-In-One Image Restoration for Unknown Corruption. CVPR 2022.

Overall, I cannot see the necessity of Agent. I think a regular LLM is enough. Also, the work is still at very early stage, lots of demonstrations are needed.

However, there are several weaknesses as follows:

1. The scope of this work is limited to only three low-level image processing tasks with just three common losses. Furthermore, the losses are combined using a simple weighted approach. To enhance the contribution of this work, it would be beneficial to explore the LossAgent's performance on a broader range of tasks with a more diverse set of loss functions and combination strategies.

2. A thorough analysis of the loss weights is missing, including their initialization and constraints to ensure the values range from 0 to 1. The paper should provide details on how these weights are initialized and managed throughout the optimization process.

3. The use of test images for external feedback generation at the end of each stage raises concerns about overfitting. It would be more appropriate to use images from a validation set rather than the test set. Additionally, using the same metric for both external expert evaluation and final loss assessment may lead to bias towards that specific metric. To address these issues and provide a more comprehensive evaluation, the authors should: 1) Use a separate validation set for external feedback generation; 2) Employ diverse external experts that are distinct from the final evaluation metrics; 3) Demonstrate how all metrics change when optimizing for single, double, and textual objectives.

4. The baseline results in Table 2 are presented with fixed loss weights. It's unclear whether these fixed weights are the same as the initial loss weights or how they were determined. Moreover, the authors don't provide information on how they ensured these weights are optimal for each task.

5. The results shown in Tables 4 and 5 fail to provide compelling evidence for the effectiveness of the proposed method, as they show only marginal improvements or, in some cases, even inferior performance compared to baselines. The evaluation is limited to optimizing only one or two objectives, which raises questions about the method's scalability and efficacy when dealing with more objectives. A more comprehensive assessment would involve testing the proposed LossAgent in multi-objective optimization contexts, incorporating both quantitative metrics and qualitative textual descriptions.