LM4LV: A Frozen Large Language Model for Low-level Vision Tasks

We thank the reviewer for recognizing our novel explorations and for reading the Appendix carefully. We also appreciate the acknowledgement of our writing. We will address the weaknesses and questions point by point:

Weaknesses:

[1] We thank the reviewer for bringing up the discussion. However, with all due respect, we disagree with the assertion that our contribution is limited. We would like to emphasize that our primary contribution is validating that LLMs possess the capability for image restoration, which is far from trivial. To ensure minimal interference from the connector between vision modules and the LLM, we deliberately use a simple linear layer. This design choice does not diminish our contribution. On the contrary, it would be less surprising if LM4LV performed well with more sophisticated modules like MLPs or Q-Former[4], as those modules may already have the capacity for image restoration, with the LLM merely acting as an identity mapping. In Section 5.1 and Appendix B.2, we conduct extensive experiments and analysis on the behavior of the linear layer. Our findings demonstrate that the linear layer itself cannot perform image restoration and instead tends to form an identity mapping. This further validates that it is the LLM, not the linear layer, that processes the image latents and performs the image restoration. Our contribution is also acknowledged by reviewer kc9o and reviewer ty68.

[2] We agree that exploring alternative ways for LLMs to process visual information is valuable. In Section 4.4, we already investigate a ViT-like method and show that a ViT-like non-auto-regressive process fails at image denoising, resulting in messy image patches. We are more than willing to conduct additional experiments if the reviewer can propose new approaches for LLMs to process and generate visual information.

[3] We thank the reviewer for bringing up the discussion. We would like to emphasize that the aim of this work is not to improve performance but to validate the capability of LLMs for image restoration. To ensure a rigorous validation, we deliberately constrain the trainable parameters to two simple linear layers. While this approach undoubtedly diminishes performance, it strengthens the validity of our findings.Therefore, we focus on demonstrating that LLMs achieve better performance than a "not doing anything" baseline, which in our case is MAE-r. As mentioned by reviewer kc9o, although we do not expect LM4LV to outperform existing methods, additional comparisons with baselines could help readers better understand the position of our paper. The degradation types in Restormer do not fully align with those used in our work, resulting in a significant performance drop for Restormer. For example, in Gaussian blurring, using the ‘Single_Image_Defocus_Deblurring’ checkpoint decreases the PSNR from 30.88dB to 25.38dB. Therefore, we only present the results for image denoising, the only task that aligns with the task in Restormer, using the same data as ours and the pretrained model from Restormer. For image denoising, Restormer achieves 32.12dB, which is better than LM4LV. We hope this comparison helps the reviewer better understand the position of LM4LV. We will test additional baselines and include them in our next draft.

[4] We thank the reviewer for bringing up the discussion about novelty. However, with all due respect, we disagree with the assertion that our novelty is limited. The composition of a fine-tuned MAE and LLM does not restrict our contributions. First, we are the first to demonstrate that MAE can be utilized for image reconstruction. While the fine-tuning process follows previous works in image reconstruction, the novel aspect lies in using MAE specifically for this purpose. Moreover, as shown in Section 4.3, other common reconstruction modules, such as VQGAN, perform poorly in comparison. In Section 6, we further explain why MAE is uniquely suitable for this task while VQGAN is not. Therefore, the inclusion of MAE is not only novel but also essential. Additionally, we are the first to explore the potential of allowing LLMs to directly generate visual tokens without relying on multimodal data or prior models. We also propose a training and sampling strategy that aligns traditional low-level vision tasks as a form of visual instruction tuning [3], an approach that has been largely underexplored. Therefore, we believe the novelty of our work is not limited.

We thank the reviewer for recognizing our work's focus on extending LLMs to low-level vision tasks without multimodal training and for appreciating the comprehensive experiments covering diverse tasks such as denoising, deblurring, and pepper noise removal. We’ll discuss the Weaknesses and Questions by point:

Weaknesses:

[1] (comparison) We want to emphasize that our work focuses on image restoration, an image-to-image task, whereas works in the vision-language domain primarily address image-to-text or text-to-image tasks. To the best of our knowledge, there are no similar approaches that employ a non-LLM text model for image restoration tasks prior to our work. We are more than willing to include additional experiments if the reviewer can propose specific baselines in the vision-language domain.

[2] (motivation) We thank the reviewer for bringing up the discussion on motivation. However, with all due respect, we believe our motivation is clear. The background of this work originates from the low-level vision community's growing interest in integrating MLLMs into low-level tasks. While MLLMs have revolutionized various fields in computer vision by leveraging the strong reasoning and understanding capabilities of LLMs, the low-level vision field has yet to fully benefit from these advancements. Using traditional modules instead of LLMs can certainly achieve good performance and has been thoroughly explored over the past decade. However, such traditional modules are challenging to incorporate into the MLLM paradigm and typically adhere to a one-turn end-to-end fashion. In this work, we aim to explore a way to integrate MLLMs/LLMs into low-level vision tasks effectively. Recent works ([1], [2]) have focused on building low-level image-to-text MLLMs, but the majority of tasks in the low-level vision domain remain pixel-level image-to-image generation tasks. Our work presents the first attempt at integrating these tasks into the MLLM paradigm. Furthermore, we demonstrate that, contrary to conventional beliefs, text-only and frozen LLMs possess the capability to process and output low-level vision features. This finding expands the boundaries of what is considered possible for LLMs in low-level vision and paves the way for future research in incorporating text-only LLMs into this domain. We welcome any further discussion or suggestions from the reviewer to refine our motivation and contributions further.

Questions:

[1] (motivation) Please see weakness #2.

[2] (high frequency details) We thank the reviewer for pointing out the incapability of LM4LV to preserve high-frequency details. We have already discussed this limitation in the manuscript (lines 528-530): "As shown in Fig. 3, LM4LV could not restore high-frequency details in degraded images. This is natural because the LLM does not have an image prior, which could be improved by adding skip-connections or multimodal data." We encourage the reviewer to provide more detailed feedback for further improvements.

[1] Q-Future/Q-Instruct: [CVPR 2024] Low-level visual instruction tuning, with a 200K dataset and a model zoo for fine-tuned checkpoints.

[2] Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, and Chao Dong. Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models.

We thank the reviewer for their thoughtful feedback and for recognizing the significance of our work in addressing a critical gap in MLLM research. We also appreciate the acknowledgement of our ablation on design choices. We’ll discuss the Weaknesses and Questions by point:

Weaknesses:

[1] (position embedding) We thank the reviewer for pointing out the misunderstanding. What we intended to convey is: a. Causal masking is not a standard practice in ViT. b. Adding positional embedding to an MAE latent is not a common practice. Since MAE already incorporates a 2D sinusoidal embedding during encoding, the latent produced by MAE inherently contains positional information. Adding an additional ROPE embedding to the MAE latent is then not a common practice.Therefore, we removed the ROPE embedding and the causal mask. But as the reviewer points out, ROPE embedding itself is indeed a common practice in ViT. We have also updated the manuscript to provide a clearer explanation in the revised version.

[2] (different LLMs) We strongly agree that exploring different model sizes is important. We have conducted such an investigation, as detailed in Appendix B.2. As the reviewer noted, experimenting with larger LLMs is resource-intensive; therefore, we utilized a diverse family of lightweight LLMs. The results are presented in Table 7 and are also summarized below.

Not only do we investigate LLMs of different sizes, but we also examine the effect of varying the size of MAE. Reducing the size of either MAE or LLM decreases performance; however, the performance remains above the baseline. This demonstrates that LM4LV can generalize well to other variants of LLMs and MAEs.

Suggestions:

[1] (baselines) We agree that comparing with traditional baselines can help readers gain a better understanding. We tested Restormer [1], a classic image restoration method designed for various degradations. The degradation types in Restormer do not fully align with those used in our paper, which brings a significant performance drop for Restormer (for Gaussian blurring, using the ‘Single_Image_Defocus_Deblurring’ checkpoint decreases the PSNR from 30.88dB to 25.38dB. Therefore, we only present the results for image denoising, the only task that aligns with the task in Restormer, using the same data as ours and the pretrained model from Restormer. For image denoising, Restormer achieves 32.12dB, which is better than LM4LV (26.77 dB). We hope this comparison helps the reviewer better understand the position of LM4LV. We will test additional baselines and include them in our next draft.

[2] (BERT) We agree that using a Masked Language Modeling model like BERT may eliminate the need for autoregressive generation. However, this work is motivated by the growing trend of applying LLMs to vision tasks, and our focus is on exploring the potential of LLMs for image restoration. But still we conducted a pilot experiment using bert-large (https://huggingface.co/google-bert/bert-large-uncased) in LM4LV. We employed the ViT-LLM method for image denoising, as described in Section 4.4, and used the same training hyperparameters as in our main experiments. We find that BERT fails to produce meaningful images, instead generating messy image patches. While further experiments could be conducted to improve performance—for example, by using a MaskGIT-like sampling method—such investigations fall outside the scope of this work. We will leave these explorations for future research.

We sincerely thank the reviewer for providing many valuable suggestions. Please do not hesitate to contact us if you have any further questions or additional advice.

[1] Zamir, Syed Waqas, et al. "Restormer: Efficient transformer for high-resolution image restoration."

We thank the reviewer for acknowledging the simplicity and clarity of our proposed method and for recognizing our work as the first to use LLMs for low-level vision tasks. We also appreciate the positive feedback on the interesting conclusions, such as the robustness of MAE visual tokens to rotation. We’ll discuss the Weaknesses and Questions by point:

Weakness:

[1] (tokens and compute): We thank the reviewer for bringing up the discussion on computation. We have included an additional section to address this in Appendix B.4 of our revised manuscript. The token amount is determined by $tokens = \frac{HW}{p^2}$, where $H$ and $W$ represent the dimensions of the image, and $p$ denotes the patch size. For our configuration, $H = W = 224$ and $p = 16$, so the token amount is 196 for a single image. The computation cost is composed of three parts: LLM computation, MAE computation, and linear layer computation, which are listed as follows:

We want to note that while the computational cost of the LLM is significant, it is an unavoidable trade-off when leveraging LLMs for low-level vision tasks. The goal of this work is not to claim efficiency, but to explore the possibility of the integration of LLMs into classic low-level restoration tasks. Our contribution lies in validating the capability of LLMs on low-level vision tasks, potentially serving as a foundational step toward deeper integration of LLMs and image restoration. Consequently, the computation of the LLM is inherently involved.

[2] (linear adapter) We strongly agree that a thorough investigation of the effect of the linear adapter in our pipeline is necessary. Tthis is what we conducted in Sec 5.1 and Appendix B.1. In Sec 5.1, we conduct the exact experiment requested by the reviewer. As shown in Fig. 6, a single linear layer fails at denoising, resulting in images with noticeable and strange patch artifacts. To further investigate the effect of the linear layer within our pipeline, we provide both numerical results and visualizations in Appendix B.1, demonstrating that the two linear adapters primarily learn an identity mapping. Therefore, it is the frozen LLM that is actually performing the image restoration.

[3] (improved baselines) We agree with the reviewer that adding related work on low-level vision is beneficial. We’ll add more discussion in our next draft. Regarding improved baselines, we want to emphasize that this work is the first successful attempt to enable a frozen LLM to perform image restoration tasks. There is no prior work comparable to ours. And naturally, the performance of our method lags behind traditional methods, as we have deliberately minimized the trainable parameters and kept the LLM frozen. This natural gap is also recognized as reasonable by reviewer kc9o. The objective of this work is not to claim superior performance but to introduce a novel possibility for deeper integration of LLMs into image restoration tasks. We impose many constraints to validate that a frozen LLM has the capability to perform image restoration, and these constraints undoubtedly diminish the performance. But we agree that comparing with traditional baselines can help readers gain a better understanding. We tested Restormer [1], a classic image restoration method designed for various degradations. The degradation types in Restormer do not fully align with those used in our paper, which brings a significant performance drop for Restormer (for Gaussian blurring, using the ‘Single_Image_Defocus_Deblurring’ checkpoint decreases the PSNR from 30.88dB to 25.38dB. Therefore, we only present the results for image denoising, the only task that aligns with the task in Restormer, using the same data as ours and the pretrained model from Restormer. For image denoising, Restormer achieves 32.12dB, which is better than LM4LV (26.77 dB). We hope this comparison helps the reviewer better understand the position of LM4LV. We will test additional baselines and include them in our next draft.

[4] (typos) We thank the reviewer for the thorough reading and pointing out the typos. We have already fixed them and updated a new revision.

I appreciate the detailed response from the authors and acknowledge the primary aim of this work: to demonstrate the feasibility of integrating LLMs into classic low-level restoration tasks. However, the experimental results seem to suggest that LLMs may not be well-suited for such tasks, as the proposed method falls short of traditional approaches in both performance and efficiency.

From a conceptual standpoint, low-level restoration tasks heavily rely on detailed appearance features, and attempting to handle such features with LLMs would result in prohibitively high computational costs. This raises concerns about the practicality of LLMs in this domain. I encourage the authors to explore potential solutions to this challenge. For now, I will maintain my current score.

We thank the reviewer for recognizing our contributions, specifically demonstrating that frozen LLMs can effectively solve low-level vision tasks and showing through comprehensive ablation experiments that the ability to process low-level information stems from text pre-training rather than the trainable linear layer.

Weakness:

We agree with the reviewer that exploring additional vision encoders could further strengthen our study. We already conducted an experiment in Appendix B.2 to validate this. We’ll expand the discussion here. Incorporating larger vision encoders such as MAE-H presents significant computational challenges, as we need to finetune them for image reconstruction on ImageNet. Our limited computation resource cannot support us to do that during rebuttal. As an alternative, we conducted a reverse experiment by fine-tuning a smaller vision encoder, MAE-Base, which is substantially smaller than the MAE-Large encoder used in our primary experiments. We evaluated this smaller model on the image deraining task, where its performance dropped from 24.62 dB (achieved with MAE-Large) to 21.79 dB. Although the smaller encoder still outperformed the MAE-r baseline (19.76 dB), the results indicate a noticeable deterioration in performance as the encoder size decreases. This observation highlights the importance of using appropriately scaled encoders to achieve optimal results, underscoring the trade-off between computational feasibility and performance.

We also conducted an additional experiment by reducing the size of the LLM. Specifically, we utilized Phi3-mini, which is 3.2B parameters smaller than the LLM used in our main experiments (LLaMA2-7B-Inst). On the same image deraining task, the performance decreased from 24.65 dB to 23.60 dB. To provide a clearer comparison, we summarize the parameter reduction and corresponding performance changes in the table below. It is evident that while the reduction in parameters for the vision encoder is much smaller than that for the LLM, the corresponding performance drop in the deraining task is significantly larger. This observation underscores the critical importance of the vision encoder in this task.

We sincerely thank the reviewer for carefully reading our paper and providing many valuable suggestions. Please do not hesitate to contact us if you have any further questions or additional advice.