TinyLUT: Tiny Look-Up Table for Efficient Image Restoration at the Edge

Response to Weakness 1 We thank the reviewers for their comprehensive review. It must be clarified that the proposed SMS strategy will not result in information loss or performance degradation due to neglecting the relationships between local indexes. As for the original depthwise convolution operation, it can be obtained as follows:

$

F_{out} = \sum_{i=0}^{n-1}\sum_{j=0}^{n-1} {I_{i,j}*w_{i,j}}

$

The reconstruction of local index relationships is achieved through a summation process. This process involves the multiplication of each pixel within the receptive field with its corresponding weight. The resulting products are then aggregated to form the final output.

After the CNN model training, we stored the results of each pixel within the receptive field multiplied with the trained weights in the corresponding LUT.

$

LUT_{(i,j)}[I_{(i,j)}] = I_{i,j}*w_{i,j}

$

Similarly, after obtaining individual input-corresponding results through separately mapped indices in SMS, the final result is constructed via summation of all outputs. The relationships between local indices are reconstructed in this process. This process is formally expressed in the next equation of our manuscript:

$

\hat F_{out} = \frac{1}{n^{2}} \sum_{i=0}^{n-1}\sum_{j=0}^{n-1} LUT_{(i,j)}[x_{(i,j)}]

$

Following the discrete retrieval in SMS, a summation of individual results is necessitated to reconstruct the relationships among local indices. To prevent misinterpretation, we will refine Figure 3 to explicitly illustrate this crucial step in the process.

Response to Weakness 2 We appreciate the reviewer's insightful question. Our motivation for DDM stems from the limitations of the previous SPLUT approach, which was constrained to a 4 MSBs and 4 LSBs symmetric decomposition scheme for 8-bit inputs. SPLUT uses a fixed quantization scale for each layer. These constraints inherently limited model performance.

Our DDM proposed an innovative asymmetric activation decomposition strategy and adaptively finds the right quantization scale for each layer, thus significantly boosting the inference accuracy and achieving a better balance between accuracy and storage. Notably, the DDM strategy is not exclusive to our method. It can be applied to other LUT-based methods to further reduce the storage consumption. 

Response to Weakness 3 The reviewer's reminder is very important and insightful. In response, we will augment the related work section with an explication of the methodologies presented in [1] and [2], the key ideas and limitations are also will be elaborated. In addition to summarizing the key ideas and limitations of SRLUT, SPLUT and MULUT, we will also summarize RCLUT's proposal of the plugin module to increase the RF of the model with slightly increasing storage overhead. SPFLUT[1] proposed a LUT compression framework to balance the performance improvement and storage growth. The incorporation of these references will enhance the comprehensiveness of our literature review.

Response to Question 1 We extend our gratitude for the reviewer's meticulous examination. The inclusion of SwinIR in the related work section was motivated by its significant performance. However, upon careful consideration of the reviewer's astute observation, SwinIR does not align with the CNN-centric theme of our work, and the inappropriate description is removed from the manuscript.

Furthermore, we restructure the categorization of methods in the related work section. Specifically, we will supplement [3] and [4] under the CNN methods subsection to provide a more comprehensive overview of CNN approaches. We reiterate our appreciation for the reviewer's thorough and insightful feedback, which has significantly contributed to enhancing the coherence and relevance of our manuscript.

Response to Question 2 We appreciate the reviewer's suggestion for improvement. We validated the suggestions of the reviewer in the 4x SR task. As shown in Table 1, the improvement resulting from using a 5x5 RF and 32 feature channels shows an increase in accuracy with only a minor linear growth in storage overhead compared to the baseline.

We extend our sincere gratitude to the reviewers for their insightful suggestions, which have illuminated additional avenues for exploration in our method.

Table 1 Quantitative comparisons on 4x SR

Response to Question 3 The reviewer's suggestion is very important and insightful. We will revise the structure of the experimental section and combine the denoising experiments with the SR evaluation. This reorganization will significantly improve the clarity and flow of our research presentation.

[1] Li et al. "Look-Up Table Compression for Efficient Image Restoration", CVPR2024

[2] Liu et al. "Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution", ICCV2023

[3] Lai et al. "Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks", TPAMI 2019

[4] Tassano et al. "FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation", CVPR2018

Response to W1 The reviewer's reminder is very important and constructive. To address the reviewer's concern, we conduct a comparative study with the abovementioned two methods[1,2]. The comparison indicates that our TinyLUT is more effective than the latest storage saving LUT methods. Literature [1] proposed a LUT compression framework named DFC for efficient image restoration with a different technological route from our proposed TinyLUT. The DFC preserves diagonal LQHQ pairs to maintain representation capacity, while non-diagonal pairs are aggressively subsampled to save storage. Method [2] proposed a reconstructed convolution to decouple the calculation of spatial and channel-wise features for maintaining large RF with less LUT size. As shown in Table 1, TinyLUT-F achieves much higher PSNR with 9× and 12× lower storage consumption than RCLUT [2] and SPFLUT+DFC [1], respectively.

Table 1 Quantitative comparisons on 4x SR

In addition, we compare SPFLUT+DFC [1] and TinyLUT in image desnoing with noise level 15. Note that RCLUT [2] did not test on image denoising task. As shown in Table 2, TinyLUT still yields about 0.2dB higher PSNR with 3× storage reduction compared to SPFLUT+DFC. The results further demonstrate that TinyLUT exhibits notable advantages in other image restoration tasks. These experiments corroborate the effectiveness of the proposed method, as noted by reviewers (G5Ti, MFFN).

Table 2 Quantitative comparisons on image denoise

Response to W2 Thanks for the reviewer's insightful suggestion. As claimed in [3,4], image denoising and SR are classical yet still active topics in low-level vision since they are indispensable steps in many practical applications. Most of the image restoration models[3,4] adopted denoising and super-resolution as representative tasks. Upon the reviewer's insightful comment, we fully agree that the limitations of focusing solely on SR and image denoising cannot comprehensively demonstrate the generality in image restoration.

To address this concern, we incorporate the deblocking task widely employed in recent literature [1] into our experiment. The results of this experiment demonstrate the value of our method for advancing image restoration on edge devices. As shown in Table 3, TinyLUT-F still achieves remarkably higher PSNR-B than other LUT methods with much lower storage consumption.

Table 3 Quantitative comparisons on image deblock

Response to W3 The reviewer's reminder is very important. In the image denoising task, our TinyLUT-S does lag far behind MULUT-SDY-X2 in accuracy, which is results from the significant reduction in storage for deploying on edge devices with very limited resource budgets, especially memory and storage [5,6]. As shown in Table 4, our proposed TinyLUT-S occupies merely 7.6% of the storage required by MULUT-SDY-X2 that is the most lightweight version of MULUT.

As for the inference efficiency, inspired by comments from reviewers G5Ti and 2kHn, we found that still some operations can be merged to reduce inference latency, such as residual addition between blocks. Meanwhile, we have introduced pthread and omp to accelerate inference in a multi-threaded parallel manner. As a result, the inference latency of TinyLUT-S is reduced to 27ms on Raspberry Pi 4B, yielding a real-time inference efficiency comparable to the current lightest SRLUT with about 1dB PSNR promotion and 4× lower storage. Hence, our TinyLUT achieves a better trade-off between accuracy and latency, as noted by reviewer G5Ti.

Table 4 Quantitative comparisons on denoise and latency

In addition, we compare our method and other LUT-based methods in image deblocking task under a quality factor of 10 in Table 3. The results demonstrate the effectiveness of our work on other image restoration tasks.

[1] Li et al. "Look-Up Table Compression for Efficient Image Restoration", CVPR2024

[2] Liu et al. "Reconstructed Convolution Module Based Look-Up Tables for Efficient Image Super-Resolution", ICCV2023

[3] Zhang et al. "Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising", TIP 2017

[4] Kim et al. "Accurate Image Super-Resolution Using Very Deep Convolutional Networks", CVPR 2016

[5] Lin et al. "MCUNet: Tiny Deep Learning on IoT Devices", NIPS 2020

[6] Lin et al. "MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning", NIPS2021

Response to Weakness 1 The reviewer's comment is very insightful. The proposed SMS and DDM methods are applicable to other LUT-based models such as SRLUT[1], SPLUT[2] and MULUT[3].

Our proposed SMS enables decomposition of the input from RF and channels of these models for fewer LUT storage overhead. For example, the storage of SRLUT-S can be reduced from 1.27MB to 16KB by SMS. Moreover, the addition of DDM will further reduce the storage from 16KB to about 4.25KB to adapt to the deployment on edge devices. Due to time constraints during the rebuttal period, comprehensive validation of this aspect will be presented in a subsequent appendix.

Response to Weakness 2 The reviewer's suggestion is very valuable. To demonstrate the advantages of our method, we have conducted comparisons on image super-resolution and image denoising tasks with method [4] in Table 1. As described in your 'Strengths' section, these experimental outcomes further substantiate the considerable potential of LUT-based approaches. The method [4], being another LUT compression framework, was published after the submission of our manuscript. Method [4] based on DFC strategy preserves diagonal pairs to maintain the representation capacity, while non-diagonal pairs are aggressively subsampled to save storage. As can be seen, TinyLUT-F achieves the highest PSNR with about 12x storage reduction compared to SPFLUT+DFC. Since DFC and TinyLUT are different technological routes, we can combine them to achieve greater storage reduction.

Table 1 Quantitative comparisons on 4x SR task

In addition, we compare SPFLUT+DFC and our TinyLUT in the image denoising task with noise level 15 on Set12 and BSD68. As shown in Table 2, TinyLUT still yields about 0.2dB higher PSNR with 3× storage reduction compared to SPFLUT+DFC. The results further demonstrate that TinyLUT exhibits notable advantages in other image restoration tasks. These experiments corroborate the effectiveness of the proposed method, as noted by reviewers (G5Ti, MFFN).

Table 2 Quantitative comparisons on image denoise

Response to Question 1 The inference speed evaluation was conducted by deploying TinyLUT on two distinct platforms: Xiaomi 11 smartphone and Raspberry Pi 4B. For the Xiaomi 11 platform, the TinyLUT was implemented in C++ and invoked by Java Native Interface. Multi-thread parallel acceleration for look-up table was achieved through the Stream API. For the Raspberry Pi 4B implementation, the TinyLUT model was also coded in C++. We employed the "pthread" library and "omp" library for multi-thread look-up table acceleration . The source code will be made publicly available on GitHub to help the readers to understand the efficiency of the LUT-based methods.

[1] Jo et al. "Practical Single-Image Super-Resolution Using Look-Up Table", CVPR2021

[2] Ma et al. "Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution", ECCV2022

[3] Li et al. "MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution", ECCV2022

[4] Li et al. "Look-Up Table Compression for Efficient Image Restoration", CVPR2024

We greatly appreciate your time and dedication to providing us with your valuable feedback. We hope we have addressed the concerns, but if there is anything else that needs clarification or further discussion, please do not hesitate to let us know. We will ensure that all edits mentioned in the rebuttal are incorporated when revising our paper. Thanks again for your participation in the discussion!

Response to Weakness 1 We appreciate the reviewer's meticulous work. Indeed, our proposed TinyLUT as well as existing LUT-based methods are mainly designed for CNN-based architectures that are widely deployed on resource-constrained edge devices. Primarily because the convolution operation is computationally efficient [1][2][3]. CNNs compete favorably with Transformers in terms of accuracy and scalability, and convolution remains much desired and has never faded[4][5]. In recent years, many CNN methods have still been proposed [4][5][6].

Meanwhile, as highlighted by the reviewer and acknowledged in our Limitations section, LUT-based compression for attention-based methods represents a critical direction for future research. This area will be a focal point in our subsequent investigations.

Response to Weakness 2 We thank the reviewer for the constructive suggestion. To address the reviewer's concern, we have conducted supplementary experiments for denoising task under 50 noise level on more color image test sets, inculding Kodak, CBSD68 and McMaster. The experiment results further indicate the effectiveness of our proposed TinyLUT, corroborating the assertions made by reviewers MFFN and G5Ti. The experimental results are shown as follows：

Table 1 Quantitative comparisons on color image denoising with noise level 50

As illustrated in Table 1, our TinyLUT-F achieved an average accuracy improvement of over 0.5dB as well as about 5x storage reduction compared to MULUT-SDYEHO-X2. Like the performance validation on color image data in super-resolution task, the quantitative comparisons on color image denoising are still needed for comprehensive assessment. Meanwhile, this additional investigation expands the scope of our work.

Response to Weakness 3 We fully agree with the reviewer's insightful comment. Following the complexity evaluation in Ref.[7], computational costs of addition and multiplication operations are employed as the metric to assess the model complexity. Following the existing literature, we conduct the complexity comparison on 4x super-resolution task with an output resolution of 1280x720. The results are presented in Table 2. As can be seen, our proposed TinyLUT-S achieves the lowest computational complexity compared to other competitors.

Table 2 The statistics of addtion and multiplication operations

To address the reviewer's concern about latency, we conducted 4x super-resolution tests on a Raspberry Pi 4B platform. At present, the mainstream high-resolution images mainly include 1920×1080 (FHD) and 3840×2160 (UHD). Given this, we illustrated the processing time of our TinyLUT and competitive methods in Table 3. As can be seen, TinyLUT-S yields the lowest inference latency, while the latency of TinyLUT-F is comparable to SRLUT-S and far superior to MULUT on high-resolution images. This further validates the high efficiency of TinyLUT on edge devices mentioned by reviewers G5Ti and MFFN.

Table 3 Latency of high-resolution image on Raspberry Pi 4B

Response to Weakness 4 Thanks for the reviewer's suggestion. Although our article received positive feedback (2kHn) in terms of writing style and chapter division, there are still some grammar issues. We have reviewed the paper again and corrected some errors to improve the quality of the article.

Response to Question 1 Thanks for the reviewer's important reminder. We comply with the reviewer's suggestion and replace the Lena image in Fig. 2 with the Baby image in the Set5 test dataset. Meanwhile, we confirmed that Lena was not included in the training set. We also removed the Lena image from the Set 14 test dataset and validated the accuracy of each method again. As the PSNR shown in Table 4, the results indicate that after removing Lena image, our TinyLUT still achieves competitive results and does not affect the ranking of our method.

Table 4 The 4x SR experiment for Set14 without Lena

[1] Shaker et al. "SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications", ICCV 2023

[2] Howard et al. "Searching for mobilenet-v3", CVPR 2019

[3] Sandler et al. "Mobilenet-v2: Inverted residuals and linear bottlenecks", CVPR 2018

[4] Woo et al. "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders", CVPR 2023

[5] Liu et al. "A ConvNet for the 2020s", CVPR2022

[6] Zhang et al. "Efficient CNN Architecture Design Guiede by Visualization", ICME 2022

[7] Li et al. "MuLUT: Cooperating Multiple Look-Up Tables for Efficient Image Super-Resolution", ECCV2022

Thanks for the authors' reply. Part of my concerns are addressed. Thanks for the efforts of the Ethics reviewers. Although the authors conducted detailed experiments to demonstrate the advantages of the proposed TinyLUT and its variants, I still believe that accelerating the CNN-based image restoration methods is not very convincing, even considering the edge device scenario where domain-specific accelerator (DSA) is still an option. In high-level vision tasks, infering ViT-based models on edge devices has been investigated [1, 2, 3].

[1] Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In European Conference on Computer Vision, 2022 [2] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Swiftformer: Efficient additive attention for transformerbased real-time mobile vision applications. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023 [3] Shashank Nag, Logan Liberty, Aishwarya Sivakumar, Neeraja J Yadwadkar, Lizy Kurian John. Lightweight Vision Transformers for Low Energy Edge Inference. ISCA 2024 workshop MLArchSys (Machine Learning for Computer Architecture and Systems).