Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design

Dear Reviewer PF2v,

Thank you for your review and suggestions on our paper.

W1:It would be more comprehensive to add ablation studies to understand the contribution of each component of the Dy-DCA framework and the compiler-level optimizations would offer a clearer picture of their individual and combined effects on performance and quality.

Thank you for the suggestion. The high video quality of Dy-DCA attributes to the connection of dynamism of the neural network to the context of video content. The most important factors that affects the video quality would be the the granularity of how we group different content to groups and feed them to different dynamic neural network paths. We analyze the different number of paths in the network structure of Dy-DCA. As shown in the following table, more paths bring higher PSNR by introducing fine-grained data patch groups with more trainable parameters, which sacrifices some latency at the user end. Due to limited time, the results in our submitted paper were mainly conducted on a two-path scenario, but we will add the ablation study in the final revision.

W2:variety of hardware architectures, including both high-end and low-end devices

Thank you for the suggestion. Our design mainly targets mobile device. In order to demonstrate high-end and low-end difference, we chose a Samsung Galaxy S10 cell phone with Qualcomm Snapdragon 855 mobile platform[R1] released 5 years ago, which can be viewed as the low-end devices in current stage. The high-end hardware device is OnePlus 11 mobile phone equipped with a Snapdragon 8 Gen 2 processor [R2] used in our paper. As shown in the following table, our method is able to achieve 1.35$\times$ speedup on Snapdragon 855 and 1.32$\times$ speedup on Snapdragon 8 Gen 2. Thus, the compiler-level optimization design is able to accelerate inference speed among devices with different hardware conditions.

[R1]Snapdragon 855. https://www.qualcomm.com/products/mobile/snapdragon/smartphones/snapdragon-8-series-mobile-platforms/snapdragon-855-mobile-platform

[R2]Snapdragon 8 gen 2. https://www.qualcomm.com/products/mobile/snapdragon/smartphones/snapdragon-8-series-mobile-platforms/snapdragon-8-gen-2-mobile-platform.

W3: The PSNR gain is marginal, though the speedup is shiny.

Our design seeks to achieve speedup and the real-time inference on the mobile devices. In our initial experimental settings, we aim to achieve highest acceleration ratio. So we set the number of paths to 2 in the Dy-DCA network structure, which is coarse-grained for video content clustering but achieves highest acceleration speed since the computation graph is less complex. We further perform more experiments, and we find that more paths can improve the PSNR significantly while only increasing latency slightly. The reason is that more fine-grained data patch groups with more trainable parameters can improve learning performance, and more paths will not affect the execution efficiency since we still use one single model for inference. Although other method achieve similar PSNR in Table 2, our on-device latency is roughly 2 to 5 times faster, as shown in Table 3. Also, other method are far from real-time requirement.

Q1: visually the performance seems close

The visual results come from the $\times2$ scaling factor. Therefore, the PSNR is very high (the PSNR of most video topics is more than 45dB) and the performance distance among different methods is narrow. Another potential reason is that we randomly pick the frames in a video to do a visual comparison, while the PSNR results are averaged over each frame in the video. We will consider using the $\times4$ scaling factor for a more pronounced comparison.

Dear Reviewer PF2v,

We highly value the time and dedication you've invested in reviewing our submission. As the deadline for the ICLR rebuttal nears on the 22nd, your prompt feedback would immensely aid us in writing a thorough response, allowing us to comprehensively address the key points you have highlighted. Thanks.

Dear Reviewer nwgm,

We sincerely appreciate your thoughtful comments about our work. Regarding the questions you raised, we believe they are important points that merit further attention.

W1: The paper lacks a detailed discussion of the limitations and potential drawbacks of the proposed solution.

Thanks for the suggestion. In our paper, we only demonstrate the compiler acceleration framework on video resolution tasks based on CNN. As the vision transformer performs well on many visual tasks, the transformer-based dynamism and video super-resolution need further discussion.

W2:The paper does not provide a detailed comparison of the proposed solution with other state-of-the-art methods in terms of model size and computational complexity.

Thank you for your suggestion. The model we choose is based on the previous works by STDO[R1], EMT[R2] and CaFM[R3], and we compare with STDO and EMT these two state-of-the-art methods with the same model architecture.
In terms of model size, we do show the model size in Table 3 and Table 4. In Table 3, we achieve lower memory consumption and end-to-end latency with a smaller model size. In Table 4, we achieve higher PSNR and lower I/O overhead with a smaller model size. As for computational complexity, we believe that it is more convincing to show an on-device end-to-end inference speed (as shown in Table 3) as a stronger demonstration beyond pure theoretical computational complexity.

[R1]Li G, Ji J, Qin M, et al. Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting, CVPR 2023.

[R2]Li X, Liu J, Wang S, et al. Efficient meta-tuning for content-aware neural video delivery, ECCV 2022.

[R3]Liu J, Lu M, Chen K, et al. Overfitting the data: Compact neural video delivery via content-aware feature modulation, ICCV 2021.

W3:The paper only provide PSNR performance. Additional subjective quality (such as MS-SSIM) should be measured.

Thank you for the suggestion. We list partial results on the UVG dataset with the scaling factor $\times2$ with WDSR backbone. We will add full SSIM results in the later revision.

W4:The paper does not provide ablation experiments.

Thank you for the suggestion. We evaluate the different number of paths in the network structure of Dy-DCA. We compare the PSNR and the latency tested on mobile GPU for different dynamic paths of Dy-DCA. As shown in the following table, more paths bring higher PSNR by introducing fine-grained data patch groups with more trainable parameters, which sacrifices some latency at the user end. Due to limited time, the results in our paper mainly conducted on a two-path scenario, but we will add the ablation study in the final revision.

We also test different video length to show our design is capable of handling longer and more complex video contents. We select 2-min game video in VSD4K [R1] and combine game-45s video and the vlog-45s video together into a 90s-long video (combine-90s). We conduct experiments using WDSR as a backbone with a scaling factor of 4. These two videos are longer and more complex when compared with the videos we show in Table 2 in our paper. As we can see in the following table, the 2-min game video is still maintaining an acceptable PSNR, and the PSNR of combine-90s is close to the average value of overfitting two videos (game-45s and vlog-45s). The results show that our proposed framework can work well even in longer videos with complex contents. We will include the ablation study on different video length in the final revision.

[R1]Liu J, Lu M, Chen K, et al. Overfitting the data: Compact neural video delivery via content-aware feature modulation, ICCV 2021.

Dear Reviewer nwgm,

Your time and effort in reviewing our submission are greatly appreciated. As the deadline for the ICLR rebuttal approaches on the 22nd, your timely feedback would greatly assist us in preparing a comprehensive response, ensuring that we can thoroughly address any points you have highlighted. Thanks.

Dear Reviewer wfAU,

Thank you for your review and suggestions on our paper.

The paper rightly points out that switching the Super-Resolution (SR) model from one chunk to another can be costly. Could the solutions proposing sparse incremental model changes address this issue (refer to [1] for an example)?

The method proposed by [R1] can adaptively fit different content with incremental model changes in a single model. However, when the video content becomes complicated, the super-resolution quality of [R1] drops dramatically. For example, in the following table, the inter-45s and game-45s have richer information. Compared with our method, the performance of SRVC drops a lot when using WDSR as the backbone with a scaling factor of $\times2$. Especially for gmae-45s, a 31.03 dB is not acceptable for a high-quality video requirement.

Furthermore, our method is based on algorithm and compiler co-design, which is more favorable to practical usage. The operators and data flow can be fully supported by mobile devices.

[R1] Khani, Mehrdad, Vibhaalakshmi Sivaraman, and Mohammad Alizadeh. "Efficient video compression via content-adaptive super-resolution."

The unique contributions in Section 2.3 are difficult to comprehend in their current form. How does this categorization of tensors encompass all use cases? What are the limitations of the proposed compiler? Is it applicable solely to SR or also to convolutional models?

We classify the operators used for modern DNNs (specifically 150 operators that are included in the widely-used ONNX (Open Neural Network Exchange) format) into 4 categories. Therefore, we do consider all use cases.

Essentially, our proposed algorithm and compiler co-design framework can target all dynamic situations. It is a general optimization framework for dynamic neural networks. Besides the SR task, we further test it on object detection tasks with YOLO-V6 [R1] and language model CodeBERT [R2] with dynamic input size. We also compare our method with MNN, and the results show that our approach works well on other types of network architectures (e.g., CNN and transformer) and tasks, and consistently outperforms MNN in terms of storage and speed.

[R1] Li C, Li L, Jiang H, et al. YOLOv6: A single-stage object detection framework for industrial applications

[R2] Feng Z, Guo D, Tang D, et al. Codebert: A pre-trained model for programming and natural languages

Dear Reviewer wfAU,

We appreciate the time and effort you have dedicated to reviewing our submission. As the deadline for the ICLR rebuttal approaches on the 22nd, we kindly request your feedback at your earliest convenience. This will allow us sufficient time to further address any potential questions or concerns you may have raised. Thanks.

Dear Reviewer J3a9,

We sincerely appreciate your thoughtful comments about our work.

The presentation of the paper, particularly in Section 2.2 and Section 2.3, needs significant improvement due to a lack of details.

Thank you for your suggestion. We understand the importance of providing comprehensive and detailed information in these sections to enhance the clarity and depth of our work. We are committed to undertaking thorough revisions for Sections 2.2 and 2.3.

Dear Reviewer J3a9,

Thank you very much for taking the time to review our submission. As the deadline for the ICLR rebuttal approaches on the 22nd, we eagerly await your thoughts on our rebuttal and any further considerations you may have. Thanks.