Lightweight Image Super-Resolution via Flexible Meta Pruning

Q5-1: The introduction of the proposed methods appears somewhat confusing. The proposed pruning method combines both structured and unstructured pruning, but it is difficult to distinguish their specific implementations and how they are integrated when reading this paper. Are channels being pruned and weights set to zero, or is it only utilizing sparse regularization for structural pruning.

A5-1: Thank you for your valuable feedback.

(1) Specific implementations. We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: First, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. Second, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

(2) How they are integrated? During the optimization, we set regularization factors $\lambda_C$=0.1 and $\lambda_W$=0.02 for both structured pruning and unstructured pruning. If the compression ratio of one pruning method is achieved, the optimization method for either the channel vectors or the weight indicators is stopped while the other one continues. The whole algorithm converges if both of the compression targets are achieved.

(3) Network pruning output. The pruned channels are removed. The pruned weights are set to zero. We conduct both structural and unstructural pruning.

We will revise the paper and make it clearer based on your comments.

Q5-2: The comparison with existing methods lacks a comparison of runtime.

A5-2: Thanks for the valuable suggestion. We provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090.

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Q5-3: Table 6 is challenging to understand.

A5-3: Thanks for the valuable comment.

(1) Table 6 is about the results of different convergence criteria. We provided explanations about convergence criteria in Sec. 4.4 of the main paper.

(2) In Sec. 4.4, we provided a brief analysis about Table 6. Meanwhile, in Sec. 2.2 of the supplementary material, we provided further analyses about convergence criteria.

Based on the reviewer's comments, we will add more content from supplementary material to the main paper, trying to make Table 6 easier to understand.

Q5-4: The advantages of this method in terms of flexibility compared to standalone structured and unstructured pruning need to be further emphasized.

A5-4: Thanks for the valuable suggestion.

(1) In Sec. 3.1 of the main paper, we give some reasons why we need flexible pruning. This is a part of our motivation, where we also briefly discuss the advantages of our FMP over standalone structured and unstructured pruning.

(2) In Sec. 1.1 of the supplementary material, we provide more clarifications and explanations about the motivation of our method.

We will revise Sec. 3.1 of the main paper to further emphasize our advantages as suggested by the reviewer.

Q5-5: The proposed pruning method combines both structured and unstructured pruning, but it is difficult to distinguish their specific implementations and how they are integrated when reading this paper. Are channels being pruned and weights set to zero, or is it only utilizing sparse regularization for structural pruning.

A5-5: Thanks for the valuable question.

(1) Specific implementations. We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: First, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. Second, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

(2) How they are integrated? During the optimization, we set regularization factors $\lambda_C$=0.1 and $\lambda_W$=0.02 for both structured pruning and unstructured pruning. If the compression ratio of one pruning method is achieved, the optimization method for either the channel vectors or the weight indicators is stopped while the other one continues. The whole algorithm converges if both of the compression targets are achieved.

(3) Network pruning output. The pruned channels are removed. The pruned weights are set to zero. We conduct both structural and unstructural pruning.

We will revise the paper and make it clearer based on your comments.

Q5-6: As a pruning method, how does it perform when applied to existing super-resolution networks?

A5-6: Thanks for the valuable question about the generalization ability of our method.

(1) In this work, we mainly apply our FMP to CNN-based EDSR and newly designed LSRB. To further demonstrate the generalization ability of the network pruning methods, we should also have applied FMP to several classic image SR models. Although LSRB is closely related to several leading lightweight networks (e.g., IMDN, RFAN, RLFN), it is more convincing to try more SR models.

(2) It is worth investigating FMP in Transformer-based methods (e.g., SwinIR).

(3) Our FMP is a general idea for flexible structured and unsructred pruning. We can apply FMP to CNN-based and Transformer-based image SR methods.

We will further investigate such a direction and provide more baselines.

Q4-5: LSRB represents a lightweight network achieved by replacing residual blocks in RLFN with simplified versions, and FMP is a lightweight method. Table 3 indicates that LSRB+FMP exhibits higher model parameters and computational complexity compared to RLFN. Could you provide an explanation for this discrepancy? Furthermore, LSRB+FMP demonstrates increased inference speed, despite the primary goal of parameter optimization not being fast adaptation in meta-learning. The reasons behind the improved inference speed is expected to be addressed and analyzed.

A4-5: Thanks for the valuable suggestion.

In Table 3 of the main paper, our LSRB has slightly larger number of parameters and FLOPs, but achieves faster speed. The reasons are mainly summarized as follows:

(1) In the building block RLRB in RLFN, there are three convolutional layers (each layer is followed by ReLU) with channel number set as 52. While, the building block in our LSRB-6-48 has two convolutional layers (only the first convolutional layer is followed by ReLU) with channel number as 48.

(2) Currently, in PyTorch, as used in our work, networks with wider width (i.e., larger channel number) will run slower than narrower one with similar model size. So, LSRB runs faster than RLFN.

(3) As we state in Effectiveness of LSRB in Sec. 4.3, it is promising to further reduce its redundant parameters with our proposed FMP. In our research, we can hardly use our common GPU with PyTorch to achieve further acceleration. But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

We will include those clarifications and explanations in the revised paper.

Q4-6: FMP introduces kernel weight pruning in addition to DHP, and the results in Table 4 is intended to show that FMP outperforms DHP. However, the combination of multiple pruning techniques may not always lead to an improved result. Besides, it seems that the higher the prune ratio, the better for the performances of both DHP and FMP. Both appear to be counterintuitive to some extents. Therefore, more detailed analysis and explanation are necessary here.

A4-6: Thanks for the valuable comments and suggestions.

(1) In the first sentence of Sec. 4.3 of main paper, we state that the training process will stop if meets convergence or reach the maximum iterations 300K. We find that our FMP would meet convergence with fewer iterations than DHP. But, this is mainly based on the validation performance to early stop the training.

(2) When we stop training our FMP with fewer iterations than DHP, our model may not achieve obvious improvements than DHP on other datasets, like Urban100.

(3) We allow early stop for the ablation study is mainly to save the training time with limited resource. We hope to investigate if it is possible for us to train the lightweight models with fewer iterations, which we think is also favorable in real practice.

(4) When the prune ratio becomes higher, the training needs more iterations. Models with more training iterations would actually get higher performance.

We will revise the paper with more detailed analyses and explanations.

Q4-7: Formula (6) is similar to formula (7) because it needs to calculate the norm as well. However, why not trying to choose the suitable norm through experiments in the same way adopted to determine the norm in formula (7)?

A4-7: Thanks for the interesting question. The main reasons are as follows:

(1) Formula (6) is for channel pruning, which has been widely investigated before in image SR, like DHP and ASSL. In previous work, we find that $L_1$ norm is a good choice for channel pruning.

(2) We choose $L_1$ norm for channel pruning so that we can keep the same as previous channel pruning works.

(3) Unstructured pruning is seldom investigated in image SR before. So, we try different choices of norm and determine the final one for our experiments.

So, in our work, we mainly investigate the choices of norm for Eq. (7), namely unstructured pruning.

Q4-8: In ESA module of LSRB, spatial attention is used instead of convolution. Is this module involved in the proposed pruning method?

A4-8: Thanks for the question. Yes, this module is involved in our pruning method.

Q4-3: The manuscript does not sufficiently address the compatibility and generalizability of the proposed FMP method. Although FMP was developed for lightweight super-resolution, experiments have only been conducted on one model, LSRB. The suitability of applying FMP to other super-resolution models remains unexplained.

A4-3: Thanks for the valuable suggestion.

(1) In this work, we mainly apply our FMP to CNN-based EDSR and newly designed LSRB. To further demonstrate the generalization ability of the network pruning methods, we should also have applied FMP to several classic image SR models. Although LSRB is closely related to several leading lightweight networks (e.g., IMDN, RFAN, RLFN), it is more convincing to try more SR models. We can mainly replace the LSRB with other SR methods and use hypernetwork to generate their corresponding parameters.

(2) It is worth investigating FMP in Transformer-based methods (e.g., SwinIR).

(3) Our FMP is a general idea for flexible structured and unsructred pruning. We can apply FMP to CNN-based and Transformer-based image SR methods.

We will further investigate such a direction and provide more baselines.

Q4-4: What optimization method was employed for the FMP method outlined in the manuscript? While the introduction section mentions SGD and proximal gradient descent, the experimental setup section suggests the use of Adam for FMP. This inconsistency is expected to be clarified.

A4-4: Thanks for pointing this part out.

(1) We optimize the channel vectors and weight indicators with proximal gradient and SGD. There are mainly two reasons: First, SGD with regularization could already lead to acceptable weight sparsity while the channel sparsity could be harder to achieve. Thus, an advanced algorithm proximal gradient descent (PGD) is used for channel pruning to accelerate the convergence of the optimization procedure. Second, by separating the optimization, we are more flexible in controlling the contribution of channel pruning and weight pruning.

(2) When we train the image SR network with flexible meta pruning jointly, we have to update the whole network parameters. In this procesure, we use the general ADAM optimizer, which is also adopted in most of other image SR methods.

We will further clarify them in the paper based on the suggestion.

Q4-5: LSFB represents a lightweight network achieved by replacing residual blocks in RLFN with simplified versions, and FMP is a lightweight method. Table 3 indicates that LSFB+FMP exhibits higher model parameters and computational complexity compared to RLFN. Could you provide an explanation for this discrepancy? Furthermore, LSFB+FMP demonstrates increased inference speed, despite the primary goal of parameter optimization not being fast adaptation in meta-learning. The reasons behind the improved inference speed is expected to be addressed and analyzed.

A4-5: Thanks for the valuable suggestion.

In Table 3 of the main paper, our LSFB has slightly larger number of parameters and FLOPs, but achieves faster speed. The reasons are mainly summarized as follows:

(1) In the build block RLRB in RLFN, there are three convolutional layers (each layer is followed by ReLU) with channel number set as 52. While, the building block in our LSFB-6-48 has two convoluional layers (only the first convolutional layer is followed by ReLU) with channel number as 48.

(2) Currently, in PyTorch, as used in our work, networks with wider width (i.e., larger channel number) will run slower than narrower one with similar model size. So, LSFB runs faster than RLFN.

(3) As we state in Effectiveness of LSFB in Sec. 4.3, it is promising to further reduce its redundant parameters with our proposed FMP. In our research, we can hardly use our common GPU with PyTorch to achieve further accelaration. But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

We will include those clarifications and explanations in the revised paper.

Q4-1: The manuscript's contribution seems to be incremental, primarily involving the addition of kernel weight pruning based on DHP [1]. While the primary distinction between the proposed method and DHP lies in kernel pruning, the results shown in Table 6 indicate that simultaneous channel and kernel pruning may not significantly show effectiveness over channel pruning alone.

A4-1: Thank you for your valuable feedback.

(1) In the first sentence of Sec. 4.3 of main paper, we state that training process will stop if meets convergence or reach the maximum iterations 300K. We find that our FMP would meet convergence with fewer iterations than DHP. But, this is mainly based on the validation performance to early stop the training.

(2) When we stop training our FMP with fewer iterations than DHP, our model may not achieve obvious improvements than DHP on other datasets, like Urban100.

(3) We allow early stop for the ablation study is mainly to save the training time with limited resource. We hope to investigate if it is possible for us to train the lightweight models with fewer iterations, which we think is also favorable in real practice.

Of course, we will revise the paper and make this part clearer based on the reviewer's comments.

Q4-2: Some of the experimental comparisons may be perceived as lacking in both fairness and completeness when it comes to demonstrating the effectiveness of the method. Notably, in Figure 1, the analysis exclusively focuses on showcasing the influence of FMP. However, it's important to note that the baseline models utilized for evaluating FMP and other comparative methods differ. This dissimilarity in the baseline models pose challenges when attempting to assess the precise impact of FMP. Besides, in subsection 4.2, the experimental comparisons fail to cover some classic lightweight SR models including RLFN and ELAN, which are also mentioned in Section 1 and Section 2.

A4-2: Thanks for the valuable suggestions.

(1) In Figure 1, we mainly want to give an overall view of our method FMP. Specifically, we show that an image SR baseline with FMP can achieve comparable or better results than recent methods.

(2) To showcase the influence of FMP, we main provide corresponding results in ablation part.

(3) We compare with more recent works, like SRPN [Ref1], RLFN [Ref2], and ELAN [Ref3]. We provide PSNR/SSIM values of compared methods in the following table.

According to the table, we can see that our FMP achieves comparable performance than recent leading methods.

We will revise the paper with more recent methods based on the suggestios.

[Ref1] Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning, ICLR, 2022

[Ref2] Residual Local Feature Network for Efficient Super-Resolution, CVPRW, 2022

[Ref3] Efficient Long-range Attention Network for Image Super-Resolution, ECCV, 2022

Q3-4: More recent methods can be compared. For example, ICLR-2022-SRPN can also be discussed and compared. [ICLR-2022-SRPN] Zhang et al., Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning, ICLR, 2022

A3-4: Thanks for the valuable suggestions. We compare with more recent works, like SRPN [Ref1], RLFN [Ref2], and ELAN [Ref3]. We provide PSNR/SSIM values of compared methods in the following table.

According to the table, we can see that our FMP achieves comparable performance to recent leading methods.

We will revise the paper with more recent methods based on the suggestions.

[Ref1] Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning, ICLR, 2022

[Ref2] Residual Local Feature Network for Efficient Super-Resolution, CVPRW, 2022

[Ref3] Efficient Long-range Attention Network for Image Super-Resolution, ECCV, 2022

Q3-5: The performance is obviously better than others. Did the authors use self-ensemble during the test phase?

A3-5: Thanks for pointing this out. No, we did not use self-ensemble during the test phase. In Sec. 4.2 of the main paper, we have stated that we provide our quantitative results without self-ensemble.

Q3-6: In practice, does FMP save more running time than channel or weight pruning? Or does FMP need specific hardware to reach obvious reduction of resource (e.g., GPU memory and running time)?

A3-6: Thanks for the valuable comments.

(1) Currently, with PyTorch and a common GPU (e.g., RTX 3090), our FMP can hardly save more running time than channel pruning. This is mainly because unstructured pruning is not specifically optimized and can hardly achieve further acceleration.

(2) But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

We will include those discussions in the revised paper.

Q3-1: The performance gains shown in Table 4 seem to be marginal. Did the results of DHP and FMP obtained by using same number of iterations or not? According to the paper, it seems that they may use different iterations based on the convergence criteria. If FMP uses less iterations and achieves similar or better results than DHP, then it makes sense. Please clarify this.

A3-1: Thanks for the valuable comments.

(1) In the first sentence of Sec. 4.3 of the main paper, we state that the training process will stop if meets convergence or reaches the maximum iterations 300K. We find that our FMP would meet convergence with fewer iterations than DHP. But, this is mainly based on the validation performance to early stop the training.

(2) When we stop training our FMP with fewer iterations than DHP, our model may not achieve obvious improvements than DHP on other datasets, like Urban100.

(3) We allow an early stop for the ablation study mainly to save training time with limited resources. We hope to investigate if it is possible for us to train lightweight models with fewer iterations, which we think is also favorable in real practice.

(4) When the prune ratio becomes higher, the training needs more iterations. Models with more training iterations would actually get higher performance.

We will clarify them in the revised paper with more detailed analyses and explanations.

Q3-2: The authors mainly apply FMP to CNN based networks, like EDSR and LSRB. It is also important to apply this method to other structures, like Transformer-based methods. So that, we can see how well its generalization ability is.

A3-2: Thanks for the valuable question about the generalization ability of our method.

(1) In this work, we mainly apply our FMP to CNN-based EDSR and newly designed LSRB. To further demonstrate the generalization ability of the network pruning methods, we should also have applied FMP to several classic image SR models. Although LSRB is closely related to several leading lightweight networks (e.g., IMDN, RFAN, RLFN), it is more convincing to try more SR models.

(2) It is worth investigating FMP in Transformer-based methods (e.g., SwinIR).

(3) Our FMP is a general idea for flexible structured and unstructured pruning. We can apply FMP to CNN-based and Transformer-based image SR methods.

We will further investigate such a direction and provide more baselines.

Q3-3: When applying the proposed FMP to other SR methods, do people have to modify too much parts of the new models? It seems that FMP can also be used for other image restoration methods. If so, it is better to discuss in the paper.

A3-3: Thanks for the interesting question.

(1) In this work, we mainly use image SR as an example to incorporate flexible meta pruning.

(2) Our pruning method is a general one and can be easily applied to image restoration methods. For other image restoration models, we can still use FMP to conduct both structured and unstructured pruning. We do not have to make too many modifications.

We will add those discussions to the revised paper.

Q2-3: Table 4 compares only one pruning method because other methods use pretrained SR backbones. However, to better justify the effectiveness of the proposed method, the authors can add more comparisons by also using a pretrained backbone. The pretrained backbone does not increase the inference cost and should not be viewed as a drawback of lightweight SR methods.

A2-3: Thanks for the valuable suggestions. We adopt ASSLN for comparison. ASSLN needs a pretrained model. We still use EDSR-8-128 as the base model and train ASSLN with the same setting as others. We provide the PSNR values in the following table.

According to the table, we find that ASSLN performs comparable to or slightly better than DHP. Pretrained models would help improve the performance, especially in the current case. Specifically, the training process will stop if meets convergence or reaches the maximum iteration 300K.

Q2-4: According to Table 3, the backbone used by FMP already brings some improvements. Thus comparisons in Table 7 may be not fair. The authors are suggested to provide the results of applying ASSLN to LSRB.

A2-4: Thanks for the valuable suggestions.

We use the LSRB as the backbone and apply ASSLN to it. As time and GPU resource is limited, we only train the SR model with scale $\times$2. We provide PSNR/SSIM values in the following table.

According to the table, we find that ASSLN+LSRB achieves improvement over the original ASSLN. This observation shows the effectiveness of LSRB. Our FMP still obtains better performance than ASSLN+LSRB. It indicates that flexible pruning (i.e., structured pruning and unstructured pruning) could better reduce network redundancy than channel pruning in image SR.

It should also be noted that ASSLN has to use a large pretrained LSRB model. In contrast, our FMP does not need a pretrained model.

Q2-1: The authors argue that unstructured pruning does not contribute to actual acceleration. However, the experiments in Table 1 do not include comparisons of latency.

A2-1: Thank you for your valuable feedback.

(1) In Table 1 of the main paper, it is not easy to provide inference time for all the compared methods. Many of them are officially implemented with different frameworks. For example, SRCNN, FSRCNN, VDSR, DRCN, and MemNet are officially implemented with Caffe. LapSRN is officially implemented with MatConvNet. To compare their inference time fairly, we should implement them under the same codebase and framework and run the models with the same GPU. Currently, we do not have such a codebase. But, it is important to have such a benchmark codebase in the future.

(2) Instead, we provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090.

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Q2-2: Some leading lightweight methods are missed in Table 1, e.g. DIPNet[1], VapSR[2], Omni-SR[3]. The authors are suggested to provide comparisons over varied model sizes to verify the effectiveness of the proposed method, because we can not justify whether it can maintain its advantage among smaller models (~400 K params).

[1] Yu et al., DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution, CVPR Workshop, 2023

[2] zhou et al., Efficient Image Super-Resolution using Vast-Receptive-Field Attention, ECCV Workshop, 2022

[3] wang et al., Omni Aggregation Networks for Lightweight Image Super-Resolution, CVPR, 2023

A2-2: Thanks for the valuable suggestions.

We compare with DIPNet[1], VapSR[2], and Omni-SR[3]. PSNR/SSIM values are provided in the following table. It should be noted that we cite the results of DIPNet, VapSR, and Omni-SR from their papers.

According to the above table, we have the following analyses.

(1) All the compared methods (i.e., DIPNet, VapSR, Omni-SR) have new network designs. Those new designs can achieve obviously better performance than our ResNet based backbone LSRB.

(2) Network parameter number matters very much. Ommni-SR achieves the best performance with the largest model size.

(3) In our work, we mainly focus on baselines with high speed, where we design new models based on the champion solution RLFN at NTIRE'22 Efficient SR. Our FMP can also be applied to new SOTA lightweight image SR methods shown above. It is important and promising to investigate applying our FMP to the above methods in the future.

As suggested by the reviewer, we will further investigate smaller models by pruning more channels and weights.

Dear Reviewer znMB,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q1-1: It's unclear how the compressed model would translate to the practical inference speed up, considering there are unstructured weight pruning.

A1-1: Thank you for your valuable feedback.

(1) Our designed lightweight image SR baseline LSRB reduces inference time than the champion solution RLFN at NTIRE'22 Efficient SR challenge. As shown in Table 3 of the main paper, our LSRB achieves much faster than RLFN.

(2) In our work, we can hardly use our common GPU with PyTorch to achieve further acceleration for unstructured pruning. But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning.

(3) We apply FMP to LSRB. With PyTorh and common GPU, the speed up of FMP mainly comes from LSRB and channel pruning. As a result, our FMP can still achieve relative speed up.

Q1-2: The ideas of channel and weight pruning have been around for a while. It is not clear what's new with the proposed approach?

A1-2: Thanks for pointing this out.

(1) Although the idea of combining unstructured pruning and structured pruning for image SR is straightforward, how to design an algorithm to achieve flexible pruning is still worth investigating. Our FMP could automatically allocate parameters and computation budget for unstructured and structured pruning.

(2) The introduced technique such as weight indicator extends the usage of hypernetworks from channel pruning to a wider scope of network (weight and channel) pruning.

Q1-3: What's the benefits of combining channeling pruning with unstructured weight pruning?

A1-3: Thanks for the question.

(1) Structured pruning and unstructured pruning are two important network compression methods that can cut down the model complexity of deep neural networks significantly.

(2) As we state in Effectiveness of LSRB in Sec. 4.3, it is promising to further reduce its redundant parameters with our proposed FMP. In our current research, we can hardly use our common GPU with PyTorch to achieve further acceleration.

(3) But, the inference time can be further improved by AI accelerators, which are specifically designed for unstructured pruning. As far as we know, there are some specific hardware designs and implementations for unstructured pruning in some institutions, where the inference time can further be reduced. However, we have no access to such hardwares right now.

(4) Instead, we provide the potential of further reducing the network redundancy with both structured and unstructured pruning.

Q1-4: In Table 3, instead of just the submodule, is it possible to show the inference speed up of the entire model?

A1-4: Thanks for the valuable suggestion.

We further provide inference time with other methods in the following Table. EDSR-16-256 means 16 residual blocks (RBs) with 256 channels for each Conv layer in RB. The inference time is tested on Urban100 ($\times$4) with an RTX 3090.

According to the inference time comparison, we can see that our FMP achieves less inference time than others.

Q1-5: What's the motivation and benefits of applying proximal gradient for channels vectors and weight indicators?

A1-5: Thanks for the interesting question.

(1) The general idea of this work is first introduced before elaborating on details of our flexible network pruning method. Structured pruning and unstructured pruning are two important network compression methods that can cut down the model complexity of deep neural networks significantly.

(2) Structured pruning leaves regular kernels after pruning, which is beneficial for the actual acceleration of the network.

(3) Unstructured pruning removes single weights in a kernel and can compress the network without sacrificing too much accuracy of the network.

(4) Since the channel vectors and the weight indicator are not intertwined by non-linear operations, we can decouple their optimization for network pruning by different methods.

Q1-6: How much performance gain can FMP harvest if using self-ensemble?

A1-6: Thanks for pointing this out. Self-ensemle in the test phase is to rotate and flip input, get different output, and then average them to get the final result.

(1) Self-ensemble can further achieve about 0.1-0.2 dB in PSNR for different scales and test dataset.

(2) However, self-ensemble would use several times of computation cost. For example, in image SR, one common practice of self-ensemble takes 8 times computational cost as the norm one without self-ensemble. So, in our paper, we do not use this strategy for further performance improvement.

Dear Reviewer Jqxo,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair