Awakening Collective Wisdom: Elevating Super-Resolution Network Generalization through Cooperative Game Theory

Computational costs incurred by our EATS

The two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. This increment is deemed acceptable, especially considering the significant improvement in generalization capability. Additionally, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase. We have added the discussion in the Section C of Appendix.

Clarification about ablation studies about the impact of varying layer ratios involved in EATS

Thank you for bringing up this concern. The involved layers in our proposed EATS are randomly sampled in advance before the training process. Subsequently, the EATS is applied to the selected layers. The experimental results consistently demonstrate that involving more layers leads to better generalization performance. This aligns with our objective of awakening a greater number of neurons and promoting equitable contributions across all neurons.

About dropout operation

The conclusions draw from the previous work [1] indicates that employing dropout only at the last convolutional layer improves generalization performance, while even causing drop in performance at other layers. Therefore, we conducted a statistical analysis of channel responses in the $2^{nd}$ and last output layers of SRResNet, SRResNet with dropout, and SRResNet-EATS (all trained on the Five5K dataset) across 100 randomly sampled unseen images.
The visualizations are detailed in Section H of the Appendix. The results indicate that while dropout operation can alleviate the co-adaptation phenomenon in the last layer, it fails to address the issue in the shallow layers. In contrast, incorporating our proposed training strategy effectively mitigates co-adaptation and achieves equitable channel responses in both shallow and deep layers.

[1] Kong X, Liu X, Gu J, et al. Reflash dropout in image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 6002-6012.

Ablation studies

Thank you for your insightful suggestion. Ablation studies on the number of filters involved in each iteration are detailed in Section F of the Appendix. Our experiments explored configurations with 1 (default), 3, and 5 filters engaged in each iteration. Compared to the default of 1 filter, involving 3 filters achieved only a slight improvement. This decline may be attributed to increased randomness and instability during the training process.

The convergence of the awakening regularization

Thanks for the suggestion. We have incorporated the visualizations of the original loss, $L_{ori}$, and our awakening regularization, $L_{awa}$, during the training process of SRResNet-EATS and RRDBNet-EATS. These visualizations can be found in Section D of Appendix. Through the plots, it is evident that the awakening regularization progressively decreases and converges with the training iteration.

Training time incurred by our EATS

The two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. This increment is deemed acceptable, especially considering the significant improvement in generalization capability. Additionally, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase. We have added the discussion in the Section C of Appendix.

Activating channel responses in the deep layer

We conducted an analysis of channel responses from a deep layer, specifically the output block, in SRResNet (trained on Five5K) on 100 randomly sampled unseen images. The visualizations are presented in Section H of the Appendix. Similar to the observed phenomenon in the shallow layer, the deep layer of the SRResNet exhibits co-adaptation, where a few channels are highly activated while others are inhibited. In contrast, SRResNet incorporated with our EATS demonstrates equitable and activated channel responses. This indicates that our proposed training strategy is effective in alleviating the co-adaptation and activating channel responses from shallow to deep layers.

Evaluation on attention-based algorithm

Thank you for your valuable suggestion. Given the time constraints, we have evaluated the effectiveness of our proposed EATS using a lightweight variant of HAN [1] with x4 scaling. This lightweight version comprises approximately 1/20 of the parameters found in the original network. To maintain consistency with the experimental settings outlined in the manuscript, we have re-trained both the lightweight HAN and HAN-EATS, incorporating high-order degradation modeling. The quantitative results are presented below. Additionally, we plan to conduct further experiments on the x2 scale and will include the updated results in the Section B of Appendix upon acceptance of this paper.

[1] Niu, Ben, et al. "Single image super-resolution via a holistic attention network." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. Springer International Publishing, 2020. https://arxiv.org/abs/2008.08767

Typos

Thank you for bringing these typos to our attention. We appreciate your keen observation. The corrections have been addressed in the revision.

Clarification regarding the parameters involved in our training paradigm

Firstly, it's essential to emphasize that our proposed Erasing and Awakening Training Scheme (EATS) does not introduce additional training parameters. Within a training iteration, we conduct a forward propagation of the original network, $f_θ$, calculating the original loss function, $L_{ori}$. Subsequently, our EATS involves randomly replacing an erased filter with a disrupted filter based on the original network, resulting in a modified network after erasure, $f_{θ^{'}}$. We then calculate the awakening regularization, $L_{awa}$, through a second forward propagation. The network is update based on the combination of these two loss functions. Therefore, it's crucial to note that our EATS only requires two forward propagations, as opposed to employing two separate neural networks.

Secondly, the two forward propagations in our training paradigm marginally increase training times, approximately by 1.4 times compared to the corresponding original networks. This increment is deemed acceptable, especially considering the significant improvement in generalization capability.

Lastly, it's important to highlight that our training paradigm is exclusively applied during the training process and does not impose any additional computational burden during the inference phase.

We have added the discussion in the Section C of Appendix.