We greatly appreciate your valuable review comments . Below is our point-by-point response to your concerns and questions. Please let us know if there's anything we can clarify further.

Weakness 1: Potential of other data augmentations in AugKD.

Reason for adopting zooming augmentations: AugKD identifies and addresses a key challenge in KD for SR that the teacher model’s function of transferring knowledge to the student is overshadowed by the ground truth labels, as the teacher’s outputs are noisy approximations of high-resolution images. By introducing auxiliary distillation samples, AugKD allows the student model to extract meaningful prior knowledge from the teacher model’s outputs. The zoom-in and zoom-out augmentations were specifically chosen because they are well-aligned with SR tasks, generating images closely related to the training data distribution and avoiding distributional shifts. They also allow the student to learn not only from direct reconstructions but also from implicit task-relevant patterns, such as fine-grained textures (zoom-in) and broader spatial consistency (zoom-out).

Other data augmentations cannot effectively address the challenges in KD for SR, and might introduce distribution shifts to the training data. Hence, their effects on improving the student SR models are limited.

Weakness 2: Regularization with non-invertible augmentations

The non-invertible augmentations, such as cropping or translation, would fundamentally alter the spatial correspondences between teacher and student models' output and make them not comparable. And since the LR images are bicubic degradation of corresponding HR labels in training of SR, the augmentations like noising and blurring are not strongly related with the task and would making images distributed different from the training data. Thus, we prioritize invertible augmentations like flipping and rotation, which can preserve the spatial integrity of input images.

Nevertheless, we recognize the importance of exploring alternative augmentation and regularization strategies. While non-invertible augmentations may not be directly compatible with SR task's requirements, for other low-level CV tasks like image denosing and debluring, the label consistency regularization can be realized by passing differently noised or blurred images to the student and teacher model.

Question 1: Combination of AugKD and FAKD

The result in Tab.9 indicates that the introduction the auxiliary distillation samples and label consistency regularization into FAKD can significantly improve the student model's performance. The performance of the combined method (FAKD + AugKD) being slightly worse than AugKD alone can be attributed to two factors. First, the FAKD is detrimental for the student model, as shown in the main experiment results. And the hyper-parameters for the combined method were not specifically tuned to optimize the balance between FAKD's feature matching modules and AugKD, potentially leading to suboptimal results.

Question 2: Heterogeneous distillation experiments

The superior performance of AugKD on the X4 scale RCAN model in Tab. 3 compared to the heterogeneous distillation results in Tab. 5 is primarily due to the architectural alignment in homogeneous distillation. In Tab. 3, both the teacher and student models share the RCAN architecture, which facilitates effective transfer of task and network specific priors. In contrast, heterogeneous distillation (Tab. 5) involves different architectures for the teacher (e.g., EDSR or SwinIR), which can introduce challenges in knowledge transfer due to mismatched architectural representations. Furthermore, the large capacity gap between models may lead to suboptimal student performance, as shown in Table 11 of the supplementary material, where the EDSR and SwinIR teachers significantly outperform the RCAN teacher in isolation. These factors collectively explain the observed performance difference.

Question 3: Issues on the typos

Thank you for pointing out the misleading typo in the manuscript. The two notations denotes the same loss term. We will carefully revise the manuscript to reduce the reader's confusion.

Question 1: Training efficiency of AugKD

As shown in the comparison on X2 EDSR, the AugKD outperforms Logits-KD by 0.55dB PSNR with an increase of only 0.21s training time per step. Considering the significant performance gains in test time, the mild extra cost in training is acceptable.

Question 2: Rationale behind the choice zoom-in and zoom-out augmentations

In knowledge distillation for super-resolution models, the teacher model's intended function, transferring knowledge to student, is shaded by the ground truth HR labels, due to the nature that the teacher model's outputs are noisy approximation to the high-resolution images. So there is barely "dark knowledge" such as the inter-class relationship or label smoothing in the classification task. The AugKD addresses the issue and re-functions the teacher model by matching student and teacher models on the auxiliary samples that are generated efficiently and distribute closely with original training data.

The zoom-in and zoom-out augmentations were specifically chosen because they are well-aligned with the goals of auxiliary distillation sample generation in SR. Besides, they also allow the student to learn not only from direct reconstructions but also from implicit task-relevant patterns, such as fine-grained textures (zoom-in) and broader spatial consistency (zoom-out).

Question 3: Alternative inverse data augmentations for label consistency regularization

While translation is indeed an invertible augmentation, it is not included in the consistency regularization module because it does not align well with the pixel-level nature of SR. Translation introduces disjointed artifacts at image boundaries, which are uncommon during testing and might mislead the student model during training.

In the training of SR model, maintaining precise spatial correspondences in input and output is critical for accurately reconstructing high-resolution images. Mild translation can disrupt this correspondence, introducing inconsistencies that would negatively impact the learning process. Thus, we prioritize augmentations like flipping and rotation, which preserve the spatial integrity required for SR tasks.

Question 5: Applicability to other SR network backbones

The proposed AugKD method is logits-based and is not constrained by specific network architectures. This flexibility allows it to be applied to various SR backbones, including Mamba and other architectures.

To further demonstrate its applicability to various backbones, we provide the experiment results on a latest SR network, DRCT[1] below. We use the large version of DRCT as the teacher model and distill the DRCT SR network by different KD methods.

AugKD consistently outperforms previous SR KD methods on different backbones.

[1] Hsu, C. C., Lee, C. M., & Chou, Y. S. (2024). DRCT: Saving Image Super-resolution away from Information Bottleneck. arXiv:2404.00722.

We greatly appreciate your valuable review comments . Below is our point-by-point response to your concerns and questions. Please let us know if there's anything we can clarify further.

Weakness 1: Unclear visualization in Figure 2.

Thank you for your suggestion regarding Figure 2. To improve the visualization for better readability, we will replace the figure with a bar chart to clearly present the differences in PSNR values between training methods.

Weakness 2: Issues on adaptive zoom-in sample selection

In our exploratory experiments, we explored and compared several auxiliary distillation sample generation strategies: 1) adaptively select the zoom-in samples based on reconstruction difficulty, indicated by the PSNR between the teacher model’s output and the ground-truth HR labels, 2) equally split the HR into several grids and use them all as auxiliary distillation samples and 3) randomly pick an auxiliary distillation sample. The result below shows that the random selection of zoom-in sample slightly outperforms the alternative approaches. And given the increased computational overhead required for adaptive selection and training with more auxiliary distillation samples, random selection remains a more efficient and practical option.

We will include these experimental results in the supplementary material to provide evidence supporting our claim and to clarify the rationale behind randomly zoom-in sample selection.

Weakness 3 and Question 4: Motivation and applicability for label consistency regularization

Label consistency regularization is motivated by the need to improve the robustness and generalization capability of the student model in KD. Normal KD matches the student and teacher models on the same input images, while a student model with strong robustness and generalizability should be able to maintain such match on augmented inputs. By enforcing consistency between the outputs of the student model for augmented input and the unaltered outputs of the teacher model, the student is encouraged to learn invariant representations that align with the teacher’s knowledge, irrespective of input perturbations. This mechanism makes the student expose to diverse data distributions, and guides it to focus on and learn SR task-related information that remains unaffected by input perturbations.

The label consistency regularization is not specific to SR but is applicable to other low-level tasks such as denoising, deblurring, and image enhancement. These tasks share the characteristic of pixel-level input-output dependencies, and maintaining teacher-student consistency under transformations is crucial. By enforcing invariance to input variations, this regularization technique can improve the robustness and effectiveness of models for various low-level tasks.

Weakness 4: Concern on architecture constraints

In the experiment section, EDSR and RCAN were selected as benchmarks because they are widely used in existing KD studies for SR tasks. While many feature-based KD methods are specifically designed for CNN-based models, AugKD demonstrates its broad compatibility with both CNNs and Transformers, such as SwinIR, as it is logits-based and independent from specific network architectures.

To provide a more comprehensive evaluation of SwinIR, we conducted additional experiments comparing AugKD with FitNet and FAKD on the X4 scale SwinIR model. The results below show that prior feature-based KD methods, such as FitNet and FAKD, fail to improve the SwinIR model effectively, whereas AugKD achieves superior performance:

To further demonstrate its applicability and effectiveness for various architectures, we evaluated AugKD on a recent state-of-the-art SR network, DRCT [1]. Using the large version of DRCT as the teacher model, we distilled the X4 scale DRCT student network using different KD methods. The results confirm that AugKD provides notable improvements compared to other methods:

Hope these results highlighting AugKD’s flexibility and effectiveness across diverse SR architectures, can address your concerns about potential architecture constraints.

[1] Hsu, C. C., Lee, C. M., & Chou, Y. S. (2024). DRCT: Saving Image Super-resolution away from Information Bottleneck. arXiv:2404.00722.

Weakness 2, 3: Motivations and insights of augmentations

Augmentation impacts of auxiliary distillation samples: AugKD identifies and addresses a key challenge in KD for SR: the teacher model’s function of transferring knowledge to the student is overshadowed by the ground truth labels, as the teacher’s outputs are noisy approximations of high-resolution images. So there is barely "dark knowledge" like the inter-class relationship or label smoothing in the classification task. By introducing auxiliary distillation samples generated by zoom-in and zoom-out augmentations, AugKD allows the student model to extract meaningful prior knowledge from the teacher model’s outputs. They also allow the student to learn not only from direct reconstructions but also from implicit task-relevant patterns, such as fine-grained textures (zoom-in) and broader spatial consistency (zoom-out), that are challenging to capture with standard KD methods.

Augmentation impacts of label consistency regularization: the label consistency regularization further strengthens student model's generalization by enforcing consistent matches with teacher model across augmented views of the same input. It's motivated by the need to improve the robustness and generalization capability of the student model in KD. Normal KD matches the student and teacher models on the same input images, while a student model with strong robustness and generalizability should be able to maintain such alignment on augmented inputs. By enforcing consistency between the outputs of the student model for augmented input and the unaltered outputs of the teacher model, the student is encouraged to learn invariant representations that align with the teacher’s knowledge, irrespective of input perturbations. This mechanism makes the student expose to diverse data distributions, and guides it to focus on and learn SR task-related information that remains unaffected by input perturbations.

Prior KD methods failed to fully utilize such task-related augmentations because most of them relied on static feature-based approaches, leading to the sensitivity to architectural mismatches or limitation to high-level vision tasks. While the AugKD identifies the issues in KD for SR and proposed augmentation-based solutions accordingly.

We greatly appreciate your valuable review comments. Below is our point-by-point response to your concerns and questions. Please let us know if there's anything we can clarify further.

Weakness 1: Concerns about model performance

Thank you for your feedback regarding the reported improvements. The improvements achieved by AugKD are consistent across all networks, datasets, and SR scales, demonstrating its robustness and practical value. These performance gains are not merely marginal but stem from the introduction of auxiliary distillation samples and label consistency regularization. As shown in Table 3, on the Urban100 test set, AugKD improves the PSNR of RCAN networks over the strongest baseline KD method (the underlined results), by 0.17dB, 0.17dB, and 0.10dB on three SR scales. The PSNR improvements over training from scratch are 2 to 7 times larger than those achieved by the strongest baseline methods.

In addition to performance improvements, AugKD offers distinct advantages in applicability and flexibility. It works effectively with both CNN-based and Transformer-based architectures, addressing compatibility limitations of many existing methods. Its versatility is further proved by its integration into diverse tasks, such as SR network quantization and real-world SR scenarios. These factors collectively demonstrate the practical value and broad potential of AugKD.

We greatly appreciate your valuable review comments. Below is our point-by-point response to your concerns and questions. Please let us know if there's anything we can clarify further.

Weakness 1: Motivation and the role of the teacher’s outputs.

Thank you for pointing out the potential inconsistency between our statements in the introduction and the motivation section.

The statement in the introduction highlights the limitations of directly aligning the student with the teacher’s output caused by the inherent noise in the teacher’s predictions in super-resolution tasks. This does not intend to suggest that the teacher’s outputs are entirely unsuitable as learning materials, but rather that they are not effectively used. When there is the GT label ($\mathcal{L}\_{rec}$ is available), matching student and teacher models' outputs ($\mathcal{L}\_{kd}$) cannot effectively transfer the teacher model's knowledge.

Section 3.2 and Figure 2 illustrate that while existing KD methods fail to fully utilize the teacher model to guide the student, AugKD makes the teacher’s outputs to serve as valuable learning references and transfers teacher's knowledge to student model through auxiliary distillation samples ($\mathcal{L}\_{augkd}$). Therefore, the student model perform more similarly with the teacher model as indicated by the higher PSNR(S,T).

We will revise the manuscript to explicitly connect these points, ensuring that our central motivation and methodology are more clearly articulated.

Weakness 2: Explanation for the performance gain of AugKD

Thank you for the series of questions that lead us to reflect more deeply on the mechanics of the AugKD.

The performance improvements of AugKD stem primarily from the combination of auxiliary distillation samples and label consistency regularization. AugKD identifies and addresses a key challenge in KD for SR: the teacher model’s function of transferring knowledge to the student is overshadowed by the ground truth labels, as the teacher’s outputs are noisy approximations of high-resolution images.

By introducing auxiliary distillation samples, AugKD allows the student model to extract meaningful prior knowledge from the teacher model’s outputs, enhancing the fidelity between the teacher and student models as a by-product. Label consistency regularization further strengthens generalization by enforcing consistent outputs across augmented views of the same input, which helps the student model adapt to diverse inputs.

While data expansion increases the training set size, it does not address the issue of knowledge transfer bottleneck. Simply expanding the data cannot mitigate the shading effect of ground truth labels on the teacher model’s outputs. AugKD, by contrast, is explicitly designed to extract and utilize the teacher’s knowledge effectively, achieving performance gains beyond what data expansion alone could provide.

Regarding augmentation strength in the label consistency regularization module, we have conducted exploratory experiments on it and the results are supplemented below. In the implementation of label consistency regularization, we independently applied four invertible augmentations (invert color, horizontal flip, vertical flip, and transpose) to $I\_{{LR}\_{zo}}$ and $I\_{{HR}\_{zi}}$  with a probability parameter  $p\_{\text{aug}}$ . The results below illustrate the impact of  $p\_{\text{aug}}$  on performance:

The results indicate that an optimal augmentation strength ( $p\_{\text{aug}} = 0.3$ ) gives the best performance of student model, and this strength parameter was used in our experiments. We will include these results and findings in the revised manuscript to clarify the mechanism and impact of AugKD.

Weakness 3: Design of label consistency regularization.

In the current implementation, invertible data augmentations are applied to the input and output of the student model while leaving the teacher model unchanged. Dropping the inverse augmentation and instead applying the same augmentation to the teacher model’s output would indeed be mathematically equivalent.

We greatly appreciate your valuable review comments. Below is our point-by-point response to your concerns and questions. Please let us know if there's anything we can clarify further.

Weakness 1: Applicability of AugKD

The idea of AugKD can indeed be extended to other low-level vision tasks by adapting the auxiliary distillation sample generation to fit task-specific data requirements. Since AugKD addresses a common issue in the KD for these pixel level CV tasks. For instance, when distill denoising or de-blurring models, task-related augmenting operations can be employed to generate auxiliary samples tailored for effective knowledge transfer. Besides, the invertible data augmentations can be directly plugged in these tasks to realize label consistency regularization. We will incorporate more examples and discussions in the manuscript and clarify this aspect further.

Weakness 2: Experiments on more advanced SR models

The EDSR and RCAN models were selected because they are widely adopted benchmarks in existing super-resolution knowledge distillation studies. While many feature-based distillation methods are specifically tailored to CNN-based models, AugKD demonstrates compatibility with both CNNs and Transformers, such as SwinIR.

To address the reviewer’s concern on insufficient results of SwinIR, we provide the results of FitNet and FAKD for the X4 SwinIR model below for more comprehensive comparison. In the experiments of the feature-based KD methods, the feature maps after uniformly picked Swin Transformer layers are matched.

Additionally, we evaluated AugKD on a recent state-of-the-art SR network, Dense-residual-connected Transformer (DRCT) [1]. Using the large version of DRCT as the teacher model, we distill the DRCT network with different KD methods. The result again confirms that AugKD provides notable improvements for the student SR model:

[1] Hsu, C. C., Lee, C. M., & Chou, Y. S. (2024). DRCT: Saving Image Super-resolution away from Information Bottleneck. arXiv:2404.00722.

Question 1: Clarification for the zoom-out operation in AugKD

While the ground truth labels for  $I\_{\text{LR}\_{zo}}$ images are available, going through the teacher model in this path serves the critical purpose of effectively transferring teacher model's knowledge. In the KD of SR models, the teacher model's intended function, transferring knowledge to student, is shaded by the ground truth HR labels, due to the nature that the teacher model's outputs are noisy approximation to the high-resolution images, and there is barely "dark knowledge" like the inter-class relationship or label smoothing in the classification task. AugKD addresses this challenge by re-purposing the teacher model to provide supervision on auxiliary distillation samples that are efficiently generated and closely aligned with the original training data distribution. By matching the student and teacher models on both zoom-in and zoom-out auxiliary samples, AugKD enables the student model to extract meaningful priors encoded in the teacher’s outputs, and achieves more effective knowledge transfer.