Visual Semantic Learning via Early Stopping in Inverse Scale Space

Thank you for your feedback and valuable suggestions regarding our work. We appreciate your comments on the clarity and novelty of our work. We address your concerns in the following.

Q1: While this method serves as a preprocessing measure for input data, there is a notable absence of comparative validation against the performance of similar procedures.

A1: Our method is not as simple as a data augmentation method, for we do not use any of the original data during our training procedure. A data augmentation method is to use noisy or corrupt image to urge the model to learn more robust information. However, our method directly discovers the most important and robust information, including structural and low-frequency information, and smooths out other suspicious information. It is much different from the data augmentation method and we do not compare the result with them.

Q2: Numerous studies, such as those found in reference [1,2], have delved into enhancing networks' capacity to process low-frequency information through a series of methods including image denoising and high-frequency noise injection. However, this paper lacks a comparative analysis with the findings of these studies or an exploration of whether there is room for further improvement building on their foundations.

A2: We have implemented [1,2] as you suggested, added comparisons with [2,3,4] in Tab.2,7, on the adversarial robustness tasks on CIFAR-10 using ResNet18. [1] performed poorly in our implementations on CIFAR-10, possibly due to its primary focus on ImageNet.

It can be demonstrated that our method outperforms other methods, whether using FGSM or PGD attacks. Also, using our method as preprocessing method, our method can further improve those methods, though it is not a specially designed defence method.

Q3: Lack of analysis explaining variances among three training methodologies.

A3: For iterative method, it can help the model learn the structural information in the images smoothly. For Fixed method, it can learn the information in images with specific sparsity. For Finetune method, it can refine the vanilla model and help it learn more shape information.

The finetune method outperform others in the adversarial robustness task because through fine-tuning, the model will learn richer information and become less susceptible to attacks, especially white-box attacks (FGSM, PGD). When a sparsity is fixed, the smooth images are very similar to low resolution images, so the fixed model trained on them performs well in low-resolution classification task. As illustrated in Fig.5, the iterative model learns low-frequency information first and then iteratively gains high-frequency information, so it can gain more stable low frequency information. Therefore, it performs well in the classification on low frequency components.

Q4: Lack of discussion concerning certain parameters.

A4: For the early stopping parameter "sparsity", in experiments we have compared the standard classification results of models trained with different sparsity levels in Tab.4, indicating that the outcome remains robust and consistent across varying levels of sparsity, provided it exceeds 0.6. This suggests a relative insensitivity to changes in sparsity, leading us to focus on a threshold of 0.6 for optimal balance in most of our experiments.

Q5: The image in Fig.7 (b) is blurred. The differences in Fig.5 are not very discernible except for the first column, suggesting there's no need for so many repetitive visuals.

A5: Thank you for pointing out the issue with Fig.7 (b). We have revised this figure for improved clarity. Regarding Fig.5, we want to show the whole training process of our iterative model, in which we can see the change in the frequency domain of feature map.

Q6: More clear and intuitive description of Graph algorithm.

A6: Thanks for the advice. We have refined this part by summarizing it in a pseudo-code format and providing an additional flowchart of the algorithm in Appendix.A. Hope these information can make it more readable.

[1]Xie, Cihang, et al. "Feature denoising for improving adversarial robustness." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[2] He, Zhezhi, Adnan Siraj Rakin, and Deliang Fan. "Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Raymond A. Yeh, Yuan-Ting Hu, Zhongzheng Ren, and Alexander G. Schwing. Total variation optimization layers for computer vision. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 701–711, 2022a.

Thank you for your valuable time and constructive feedback. We are glad you find our method interesting and have corrected typos and clarify explanations to enhance readability. We address your concerns in the following.

Q1: The connection and difference between the existing theory and the method introduced in [1][2] is not clear.

A1: In [1], a method which can generate a sparse regularization path for $l_1$ regularization is proposed. In [2], the method is used to prune neural networks by generating a sparse regularization path for network parameters. In this work, we use this similar method to generate sparse regularization path on image data, to discover and make use of the structural sparsity in them. Using it, we can smooth the image and help improve the robustness of model by training them on smooth images. Also, we propose an acceleration algorithm using Graph algorithm to make the instance smoothing algorithm tractable. More detail can be found in the introduction part.

Q2: Many notations are not explained well.The role of $\beta$, $\gamma$ is unclear. In section 3.1, the claim "with t playing a similar role as $1/\lambda$ in Eq1." is confusing. And the definition of $\Vert D\beta \Vert$ seems has typo.

A2: We have clarified our notations in section 3.1. $\beta$ represents the TV-regularized image obtained by minimizing the loss function in Equation 1. $\beta_t$ denotes the regularized image at the $t$th iteration step. $\gamma$ serves as the variable splitting term for this regularization to control the Total Variation of images. For further clarity, each element of $\gamma$ refers to an edge of Total Variation graph, which consists of pixels as nodes and $D$ as edge adjacency matrix. Regarding the role of $t$, as mentioned in Section 3.1, the path $\gamma_t$ transitions from sparsity to density as $t$ increases. Additionally, we have corrected the typo identified in your review.

Q3: In experiments, the compared methods are not enough.

A3: Addition to TV-layer [3], we've added comparisons with PNI [4] and the best results presented in [5], which involve smoothing the kernel (filter out high frequency components) and adversarial training to improve the robustness. It can be seen in Tab.2,7 that with FGSM attack, our method can outperform them without preprocessing. With PGD attack, our method can outperform them with preprocessing with a large gap when the attack strength is large.

[1] Huang et al. "Boosting with structural sparsity: A differential inclusion approach."

[2] Fu et al. "Exploring Structural Sparsity of Deep Networks via Inverse Scale Spaces"

[3] Raymond A. Yeh, Yuan-Ting Hu, Zhongzheng Ren, and Alexander G. Schwing. Total variation optimization layers for computer vision. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 701–711, 2022a.

[4] He, Zhezhi, Adnan Siraj Rakin, and Deliang Fan. "Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[5] Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

Thank you for your efforts and positive assessment of our paper. We address your concerns in the following.

Q1: Validation on larger datasets.

A1: We have extended our training to include ImageNet100, with the results presented in Tab.5 and achieving consistent results in standard classification.

Q2: Applying TV regularization on feature maps.

A2: Exploring TV regularization on feature maps is one of our future goals. To date, preliminary trials using TV regularization with our method on feature maps have been made. We train ResNet18 on CIFAR-10 with TV regularization on feature map of the first convolution layer. Evaluate the model using the same experiments in Sec 4.3, we get the results in the following table. As illustrated, the model with TV regularization on the feature map exhibits higher accuracy in capturing low-frequency components while demonstrating lower accuracy in high-frequency components. This observation suggests its capability to effectively learn low-frequency information.

Q3: Comparison with other structure-based/TV regularization methods, such as TVM (Yeh et al., 2022b) or LRP (Bach et al., 2015)?

A3: We have compared our method with TVM [1] in Tab.1,2 (named as "TV layer") for noisy and adversarial robustness tasks. Our approach applies TV regularization to image data, distinguishing it from TVM, which requires alterations to the model's structure. For further comparison, we have added comparative analyses with [2,3] in Tab.2,7, focusing on adversarial robustness tasks using CIFAR-10. Additionally, we have incorporated LRP [4] visualizations in Appendix M, as per your suggestion.

Q4: Sensitivity analysis on different TV regularization parameters.

A4: We've compared the standard classification results of models trained with different sparsity levels in Tab.4, indicating that the outcome remains consistent across different levels of sparsity, particularly when it exceeds a threshold of 0.6.

Q5: How do you choose the optimal sparsity?

A5: We clarify that our study did not specifically aim to identify the optimal sparsity for each task. Our method has shown robust performance across various sparsity levels, indicating that the experimental outcomes we observed are largely independent of the chosen sparsity. While most of our experiments were conducted with sparsity settings of 0.6 and 0.8, we anticipate similar results with other levels, such as 0.7.

Q6: How do you evaluate the quality and diversity of the generated image path?

A6: Qualitatively, the visualizations of image paths in Fig.1 and Fig.9 illustrate the trade-off between structural and detailed information. With lower sparsity, structural information, such as shape, dominates. While with higher sparsity, more detailed information such as texture shows up. Quantitatively, we observed that higher sparsity correlates with enhanced accuracy in standard classification tasks, while our experiments in section 4 reveal a decrease in robustness, indicating the trade-off between them. With incomplete information such the lack of detailed information when sparsity is low, the model cannot achieve high accuracy. With excessive detail learned, as in the vanilla model, the model lacks robustness.

Q7: How do you handle the cases where the structural information is not sufficient or reliable for the task?

A7: Sections 4.1, 4.2, and 4.4 detail experiments demonstrating the resilience of our approach in these challenging conditions with noisy images, adversarial attacks, or low-resolution data. It is shown that structural information is robust in these cases and can help improve the model's robustness.

[1] Raymond A. Yeh, Yuan-Ting Hu, Zhongzheng Ren, and Alexander G. Schwing. Total variation optimization layers for computer vision. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 701–711, 2022a.

[2] He, Zhezhi, Adnan Siraj Rakin, and Deliang Fan. "Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Bach S, Binder A, Montavon G, Klauschen F, Müller K-R, Samek W (2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE 10(7): e0130140. https://doi.org/10.1371/journal.pone.0130140

We express our gratitude to the reviewer for your efforts and valuable insights provided for our work. We are glad that you appreciate our novelty, and we address your other concerns below.

Q1: The symbols in formulas need to be specified.

A1: We have improved the explanations for symbols in section 3.1. Specifically, $\beta$ represents the dense parameter for the loss function in Eq. 1. When $\beta$ minimizes the loss, it represents the TV-regularized image. Meanwhile, $\tilde{\beta}$ in Eq. 4 denotes the sparse TV-regularized image derived by projecting $\beta$ onto the subspace formed by the support set of $\gamma$.

Q2: Other improvements of our projection method compared with SVD and LSQR.

A2: Our main contribution is to propose the instance smoothing method, and the efficient sparse projection method is to make the time cost tractable. For performance, since our method can derive the analytical solution just as SVD, so theoretically no difference, even results in smaller loss in practice due to numerical issue. LSQR may struggle with convergence speed or precision, relying on iterative solutions.

Q3: The choice of early stopping time.

A3: In experiments, we have tried several sparsity levels as the criterion for early stopping, and compared the standard classification results in Tab.4, indicating that the outcome remains robust and consistent across varying levels of sparsity, provided it exceeds 0.6. This suggests a relative insensitivity to changes in sparsity, leading us to focus on a threshold of 0.6 for optimal balance in most of our experiments.

Q4: The comparison between this method and existing methods is missing.

A4: TV-layer [1] is a TV regularization method in which we have compared our methods in Tab.1, Tab.2. We also provide new results with [2,3] in Tab.2,7 in the adversarial robustness task. It can be seen that our method can outperform it in adversarial robustness tasks with FGSM and PGD attacks on the CIFAR10 dataset. Also, as a preprocessing method, our method can further improve the accuracy of adversarial images. These results show the effectiveness of our method, though it is not a specially designed defense method. Additionally, we have incorporated LRP [4] visualizations in Appendix M.

[1] Raymond A. Yeh, Yuan-Ting Hu, Zhongzheng Ren, and Alexander G. Schwing. Total variation optimization layers for computer vision. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 701–711, 2022a.

[2] He, Zhezhi, Adnan Siraj Rakin, and Deliang Fan. "Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Bach S, Binder A, Montavon G, Klauschen F, Müller K-R, Samek W (2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE 10(7): e0130140. https://doi.org/10.1371/journal.pone.0130140