Noise-Augmented Deep Neural Networks for Image Classification: Insights from Information Theory

The paper introduces NoisyNN, a novel approach that leverages positive noise to improve deep neural networks' performance on tasks like image classification and domain adaptation. The authors categorize noise into positive noise (PN) and harmful noise (HN) based on their effect on task complexity, using information-theoretic metrics. The analysis is conducted on various noise type including Linear transformation noise, Gaussian and salt-and-pepper noise. Extensive experiments validate NoisyNN's effectiveness using CNNs and Vision Transformers (ViTs) across multiple datasets.

However, a major red flag is the suspicious ImageNet-1K accuracy. No model (22K pretrained + 1K finetuned) has ever achieved 95% top-1 accuracy on the ImageNet-1K validation set. Even with extensive web-scale pre-training, the best state-of-the-art performance remains below 92.5% (https://paperswithcode.com/sota/image-classification-on-imagenet). I intend to reproduce the reported performance. However, the README in the anonymous submission is completely missing, raising concerns about the completeness of the codebase. This lack of professionalism further reflects the questionable nature of your reported performance. I am willing to proceed, but please provide detailed instructions first. Given the unusually high performance you reported, you should be thoroughly familiar with the codebase, so there should be no reason for not providing this.

This paper analyzes the proactive noise injection into deep learning models from both theoretical and empirical perspectives. This paper theoretically characterizes the positive noise that reduces task complexity based on information entropy, and demonstrates the presence of positive noise through experiments. The approach ‘NoisyNN’ leveraging positive noise is proposed, and its efficacy is shown through empirical experiments.

The paper proposes a new type of noise for injecting in the latent dimension space to regularize the network. It shows that the popular Gaussian and Salt-Pepper noises are harmful and degrade performance. The paper shows the requirement for noise to have a positive effect. Based on this, a linear transformation noise is proposed, which is created by a circular shift Q matrix.

The paper explores how adding noise to deep neural networks can improve image classification and domain adaptation. It differentiates between positive noise, which helps, and harmful noise, which hurts model performance. The authors introduce NoisyNN, a method that uses positive noise to enhance model outcomes. Theoretical and empirical evidence supports the effectiveness of NoisyNN, showing it can achieve better results than traditional approaches. This study offers a new view on noise in machine learning.

[AC feedback] Even though any of us checked the implementation, it will not be possible to rigorously check the training dataset. Here is the checksum script for checking the validation split leakage (bash script). I assume that your training image directory looks like this: ..... (many scripts)

I sincerely thank AC for providing the Checksum script for verification. Across my experience as a reviewer in hundreds of papers, this marks the first time an AC has offered support at the script level. I deeply appreciate your expertise and commend this thoughtful gesture. Such actions contribute significantly to fostering a healthier and more robust research community.

[author feedback] Thank AC for the suggestions. We ran the checksum script, there is a 797 overlap between training and validation images.

I appreciate the authors' honesty. That said, it is evident that this paper has substantial room for improvement overall. With more rigorous experiments, there could still be an opportunity to clarify its contributions and pursue publication.

Best,

Reviewer

This paper proposes NoisyNN, a regularization method that injects noise into the latent space. According to the paper, noise is typically harmful to neural networks, but noise can be positive under some conditions. Based on the derivation of the task entropy, this paper proposes to use positive linear transformation noises for neural network regularization. Experiments are conducted on image classification and domain adaptation.

This paper has mixed opinions, mostly negative ones. Also, the AC acknowledges that three out of four reviewers did not engage in the discussion, hence, the decision is not solely made by the average score.

The biggest problem of this paper is the reliability of the experimental results. As also pointed out by Reviewer xiNE, "Even with extensive web-scale pre-training, the best state-of-the-art performance remains below 92.5% (https://paperswithcode.com/sota/image-classification-on-imagenet)." I also agree that this paper reports suspicious experimental results (not only for 94.8% but also for numbers in Table 1-3). In my opinion, ResNet-50 with 81.3% top-1 accuracy is also suspicious. Even highly optimized ResNet-50 achieves 80.4% [A] top-1 accuracy -- Note that this model did not even follow the common training recipe, but was based on a highly optimized new recipe.

• [A] ResNet strikes back: An improved training procedure in timm

As mentioned in my previous comment, from my perspective, this paper should be validated very carefully and rigorously, including a thorough review of the implementation, trained weights, and related aspects. Given the unusually high numbers reported here, it is reasonable to expect the authors to demonstrate that their code and methodology are free of issues. However, as of now, I do not think that the experimental results are reliable. One possibility of the unusually high accuracies could be that noise injection is even performed during the inference, and the noise injection is performed batch-wise. There could be two potential bugs. (1) we do not shuffle mini-batch for validation, namely, the same mini-batch likely has the same classes (2) if the operation is affected by different samples in the mini-batch, then it uses other "test" samples for making a prediction. One possible simple validation will be to run the validation code with batch size 1. I tried to test this one while writing the meta-review, but unfortunately, the weight provided by the authors was not accessible during the meta-review phase.

There are also other concerns, such as the lack of comparisons with the other augmentations (vxMD, xiNE) the lack of implementation details (xiNE), the lack of instructions for the code (MpsV, xiNE), writing quality (xiNE). In my perspective, these concerns were not fully addressed because the reproducibility and reliability issues still remain.

I strongly suggest releasing the code publically based on timm library, which is the rule-of-thumb to ImageNet classification. If the result is truly reproducible, then it will be a great contribution to the community. However, if the implementation details are still unreproducible and hidden from open-source users, then I think the contribution of this submission will be limited.

As a minor comment, all the discussions between the authors and the reviewers are hidden from general audiences. The AC requested to change the visibility of the authors' comments to the authors, but the authors did not change the visibility. Although there is no specific policy for visibility (as far as I know), I think that open discussion is an important value to our community. I strongly recommend the authors change the visibility of the responses after decisions are made.

Reject