Dear reviewer PcoK,

Thank you for your valuable feedback and comments. We appreciate your recognition of our paper's clear presentation of the problems with non-normalized filters, its connection to classic results from image processing, and the formal proof of our claims. We are also grateful for your praise of our proposed method's simplicity, elegance, and potential for widespread adoption. Additionally, we appreciate your acknowledgment of the thoroughness of our evaluation, including the insightful behavioral analyses that provide a deeper understanding of the networks. We address your concerns and questions in the response below.

I think section 3 could be streamlined a bit more to make it even easier to digest and follow. For some ideas, see my questions below.

We appreciate the reviewer's suggestions below for streamlining section 3 and will incorporate them to improve clarity and readability.

The mix of evaluation plots and tables, also including the supplementary material, seems a bit messy.

Thank you for your feedback about this. We agree that the current presentation of evaluation plots and tables, including the supplementary material, could be improved. With a large number of results to report, we strive to strike a balance between clarity and concision. In the final version, we will make every effort to polish the presentation, potentially moving some results to the appendix to enhance the overall flow and readability.

Questions:

Looking at Eq. 1, I wonder whether and how these results also apply to fully connected layers?

Great suggestion. Since fully connected (FC) layers can be viewed as a special case of convolutional layers, we ran a simple experiment to explore the effects of normalizing the linear layer in the classifier of an R20 trained on CIFAR-10, where our initial findings show a slight increase in accuracy. Interestingly, this raises connections to classical wisdom on classifier design, such as maximizing margin with weight regularization, similar to SVMs. This potential link highlights opportunities for future work, where we can further analyze and explore the relationships between normalization and classifier performance.

Line 174f: To a cursory reader, it might be non-obvious that after the normalization, is s guaranteed that w.l.o.g., and either 1 or 0, and hence is either 0 or 1, thus the filter is either strictly averaging or strictly differencing, and Eq. 11 can be used to prove the equivariance and invariance guarantees. I suggest clearly describing this reasoning and really "hammering it in", as I believe it is a key step to understanding your contribution.

We will follow the reviewer’s suggestion and describe this part more clearly to facilitate the understanding of our contribution.

If I'm not mistaken, the theoretical equivariance and invariance results only cover spatially uniform gain and bias. As you're also evaluating on spatially varying gain and bias, is it possible to elaborate on that? Do you just assume local spatial uniformity since filter kernels are usually small? If yes, I think this should be mentioned in the paper.

Thank you for pointing this out! We indeed assumed local spatial uniformity, which holds reasonably well given the small size of filter kernels. You are correct that this assumption might not strictly hold for spatially varying gain and bias. Notably, our experiments show that the gain in accuracy from our method compared to vanilla ResNets is more pronounced for spatially varying cases. For instance, our R34N obtains a 4.2% gain in accuracy on $D_C$, whereas the gains on $D_L$ and $D_B$ are 7.2% and 31.3%, respectively. This suggests that our method provides more significant benefits when dealing with spatially varying corruptions, particularly those with faster spatial variations like $D_B$. We'll make sure to clarify this assumption and its implications in the paper to provide further context.

What is meant by "co-domain symmetry", mentioned in various places in the paper?

In classical signal processing, spatial equivariance is well understood: A convolutional filter responds consistently to translations of the input. Our filter normalization extends this structured response into the co-domain, i.e., the filter output space under intensity changes.

We define co-domain symmetry as follows: A filter exhibits co-domain symmetry if its response transforms in a predictable, structured way under affine transformations of the input intensity $x$, via gain ($g$) and offset ($o$).

We will clarify these definitions in the revision and illustrate them more explicitly in the discussion around Eq. (11).

Line 212f: Is there a reason why you introduce sort pooling layers? In general, I would appreciate a more complete description of all tested architectures, including network width, depth, activation functions, pooling layers, residual connections, and the use of batch normalization in the interest of reproducibility.

We added affine convolutions and sort pooling layers for a fairer comparison with norm-equivariant nets [15]. We will add a section in the appendix giving more details about the architecture used for this experiment.

[15] Herbreteau, et al. Normalization-equivariant neural networks with application to image denoising. NeurIPS, 2023.

Lines 214 & 243: Is there a specific reason why you chose SGD over Adam?

Not really. We followed implementations from PyTorch official examples.

Lines 326f: "Artifacts reduction" and "a more targeted correction" are rather handwavy justifications for the superiority of filter normlization over instance normalization. In addition, no intuition to what this actually means is provided. I suggest you either elaborate on this or remove these claims.

Regarding "artifacts reduction", we observed that instance normalization can sometimes introduce undesirable artifacts in the output, particularly when the instance statistics are not representative of the atmospheric corruptions. On the other hand, by "more targeted correction", we mean that normalizing the weights rather than activations allows our approach to directly correct the filter's response to atmospheric corruptions, rather than indirectly adjusting the activations. We will further analyze our claims and expand on these points in the final version of the paper.

Minor comments: We will carefully consider all minor comments and revise the paper accordingly for the final version.

Dear reviewer j6NF,

Thank you for your valuable feedback and comments. We appreciate your recognition of our method’s simplicity and the gains it provides in robustness and interpretation. We address your concerns and questions in the response below.

Experiment could include other corruptions beyond these customized datasets.

Thank you for the suggestion to strengthen our experimental validation using established robustness benchmarks. Our method is specifically designed to address atmospheric transformations—i.e., global additive and multiplicative corruptions such as changes in brightness, contrast, and haze (see Eq. 10 and Fig. 2). We already evaluate our method across a range of corruption types and intensities (see also Fig. A.1.1 in the Appendix).

ImageNet-C includes severe and often localized distortions (e.g., noise, blur, digital artifacts) that fall outside the atmospheric transformation family we explicitly target. Our method outperforms on some categories such as brightness and underperforms on others such as snow, all with a very small margin. While we acknowledge these results, we emphasize that ImageNet-P, proposed in the same paper, offers a more direct test of our model’s design goals: ImageNet-P applies smooth, progressive perturbations, especially in the "weather" category (e.g., brightness and snow), aligning well with our focus on global intensity transformations.

Robustness is measured using the Flip Rate (FR)—the fraction of samples whose predicted class changes at the highest perturbation level compared to the clean version: $FR = \frac{1}{n}\sum_{i=1}^n \mathbb{1}(f(x_i) \neq f(x_i^{30}))$, where $x_i^{30}$ is the corrupted image at the maximum perturbation level (30), and $n$ are all images included in the ”weather” category, which includes brightness and snow modifications (See results below).

The flip rate of a vanilla R34 is 41.3%, while our R34N with normalized convolutions achieves 40.0% flip rate (lower is better).

Lastly, we further assess generalization to natural variation by binning the ImageNet-1k test set into 9 contrast levels (based on pixel standard deviation). R34N maintains more stable accuracy across the contrast spectrum compared to the baseline, which drops sharply at extreme values (see Fig. X). This confirms the model's improved robustness to natural realistic atmosphere conditions.

We will include new plots, statistical analyses, and representative image examples in the final version of the paper.

[1] Hendrycks, et al. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019.

Comparison of ResNet fine-tuned with CLIP on zero-shot is somewhat misleading, and perhaps not the best choice. We know that CLIP does not do well on robustness to covariate shift, I am not sure what the experiment shows for the actual topic of the paper.

We appreciate the reviewer’s point and fully acknowledge that comparing our R34N model—trained on ImageNet-1k for supervised classification—to CLIP, a foundation model trained on the much larger LAION-5B dataset using contrastive vision-language pretraining, involves substantial differences in scale, objective, and intended use.

That said, if the comparison is “unfair,” we would argue it is unfair to our disadvantage: CLIP is trained on several orders of magnitude more data with broader supervision, yet still shows notable vulnerability to simple intensity corruptions, where our much smaller model remains robust. From this perspective, the comparison is not only informative but also conservative, demonstrating that large-scale pretraining does not guarantee robustness to common low-level variations.

Our intent is not to claim general superiority over CLIP, but to highlight complementary strengths: CLIP excels in semantic generalization and zero-shot transfer, while our method targets physical robustness to real-world image degradations. We believe this suggests that classical filtering principles—like those embedded in our filter normalization—may provide value even when incorporated into large-scale models, particularly CNN-based ones such as CvT [2] and ConvNeXt [3].

We will revise the manuscript to more clearly acknowledge the training scale differences and provide a balanced discussion of this comparison.

[2] Wu, et al. CvT: Introducing Convolutions to Vision Transformers. ICCV 2021.

[3] Liu, et al. A ConvNet for the 2020s. CVPR 2022.

Not widely accepted use of the word "atmosphere" effect, The paper even shows some experiments on astronomy data taken from space.

We borrowed the term 'atmosphere' from Adelson’s work on Lightness Perception and Lightness Illusions [4], where it is defined as 'The net effect of the viewing conditions, including additive and multiplicative effects, may be termed an “atmosphere.”' We find this term suitable for our purposes. However, as the reviewer pointed out, we agree it may cause confusion when applied to domain-specific data, such as astronomy images. To clarify, we will specify in the paper that the term 'atmospheric effects' is particularly relevant to natural color images.

[4] Adelson. Lightness Perception and Lightness Illusions. MIT Press, 2000.

Questions:

Would it be possible to also add a fine-tuned CLIP for comparison

Unfortunately, fine-tuning CLIP is not feasible for the length of the discussion period. We included zero-shot CLIP as a baseline since it is a widely used approach for image classification.

Thanks much for your careful and thoughtful reviews.

We would like to clarify that our comparison between R34N and CLIP focuses on robustness to atmospheric corruptions, not semantic generality.

In the table below, we can see that specialist models like ResNets and ViTs trained on ImageNet for supervised classification (first three rows) perform well on the ImageNet test set ($D$) but struggle with corrupted datasets (for instance, see ~1-2% accuracies on $D_S$). CLIP, a generalist model trained on 400M images, performs better on corrupted datasets due to its exposure to more variations (See accuracy of up to 49% on $D_S$). However, our model, R34N, outperforms CLIP in robustness to atmospheric corruptions (67% on $D_S$), despite not seeing any such transformations during training.

Despite CLIP’s broader training, it suffers a significant drop under intensity corruptions, while our much smaller and more narrowly trained model remains robust—even to perturbations it never saw. From this perspective, the comparison is informative: it highlights that large-scale pretraining does not automatically confer robustness to low-level corruptions, and that classical principles like filter normalization can still play a valuable role.

We consider our comparison with CLIP is not unfair as our goal is not to claim general superiority over CLIP, but to illustrate an aspect where even foundation models may benefit from architectural improvements. Filtering-based normalization remains relevant, particularly for CNN-based models such as CvT [1] and ConvNeXt [2].

We will revise the paper to reflect this nuanced comparison and clarify that our claims are specific to robustness.

[1] Wu, et al. CvT: Introducing Convolutions to Vision Transformers. ICCV 2021.

[2] Liu, et al. A ConvNet for the 2020s. CVPR 2022.

Dear reviewer qNTu,

Thank you for your valuable feedback and comments. We appreciate your recognition of our method's clear and elegant formulation, solid derivation, strong motivation, well-structured presentation, and comprehensive experiments. We also appreciate your acknowledgment of the importance and usefulness of our observation regarding unnormalized filters in deep networks. Below, we address the concerns and questions you raised.

The header of section 3.1 "Any filter is both averaging and differencing" is not guaranteed. It should be better rephrased as "any linear / convolutional filter ... "

Thank you for the suggestion. We completely agree that specifying 'linear filter' adds precision and will improve clarity to the title of section 3.1. We will adopt this rephrased title in the final version.

The experiment on data augmentation on CIFAR-10 with only atmospheric augmentations may not be very representative. First, the data size of CIFAR and the image size are both small. Recent advances in image processing are all conducted on large scale high resolution data. It is suggested to make more thorough experiments on those dataset to make augmentation conclusion. Second, the experiment results do not show a clear trend that data augmentation techniques make model suffer, even though the performance on D_s alone is not as good as expected.

Thank you for your insightful comment. While our results on CIFAR-10 don't show a clear trend of data augmentation techniques hurting model performance, our approach achieves similar robustness without the need for data augmentations. This demonstrates a key advantage of our method, as it eliminates the need for data augmentation and the associated computational costs. Additionally, our approach avoids the challenges of designing effective augmentations, which can require domain-specific knowledge. To further validate our findings, we will include experiments on ImageNet in the final version, when computational resources permit.

Thank you for your valuable feedback and comments. We appreciate your acknowledgement that our work addresses a critical problem in Deep Learning, its novelty, effectiveness, simplicity, and broad applicability. We address your concerns and questions in the response below.

Clarify Theoretical Concepts: Provide clear mathematical definitions for "co-domain symmetry" and "atmosphere-equivariance." Elaborate on how Equation (11) directly translates to the broader concept of atmosphere-equivariance for deep networks.

Thank you for raising the concern about the clarity of our definitions of co-domain symmetry and atmosphere-equivariance. We agree these terms need more precise explanation.

In classical signal processing, spatial equivariance is well understood: A convolutional filter responds consistently to translations of the input. Our filter normalization extends this structured response into the co-domain, i.e., the filter output space under intensity changes.

We define co-domain symmetry as follows: A filter exhibits co-domain symmetry if its response transforms in a predictable, structured way under affine transformations of the input intensity $x$, via gain ($g$) and offset ($o$).

Our method ensures that normalized filters decompose into two interpretable components:

1. Averaging filters respond linearly to both gain and offset: $f(g \cdot x + o) = g \cdot f(x) + o$, where $|w| = 1$ and $r(w) = 1$.

2. Differencing filters are invariant to offset and equivariant to gain: $f(g \cdot x + o) = g \cdot f(x)$, where $|w| = 1$ and $r(w) = 0$.

This decomposition yields atmosphere-equivariance, meaning that under global illumination changes (such as haze, shadows, or lighting shifts), the response of each filter changes in a predictable and interpretable way. For averaging filters, both contrast and brightness shift the output, while differencing filters respond only to contrast. This behavior is particularly desirable in scenarios where the semantic structure of the scene should be invariant to lighting conditions.

We will clarify these definitions in the revision and illustrate them more explicitly in the discussion around Eq. (11).

Address Non-linearity Impacts: Discuss the implications of non-linear activation functions (e.g., ReLU) on the proposed filter normalization's effects and the concept of "interpretable responses" throughout the entire network

Thank you for raising the question regarding the role of non-linearities (e.g., ReLU) and their impact on the proposed filter normalization and interpretability of responses.

Our proposed method ensures atmosphere-equivariance at the convolutional layer level. That is, prior to any non-linear activation, filter responses transform in a structured and predictable way under global intensity shifts (gain and bias). This forms the foundation for building robustness to such transformations throughout the network.

It is important to clarify that, while individual layers are equivariant, the goal of the full network is to achieve invariance to atmospheric transformations—just as classical CNNs preserve spatial equivariance locally but aim for global translational invariance in their final semantic outputs. Non-linearities such as ReLU serve this purpose: they progressively reduce sensitivity to irrelevant variations (e.g., illumination), transforming equivariant features into invariant representations through depth and learned hierarchy.

What makes our approach effective is that starting from an atmosphere-equivariant basis makes the job of learning invariance easier and more structured. This is evident in our experiments (Lines 311–318 and Table 2), where models using filter normalization without any data augmentation outperform baselines with augmentation, by a large margin (e.g., +14% accuracy on $D_S$). This suggests that the network is better able to learn invariances downstream when the early layers encode meaningful, interpretable transformations.

In short, while non-linearities do break strict equivariance, they play a necessary and constructive role in turning structured responses into invariant representations. Our method complements this process by providing a more robust and interpretable foundation at the convolutional level.

We did further experiments on different types of activation functions, specifically Tanh and Sigmoid, in place of ReLU. Our results on CIFAR-10 $D_C$ with R20 and our R20N model show that our method consistently outperforms the vanilla R20 model, regardless of the activation function used:

This suggests that our filter normalization approach provides robustness benefits that are complementary to the choice of activation function. We will include these experiments in the final version of our paper.

Refine CLIP Comparison: Rephrase claims regarding CLIP's "outperformance" to explicitly acknowledge and emphasize the vast disparity in training data scale (ImageNet-1k vs. LAION-5B).

We appreciate the reviewer’s point and fully acknowledge that comparing our R34N model—trained on ImageNet-1k for supervised classification—to CLIP, a foundation model trained on the much larger LAION-5B dataset using contrastive vision-language pretraining, involves substantial differences in scale, objective, and intended use.

That said, if the comparison is “unfair,” we would argue it is unfair to our disadvantage: CLIP is trained on several orders of magnitude more data with broader supervision, yet still shows notable vulnerability to simple intensity corruptions, where our much smaller model remains robust. From this perspective, the comparison is not only informative but also conservative, demonstrating that large-scale pretraining does not guarantee robustness to common low-level variations.

Our intent is not to claim general superiority over CLIP, but to highlight complementary strengths: CLIP excels in semantic generalization and zero-shot transfer, while our method targets physical robustness to real-world image degradations. We believe this suggests that classical filtering principles—like those embedded in our filter normalization—may provide value even when incorporated into large-scale models, particularly CNN-based ones such as CvT [2] and ConvNeXt [3].

We will revise the manuscript to more clearly acknowledge the training scale differences and provide a balanced discussion of this comparison.

[2] Wu, et al. CvT: Introducing Convolutions to Vision Transformers. ICCV 2021.

[3] Liu, et al. A ConvNet for the 2020s. CVPR 2022.

Strengthen Experimental Validation: Evaluate the proposed method on well-established public robustness benchmarks, particularly relevant subsets of ImageNet-C (e.g., brightness, contrast, haze). Expand the natural data experiments beyond just inline text, presenting them in comprehensive tables that compare all baselines where applicable.

Thank you for the suggestion to strengthen our experimental validation using established robustness benchmarks. Our method is specifically designed to address atmospheric transformations—i.e., global additive and multiplicative corruptions such as changes in brightness, contrast, and haze (see Eq. 10 and Fig. 2). We already evaluate our method across a range of corruption types and intensities (see also Fig. A.1.1 in the Appendix).

ImageNet-C includes severe and often localized distortions (e.g., noise, blur, digital artifacts) that fall outside the atmospheric transformation family we explicitly target. Our method outperforms on some categories such as brightness and underperforms on others such as snow, all with a very small margin. While we acknowledge these results, we emphasize that ImageNet-P, proposed in the same paper, offers a more direct test of our model’s design goals: ImageNet-P applies smooth, progressive perturbations, especially in the "weather" category (e.g., brightness and snow), aligning well with our focus on global intensity transformations.

Robustness is measured using the Flip Rate (FR)—the fraction of samples whose predicted class changes at the highest perturbation level compared to the clean version: $FR = \frac{1}{n}\sum_{i=1}^n \mathbb{1}(f(x_i) \neq f(x_i^{30}))$, where $x_i^{30}$ is the corrupted image at the maximum perturbation level (30), and $n$ are all images included in the ”weather” category, which includes brightness and snow modifications (See results below).

The flip rate of a vanilla R34 is 41.3%, while our R34N with normalized convolutions achieves 40.0% flip rate (lower is better).

Lastly, we further assess generalization to natural variation by binning the ImageNet-1k test set into 9 contrast levels (based on pixel standard deviation). R34N maintains more stable accuracy across the contrast spectrum compared to the baseline, which drops sharply at extreme values (see Fig. X). This confirms the model's improved robustness to natural realistic atmosphere conditions.

We will include new plots, statistical analyses, and representative image examples in the final version of the paper.

[1] Hendrycks, et al. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019.

The authors' responses adequately address the theoretical and non-linearity concerns, and their re-framing of the CLIP comparison is reasonable. I strongly recommend that the authors change the claim on the CLIP comparison.

However, the experimental validation remains a significant concern. While the authors present promising Flip Rate (FR) results on ImageNet-P, the result does not include accuracy, which is a more fundamental metric.

I understand that ImageNet-P is more appropriate than ImageNet-C, but is there any reason why the accuracy performance for ImageNet-P is not included in the rebuttal?

Thank you for your request. We are happy to provide the accuracy results along with the Flip Rate (FR) below. The accuracy of our R34N is higher than R34, which is consistent with its lower FR.

We used FR as the metric in this experiment, following the convention established by the ImageNet-P benchmark [1] (Section 4.2). While both metrics show consistent trends, the FR gap is more pronounced than the accuracy gap, making it a more sensitive indicator of perturbation robustness — most likely the rationale behind its original proposal [1]. We are happy to include both metrics in the final version and hope this satisfactorily addresses your concern.

[1] Hendrycks, et al. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR 2019