Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels

We are grateful for your detailed review. In the following response, we address each of the concerns you have highlighted.

Weaknesses:

1. As Reviewer GKik pointed out, the originality of this work consists in noticing that only a handful of DoG-like filters are learned in all subsequent layers of the DS-CNNs. The ultimate goal and the significance of this paper is to show, for the first time, that on average, more than 80-90% of the state-of-the-art DS-CNN filters are explainable by a single function, which is DoG. It is our hope that this paper will drive other future works in this area in the direction of a potential canonical and foundational vision model.

2. While K-means is good at revealing overall trends, it lacks the precision needed for our research goals. There are several reasons why k-means clustering is not functional for our case:

• High-Dimensional Efficacy: Our method using an autoencoder outperforms K-means in clustering high-dimensional filters, a task where K-means struggles in our case.
• Identifying Unclassified Filters: K-means cannot categorize filters as "unclassified," leading to a lot of noise, as evidenced in Section N of our appendix. In contrast, our autoencoder uses a loss to effectively identify and label unrecognized filters. For an example of unclassified filters, please see Section E of the appendix in the revised manuscript.
• Consistent Criteria Across Models: The autoencoder provides a uniform clustering standard for all models, ensuring a fair comparison. K-means, however, treats each model separately with varying cluster centers, restricting consistent analysis.
• Clustering in Underfit Models: K-means fails to effectively cluster in underfit models due to a shortage of complete filter convergence. Our autoencoder approach is robust enough to handle such scenarios, demonstrating its adaptability and clustering capability. For an example of underfitted models, please see Section G of the appendix in the revised manuscript.

It is speculated that the denoising effect may smooth the actual filter's learned patterns, so the results may not be convincing.

We are not sure if we have understood the reviewer's concern properly, but to address the speculation we would like to clarify that all the clustered filters presented in our paper are original and directly extracted from the models and not reconstructed by the autoencoder. Our use of the autoencoder was exclusively for clustering purposes, without altering the filters.

We hope our explanations have justified the use of autoencoders in our analysis. If there are still doubts regarding their superiority over K-means for this specific application, we welcome further discussion to clarify any remaining concerns.

3. We would like to clarify that the autoencoder’s training was unbiased and unsupervised, involving more than one million filters from diverse models. We employed a strict loss threshold for the autoencoder reconstructions to prevent underfitting and false-positive classifications. Had there been any bias in the autoencoder model, it would have resulted in a significant number of unrecognized filters in most models, which was not observed in our analysis. The accuracy of our classification is evidenced by the random cluster samples in Figures 5 and 6, and Figure 20 in the appendix further demonstrates our method's effectiveness in correctly identifying 'unrecognized' filters.

1. We thank the reviewer for suggesting larger kernel DS-CNNs. In response, we extended our investigation to include larger kernel DS-CNNs. Specifically, we examined the RepLKNet-XL model, referenced in [1], which utilizes 27x27 filters. Our observations confirmed the presence of similar DoG-shaped filters in this model as well. To illustrate this, we have included selected examples of these larger filters in Section I of the appendix in our revised manuscript. Despite their larger size, some of these filters also exhibit a central focus characteristic of DoG shapes and their derivatives, although they are much higher in dimension and have fewer proportions of these kernels compared to models with smaller kernels. Regarding the paper [2] you mentioned, it utilizes sparse 51x5 and 5x51 kernels, which differ from the square kernels we focused on in our study.

The DoG filters and their derivatives have long been proposed to model the biological visual system, as well as their widespread application in traditional image processing. In this paper, we conducted an extensive analysis of the trained filters across several DS-CNN models, and we discovered that many of these filters are similar to the shape of DoGs and their derivatives, without any direct supervision to this effect. We interpret this high frequency of occurrence as more than mere coincidence. We hope our additional clarifications and the provided examples address your concerns. If the reviewer holds a possibility of this resemblance being coincidental, we are more than willing to engage in further discussion of this topic.

2. The filter classification score in Table 1 represents the percentage of filters in each model that we categorized into a main DoG filter family. Some filters, marked as 'not classified', showed higher reconstruction loss and were more cluttered, although some still resembled our main classes. A higher classification score indicates a greater proportion of filters in that model were successfully classified.

3. Thank you very much for pointing this out. We have corrected this table in our revised manuscript.

We hope that our response adequately addresses the reviewer's points and hope it positively influences their assessment of our work.

1. To further investigate this, we trained two new instances of Efficientnet-B4 with different random seeds, as mentioned above. We consistently observed the same patterns in the 10th 5*5 layer of this model, which is notably the first layer in a block of eight layers following a six-layer block. Additionally, we found a higher proportion of clustered filters in the first layer of the preceding block. This pattern recurrence in the initial layers of different blocks suggests that the position of a layer within the network's architecture, specifically at the beginning of a block, has a more pronounced influence on the filters learned in that layer. However, we still do not know what might be the exact reason behind it.

2. We currently do not have a definitive explanation for layer 14's behavior. Upon closer examination of its filters, we observed that they lack any clear, discernible patterns. To provide a better understanding, we have included samples of filters from this layer, as well as from the adjacent layers, in Section E of the appendix. This comparative illustration highlights the unique characteristics of layer 14 in contrast to its surrounding layers.

3. We tested the models ConvnextV2 Huge and ConvnextV2 Large, both trained on ImageNet-1k and ImageNet-22k, on the ImageNetV2 test sets (link:https://imagenetv2.org/). The table below summarizes the findings, illustrating the relationship between the proportion of identifiable filters (as clustered by our algorithm) and the generalization performance of the network on the new test datasets:

We hope that our revision and additional experiments we have provided in response to your comments will increase the confidence in your assessment of our work. We believe these enhancements significantly strengthen our submission.

Thank you very much for your positive assessment of our work, detailed review, and constructive suggestions. We believe we can address all the issues to make a stronger paper. Below are the detailed responses to your feedback.

Weaknesses:

1. We agree that a potential outcome of our findings can be in developing novel initialization and regularization techniques. An initial experiment, detailed in Section F of the appendix, shows promising results where DoG filter initialization not only enhances accuracy but also increases the number of classified filters. While preliminary, this suggests potential for further improvement. With respect to regularization, we are looking into imposing low-order DoG constraints by employing our classification technique as a distance to be added to the loss function on DS-CNNs. However, this requires more thought and extensive experimentation.

2. We appreciate the reviewer's insightful suggestion. In response, we have undertaken two additional analyses. Firstly, following your Question 1, we retrained the efficientnet_b4 model twice using different random seeds. The findings from these trainings are detailed in the appendix, Section D. Our observations showed that even with varying random seeds, the layer proportions remained consistent with the original model as released by PyTorch. This strengthens the hypothesis that these filter proportions are an inherent characteristic of the architecture, rather than a product of specific training instances. Secondly, we included Figure 11 in our revised paper, which displays the total proportions of clusters across all layers in different models. This revealed a compelling trend: models from the same family tend to learn similar filter proportions, irrespective of their size or training data.

3. Our experiment on initialization with low-order DoGs supports the observations mentioned. We found that the proportions of filters in each low-order DoG cluster remained largely consistent even after 50 epochs of training the model (Please see Figure 23 in Section F of the appendix in our revised manuscript.)

4. Thank you for your valuable suggestions.

For under-fit models, we monitored the evolution of filters in the Convmixer768-32 model throughout its training, from the initial state up to epoch 50. This duration is shorter than the original model's 300-epoch training, thus representing an under-fit scenario. Our observations indicate a gradual development of DoG patterns, beginning from the earliest layers. Notably, the on-center DoG pattern is the first to emerge. These observations are detailed in Section G of the appendix in our revised manuscript.

Regarding over-fit models, in line with your suggestion, we attempted to intentionally overfit the Convmixer768-32 model. We achieved this by training the model on a randomly selected 10-class subset of ImageNet for 200 epochs without any data augmentation. The outcome, detailed in Section H of the appendix and illustrated in Figure 37, revealed that the filters in the overfitted model lacked discernible patterns, appearing random. This result aligns with your prediction and supports our hypothesis that the emergence of DoG family patterns is linked to learning more general patterns, and hence the generalization capabilities of the models.

We greatly appreciate your positive evaluation and insightful suggestions. Here are our detailed responses to the points you've raised.

Weaknesses:

1. Thank you for recommending the exploration of models trained on different datasets. In response, we sought DS-CNN models with publicly available weights trained on diverse datasets. We found the InternImage [1] models trained on the ADE20K Semantic Segmentation task. The clusters of the trained filters of the InternImage-T can be found here. Notably, this model employs 3x3 kernels and uses Deformable depthwise convolutions. Despite the model's use of deformable convolutions and its training on semantic segmentation with scene-centric images, the filters it learns are quite similar to those of standard depthwise convolutions trained on ImageNet for classification. This similarity is evident when comparing them to the 3x3 MobileNet kernels detailed in our Appendix. This finding underscores the generalizability of our observations across different datasets and tasks.

2. We agree with the reviewer that the squared filters may be a skewed version of DoGs. However, our analysis suggests that one-sided padding is not a primary factor in the emergence of these patterns. To further investigate, we trained two new instances of Efficientnet-B4 with different random seeds, detailed in Section D of the appendix. We consistently observed the same patterns in layer 10 with 5*5 kernels of this model, which is notably the first layer in a block of eight layers following a six-layer block. Additionally, we found a higher proportion of clustered filters in the first layer of the preceding block. This pattern recurrence in the initial layers of different blocks suggests that the position of a layer within the network's architecture, specifically being at the beginning of a block, has a more pronounced influence on the filters learned in that layer. This finding points towards architectural design, rather than padding techniques, as a more significant factor in the formation of these filter patterns.

3. Thank you for bringing up the impact of global pooling on the learning of weights. The patterns observed in DS-CNNs appear consistently across all layers, not just before global pooling. This pervasive presence of patterns in DS-CNNs, irrespective of the layer, suggests that their formation is less influenced by global pooling compared to traditional CNNs.

4. Thank you for the suggestion to compare our findings with those from this work. Their research, inspired by neuroscience, involves training continuous depth CNNs with filters that are a weighted sum of DoG and its derivatives, with the sum's weights being trainable. Recognizing the relevance of this work, we have added it to the related work section in our revised manuscript.

Language Issues:

Thank you very much for noticing these issues, we revised our manuscript and fixed them.

[1] InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2022).

We would like to thank you for the thorough review and for raising important points. Below, we address the points raised by you.

Weaknesses:

1. As Reviewer GKik pointed out, the originality of this work consists of noticing that only a handful of DoG filters are learned in all subsequent layers of the DS-CNNs. It is our hope that this paper will drive other future works in this area in the direction of a potential canonical and foundational vision model.
   A potential outcome of our findings can be in developing novel initialization and regularization techniques. An initial experiment, detailed in Section F of the appendix, shows promising results where DoG filter initialization not only enhances accuracy but also increases the number of classified filters. While preliminary, this suggests potential for further improvement. With respect to regularization, we are looking into imposing low-order DoG constraints by employing our classification technique as a distance to be added to the loss function on DS-CNNs. However, this requires more thought and extensive experimentation.

2. We agree that performance is not necessarily related to interpretability. While we recognize the significance of performance comparisons, the aim of this paper was not to compare performance metrics across different architectures, like vision transformers, but just to deepen the understanding of DS-CNNs' structural features.

3. You rightly point out that models like ResNets, using standard convolutions, may learn a basis of functions or filters that are less interpretable but still efficient. The goal of this paper was not to claim that interpretability is an outright advantage in performance but to contribute to the understanding of how different architectures process information. We believe this adds valuable perspective to the ongoing discourse on neural network design and function.

4. Our research identifies the presence of DoG and derivative filters in the deeper layers of DS-CNNs, beyond their common application in image processing. DoGs are known for edge detection in images, while derivative filters track changes in brightness. However, in DS-CNNs, these filters operate on more abstract feature maps in deeper layers. Further study is required to determine the specific roles and advantages of these filters at each layer of a CNN model.

5. Thank you for pointing out the inconsistency in our terminology. To clarify, "DSC" refers specifically to depthwise separable convolutions, a term encompassing both depthwise convolutions (abbreviated as "DC") and pointwise convolutions. In response to your feedback, we have revised the manuscript to ensure consistent and clear use of these terms throughout.

6. Thank you for your valuable feedback. We have updated all figure captions to be more self-explanatory and clear. We would be grateful if you could let us know whether these revisions meet your expectations or if further improvements are needed.

Questions:

1. To further investigate this, we trained two new instances of Efficientnet-B4 with different random seeds, detailed in Section D of the appendix. We consistently observed the same patterns in the 10th 5*5 layer of this model, which is notably the first layer in a block of eight layers following a six-layer block. Additionally, we found a higher proportion of clustered filters in the first layer of the preceding block. This pattern recurrence in the initial layers of different blocks suggests that the position of a layer within the network's architecture, specifically at the beginning of a block, has a pronounced influence on the filters learned in that layer. However, we still do not know what might be the exact reason behind it.

2. There are several reasons why k-means clustering is not functional for our case:

• High-Dimensional Efficacy: Our method using an autoencoder outperforms K-means in clustering high-dimensional filters, a task where K-means struggles in our case.
• Identifying Unclassified Filters: K-means cannot categorize filters as "unclassified," leading to a lot of noise, as evidenced in Section N of our appendix. In contrast, our autoencoder uses a loss to effectively identify and label unrecognized filters. For an example of unclassified filters, please see Section E of the appendix in the revised manuscript.
• Consistent Criteria Across Models: The autoencoder provides a uniform clustering standard for all models, ensuring a fair comparison. K-means, however, treats each model separately with varying cluster centers, restricting consistent analysis.
• Clustering in Underfit Models: K-means fails to effectively cluster in underfit models due to a shortage of complete filter convergence. Our autoencoder approach is robust enough to handle such scenarios, demonstrating its adaptability and clustering capability. For an example of underfitted models, please see Section G of the appendix in the revised manuscript.