We want to thank the reviewer for their review and for accepting our paper. We have to mention that although our paper can be viewed as a kind of pruning, ours is more of an analytical paper and we are more focused on understanding the DS-CNN filters in this paper regardless of what the application can be.

---

1. Weaknesses:

• “confidence on the smaller datasets”: For sure, For Flowers and Pets datasets we repeated the experiments 4 times to serve as a confidence, but we can complete them even more with confidence intervals. We repeated the CFAR10 for you and our results seem to be consistent. Each experiment was repeated 4 times, and we now report the results with error bars (mean ± standard deviation). The experiments are on-going that we will complete and update that table. Please find the confidence table below: | Dataset | Setting | Accuracy | |----------|------------------------------|--------------------:| | CIFAR-10 | Original | 96.95 ±0.10 | | CIFAR-10 | ImageNet Filters | 97.07 ±0.18 | | CIFAR-10 | Acc with 8 unique Filters | 96.36 ±0.13 |

• “filters still hold under more complicated visual tasks”: This is a valid argument! We definitely should try our filter on other tasks. We would try the COCO object detection task but it takes time. Is there any task or dataset that you are interested in?

---

2. Typos:

• Thank you very much. We would correct them.

---

3. Questions:

• In both Eq (1) and Eq (2) It’s the length of the vectors, here 7×7 = 49. But I totally get where your confusion comes from! In the next lines “n” is different and it’s the number of different vectors (here number of filters we have). It’s a mistake on our part and we have to fix the notation. A genuine thank you for noticing this.
• “What is different if one finds a linear combination of all base filters for each trained filter?”: First, we are not sure if we should use “affine transform” here because most of the time affine transform means matrix multiplication transform (Are we wrong?) but here we are just using a linear shift (ax+b) where “a” and “b” are just scalars. Second, about linear combinations, we should remember how powerful linear combinations are. To give you a simple example, linear combination of just 9 random 3×3 filters would span the space of all possible 3×3 filters because 3×3 filters have 9 dimensions. But that is not the case for linear shifts. So mathematically, if we show the filters are converging to linear shifts, we are finding a far greater constraint to the space of filters compared to if we’ve used linear combination.
  TLDR: Using linear combinations would definitely increase the accuracy, but the scientific importance of it is very low compared to linear shifts.
• You are completely right. Actually we are adding a loss comparison chart that would show that. For fast convergence, we recommend initializing your DS-CNN models with these filters; they do make a real difference for the starting epochs.
• Sorry, we thought about it a lot, our guess would say “Yes”; however, we couldn’t come up with a scientific experiment or explanation to answer this question the professional way.
• Oh! Those papers look super interesting! Especially “Which Frequencies Do CNNs Need?” Indeed there could be a connection because each DS-CNN is mathematically a classical CNN. We have to dive into that to figure out the connection. About the “base filters” you mentioned, I wish I could send pictures in this rebuttal, but we suspect they are bases for natural images, because we could construct them by finding the eigenvectors of the covariance matrix of any natural image we tested. It would be hard to explain without figures unfortunately. But anyways, thanks for the papers.

import numpy as np
import matplotlib.pyplot as plt
from skimage import io, color
from skimage.color import rgba2rgb
from sklearn.preprocessing import normalize
from scipy.linalg import eigh

k = 7
A = io.imread("your_image.jpeg")
if A.ndim > 2:
    A = color.rgb2gray(rgba2rgb(A) if A.shape[2] == 4 else A)
A = A.astype(float)

h, w = A.shape
patches = [A[y:y+k, x:x+k]
           for y in range(2, h-k, k)
           for x in range(2, w-k, k)]
d = np.arange(k) - (k//2)
g = np.exp(-(d[:, None]**2 + d**2) / 2)
g /= g.sum()

X = (np.array(patches) * g).reshape(-1, k*k)

X -= X.mean(0)

eigvecs = eigh(X.T @ X)[1]
V = normalize(eigvecs[:, -8:].T)
eigen_patches = V.reshape(8, k, k)

fig, axs = plt.subplots(2, 4, figsize=(10, 5))
for i, ep in enumerate(eigen_patches):
    axs.flat[i].imshow(ep, cmap='gray')
    axs.flat[i].axis('off')
plt.show()

c = 5
offset = (k - c) // 2          # = 1
cropped = eigen_patches[:, offset:offset+c, offset:offset+c]
V = cropped.reshape(8, -1)       
patches = [A[y:y+c, x:x+c]
           for y in range(2, h-c, c)
           for x in range(2, w-c, c)]
X = (np.array(patches)).reshape(-1, c*c)
X -= X.mean(0)

weights = X @ V.T
recon_patches = (weights @ V).reshape(-1, c, c)
recon = np.zeros_like(A)
count = np.zeros_like(A)
idx = 0
for y in range(2, h-c, c):
    for x in range(2, w-c, c):
        recon[y:y+c, x:x+c] += recon_patches[idx]
        count[y:y+c, x:x+c] += 1
        idx += 1
count[count == 0] = 1
recon /= count

plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.imshow(A, cmap='gray')
plt.axis('off')
plt.subplot(122)
plt.imshow(recon, cmap='gray')
plt.axis('off')
plt.show()

We thank you for your time and comments. We have carefully considered each point raised and have done our best to address them in detail below. Thank you for helping us improve the paper.

---

Weaknesses:

1.“'linear shift' without a precise description”: We tried to define linear shifts as soon as line 6 of the abstract as a simple (ax + b) operation where “a” and “b” are scalars. For example (7, 7) = 2 × (3, 3) + 1. We mentioned this once again in line 264 “mathematically expressed as a(x + b).”. We would clarify this more by writing an example in the paper. This is a very simple operation and linear shift was the best term we could come up with. If you have any suggestion about using a better term we would appreciate your input. Affine transformation requires matrix multiplication to move a geometric shape, but here “a” is only a scalar.

2.VAEs are not suitable for this task. VAEs are used as a probabilistic generative model to produce new samples and generate interpolations between samples, but here we strongly do not want any new sample. And we strongly need to keep the code dimension to be 1, because then we can select limited samples (in this paper 50). But 1D latent space is much more restrictive for VAE and doesn’t make sense. We’ve spent months and tried many different models to finally come to auto encoders, I think you are underestimating the power of AE in this task to just call them “vanilla”. By the way, we would be happy for sure if we can find a better model/method to do this task. If there is any other method that is not the most solid we are happy to hear.

---

Writing:

• We believe the reviewer is not familiar with the literature of DS-CNNs. We are indeed studying DS-CNNs models [1], We call the convolution part of DS-CNNs “Depthwise Conv” and when it is combined with the next fully connected layer we call the whole model Depthwise Separable Convolution model. And No! 7×7 filters indeed have 49 parameters, not 14. Working on this topic for long, we really don’t know which reference you are pointing to here (49 parameter depth-wise CNN) vs (14 parameter DS-CNN). Could you please cite a model that is DS-CNNs and has 14 parameter for its 7×7 filter?

• Thanks for helping us improve the paper. We would change “length” to “norm” to not make this confusion. In our experiments center-norming helped the autoencoder to encode better.

• Figure 2 is just for visualization purposes not the actual model. Our model also inputs 7×7=49 parameters but we showed a 3×3=9 input, because the other one was just too big to show.

• Matrix F is actually defined in line 140, and it is used later, in line 159. Matrix F serves as a compact representation of all the depthwise filters in a layer.

• “In Line 142” You are totally right in this matter this is a valid confusion that might accrue. We have to rewrite that section carefully to clarify this. Thank you for pointing that out.

• We wrote in 147 that “we use linear regression” and in 149 “ This problem has a well-known solution.” But we needed to put the formula to make the other steps clear to the reader, as we say in 151 “Calculating Equations (1) can be computationally intensive” then we follow by our own method to reduce the computation by normalising the candidate filters first. We don’t think this is trivial for all readers.

• “greedy algorithm” At first we considered this as trivial, but now we would add pseudo code to the appendix to improve the clarity.

• “ three pages long without subsection”: thank you for pointing this out. We would try our best to make that long section more clear.

---

Method:

• We have addressed this above.

• “Does it mean the phenomenon can only be observed in depth-wise but not a standard one?” This is an excellent question that we can write a lot about. There is evidence that classical CNNs are converging towards some “Master Key filters” too, but the evidence for DS-CNN filters is much stronger. On the other hand, unlike classical CNNs, DS-CNN filters have been shown to follow gaussian patterns, and that was our motivation for this paper to see how much can we shrink this pool? Although maybe “Master Key Hypothesis” is true for classical CNNs, we don’t think we ever can find a set as small as 8 filters for them.

---

Experiments:

• This is completely true and valid, and we did not claim we do parameter reduction in this paper. This is an analytical paper for scientific understanding.

• This hypothesis is wrong. First, we already put a random experiment in Table 1, in which we showed that the trained filters are converging to a linear shift of one of our 8 filters out of nearly infinite possible filters in the space. Second, for you, we conducted the experiment of Table 2 with 8 random filters, and after 80 epochs we stopped the experiment due to low performance. The accuracy was 23.7 at epoch 80. For your reference, our accuracy was 78 at this epoch.

• We doubt similar results can be achieved for classical CNNs, but it is worth trying, and we will explore this.

---

[1] [Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1251–1258. https://doi.org/10.1109/CVPR.2017.195]

Dear reviewer,

Thank you for reviewing this paper. The authors have provided a detailed rebuttal to your review, but I cannot see any edits to your review that take the author reply into account, and also your "Final justification" is missing. Please engage with to the rebuttal asap. Note I flagged you as non-responsive, but this flag will be removed once you posted a satisfactory reply

Thank you, the AC

We sincerely thank you for the thorough review, especially for providing clear action points. This form of review is something we truly appreciate, and we were even inspired to adopt a similar style in our own future reviews. Below we provide our responses:

---

Weaknesses:

1.1 We gathered around 1 million filters in the bank from 54 models of different architectures, all with 7×7 depthwise filters, and with varying model sizes, dataset sizes (ImageNet-1K or 21K), and input image resolutions. We will add a summary to the main text and a comprehensive list to the appendix of the paper to cover this information.

1.2 Our observations of the proportions of filters in each model revealed that each model family has rather consistent proportions of each filter type in its trained filters, regardless of model size (more details are presented in the next action item). The proportions vary slightly between layers, but our experiments suggest that a uniform distribution per layer works similarly. We used the proportions derived from the trained models for initializing the networks.

---

2. Depending on the architecture, the most popular filter type differs. For instance, in ConvNeXtV2, the most frequent was the Gaussian filter (Type 8), whereas in HorNet, it was Type 1. We have prepared plots showing statistics in total and per layer for each model. Unfortunately, we cannot include these here due to rebuttal rules, but we will add them to the paper and appendix. Below is an illustrative table:

As shown, ConvNeXtV2 has proportions similar within its family but distinct from the HorNet family. Despite the lower percentages for filter types 1–4 in ConvNeXtV2, they are essential for model accuracy. We performed experiments removing those four types, and the accuracy dropped noticeably.

---

3. Thank you for the suggestion. For the original models, we took trained weights and accuracies directly from the official repositories, so we did not initially have the training logs. We have now started training the original models on our own GPUs and have plotted the loss dynamics for training and validation for one model, both with and without frozen initialized filters. While we cannot provide figures and plots here, we will include them in the final paper.

---

4. DS-CNNs with 3×3 filters (such as MobileNet) also show similar patterns in their trained filters. Our autoencoder method (Figure 2) does not converge well on 3×3 filters, even though Gaussian patterns are visible. In this paper, we performed our search on 7×7 filters since we could gather a large enough bank of filters and our autoencoder converged to a reach code layer.

---

Small:

1. Based on our experience, not all readers are familiar with DS-CNN architectures. Regarding scale-space theory, there is a fascinating link to our paper: the theory mathematically proves that convolution with Gaussian or derivative filters is the only operation that can scale an image. This classical computer vision theory proposed the Gaussian filters we found in an unsupervised manner, years before modern deep learning. We agree this related work section can be shortened, and we will do so to make room for the new suggested details and plots.

2. We have added the missing citation. Thank you for pointing this out.

3. We have fixed the noted issue. Thanks again.

---

Questions:

1. We do not have prior experience with quantization, but the idea is interesting and our results may indeed be useful in this direction.

2. Yes, all code will be released in a public repository. Also, the 8 filters are already provided in full in the appendix, and can be directly used for initialization or other experiments.

1. "Do you have any suggestions on how we can share the plots with you?" It seems that it is impossible this year, sorry for misleading you.
2. “Do you have any ideas why the autoencoder does not converge for 3 x 3 filters?” Please add your explanation to the paper.

I have also read your conversation with reviewer MKLF. Please note that while they might be overreacting, your writing in rebuttal comment to them is not as neutral and polite as it should be (and as it was in your other rebuttal answers here).

1. Please note that in formal arguments you should not consider emotional judgments, such as “This was a surprising result for us and very exciting.”, being a good argument.
2. Your answers also should not get personal, as in “You clearly didn’t understand the "master key filter hypothesis” paper.” – this phrase does not add any relevant information to the comment.
3. Please try not to be seriously offended by the reviewer writing “So, what?” to your paper: 1) English might not be their native language, so they might not even see that as offensive 2) It is absolutely impractical to try to get back at them.
4. If a reviewer suggests you to change the wording when they did not understand something, it is actually an important input – if ¼ of reviewers did not understand what you wrote, then probably ¼ of other readers would not understand it as well.

Overall, you addressed the points from my initial review, so I will raise my score to 4.

We genuinely appreciate the time you have put to read and review our work.

Weaknesses:

---

1. Contribution overlaps / “limits the novelty”

This is definitely not correct and actually surprised us. The paper starts by stating:

“This paper extends this hypothesis by radically constraining its scope to a single set of just 8 universal filters.”

Confining a hypothesis that could accept sets of hundreds of thousands of filters to an experimentally verified 8 filters is undeniably novel. By the same logic, any paper building on the “lottery ticket hypothesis” or any other hypothesis would not be novel.

We appreciate that the reviewer acknowledged:

“I acknowledge that implementing the master keys as eight filters is novel and makes a contribution.”

Thank you for stating this.

However, regarding:

“But it is a bit insufficient for me.”

If the implication is that after reading the previous two papers it was trivial to conclude that all filters could be replaced by just 8 without significant accuracy drop (Table 2), we must respectfully disagree. This was a surprising result for us and very exciting. Moreover, filters 1–4 were discovered in this paper and are not mentioned in prior work.

---

2. Limited practical contribution

Great practical values are a major potential of theoretical and analytical understanding, we shouldn't shut down analytical papers because they aren’t engineering papers. Despite this paper not being a practical paper, we showed our 8 filters have practical initialization values in Table 3 that they even beat ImageNet transfer learning on smaller datasets (isn’t that a surprise?). We hope the future “master Key” filters can show a much better practical value.

---

3. Additional parameters A, B, shift

We believe there is a misunderstanding about both our paper and the “Master Key Filters Hypothesis.”

• First, we are not introducing additional parameters to represent our 8 filters. If you are referring to linear shifts (ax+b) in Table 1, we are showing that the filters of all those trained networks are closely converged to the linear shift of one of these 8 filters, out of the near infinite space they could converge to. And this is because DS-CNN filters are scale-free due to their architecture. And then, in Table 2, we are not using linear shifts. The master keys set we are introducing are these 8 filters.

• Second, there should be a misunderstanding about the master keys hypothesis too, using parameters does not conflict with the master keys hypothesis. This hypothesis says that there exists sets (not necessarily one set) of master key filters (can be thousands of different filters) that are universal for any input data, in contrast to previous beliefs. To put it simply, it says the previous hypothesis of “specialized(data-specific) filters in deep layers” is not true.

Regarding:

“I think A, B are necessary”

If by A, B you mean linear shifts, they are not necessary. Table 2 shows that a model with only 8 unique filters (no shifts) can achieve comparable accuracy.

On:

“bilinear interpolation”

We do not use interpolation. In Table 1, we simply apply linear shifts (aX + b). For example, (7, 7) is a shift of (3, 3) because (7, 7) = 2 × (3, 3) + 1. In Table 1 we are showing with an unsupervised experiment that filters of all models are closely converging to a linear shift of one of those 8 filters.

---

4. Limited applicable architectures

This is a DS-CNN analysis paper, but DS-CNN models are far from “limited.” Almost all top CNN-based models today are DS-CNNs. The last state-of-the-art classical CNN we recall is ResNeXt-101 from 2017, seven years ago.
It is not a paper about ViTs, and this is not a limitation, it is focus.

---

5. Finding “Master Keys” requires evaluation on the test set

To address your concern, we’ll use a validation set from the training set to perform the search for the samples. The results are the same. Actually you can repeat the same experiment on any large test set you want, the results would be similar.

---

Questions:

1. We hope the difference between our work and prior work is now clear. If not, we ask the reviewer to show:

   • (a) where in those papers it is demonstrated that trained filters converge close to a linear shift of one of our 8 filters;
   • (b) where it is shown that you can train a model with only 8 unique filters and achieve comparable accuracy;
   • (c) where 50% of our filters (filters 1–4) appear in those papers.

2. We hope future works can find practical benefits, but right now finding the minimum set is more than just interesting to us, it’s scientifically important.

3. Linear shifts are discussed in our previous discussion as (aX+b) where “a” and “b” both float scalars. We don’t learn them, we only calculate them in Table 1 with linear regression discussed in line 147.

4. As we discussed, we don’t use linear shifts for training these 8 filters, the linear shift is only for Table 1.

5. In both cases It’s the length of the vectors, here 7×7 = 49. But I totally get where the confusion comes from! In the next lines “n” is different and it’s the number of different vectors (here number of filters we have). It’s a mistake from our part and we have to fix the notation. A genuine thank you for noticing this.

6. On “without pre-LayerNorm”: Our intuitional answer would be yes. If you visualize the filter of all these models you’d see clear gaussian patterns no matter the task. There is no reason for us to believe this wouldn't work on non pre-LayerNorm when we can see the filters look the same. But can we show experimental results right now? No, is it because of the LayerNorm? No! It’s because of the filter size. The models you are introducing preliminary use 3×3 filters and our autoencoder method (Figure 2) just does not converge well on 3×3 filters and we couldn’t fix it although the gaussian filters are there. That’s why we have to go with 7×7 filters. If you know any non pre-LayerNorm 7×7 DS-CNN model we would be happy to get the test right now with the same 8 filters we extracted from ConvNext-base.

PART 1

We first want to say that we did not intend any disrespect towards the reviewer, and if in any way our responses have caused this feeling, we sincerely apologize. Before responding to your further concerns, we would also like to point out that in the peer-review process it is expected that both of us are “peers”. Disagreeing and expressing what we believe is right is not disrespect, and we actually appreciate disagreements. In this regard however, expressions like “So what?” are condescending, or using terms like “frustrated” are disrespectful, which we never expressed in our responses.
We want to have a meaningful, peer-to-peer discussion, freely expressing what we regard as facts, to come closer to a common ground.

---

1. Contribution overlaps / “limits the novelty”

• “The important point is that DS conv filters are redundant and can be reduced.”: We believe there is a misunderstanding around the "master key filter hypothesis” paper. That paper is not about filter redundancy, it says nothing about it. Actually this hypothesis lets all filters of a DS-CNN model be different. What that paper is about, is filter generalization. Previously, it was believed that the deep filters of CNN models are not general and they are dataset-dependent, the "master key filter hypothesis” paper hypothesised otherwise, all filters are general and there exist universal sets for CNN models. But such a set can be as large as all filters of a model, no redundancy is needed.
  In contrast, the important point that you are mentioning as a contribution of that paper, is one of the most important contributions of our paper! You can see it for yourself in our paper, look at line 103, our bold title is “Do We Need Thousands of Distinct Filters?” This is actually the single motivation of this paper! Exactly in the next line we write ““we investigate whether employing thousands of unique filters is essential for maintaining the performance of DS-CNNs.” That is why our paper is indeed novel, and as pointed by you, makes an important point.
  These aside, we want to make a heart-to-heart statement here: Do you see the misunderstanding gap between you and us here? We know these gaps can be frustrating for you, but note that it’s hard-to-handle for us as well. Sorry if you were bothered before.
• “but I don't think the number of filters makes it interesting”: That paper is not talking about the number of filters or filter redundancy, but for the point that number of filters doesn’t matter we have to strongly disagree. How come it doesn't matter if a basis filter set of master key filters are 10,000 filters or 8? Of course finding a limited set, identifying them, and reporting them is important.
• “This paper reduces the number of keys to 8 filters. So, what?” We don’t know how to respond to this question. A paper not being engineering, doesn’t mean its scientific findings are “So, What?”. You can ask this question from many fundamental papers of deep learning. For example, if you were an author of >5K cited paper ”Understanding deep learning requires rethinking generalization”, and a reviewer asked you “so, what?” How would you answer? (and about the incremental part, we hope we have cleared it.)

2. Limited practical contribution

• This paper contributes to our understanding of inner-working of DS-CNNs. This is clearly one of the subareas of the subject area “Deep Learning” in Neurips: “Analysis and Understanding of Deep Networks”. If this topic is not your area of interest, we understand, but it definitely is not a weakness. Also, as shown in Table 3, we reference the baseline Imagenet filters transfer which is the most common transfer learning approach.

---

To be continued on the next answer.