The Power of Linear Combinations: Learning with Random Convolutions

W1: [...] practical impact of this approach [...] Q3: Could you clarify what you see as the main practical impact of this approach? (see above)

Our practical relevance is showing, that not learning the spatial convolutions obtains similar or even better accuracy/robustness in large networks. This comes at the benefit of a significantly reduced parameter count, and thus lower VRAM, faster training etc.

Are such large networks realistic? Generally, we are all aware of the trend for larger models, but even today adversarial robustness SOTA (see https://robustbench.github.io/ under CIFAR-10, $\ell_\infty$) uses WideResNets-70x16 [1] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments.

Just to clarify: generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The LC-ResNets are just proxies for linear combinations in real architectures.

Additionally, we deliver “food for thought”: i) pointwise convolutions after learnable spatial convolutions can impair performance - does this limit current architectures (e.g., ResNet-50)? ii) currently practiced initializations are not ideal when scaling kernel size (as commonly done) - spatial considerations are one thing to consider; iii) any studies of convolution filters must consider the presence of linear combinations.

Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

W2.1: Second, the theoretical analysis of their approach seems somewhat limited. Although the authors list as one of their contributions the theoretical explanation for the impressive performance of their factorization, I'm not sure where in the paper this happens. Q4: Could you clarify what you list as theoretical analysis for the performance of your approach? (see above)

We refer to the theory at the beginning of Sec. 4, where we show that a pointwise convolution following a spatial convolution is equal to a single convolution with a linear combination of the filters. Then it follows, that in such a scenario, freezing spatial filters to random inits and learning LCs is as expressive as learning them (if you have sufficiently many). Since random LC networks can learn approximations of fully learnable filters they perform similarly. We believe that this theory sufficiently explains why some networks can train to near-same accuracy with random spatial filters. Does this clarify the theoretical analysis?

W2.2: On the one hand the authors say a large enough basis of random spatial filters provides enough expressivity to represent any other fully learnable spatial kernel (lemma 4.1), on the other hand they indicate that their approach has a clear regularizing effect, improving robustness. Are these not two contrasting observations?

Actually, they are not! The lemma only holds if you have sufficiently many kernels. In practice, you will have a limited number and only approximate the learnable filters (e.g., see Fig. 8 in the Appendix) - keep in mind, that the 1x1 operates on the $k\times k\times c_{in}$ filter and not kernel level and must create all $c_{out}$ filters ($k\times k\times c_{in}$) from the random filters.

W3: Third, the experiments conducted by the authors in the main body of the paper are quite limited in scope, in that they are all classification problems. Given that the authors themselves indicate that linear combinations of random weights have a hard time forming sharp spatial patterns, I think more attention should also be given to e.g. fine-grained segmentation (I’ve seen the results in the appendix but the authors might consider moving them to the main body and providing a more detailed experimental setup). Q2.3: … but would this mean your model is unable to perform more fine-grained tasks such as segmentation? Why not?

Due to space limits, we decided to put these results in the appendix (as the trend is similar). Including these results would mean that we have to remove or shorten some other section. Do you have any suggestions on what we should replace?

Regarding the sharp spatial patterns: This only affects larger kernels. Common models such as U-Net with 3x3 are not affected. Besides, “sharp” transitions in masks are generated by the high-frequency information on skip-connections, not the up-convolutions [1]. In fact, the results in the appendix are obtained on a U-Net.

[1] Olaf Ronneberger, Philipp Fischer, Thomas Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI, 2015.

Q1: how do the modified models and their baseline counterparts compare in terms of computational requirements?

We deliberately did not report computational requirements, because we do not propose to use the LC-ResNets instead of off-the-shelf architectures (see above).

Our message is freezing spatial filters reduces training time and does not affect accuracy (or even decreases) it in large networks. Here is an example of common networks. We report the number of trainable parameters (as a proxy for VRAM requirements) and latency (required time for one forward and backward pass). Please note that the inference time will depend on the framework, driver, hardware etc. FLOPs remain unaffected.

Q2.1: I assume the visualisations of the kernels shown for the frozen model are obtained after training?

Yes, all visualized weights are shown after training.

Q2.2: You indicate that there is a very clear spatial pattern in trained non-frozen kernels that isn’t present in the frozen model, and you indicate this as the main reason for trailing performance. This seems like a reasonable hypothesis (although somewhat in contrast to Fig 2), …

Fig. 2 shows a smooth resulting kernel - for an example of a “sharp” filter please see Fig. 8 in the Appendix. However, if you had enough filters/LCs, you could reproduce the sharp filters as such there is no contrast to Fig. 2.

Please let us know if we have addressed all of your concerns. If you have any other questions/concerns please feel free to ask. We would be grateful if you would reconsider your score in light of our updates. Thank you again for your time and consideration.

W1: The idea is trivial and well-known.

With all due respect, we remind you of the reviewer's instructions for raising weaknesses: “Be specific, avoid generic remarks. [...] if you believe the contribution lacks novelty, provide references and an explanation as evidence;”

If you believe that our results are well-known, please provide references and explanations as evidence. As a reminder here are our key statements from the conclusion section:

1. Random frozen CNNs can outperform fully trainable baselines in clean and robust accuracy.
2. In modern off-the-shelf networks, learned spatial convolution filters only marginally improve the performance compared to frozen random models in small kernel regimes. The reason for this is an implicit computation of linear combinations between pointwise and spatial filters.
3. Adding learnable pointwise convolutions immediately after spatial convolutions can decrease performance and robustness.
4. Only, under increasing kernel size, learned convolution filters begin to increase in importance but only due to a different spatial distribution of weights that cannot be reflected by currently practiced i.i.d. initializations.

Needless to say, we disagree that these statements are well-known or trivial.

W2: the experiments are done on toy models and datasets, which are not convincing. Q1: Experiments on state-of-the-art models

It is not true that we only experiment with toy models and datasets. We ablate the effects on various ResNets scaling linear combinations, width, and depth on CIFAR-10. Following that, we show that the results transfer to multiple other datasets including ImageNet (Sec. 5.2) which is arguably the benchmark in vision, as well as other tasks such as Object Detection and Segmentation in Appendix A.1 containing real-life problems from Kaggle Challenges. Fig. 1 also shows the behavior of popular benchmark CNNs: ResNe(X)t-50 and Wide-ResNet-50. In all of these experiments, we see the same observations - how can the results be unconvincing?

Please let us know if we have addressed your concerns. If you have any other questions/concerns please feel free to ask. We would be grateful if you would reconsider your score in light of our updates. Thank you again for your time and consideration.

S1: Altogether I believe this is a very strong submission with a little flaw in its evaluation.

Thank you for stating that our submission is very strong! Could you please clarify why you think the evaluation has a flaw? We would be grateful if you could tell us what we need to improve for you to increase your score.

W1: The only weakness I see is that the experiments in Section 5.2 are not conclusive. I would encourage the authors to try and improve this part as it weakens the analysis of the paper –which is very strong up to this point.

Sec. 5.2 (“Scaling to other problems”) shows that our results are not limited to CIFAR-10 but scale to other datasets and tasks. We think it is a very important section - otherwise, readers may be concerned that the findings don’t scale (e.g., see Reviewer gGqG). Could it be that you meant to some other section?

Q1: Do you have any insights wrt modus operandi of long convolutional models and how it differs from that of conventional (small kernel) architectures?

Long convolutions are more popular in NLP tasks, where it seems reasonable to assume that there is a correlation between distance in the context and relevance, i.e., far tokens are less relevant than closer ones. As such, long convolution kernels often show a decay of magnitude with depth (e.g., see [1]), which is very similar to the large kernel findings on vision we have in Sec. 5.2. Thus we would expect similar implications: I) current init methods won’t allow to us to reconstruct these filters only via LCs from random filters; II) we need better init methods for large/long convolution weights.

[1] Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey, “What makes convolutional models great on long sequence modeling?”, arXiv preprint arXiv:2210.09298, 2022.

W1: The hypothesis and results are not surprising at all given how many linear combinations the authors learn in replacement of learned spatial filters. [...] Q1: If you believe this is not explained simply as learning from an orthogonal basis, as explained above, how many of the randomly sampled spatial filters are not orthogonal? [...]

The central assumption of the reviewer’s rejection rating regarding the orthogonality does not hold - neither in theory nor in practice! Theoretically (by the law of large numbers), the reviewer's orthogonality assumption only holds for very high dimensional vectors ( $C \rightarrow \infty$ ), but the actual dimensionality of C used in our evaluation (and realistic applications) is far from that. We have $C \leq 64$ in our experiments.
To verify this, we have followed your suggestion and computed this on the largest layer in a ResNet-LC20x32 under both, uniform and normal inits. This model breaks even with the baseline - as such it should contain a significant ratio of orthogonality. Yet, even when we conservatively define orthogonality with cosine similarity < 0.2, 0% of the filter pairs are orthogonal in both cases.

W2: [...] is it useful to train a CNN in this manner over learning spatial filters? It's obvious that, as explained by the authors in the paper, using frozen LC models that match learned LC generalization performance are much more expensive [....] Q2: what does your paper contribute that is not simply empirical validation of this understanding? Q3.1: Why is it useful to train a frozen LC model v.s. a learned spatial model that achieve the same generalization?

Just to clarify: Generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The ResNets-LC are just proxies for linear combinations in real architectures.

Instead, we ask: why would you train all parameters, if not learning the spatial convolutions obtains similar or even better accuracy at a significantly reduced parameter count and lower VRAM, faster training etc.? Our results show that, eventually, the utility of learned convolutions saturates and beyond that deteriorates performance - if networks are sufficiently large.

Are such networks realistic? Yes. SOTA research on adversarial robustness (see https://robustbench.github.io/ under CIFAR-10, $\ell_\infty$) already uses WideResNets-70x16 [2] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments. And, generally, we are all aware of the trend for larger models.

Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

W3: Many of the empirical results compare models with drastically different numbers of trainable parameters, while using the same training hyperparameters. [...] Q4: Did you do any hyperparameter turning for frozen LC v.s. learnable LC models you compared to ensure using the same hyperparameters/training setup is appropriate?

We have opted for an extra-long schedule relative to the small datasets (200 Epochs) to ensure appropriate convergence. We have ensured that the training accuracy of learnable LC-ResNets at any expansion is near 100%, i.e. they have converged. Further, earlier experiments with shorter schedules (75 Epochs under cosine decay) showed similar trends. Thus we are confident that we appropriately train the models. Tuning is mostly trial-and-error, of course, it may be possible to skew the results in either way - we are confident that using these parameters for all experiments allows for a fairer comparison.

W4: No explicit comparison of the compute and VRAM requirements for training models that are compared, in many cases there is quite a drastic difference it would seem. This is especially stark in section 5.1 where models with exponential increasing compute/param are compared with a fixed baseline model, i.e. ResNet-20-16. Q3.2: What is the tradeoff in compute and working memory (i.e. GPU VRAM) utilization for achieving this with frozen LC models v.s. a baseline learned spatial model for the same generalization performance?

We deliberately did not report computational requirements, because we do not propose to use the ResNet-LCs instead of off-the-shelf architectures (see above).

Our message is freezing spatial filters reduces training time and does not affect (or even decreases) accuracy in large networks. Here is an example of common networks. We report the number of trainable parameters (as a proxy for VRAM requirements) and latency (required time for one forward and backward pass). Please note that the inference time will depend on the framework, driver, hardware etc.

W5; While I like the robustness analysis and think it warrants further attention, this is yet another reason why even when frozen LC models match generalization, they are poorly motivated.

We aim to explore the robustness aspect in future work. We have also observed that by using label smoothing we can further amplify the effect. One idea that we can give in this rebuttal is that due to the linear combinations, filters tend to be smoother, and at least for the initial layer this has been linked to improved robustness [1].

[1] Haohan Wang, Xindi Wu, Zeyi Huang, Eric P. Xing, “High-frequency Component Helps Explain the Generalization of Convolutional Neural Networks”, CVPR, 2020.

[2] ShengYun Peng, Weilin Xu, Cory Cornelius, Matthew Hull, Kevin Li, Rahul Duggal, Mansi Phute, Jason Martin, Duen Horng Chau, “Robust Principles: Architectural Design Principles for Adversarially Robust CNNs”, BMVC, 2023.

Please let us know if we have addressed your concerns. If you have any other questions/concerns please feel free to ask. We would be grateful if you would reconsider your score in light of our updates. Thank you again for your time and consideration.

I'd like to thank the authors for their rebuttal, and apologize for my late participation in the rebuttal period - this was due to exceptional circumstances.

The central assumption of the reviewer’s rejection rating regarding the orthogonality does not hold - neither in theory nor in practice! Theoretically (by the law of large numbers), the reviewer's orthogonality assumption only holds for very high dimensional vectors ( ), but the actual dimensionality of C used in our evaluation (and realistic applications) is far from that. We have in our experiments. To verify this, we have followed your suggestion and computed this on the largest layer in a ResNet-LC20x32 under both, uniform and normal inits. This model breaks even with the baseline - as such it should contain a significant ratio of orthogonality. Yet, even when we conservatively define orthogonality with cosine similarity < 0.2, 0% of the filter pairs are orthogonal in both cases.

This has little to do with the law of large numbers. It's just simply a fact that sampling random vectors from high-dimensional spaces is unlikely to result in vectors that are collinear, and yes even 3x3x64 is still a relatively high-dimensional space where I believe that will hold. If all your basis vectors (i.e. spatial filters) are co-linear as you are claiming above, you could never represent anything very different with any of your LC models's filters... so that makes very little sense! I believe it sounds like you have not computed the orthogonality of the spatial filters correctly (note this is the random spatial filters only you should be looking at).

Just to clarify: Generally, we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless (although in some cases it is an easy trick to improve performance - e.g., see Tab. 4 in the Appendix). The ResNets-LC are just proxies for linear combinations in real architectures. Instead, we ask: why would you train all parameters, if not learning the spatial convolutions obtains similar or even better accuracy at a significantly reduced parameter count and lower VRAM, faster training etc.? ... Are such networks realistic? Yes. SOTA research on adversarial robustness (see https://robustbench.github.io/ under CIFAR-10, ) already uses WideResNets-70x16 [2] (16 times wider than a normal ResNet)! We assume that we would obtain cheaper and better training with freezing. Sadly we cannot verify this, due to the necessary GPU VRAM to run experiments. And, generally, we are all aware of the trend for larger models.

Does this clarify our intent? Would the reviewer suggest that we update our conclusion with this text?

Unfortunately that doesn't clarify anything for me, in many ways it confuses me further. If your question is "why would you train all parameters..", and simultaneously you are claiming "we do not promote training a frozen LCified network over a non-LC baseline. That would be indeed useless", then hasn't that answered the question you claim to pose? Also since VRAM is the most bottlenecked resource for DNN training on GPU, any tradeoffs you show for "efficiency" in training LC models at the sacrifice of VRAM is very poorly motivated indeed.

Our results show that, eventually, the utility of learned convolutions saturates and beyond that deteriorates performance - if networks are sufficiently large.

This is indeed the most interesting bit of your results I'll grant you if it holds up, but I'm not convinced that the results do hold up given the other issues I highlighted on using the same hyper-parameters across models of very different numbers of learnable parameters.

We have opted for an extra-long schedule relative to the small datasets (200 Epochs) to ensure appropriate convergence. We have ensured that the training accuracy of learnable LC-ResNets at any expansion is near 100%, i.e. they have converged. Further, earlier experiments with shorter schedules (75 Epochs under cosine decay) showed similar trends. Thus we are confident that we appropriately train the models. Tuning is mostly trial-and-error, of course, it may be possible to skew the results in either way - we are confident that using these parameters for all experiments allows for a fairer comparison.

An extra-long training schedule does not make up for potentially poor choices of hyper-parameters in training models with very different numbers of trainable parameters. While I appreciate this is a difficult thing to do, giving us some idea of how you've found the hyper parameters for different models (or trying some tuning to verify they are reasonable) is necessary when you are comparing two training runs of very different models.

Unfortunately this rebuttal has only made me question some of the analysis and results further. I apologize for being so late to the review process, but I hope the authors can further clarify.