Hypernetwork-Based Equivariant CNNs

Thank you for your constructive comments. Below we have made responses to your comments. If you have any further comments, please feel free to let us know and we are more than glad to discuss with you.

Question 1: Could you please expand on how your work relates and contrasts from the work of Zhdanov et al[5], who also use a hypernetwork-based approach to parameterize steerable convolutional kernels? I think it might be good to mention this work.

We thank you for referencing this paper. In this work by Zhdanov et al. [r1], they used a MLP to generate $G$-Steerable kernels, instead of computing kernel basis for each particular group $G$. This definitely falls into the category of a hypernetwork. While their work simplifies the process of finding steerable kernels, the main difference from our approach is that their main network is still steerable CNNs. In contrast, our HE-CNN uses a regular CNN as our main network, and does not incorporate any G-CNN based structures. This related work will be added to the final version of our paper.

[r1] Zhdanov, M., Hoffmann, N., & Cesa, G. (2024). Implicit convolutional kernels for steerable CNNs. Advances in Neural Information Processing Systems, 36.

Question 2: Could you provide details on the computational complexity of your method, i.e. its runtime and memory efficiency compared to the base CNN and G-CNN models? I think this is an important factor to consider in the evaluation of your work, as it might help the reader gauge how well your method would scale to larger-scale data.

In the following table, we provide our model's training time per iteration comparing to base CNN and G-CNN on the CIFAR10 dataset based on the 13-layer CNN. We further trained G-CNN with a dynamic hypernetwork for comparison.

Weakness 1: My main concern is with respect to the computational overhead that your method imposes, passing inputs through the NEP $|G|$ times. It would be good to indicate how your method compares in terms of memory and computational efficiency to base CNN and G-CNN models. It seems to me this computational requirement might be prohibitive in applying your model to larger-scale data (e.g. imagenet which has 224x224 resolution). Could you comment on this?

We agree with you that our current limitations on scalability, primarily due to the usage of hypernetworks and generating input-dependent parameters. Our current solution utilized LoRA to reduce the parameter of the last linear layer in the NEP generator. The comparison of parameter count and training time are detailed in our response to Question 2. The reason for not applying our method to larger-scale datasets is that there are few applications for baseline equivariant CNNs on ImageNet/Tiny ImageNet. As of this response, we found no reports based on G-CNN or steerable CNNs on the Paper with Code website for ImageNet/Tiny ImageNet. We choose Rotated MNIST, STL10 and CIFAR10/100 to match datasets chosen in group equivariant CNNs [r2] and steerable CNNS [r3,r4].

[r2] Cohen, T., & Welling, M. (2016, June). Group equivariant convolutional networks. In International conference on machine learning (pp. 2990-2999). PMLR. [r3] Cohen, T. S., & Welling, M. (2016). Steerable cnns. arXiv preprint arXiv:1612.08498. [r4] Weiler, M., & Cesa, G. (2019). General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32.

Weakness 2: The experimental validation of this method is somewhat limited. Especially the results in Sec. 5.1 are hard to valuate, the authors seem to intentionally have chose a very weak baseline (96% on MNIST can easily be achieved by simple MLPs), but i'm not sure why they chose this base model. It makes the results in this table very hard to judge on how informative they are. I think it would be worthwhile to additionally include results for non-invariant tasks (e.g. segmentation).

We appreciate your concern. A more detailed comparison of our methods on Rotated MNIST is done in section 5.2, utilizing a significantly larger models. As of section 5.1, such experiment are designed not to show the full capacity of each methods. Rather, we want to empirically show that constraints on the $Z4$-filters negatively impact performances, especially when trained and tested on the original MNIST. We believe a smaller model would effectively highlights these constraints, and we ensured that the network sizes are consistent across all setups for a fair comparison.

Thank you for your constructive comments. Below we have made responses to your comments. If you have any further comments, please feel free to let us know and we are more than glad to discuss with you.

Weakness 1: The 90-degree rotation settings is over-simplified to some extent. It is easier for the hypernetwork to learn the G-equivariant parameters compared with a larger rotation group. Especially in the linear layer setting, if the rotation degree is not in {0, 90, 180, 270}, the rotated feature cannot be easily represented as permutation which is the premise of the hypernetwork design.

We appreciate your insight regarding our HE-CNN, which currently supports a subset of common transformations. In Section 4.5, we extend the 90-degree rotation to rotations of 90/n degrees. Similar to the G-CNN, we are currently limited to translation and discrete rotations. However, we are optimistic that these limitations can be addressed in future work. As the E(2) steerable CNN [r1] extends the G-CNN to the continuous case, we look forward to learning from it to enhance our HE-CNN for future exploration.

[r1] Weiler, M., & Cesa, G. (2019). General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32.

Weakness 2: The current presentation, especially equations and figures, is a bit hard to interpret. For example, in Line 204, what’s the dimension of the output $a^i$.

Thank you for pointing out potential sources of confusion. In Figure 2, we intended to show that each $a^i$ is a quarter of the full filter. When the full filter is in dimension $K$, the exact dimension of each $a^i$ is (out channel, in channel, $\lceil \frac{K}{2} \rceil$, $\lceil \frac{K}{2} \rceil$), as mentioned in line 198 on page 4. We agree that we should make the connection more clear, and adjustments will be made in the final version.

Weakness 3: The proposed method does not show competitive results compared with the given limited benchmarks, especially in experiments with deeper CNNs on the CIFAR-10/100 datasets.

We appreciate your concerns regarding the comparison of our method with researches building upon G-CNN. We would like to clarify that our HE-CNN aims to achieve equivariance through a novel approach while reducing the constraints present in G-CNN-based structures. Thus, main baseline methods in comparison are G-CNN-based methods such as partial equivariance [r2], which are largely derived from G-CNN. Since steerable CNNs, along with most equivariance structures in image processing, are based on G-CNN, we believe our novel alternative has significant potential for future extensions.

[r2] Romero, D. W., & Lohit, S. (2021). Learning partial equivariances from data. arXiv preprint arXiv:2110.10211.

Question 1: Could you please elaborate on the detailed hypernetwork design used in the experiments?

For a general guide line, our hypernetwork should output approximately a fourth of the main network's parameters. For our implementation, All our Hypernetwork are CNN based. For experiment in section 5.1, our NEP generator is composed of 2 hidden convolutional layers with 16 hidden channels and three by three kernels. We have one linear head for each layer in the main network, and used LoRA for each layer with intermediate rank of 4. For experiment in section 5.2, the NEP generator is composed of a 3-layer convolution with [16,32,32] intermediate channels,[3,3,4]filter size and [2,2,1] stride. If LoRA is utilized, the intermediate rank is set to 4. For the experiment on CIFAR10/100, the NEP generator has 3 convolution layer with [32,64,64] channels with all filter sizes 3. Batch normalization, relu and max-pooling layers are inserted between convolutional layers. Stride is set to 2 for the last convolutional layer. When LoRA is used, intermediate rank is set as 7. For STL10, NEP generator is composed of 4 convolutional layers with [64,128,128,64] intermediate channels with filter size 3. Batch normalization, relu and max-pooling layers are inserted between convolutional layers. Pooling are down after first three convolutional layers.

Thank you for your constructive comments. Below we have made responses to your comments. If you have any further comments, please feel free to let us know and we are more than glad to discuss with you.

Question 1: What is the train/test split for the MNIST datasets used in this paper? Is it 12k/50k?

For the original MNIST and the randomly rotated MNIST, we use the default split from torchvision MNIST dataset, which is 60k/10k for training and testing, respectively.

Weakness 1: Outdated and Inconsistent Baselines.

We want to clarify that our main contribution is to propose the HE-CNN, achieving equivariance with a novel approach and alleviating the constraints presented in methods based on G-CNN. Thus, main baseline methods in comparison are G-CNN-based methods such as partial equivariance [1]. Since most equivariance models such as steerable CNN are based on G-CNN, we believe that our proposed method has lots of potential and extensions.  Steerable CNN, as a more generalized version of G-CNN, is very powerful, and our method shows comparable performance with Steerable CNNs. In our future work, we will apply our method to steerable CNNs. 

Weakness 2 and 3: Missing Parameter Counts, and Questionable Results in Table 3.

In the following table, we provide the comparison among different models in terms of the number of parameters. We agree with you that the performance with LoRA is a more fair comparison. We never claim to be the new state-of-the-art in our paper, and we have further clarify that in the paper to avoid misunderstanding.

Additionally, for the 13-layer-CNN on CIFAR10, we provide a comparison of the number of parameters and training time.

Weakness 4: Limited Generalizability to Other Transformations.

In section 4.5, we expand to 90/n-degree rotations and reflection, which is the same as G-CNN. However, G-CNN has managed to extend to Steerable CNN which is more versatile in terms of chosen groups. Although we are still limited to certain groups, we believe that our HE-CNN has the potential of extending in our future works through the study of group theory and parameter space symmetry. 

Weakness 5: Issues with Filter Generation from Rotated Inputs, and concerns on translation robustness.

We would like to point out that our HE-CNN imposes no restrictions on the generated filters for images with the same labels. For example, two different images of "7" may produce different sets of filters. If two images are exactly rotated versions of each other, their generated filters will also be related through the same rotation, guaranteed through the equi-combiner.

To address your concern on small translations, we provide an additional experiment. For our model trained with Rot-MNIST in section 5.2, we add random translation augmentation to the Rot-MNIST testing set and then test each trained model on it. (No translation augmentation is added during training.) We observe that your assumption is correct: $Z4$-CNN is more robust to translation, with performance dropping from 97.72% to 42.89%. Our method, dropping from 97.91% to 29.40%, exhibited a similar performance decline compared to the regular CNN structure. We thus conclude that our method is as robust as the regular CNN. As you mentioned, our motivation did not include translation invariance, and in future works, we aim to look deeper into this question and study translation equivariance in greater details.

[1] Romero, D. W., & Lohit, S. (2021). Learning partial equivariances from data. arXiv preprint arXiv:2110.10211.

[2] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184):1–25, 2019.

To address your concern on small translations, we provide an additional experiment. For our model trained with Rot-MNIST in section 5.2, we add random translation augmentation to the Rot-MNIST testing set and then test each trained model on it. (No translation augmentation is added during training.) We observe that your assumption is correct: CNN is more robust to translation, with performance dropping from 97.72% to 42.89%. Our method, dropping from 97.91% to 29.40%, exhibited a similar performance decline compared to the regular CNN structure. We thus conclude that our method is as robust as the regular CNN. As you mentioned, our motivation did not include translation invariance, and in future works, we aim to look deeper into this question and study translation equivariance in greater details.

I appreciate the extra experiment that confirms my thought.

The proposed method achieves rotation equivariance because the group has the semidirect product structure: $(R^2, +) \rtimes G$, where $G$ is Z4 or Dn in the context of this paper. The method uses the NEP generator and Equi-Combinator to take care of the transformations in group $G$, and the main network to take care of the translation in $R^2$. The decrease of accuracy for both G-CNN and HE-CNN is caused by the imperfect translation equivariance. Note that G-CNN is affected by this issue only once whereas twice for HE-CNN. One in the NEP generator and one in the main network. Thus, the same issue will arise even if HE-CNN is upgraded to incorporate steerable CNNs.

The paper argues that the HE-CNN can reduce constraints imposed on filters so that HE-CNN is more robust to little rotational variation than G-CNN and more versatile . I think, what is happening is that the constraints imposed on filters w.r.t. the group $G$ are transferred to the constraints imposed on filters w.r.t. $(R^2, +)$.

This method would be better off with CNNs having improved translation equivariance in the NEP generator. However, this means that the authors need to redo all experiments. Alternatively, the paper should show a trade-off between being versatile and being vulnerable to translation errors (with more parameters than baselines). Unfortunately, I don't see current experiments can show such a trade-off.

Thank you for your constructive comments. Below we have made responses to your comments. If you have any further comments, please feel free to let us know and we are more than glad to discuss with you.

Question 1: Is it correct that the method assumes that the orientation of the input is know beforehand? If not, how is it that the $a^i$ can be described relative to a canonical direction?

We apologize for the confusion and thank you for the opportunity to clarify. We do not assume the orientation of the inputs or the group action is known beforehand. What we meant at the end of section 4.6 is that, due to the fact that we take the average approach when generating parameters, equivariance in our method is satisfied if and only if the group action is applied onto all images in the batch. We would like to emphasize that this relaxation of the definition of equivariance is also true for G-CNN when batch normalization is applied within the layers.

To answer your question about $\{a^i\}$: For an input $x$, we collect all rotated versions of $x$. This entire collection is sent to the NEP generator, resulting in all $\{a^i\}$. Note that input $x$ can be in any orientation, not necessarily the canonical position. Our equi-combiner ensures that the final generated filters follow any rotation of the inputs, regardless of the initial position of $x$.

Weakness 1: The scalability of the method is really problematic.

1a. In terms of computation, this method basically uses a neural network to generate the weights of another neural network. Consequently, one could argue that this imposes important constraints on the size that the hypernetwork may have.

1b. The method requires passing N versions of the input through the network, becoming a clear constraint for large groups.

1c. The hypernetwork must generate weights for the whole of the other network, purposing scalability issues of the method.

Weakness 1a+1c: In the table below, we provide a comparison of the number of parameters and training time of the 13-layer CNN used on the CIFAR10 dataset. We also trained the Z4-CNN with a similar dynamic hypernetwork for comparison. In the revision, we will include a dedicated section discussing our current limitations in depth.

Weakness 1b. We agree that passing inputs $N$ times is currently a limiting factor, particularly when $N$ is large. The G-CNN also faces this limitation, required to process one input with N rotated filters. The Steerable CNN [r1] helps mitigate this limitation, and E(2)-Equivariant Steerable CNNs [r2] extends basic G-CNN to continuous settings. Therefore, we believe in the potential of HE-CNN to overcome this limitation in future work.

[r1] Cohen, T. S., & Welling, M. (2016). Steerable cnns. arXiv preprint arXiv:1612.08498. [r2] Weiler, M., & Cesa, G. (2019). General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32.

Weakness 1d. The authors mention that the method is equivariant only if all inputs in a single batch have the same orientation. This is very unrealistic and an extremely constraining limitation. Weakness 2: Similar to weakness 1d, from what I understand, the method assumes that it is known what g transformation is applied to the input.

Weakness 1d and 2: As mentioned in the response to Question 1, we do not assume that all inputs are presented in the same orientation, and we do not require information about the exact group action applied to the inputs.