GRepsNet: A Simple Equivariant Network for Arbitrary Matrix Groups

We thank the reviewer for their detailed feedback. We answer all the concerns raised below.

Reviewer: Why did you limit yourself to orthogonal group experiments where e.g. Finzi et al. (2021) also consider other matrix groups?

Response: Please note that our model design is inspired from the EMLP model (Finzi et al. (2021)). Our formulation can handle general matrix groups just like EMLP. Hence, we closely followed many of the experiments of EMLPs to provide a fair comparison with EMLP. Please note that Finzi et al. also provides experiments only corresponding to orthogonal groups in Sec. 7 of their work. In Fig. 7 of their work, Finzi et al. only provides the construction of their convergence algorithm for non-orthogonal groups, but do not provide any experiments for performance evaluation corresponding to those constructions.

Reviewer: It is not quite clear ... I don't see how that is used in the image case. Similarly for the FNO comparison.

Response: For any data domain (e.g., images, graphs, pdes), we assume that the input to the network is a tensor of certain known type for some known group. For example, 5 (T_0 + T_1) for the O(5) used in some experiments in Sec. 5.1. Similarly, for image datasets in Sec. 5.2 and FNO datasets in Sec. 5.3, we assume that the inputs are first preprocessed to obtain data of type T_1 corresponding to the C4 group of 90-degree rotations.

This preprocessing can be performed in many ways, such as rotating parts of the inputs as we do for images or rotating entire images as done for FNOs as described in Sec. D.2. Note that any preprocessing that outputs T_1 representation would work for our experiments. Also note that for images, we first divide the image into 4 parts from the center and rotate each with different multiple of 90 to obtain a T1 representation of image, where this T1 representation has the same size as the original image. On the other hand, for FNO, we simply rotate the entire image to obtain the T1 representations. This is because dividing the network into smaller parts would affect the frequencies considered for Fourier transform in the FNO. In both these cases, we simply worked with T1 representation across the network.

For second order finetuning in Sec. 5.5, we first use T1 representation, followed by their conversion to T2 representation. We find that T2 representation when used with features obtained from pretrained models gives better classification performance when compared simply using T1 representation. The motivation here is similar to [1] where outer product in the space dimension helps provide better features. Instead, we take outer product in the group dimension resulting in a T2 representation that is equivariant to the same group C4.

[1] Lin et al. "Bilinear cnn models for fine-grained visual recognition." ICCV

Reviewer: What do the equivariant layers exactly look like. Could you write down what the conversion layers and linear layers comprise mathematically?

Response: We have now updated the details of our architecture in Sec. D.1 to describe how exactly we form the conversion and linear layers. In short, the key to the working of our architecture lies in the representation of the input, output and hidden layers. Input representation: In Sec. 5.1, 5.4 we are given the input in the form of tensor representations, whereas, in Sec. 5.2, 5.3, and 5.5 we convert the inputs (images or pdes) to appropriate tensor representations before feeding into the network.

Hidden layer and output representations: In each of the applications, the networks are designed based such that the hidden representations are of appropriate tensor types. All details are now provided in Sec. D.1.

Reviewer: How is equivariance achieved for the image experiments? Could you provide more details into your architectures, experimental setup, and so on?

Response: Please note that for images (as well as PDEs), we convert the input to T1 tensors corresponding to the regular representation of the C4 group (please check the details in Sec. D.2, D.3). For the image experiments with MLP-MIxers in Sec. 5.2, we keep T1 tensors for all the hidden representations, except for the output that is converted to T0 tensors. Whereas for the finetuning experiments in Sec. 5.5, we start with regular T1 representations, then convert them to T2 representations to obtain better features as explained above. We don't find much advantage with the use of T2 representations with MLP-Mixers features, hence, we only use them with CNNs.

Reviewer: Why do N tensors lead to N^2 invariants?

Response: Because in the First Fundamental Theorem of Invariant Theory for the Euclidean group O (d) [2], the invariants correspond to the inner products of input vectors. Hence, for N input vectors, we have N^2 inner products or invariants. We hope this clarifies the question.

[2] Weyl. The classical groups: their invariants and representations, 1946.

We thank the reviewer for their detailed comments and suggestions. We address the concerns raised below.

Reviewer: As I've mentioned, I see this work as a generalization of the vector neurons architecture. The generalization is, however, very straightforward and requires no technical work. Even the proof of equivariance carries over almost directly.

Response: We agree with the reviewer that this work can be seen as a generalization of the vector neuron architecture as mentioned in Sec. 4.2.

We agree that the generalization is simple in theory. But, it is a crucial generalization that makes GRepsNet work with any general data type and makes it competitive to more sophisticated architecture such as EMLPs. Further, we apply our architecture in a wide variety of domains achieving competitive performs with SOTA models across many of them, e.g. with GCNNs and E(2)-CNNs for image classification (now added in Tab. 7 and 8), etc.

Hence, we believe that the simplicity of our model makes it appealing for use in a wide range of domains.

Reviewer: Even though the architecture is quite general, the authors use ad-hoc methods for designing the architectures in the experiments ... The paper could be more clear in describing which sub-symmetries are being handled by their model.

Response: We have now updated Sec. 4.3 to give an overview of our design principle. The main design choices made are simply the input representations types, i.e. regular vs. non-regular group representations. We hope this gives a more unified view of the design choices made in our work.

We have now also added the groups of symmetries considered for each task in Sec. 4.3, e.g., group of 90-degree rotations for pde solving with FNOs, O(3) for N-body dynamics, just like in previous works G-FNOs and EGNNs, respectively. We have only considered these subsymmetries so as to have a fair comparison with the baselines, e.g. GCNNs use group convolutions only to add the symmetries corresponding to the 90-degree rotations (C4 group) on top of CNNs that are already translation equivariant. Hence, when comparing with GCNNs, we simply use the C4 group symmetry for GRepsCNNs on top of CNNs that are already translation equivariant.

Reviewer: The work shouldn't be compared to EMLP (Finzi et al.). As I've mentioned, comparing to vector neurons is much more appropriate. What EMLP does is that it computes all equivariant bases, which is much more challenging than providing some equivariant bases, as in this work. EMLP is a valid baseline of comparison in experiments, but the goal of that paper is different from this one.

Response: We agree with the reviewer. Our architecture is not primarily meant to be compared with EMLPs. Instead, our goal is to design a general purpose equivariant network inspired by EMLPs, but designed to be much simpler, faster, and competitive in performance. This is also justified by our experiments, where comparison with EMLPs is only one subsection. Whereas a majority of the experiments focus on larger datasets (that EMLPs cannot be used with due to its complexity), and leverage the benefits of equivariance across many domains, often performing competitively with the SOTA models. Since VNs are already a special case of our design, as described in Sec. 4.2, we do not compare them in experiments.

Reviewer: For the image experiments there should be a comparison with group convolutional networks (Cohen & Welling).

Response: We have now added baseline comparisons for image classification with GCNNs and E(2)CNNs. We find that not only is GRepsCNNs are competitive with both baselines, but when we use second-order tensors in these SOTA models, they outperform their original first-order counterparts. That it, when we introduce second-order features in GCNNs and E(2)CNNs, let's call them T_2-GCNNs and T_2-E(2)CNNs, respectively, then they outperform the original GCNNs and E(2)CNNs in image classification. This is similar to our finding with second order equituning in Sec. 5.5.

Reviewer: Are there non-linearities being applied to T_0 representations (scalars)? ...

Response: Yes, we do add non-linearities in the T_0 layer. We have now written this explicitly in Sec. 4.1.

Reviewer: Does the model provide universal approximation guarantees?

Response: Yes, we have now added a simple constructive universality proof of GRepsNet for the O(d), O(1, d) groups in Sec. G of the appendix.

We thank the reviewer for their detailed feedback. We address the concerns raised by the reviewer below.

Reviewer: "... it does not provide enough information to form a strong understanding of the proposed method ...", "The structure of the paper should be changed in order to improve its readability. The standard "Intro, Related Work, Theory, Experiments, Concluiosn" can serve as an example."

Response: We have added a "Related work" section after the "Intro" and moved the detailed discussion with EMLPs and Universal Scalars to the appendix. We believe this helps with better readability and provides a better comparison with various works in literature.

We have now added a paragraph in Sec. 4.3 to better explain how our general construction provided in Sec. 4.1 is used to design the architectures for various applications. Further, we have added more details on the exact networks designed for our experiments in Sec. D in the appendix. We hope these changes help clarify our method better.

Reviewer: It's not clear what type of neural network is being built. What is the main contribution of the paper? Is it proposing the use of different features, or is it introducing a new type of neural network? Is it a marginal improvement over EMLP? Is it the next interation of Villar et al? Is it an absolutely new approach for building equivariant models? If so, why don't you compare it against SOTA competitors?

Response: The main goal of our work is to provide a simple, general, and efficient algorithm to construct equivariant architectures: GRepsNet is easy to construct, works across a wide range of domains, and provide competitive performance to SOTA equivariant models.

It is not a marginal improvement over EMLP, rather, it provides a much simpler architecture for the same type of equivariance. It is related to Villar et al., but the emphasis is to design more practical networks than Villar et al. Note that Villar et al. mostly focus on construction of universal architectures, but only experiment with synthetic data since it is not trivial to apply it on larger datasets of more practical importance such as image datasets. Hence, we design equivariant networks that are computational efficient and hence of practical importance.

The key takeway from our work is the importance of feature representations in equivariant networks. We show that by simply using suitable feature representations, we can obtain competitive performance with SOTA models that use much complicated architectures and can be computationally expensive.

Moreover, the use of higher order tensors in equivariant finetuning (Tab. 5), GCNNs (now added Tab. 7) and E(2)-CNNs (now added Tab. 8) outperform their traditional first order representation based architectures. Hence, again, emphasizing the importance of group representation in constructing group equivariant networks.

Reviewer: The choice of datasets and models in the experiments is not clear. Why was MLP Mixer chosen instead of a group equivariant CNN?

Response: We have now added comparison of GRepsCNNs with GCNNs in Tab. 7 finding that GRepsCNN perform competitively with GCNNs. Moreover, we extend GCNNs, which use first order regular representations, to second order representations. By doing so, we outperform the original GCNNs of Cohen et al. in image classification task. This shows that using higher order tensor representations is a powerful tool in equivariant representation learning.

Reviewer: Can you compare your method to a group equivariant neural network with a similar number of parameters ... CNN from Weiler M., Cesa G. General e (2)-equivariant steerable cnns NeurIPS 2019

Response: We have now added comparison of GRepsCNNs with E(2)CNNs in Tab. 8 finding that GRepsCNN perform competitively with E(2)CNNs. Moreover, we extend E(2)CNNs, which use first order regular representations, to second order representations.

By doing so, we outperform the original E(2)CNNs of Weiler et al. in image classification task. This shows that using higher order tensor representations is a powerful tool in equivariant representation learning.

Reviewer: Page 4, "Process converted tensors", you write that "we simply pass it through a linear neural network with no point-wise non-linearities or bias terms to ensure that the output is equivariant". What is a linear NN here? Is it a simple matrix multiplication?

Response: By linear NN, we simply mean a sequence of matrix multiplications.

We thank the reviewer for their positive comments and suggestions.

Weakness: We agree that our framework is very general and can handle any tensors of higher orders. However, all of our experiments only required tensors up to order 2, hence, we restrict ourselves with second-ordered tensors. We leave applications to higher-ordered tensors for future work.

Question: We thank the reviewer for pointing us to the universality of our method. We now add a simple constructive proof showing that our model is indeed a universal approximator of equivariant vector functions and invariant scalar functions for O(d) and O(1, d) groups in Sec. G in the appendix.