Dear Reviewer Wppf,

We thank you very much for your review and for your time. We address your comments and questions below.

Experiments on larger datasets

We address this concern in our global response to all reviewers.

Accuracy versus computational cost

We agree that showing accuracy versus computational cost is necessary to make our main point much stronger. We have added a scatter plot quantifying this trade-off for the different pooling methods. We describe it in our global reponse to all reviewers. We thank you for the great suggestion.

Question on evaluation of robustness/completeness

Thank you for raising this important point. We appreciate your question, as it helped us realize that the purpose of Figure 6 was not clearly presented. We clarify it here, and also refer you to the additional experiments on robustness/completeness that we discuss in our global response to all reviewers.

Figure 6 is not meant to be a validation of the completeness of the G-Bispectrum, but rather an illustration. This is why, as you correctly noted, Figure 6 is qualitative and not quantitative.

Our point here is that Theorem 4.1 (completeness of the selective G-bispectrum on $C_n$) is known to be true since it is proved: all signals that have the same selective $G$-bispectrum are known to be identical up to group action. Hence, numerical experiments can only validate it. In consequence, Fig. 6 is not a statistical validation but an illustrative example to convince the reader. This is why we visualize examples and do not report statistics aggregated over a much larger dataset of random signals.

Besides, to provide a more comprehensive picture of the robustness of the selective $G$-bispectrum, we have added in the "Answer to all reviewer" and the PDF attached an experiment where, this time, input image are recovered.

We hope this re-framing of Figure 6 clarifies its purpose. We will integrate this wording into our revised manuscript.

Additionally, we provide novel experiments illustrating robustness and completeness in our global response to all reviewers.

Question on memory output: report the number of scalar coefficients

Yes, the G-bispectral (Bsp) coefficients are matrices in general, although they are scalars for commutative groups. We originally chose to report the number of G-bispectral coefficients to report a single value across groups, and also because the size of these matrices does not change the O complexity. Yet, you are absolutely right that it would be very useful (and potentially less misleading) to report the number of scalar coefficients as well. We do so below.

The number of scalar coefficients in the G-TC is $K * |G| * \frac{|G| + 1}{2}$. This formula applies to all the groups discussed on our paper: $C_n$, $D_n$ the octahedral group $O_h$ and the full octahdral group $FO_h$.

The number of scalar coefficients in the Max G-pool is $K$, also the same across the groups $C_n$, $D_n$, $O_h$ and $FO_h$.

The number of scalar coefficients in the selective G-Bsp varies depending on the group.

• $C_n$: $K * |G|$ scalar coefficients,
• $D_n$: $\approx K * 4|G|$ scalar coefficients,
• $O_h$: $K * 172$ scalar coefficients ($\approx K* 7 |G|$),
• $FO_h$: $K*334$ scalar coefficients ($\approx K *7 |G|$).

These numbers are computed by considering which the G-Bsp coefficients are used in the selective G-Bsp (which is given in our theorems and proofs), and by summing the scalar coefficients of their respective matrices.

Discussion on practical challenges in parallel implementation

Thank you for raising this point. Our main contribution is about computational gains, made possible thanks to new mathematical theorems on group theory. Thus, we agree that it is important to discuss other ways to obtain computational gains. Parallel computations is one of them. Following your suggestion, we discuss this point below. (Notation: Bsp = Bispectrum)

To further gain in computational efficiency, we could compute the G-Bsp coefficients in parallel. This can be done both for the full G-Bsp and for the selective G-Bsp.

Indeed, for both, we know which G-Bsp coefficients are needed: we need all of them for the full G-Bsp (or, half of them, given the symmetry rho_i, rho_j <> rho_j, rho_i), and a subset of them for the selective G-Bsp where the subset is given by our theorems. Consequently, each G-Bsp coefficient can be computed with a separate process. One would need to account for the fact that each G-Bsp coefficient has a different memory footprint. Indeed, each G-Bsp coefficient is a matrix with a size depending on the nature of the coefficient.

Since this parallelization would speed up both the full G-Bsp and the selective G-Bsp, the selective G-Bsp would still be faster. Lastly, since the matrices defining the G-Bsp coefficients are generally non-sparse, no further computational gain that can be obtained there.

Notations

Thank you for reporting the typos. We have corrected these.

Conclusion

Again, we thank you for the thorough review and the very interesting questions. Your feedback has been beneficial for us, and we believe that the additional experiments and discussions significantly strenghten our paper. If you have any remaining questions or comments, we would be happy to discuss further!

We hope that we have addressed your concerns and we look forward to discussing these topics further.

Dear Reviewer ZVp5,

We thank you for your time and thoughtful review. We address your comments and suggestions below.

Experiments on larger datasets

Thank you for raising this point. We comment on it in our global response to all reviewers.

Showing accuracy versus computational cost

We agree that showing accuracy versus computational cost would make our points much stronger. We have added a scatter plot quantifying this trade-off for the different pooling methods. See global response to all reviewers.

Showing "selectivity"

Indeed, we mention that the G-Bispectrum offers greater selectivity in our sentence: "the $G$-Bispectrum has been incorporated into deep neural network architectures as a computational primitive for $G$-invariance - akin to a pooling mechanism, but with greater selectivity and robustness."

In this sentence, we use the word "selectivity" to refer to the completeness property of the G-Bispectrum. The G-Bispectrum of a signal is complete in the sense that it removes only the variation due to the action of a group G on the signal, while preserving all information about the signal's structure: it selectively removes the group action without destroying the representation. We have provided empirical evidence to validate this claim in our results presented in Figure 6 and the new experiment from the PDF attached.

In contrast, we believe that you understood the word "selectivity" as it is commonly used in machine learning, where it refers to "specificity", the true negative rate (TNR) of a classification task. We apologize for our poor wording that led to this confusion.

To avoid misunderstanding, we propose to rephrase our original sentence as follows: "the $G$-Bispectrum has been incorporated into deep neural network architectures as a computational primitive for $G$-invariance - akin to a pooling mechanism, but with greater selectivity and robustness."

Since we do not claim greater "specificity" in the classification sense, we have not provided results on this front. However, we can show the confusion matrices of our classification results in supplementary materials. We hope that we have addressed this concern. Let us know if we should provide additional details here.

Showing robustness

Thank you very much for this suggestion. We have added an experiment in our global answer to all reviewers accordingly.

Conclusion

Again, we thank you for your time and attention. We hope that we have addressed your concerns and we look forward to discussing these topics further.

Dear Reviewer UtSC,

We thank you for your review and insightful comments and suggestions. We address your points below.

Continuous Groups

We thank you for asking the very interesting question of how to generalize to continuous groups.

The formulas defining the full and selective G-Bispectra (G-Bsp) only depend on the G-Fourier transform. Since the G-Fourier transform can be defined for continuous groups, so can the G-Bsp. However, practical difficulties arise upon implementation.

Indeed, the full G-Bsp has one G-bispectral coefficient per irreps of the group G. While discrete groups have a discrete number of irreps, continuous groups can have an infinite number of them. For example, SO(3) has an infinite number of irreps. Accordingly, because of memory constraints, we cannot compute the full G-Bsp for SO(3), nor for other continuous groups with an infinite number of irreps.

Interestingly, the selective G-Bsp that we propose does not require all of the G-bispectral coefficients. Thus, it does not require to use all the irreps of the group G. As a result, it is possible that we could compute the selective G-Bsp for continuous groups.

Let us, therefore, assume that the selective G-Bsponly requires a finite number of irreps for a given continuous group G. In this case, we face another obstacle. This obstacle appears when we want to use this G-Bsp as the pooling mechanism of a G-convolutional neural net - as done in our classification experiments. Indeed, the G-Convolution that preceeds the (selective) G-Bsp is a so-called regular G-convolution (Group Equivariant Convolutional Networks. Cohen & Welling, 2016), which uses a discretization of the group G. Thus, the output $\Theta$ of the G-convolution would be a signal over a discrete/discretized group. Consequently, we would not be able to use the continuous selective G-Bsp there.

To overcome this obstacle, one could think of using a steerable G-Convolution, instead of a regular one, since steerable G-convolutions apply to continuous groups (Steerable CNNs. Cohen & Welling, 2017). In our paper, we chose to use regular G-Convolutions because: (i) we are in fact interested in discrete groups and, (ii) a regular G-convolution outputs a signal $\Theta$ defined over a group - much needed to apply the definition of the G-Bispectrum - whereas a steerable G-Convolution outputs a more complicated mathematical object for which the G-Bispectrum needs to be generalized. We discuss this aspect below.

Consider a group $G$ that is the semi-direct product $G=\mathbb{Z}_2 \ltimes H$ of the group of translations $\mathbb{Z}_2$ and a group $H$ of transformations that fixes the origin $0 \in \mathbb{Z}_2$. Consider the output of the steerable G-Convolution, which is a map $\Theta: \mathbb{Z}_2 \rightarrow \mathbb{R}^K$, where $K$ is the number of filters in the convolution. The signal $\Theta$ has the following mathematical property: $\Theta$ is a $K$-dimensional field over the grid $\mathbb{Z}_2$ that transforms according to a specific representation $\pi$ of the group $G$. This representation is induced by a representation $\rho$ of its subgroup $H$ on $R^K$ (Steerable CNNs. Cohen & Welling, 2017), such that it is common to write $\pi=\mathrm{Ind}\_H^G \rho$. Consequently, the interaction between the signal $\Theta$ and the group $G$ is very different for the steerable case, compared to the regular case.

• For a signal $\Theta$, the action of an element $(t, r) \in G = \mathbb{Z}^2 \ltimes H$ is: $[\pi\_{t, r} \Theta] (x) = \rho(r)\Theta((t, r)^{-1} x) \in \mathbb{R}^K.$

• By contrast, when we use regular G-convolution, the action of an element $g \in G$ on each coordinate of the signal $\Theta$ is: $[g \ast \Theta] (g') = \Theta((g^{-1} g) \in \mathbb{R}.$

The interaction between the signal $\Theta$ and the group $G$ is central to the definition of a G-Bsp. In fact, there is currently no definition of a G-bispectrum for the general steerable case. However, there is a definition of G-Bsp for signals defined over the homogeneous space of a group $G$ by Kakarala, which corresponds to a special steerable case for which $\rho$ is the trivial representation: $\rho(r) = r$ for all $r \in H$.

Therefore, generalizing to continuous groups involves i) determining whether the selective G-Bsp only requires a finite number of coefficients, and for the classification experiments: ii) using a steerable G-convolution with trivial $\rho$ together with the G-Bsp for homogeneous spaces.

These ideas could provide fruitful paths for exploration in future work. We believe, nonetheless, that presenting the theory for discrete groups is an important first step and should remain the focus on our work. Relatedly, in geometric deep learning, the paper Group-Equivariant Convolution Networks (Cohen & Welling, 2016) introduced the G-CNN for discrete groups. Only later did the paper Steerable CNNs (Cohen & Welling 2017) present a theory that applies to continuous groups.

We will gladly add this discussion to the final version of the paper, thank you for this suggestion.

Larger datasets

Thank you for raising this point. We discuss it in the global answer to all reviewers.

Minor

• You are right to mention that numerical zeros are a recurrent issue in the field of numerical computations, especially when used for division. However, the G-Bsp relies on additions and multiplications. Hence, we did not encounter such problems in our experiments.
• Thank you for raising this point about the number of scalar coefficients. In the revised paper, we add a paragraph clarifying the number of bispectral matrices and the related number of scalar coefficients: see our answer to reviewer Wppf.

Specifically, the approximation you mention comes from the fact that $|D_n |=2n$. Hence, $16\frac{n-1}{2} \approx 4|D_n|$.

Conclusion

Again, we thank you for your time and attention. We hope that we have addressed your concerns. We look forward to further discussing these topics.

Dear reviewer P7yz, we wish to thank you again for your time. In what follows, we address the points you made in your review.

Issues with ill-defined notions:

The term "Selective G-Bispectrum"

You are right to mention that the "Selective G-bispectrum" admits degrees of freedom, that is, it is not uniquely defined and there are several "Selective G-bispectra". As you correctly pointed out, each choice of $\rho_1$ will yield a different "Selective G-bispectrum".

In the revised version of the paper, we propose to clarify this point as follows. For the most common groups appearing in our theorems, we call "Selective G-bispectrum" the bispectrum proposed in their respective proofs. Indeed, each theorem provides a precise set of G-bispectral coefficients. The Selective G-bispectrum will refer to that set. For other groups, we call Selective G-bispectra the G-bispectra obtained via our algorithmic procedure that yield the maximum number of G-Fourier coefficients.

We thank you for this important remark on the main term of our paper. We will make sure to clarify this in the final version of our work.

Computing the Clebsch-Gordan matrices

You are correct mentioning that obtaining the Clebsch-Gordan (CG) matrices is not an easy task. However, this work needs to be done only once before the training. This is because these matrices are not data-dependent and can be stored. In consequence, obtaining the CG matrices is not a bottleneck.

Additionally, we point out that - for the octahedral and the full octahedral groups - we only use the CG matrices in the proofs of our theorems. Specifically, we only use them to find out the minimal set of G-bispectral coefficients that is needed. In practice, when we implement the selective G-bispectra for these two groups, we do not even use the CG matrices. This is because the G-bispectral coefficients can be written in the following alternative form:

\beta(\Theta)\_{\rho_1,\rho_2} = \sum\_{g_1 \in G} \sum_\{g_2 \in G} \sum\_{g \in G} \Theta^*(g) \Theta\left(g g_1\right) \Theta(g g_2)[ \rho_1(g_1)^{\dagger} \otimes \rho_2(g_2)^{\dagger}].

This expression comes from the definition of the G-bispectrum as the G-Fourier transform of the G-triple correlation.

Since computational gain is an important part of our contribution, we feel that it is important that we discuss and clarify this important point in the final version of the paper. We thank you for this feedback.

Issues with motivations:

Completeness of avg pooling versus max pooling

We respectfully think there might me a confusion with the notion of completeness. You are right to notice that average pooling uses all data to be computed, however so does the max pooling since all data must be compared to retrieve the maximum. In consequence, the average of a list is not more complete than the maximum, one does not give more information about the distribution of a list than the other.

Distinction between the G-bispectrum and the G-Fourier transform

We are glad to precise the interest of the G-bispectrum w.r.t the Fourier transform (FT). The advantage of the G-bispectrum is its invariance to group action while the FT is not invariant: the FT is only equivariant. To achieve group-invariance of a neural network, the FT is thus not a interesting quantity, at least it does not provide specific improvement towards this goal.

Kakarala (1992) proved the invariance of the $G$-Triple Correlation (G-TC) for any compact group $G$, and by extension, the invariance of the $G$-bispectrum. This line of research also proves that the G-TC is the unique, lowest-degree polynomial invariant map that is also complete. Consequently, to achieve completeness, the simplest approach is the use of the G-TC or of its Fourier transform: the G-bispectrum. This comment raises the follow-up question: why using the G-Bispectrum over the G-TC? This question is linked to your question "why a spectral approach?", which we answer below.

Why a spectral approach?

This is a very interesting question. Our motivation for proposing, and using, the "Selective G-bispectrum" is motivated by the $O(n\log(n))$ complexity that it allows to obtain. By comparison, the complexity of the G-TC is $O(n^3)$ and we were not able to design a "selective G-TC" to reduce it. Consequently, the selective G-bispectrum is the unique, lowest degree polynomial invariant map that is also complete and that is the most computationally efficient. We will insist more on this point in the final version of the paper.

Conclusion

Again, we thank you for the attention given to our work and for your thorough review. We have seeked to address your concern and comments with this rebuttal. If the elements provided above enhance your appreciation for our work, may we kindly ask whether you would consider defending our paper and increasing your score?

Dear reviewers, please find below the results of an additionnal experiment.

New Results on CIFAR-10

Our results on CIFAR-10 present the same patterns as those on MNIST/EMNIST in the original manuscript. Here, we compare the performances of the G-Bispectrum layer with respect to the G-TC, the Max G-pooling and the Avg G-pooling models, trained on the SO(2)-CIFAR-10 datasets and we assess the accuracy by averaging the validation accuracy over 10 runs (see Table).

The architecture of the neural network is kept minimal and is the same across experiments. As for our other experiments, we choose a simple architecture to isolate the contribution of the pooling layers. Here, we use a neural network with 1 convolution, 1 G-convolution and 1 G-pooling where the G-pooling varies across Avg G-Pooling, G-Bsp Max G-pooling and G-TC. We use $C_8$ and $D_8$ for the discretization of the groups. We use 16 filters.

As for MNIST/EMNIST, the following patterns hold: at equivalent number of parameters, the more computationally expensive the pooling layer, the better the accuracy. The use of the G-TC becomes prohibitive when |G| increases.

We hope that this additional result are convincing and motivate you to increase your score.