Efficient Model-Agnostic Multi-Group Equivariant Networks

We thank the reviewer for their feedback. We aim to address the weaknesses here.

Reviewer: Weaknesses: I think the group-equivariant network that is designed for inputs having large product groups acting on them is claimed to be less expressive but more efficient than equitune (page 2). However the loss of expressivity is not discussed in section 3. I think this part is fundamental and should be stressed.

Response: Thanks for this comment. We have now added a discussion on the loss of expressivity in the newly added Sec. G in the appendix of the paper to provide an intuition describing where the trade-off between efficiency and performance comes from.

Reviewer: Questions: For the results in Table 1, why do you consider the input as ordered? What if instead we consider the input as a set of images, and apply Maron et al 2020? How would it compare?

Response: If we ensure that the fusion layers are identical and share across channels, then our framework directly extends for sets instead of sequences. But Maron et al. 2020 is not applicable here since they assume that the group elements acting on each image is the same. Thus, Maron et al. is not equivariant to the group actions considered here. Whereas, in Tab. 1, the group elements applied are chosen independently.

In the appendix of Maron et al. Sec. B, they also provide some results on applying independent group actions to each image/data point, but they assume the group actions are transitive. We make no such assumptions. In fact, the transitive assumption does not hold on the image classification tasks considered.

Reviewer: Related to 1, I think the multi-image task does not really test the first scenario, as the input groups are the same and therefore Maron et al 2020 can be directly applied.

Response: As discussed above, please note the framework of Maron et al. 2020 does not work in our problem setting.

We thank the reviewer for their feedback. We aim to address the weaknesses here.

Reviewer: The related work could be expanded to highlight the difference between this work and the existing ones. The novelty and contributions could have been highlighted.

Response: Thanks for the suggestion on emphasizing novelty and contributions, which makes it easier for the reader. As we have now added in Sec. 2, our work follows the very recent trend of works in equivariant deep learning that focuses on making pretrained models equivariant [1-4].

Since it a new area of research, there are very few works in this direction. Most of the works described in Sec. 2 focus on training from scratch and are therefore very different from our setup.

Compared to [1] that uses a simple averaging over the entire group to obtain symmetrization, we simply perform averaging over subgroups when the group can be decomposed as products. [2-4] use weighted averaging over group elements to obtain symmetrization, which are complementary to our work and can be used on top of our work for future work.

[1] Basu et al. "Equi-tuning: Group equivariant fine-tuning of pretrained models." AAAI 2023.

[2] Basu et al. "Equivariant few-shot learning from pretrained models." NeurIPS 2023

[3] Kim et al. "Learning probabilistic symmetrization for architecture agnostic equivariance." NeurIPS, 2023.

[4] Mondal et al. "Equivariant Adaptation of Large Pre-Trained Models." NeurIPS 2023.

Reviewer: More institutions and explanations of the theoretical findings could be provided after each theorem. It is not easy to figure out how important these theoretical results are and how they could be used to guide the practical designs. In addition to the comparison of computational complexity, it is unclear how and to what extent the proposed designs are practically useful.

Response: We have now added further description in Sec. 3.2 about the connections between Thm. 1 and Thm. 2. Thm. 1 proves the equivariance of the construction in (1) and Thm. 2 shows that the construction covers the entire linear equivariant space. Hence, (1) characterizes the entire space of linear equivariant networks.

Thm. 3 proves the universality of the IS fusion layer and Thm. 4 proves the equivariance of the general non-linear construction in (6), as described in Sec. 3.

The main goal of the work is to provide computational efficiency over [1] with little loss in performance, as described above. Further details on the trade-off between efficiency and performance is described in the newly added Sec. G in the appendix.

We clarify the weaknesses and questions raised by the reviewer. With these clarifications, we sincerely hope the reviewer will kindly reconsider reading through our paper. We strongly believe the reviewer will find it a compelling next step in designing efficient model agnostic equivariant networks, following the recent emergence of works in this area [1-4].

[1] Basu et al. "Equi-tuning: Group equivariant fine-tuning of pretrained models." AAAI 2023.

[2] Basu et al. "Equivariant few-shot learning from pretrained models." NeurIPS 2023

[3] Kim et al. "Learning probabilistic symmetrization for architecture agnostic equivariance." NeurIPS, 2023.

[4] Mondal et al. "Equivariant Adaptation of Large Pre-Trained Models." NeurIPS 2023.

Reviewer: "... the model is vacuous, and can only apply for the trivial symmetry."

Response: Please note that this is a misunderstanding since we design group equivariant models (not invariant-symmetric) for product groups G1xG2 for any two groups G1 and G2. In Thm. 2, we prove that our model design for the linear case covers the entire linear equivariant function space corresponding to product groups. Hence, clearly, our model does not only apply to trivial symmetries. Further, the competitive performance of our model design while respecting all the required equivariances also implies its meaningfulness. Please find more details on the nature of the invariant-symmetric fusion layers in the response below.

Reviewer: "In page 2, the notion of “invariant-symmetric” seems strange."

Response: Please note that we do not invent the notion of invariance-symmtery, we simply discover that it is present in linear equivariant layers of product groups and give it a name. This observation/discovery helps us design simpler networks going beyond the linear framework.

As we show in Thm. 2, it is naturally present in linear models of product groups. When a linear group equivariant model for the first design is constructed for multiple groups, this notion of invariance-symmetry naturally arises as a fusion layer between the different inputs. This is similar to different fusion layers with symmetries that arise in Deep Sets [5], and DSS [6]. Unlike Deep Sets and DSS, we do not stack up linear layers to construct our full model. Instead, we give a model-agnostic design, which goes beyond linear design and works even for pretrained models, leading to the methods discussed in Sec. 3.3, 3.4.

[5] Manzil et al. "Deep sets", NeurIPS 2017 [6] Maron et al. "On learning sets of symmetric elements", ICML 2020.

Reviewer: "Note that you simply define the identity action ... just the trivial symmetry."

Response:

• a) Please note that Pg. 2 only provides the def of invariance-symmetry (IS) that we discover in the fusion layer of the equivariant network corresponding to product groups. Thus IS only represents a subcomponent of our model design, it does not define the symmetry we are aiming for, which is group equivariance to product groups.

• b) Please note that the reviewer seems to have misunderstood the definition of IS and confused it with trivial symmetry. The two are different concepts as described below. Also, note that we make no assumptions on the IS function; instead, we provide functions f that precisely satisfy the IS constraint.

  • trivial symmetry (https://en.wikipedia.org/wiki/Symmetry_group) means the group action corresponds to a group with one single element that does not perform any transformation to the object of the action.
  • A function f: X -> Y has the property of IS with respect to ordered groups (G1, G2) where G1 acts on X and G2 acts on Y, if f(g1 x) = x (invariance) and g2 f(x) = f(x) (symmetry) for all x in X, g1 \in G1, g2 \in G2. From what we can gather, the reviewer may be confusing the notion of symmetry in our work f(x) = g2 f(x) to trivial symmetry group. Trivial symmetry group means a group with a single element which when acts on an object leaves the object unchanged. On the other hand, our notion of symmetric function simply means any function that remains unchanged with respect to group actions on the output of the function. This is very closely related to the notion of invariance (we call a function invariant when the output remains unchanged with respect to transformations on the input). On the other hand, we call a function symmetric when it remains unchanged with respect to a group transformation applied to the output. Consider the following example. Let f1: R^2-> R^2, f2: R^2->R^2 be functions defined as f1(x, y) = (1, 1), f2(x, y) = (1, 1) if (x, y) in {(0, 0), (0, 1), (-1, 0), (0, -1)}, else (0, 0). Here f1 is invariant whereas f2 is symmetric, with respect to 90 degree rotations of the input about the origin. Finally, we reemphasize we did not invent this notion of IS it naturally arises in equivariant network design for product groups, which we discover, name, and exploit for efficient model designs.

Thank you. Note that semidirect product is not symmetric. One subgroup is normal and the other is not in general. So, if you define a nested semidirect product you need to use parenthesis that indicate in which order you apply the semidirect products.

For example, does $G=G_1 \rtimes G_2 \rtimes G_2$ mean $(G_1 \rtimes G_2) \rtimes G_3$, where $G_1 \rtimes G_2$ is normal in $G$, and $G_1$ is normal in $G_1 \rtimes G_2$, or does it mean $G_1 \rtimes (G_2 \rtimes G_3)$, where $G_1$ is normal in $G$ and $G_2$ is normal in $G_2 \rtimes G_3$?

Thank you for you response.

Let me see if I can rephrase things more accurately so we can agree.

For $f$ to be IS, you assume that for every $y$ in the range of $f$, and every $g_2\in G_2$, you have $y=g_2f$.

This means that the range of $f$ is a subset of the following set of points in $Y$: the set $S$ of all $y\in Y$ such that $g_2y=y$ for every $g_2\in G_2$. So $G_2$ acts trivially on the range of $f$. Namely, the action of $G_2$ is the identity there, and the range of $f$ is called fixed under $G_2$. In $S$, the action of $G_2$ is the trivial action (this is what I called informally a trivial symmetry.).

Do we agree now on that?

I do see now that it is meaningful to assume that the rage of $f$ is fixed under the action. For example, under the reflection action on images (action of the group $\{-1,1\}$), the output of $f$ must be in the set of symmetric (or even) images.

Regarding replacing direct product with semidirect product, the image application indeed has a semidirect product group structure.

Regarding the other applications, I cannot localize in the paper the definitions of the semidirect product group and its action in these applications. Can you write these definitions explicitly so we see that these are indeed semidirect product groups?

I apologize for not reading in detail the whole paper the first time. The inconsistencies in group theory (the fact that the theory was for direct product groups but you used non-commuting groups in applications), and the confusion about triviality of the action led me to immediately reject the paper.

Now, I need to see the revised paper so I can re-evaluate it.

We thank the reviewer for their detailed feedback to improve the contributions of the paper. We aim to clarify the concerns raised by the reviewer.

Reviewer: The first one is on the motivation. It is unclear to me why we would want to make an existing trained model equivariant.

Response: Please note that making pretrained models equivariant is of crucial importance with the rise of large pretrained models. These large models are usually not trained with equivariances required for certain downstream tasks, but training them from scratch with group equivariance is extremely expensive and hence not often an option. Yet, using these large foundation models in downstream tasks shows excellent performance. Thus, the motivation of this work (focused on making pretrained model equivariant) is of great importance. Indeed, this research direction has gained recent importance as evident from several papers [1-4] in this direction. These are clearly mainstream papers on group equivariance published in respected conferences (NeurIPS and AAAI). Hence, we respectfully disagree that the considered task does not fit in the broader equivariant literature.

Further, we do not believe that the language task is contrived. The problem of addressing social biases in generated text from language models and the used framework [5] is widely studied. Further, addressing the issue of bias in natural language generation is not trivial, as shown in several works such as [6]. Taking inspiration from the formulation of [1] that shows that equivariance can provably debias natural language with respect to defined groups, we aim to extend these equivariance techniques to larger social groups much more efficiently to help safer deployments of language models. One can further note that intersectionality has been a pernicious problem in debiasing, and our mathematical framework is very natural therein.

[1] Basu et al. "Equi-tuning: Group equivariant fine-tuning of pretrained models." AAAI 2023.

[2] Basu et al. "Equivariant few-shot learning from pretrained models." NeurIPS 2023

[3] Kim et al. "Learning probabilistic symmetrization for architecture agnostic equivariance." NeurIPS, 2023.

[4] Mondal et al. "Equivariant Adaptation of Large Pre-Trained Models." NeurIPS 2023.

[5] Sheng et al. "The Woman Worked as a Babysitter: On Biases in Language Generation." EMNLP-IJCNLP 2019

[6] Steed et al. "Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models." ACL 2022

Reviewer: Large discrete product groups? In addition, to the lack of coherent motivation, I also found the claims in the paper to be unsupported. A large theme of this work is on building equivariance across large discrete product groups. In the multi-input experiments (Table 1) you have N=4 and the C_4 group. This is not a large product or large individual group. Similarly, in the compositional generalization experiment ... largest product group considered is of size eight". Again this is not a large product group. A similar criticism can be attributed to the intersectional fairness experiments. So I find the entire claim and motivation for doing this work lacking.

Response: Our main contribution is intended to be theoretical, where the experiments are given to validate our theoretical claims. Hence, we considered products of small groups rather than large discrete product groups, and show that the gap in computational complexity (in terms of memory consumption) is as comes from our theory. We are happy to rewrite the experiments section to indicate that the experiments are meant to validate the theoretical findings and that we did not work with very large product groups since that extra computational cost was unnecessary for theory validation.

Thank you again for the second part of your response. I remain unconvinced as before and I will maintain my rating. I encourage the authors to think about my initial rebuttal---especially the experimental design---more carefully. I think the authors would greatly benefit from departing and leaning on the line of work related to Equitune.