Response to Reviewer puxG

The motivation is not so clear to me. The applications of group theory are everywhere in the real-world. Thus, how to choose a good representation for the group which is associated with the learning task is more important. In this paper, authors formulate this problem but do not give a clear answer.

In the $S_{10}$ experiment in section 5.1 we compare against using precomputed representations and show significantly worse performance compared to the learnable representations of MatrixNet. This ablation shows MatrixNet can automatically learn more useful group representations for a given task. Our results parallel other results in deep learning showing that learned feature representations often outperform expert-engineered features. We discuss our motivation below with your question.

I didn't see any insight behind the paper. For the experiment, I think it only solved a mathematical problem at simple setting.

The task used for our experiment on the Artin braid group in section 5.2 is an open mathematical research problem. This task is limited to $B_3$ since the multiplicity counts are not known for any larger braid groups. The experiment over $S_{10}$ in section 5.1 was chosen to provide a well-studied group with known representations to compare precomputed representations against the learned representations of MatrixNet.

Questions:

Q1: The motivation is not so clear to me. The motivation is to solve a mathematical problem or real-world learning task?

A1: Thank you for the feedback. We will make sure to make the motivation and application more clear in the paper. The second task used in our experiments over the Artin braid group is a current open mathematics research problem. Our goal is to create a model that can help mathematicians build intuition and formulate conjectures by 1) computing additional data points and 2) providing new insights through inspecting the learned representations. We believe this application is relevant to real mathematical research.

Q2: In line 166, can $\mathcal{f}$ be seen as a representation of \mathcal{G} or not?

In line 166 the function $\mathcal{f}$ does not constitute a representation of the group since the output space does not obey the group composition operation. Let $v_i = \mathcal{f}(g_i)$ be the vector target for a group element. Then if $g_k = g_i \circ g_j$ it is not the case that $v_k = v_i + v_j$.

Q3.1: Can you show more experiment results? Such as larger symmetric group?

A3.1: Yes, thank you for the suggestion. Using the same data generation scheme and data splits detailed for the $S_{10}$ experiment, we generated 800,000 samples for the group $S_{12}$. $S_{12}$ is roughly two orders of magnitude larger than $S_{10}$ (12! vs 10! elements). MatrixNet achieves an MSE of 3e-4 across all dataset splits with a classification accuracy of over 99%. The only increase to the network size necessary in this case was altering the rep size from 10 to 12.

Q3.2: Also I didn't understand the meaning of predict the order of the group element? I think we should determine it instead of estimating it?

A3.2: Sorry for confusing terminology. We will change “predict” to “determine.” Our model outputs an exact order, not an estimate. We used the term predict in a colloquial way to refer to the model output.

Response to Reviewer KZHf

Parts about the braid group are very difficult to follow and probably need a lot more background for readers and attendees for this conference.

Thank you for this feedback. We will make sure to revise this section to add more clarity. Intuitively the braid group defined in the paper represents all of the possible ways to braid a set of $N$ ropes. The braid group is closely connected to many fields of math but in particular we are interested in how it acts on mathematical categories through “categorical braid actions”. Mathematicians are interested in studying “filtrations” of the objects of categories but there isn’t a unique filtration for all objects within a category. The categorical braid actions however act on all such filtrations in the same way and so the multiplicity counts provide a canonical way to describe object filtrations regardless of the specific filtration. We include a more in-depth discussion of the braid group and categorical braid actions in section A of the appendix.

The experiments are a little weak in my opinion. It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task.

Our results parallel other results in deep learning showing that learned feature representations often outperform expert-engineered features. The precomputed representations we used are a natural choice for matrix representations for the groups used. For example, the group $S_{10}$ represents all of the ways to permute 10 objects and can intuitively be represented by 10x10 permutation matrices, which is the precomputed representation we use in the experiment. However, depending on the task, this may not be the most useful representation. For example, the sign representation of a permutation is simply $\rho(\sigma) = \pm1$ depending on the parity of $\sigma$. This would be a useful feature for determining a group element's order since every permutation of odd order has even parity. Our approach is designed to automatically learn a representation that is useful for the given task.

What happens when the size of the group representations is larger. I am not sure if the proposed architecture and learning mechanism can learn good representations when the size is large.

Our method can scale to large representations using MatrixNet-MC. MatrixNet-MC assumes large representations have a block diagonal structure, which is efficient since it means the number of trainable parameters grows asymptotically linearly instead of quadratically. For well chosen block sizes, this does not harm expressivity since for many groups, all representations will be block diagonal with respect to a good choice of basis with the maximum block size equal to the largest dimensional irreducible representation. Even for very large groups these irreducible representations are low dimensional which can be learned by MatrixNet-MC.

We omitted results with larger representation sizes as the performance did not change with matrix size. We include additional results for MatrixNet on the braid group task with doubled representation sizes in Table 2.

Response to Reviewer JC7A

Limited applicability to real tasks. The current approach is only applicable to finite groups.

Our method is not limited to finite groups. One of our experiments focuses on the infinite Artin braid group. You are correct that our approach is not formulated for continuous groups as it is limited to discrete groups.

Also, I think the tasks conducted in the experiments are not directly bridged to real problems, and I feel uncertain about how the proposed method is practically valuable.

The second task used in our experiments over the Artin-Braid group is a real current research problem in pure math. Mathematicians have only found a way to compute the answer for the simplest braid group $B_3$ – and they do not have a simple or intuitive formula. Our goal is to create a model that can help mathematicians build intuition and formulate conjectures by 1) computing additional data points and 2) providing new insights through inspecting the learned representations. We believe this application is relevant to real mathematical research.

It is unclear which component significantly contributed to the final performance gain: decomposition of g into generators or a trainable representation of a generator.

We do test the impact of a tranable representation independent of the decomposition in our first experiment in section 5.1 of the paper by comparing against the precomputed representation. This ablation replaces the learnable representations with the permutation representation of $S_{10}$ and we see significantly worse performance when compared to learnable representation.

We also show that just decomposing g into generators but not using a learned or fixed group representation does not result in improved performance. We compare against two sequential baselines, a transformer and LSTM, which both take the decomposition of g as input. These models however do not learn a group representation showing that just the decomposition of g into a generator sequence does not explain the improved performance of MatrixNet.

The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).

We disagree that only synthetic tasks are used. The task used in our Artin braid group experiment is an unsolved math problem from a recent pure mathematics publication [39]. Mathematicians have only found a way to compute the answer for the simplest braid group B_3 – and they do not have a simple or intuitive formula. Even so, we believe the two groups used, $S_{10}$ and $B_3$, are sufficiently large with $|S_{10}| = 10!$ and $B_3$ being an infinite group.

Questions: Q1: Given g, is its decomposition into the generators always uniquely determined?

A1: No, the decomposition of g is not unique. This is a great question and is a huge part of the motivation behind our approach. MatrixNet is designed to be invariant to the choice of decomposition.

Q2: In the experiment of 5.1, what will happen when we employ the group decomposition while using the fixed representation? I mean, the feature of g is given as concat[$\rho(g_1)$, ..., $\rho(g_n)$] where $g = g_1 \circ ... \circ g_n and \rho(g_i)$ is some representation of $g_i$ (e.g. irreducible rep).

A2: You could do this, but concatenating in this way means the feature size grows with the length of the decomposition. Since the decomposition of is not unique, another drawback is that this method would result in different features for two different but equivalent decompositions of g. The length of a decomposition is also unbounded. For example, in the braid group an element can be decomposed into arbitrarily many generators which would result in unbounded feature sizes. Our method by contrast is more efficient since feature sizes are fixed and with low relational error produces unique representations for a group element.

Typo: One of $M_{g_ik^-1}$ would be ${M_{g_ik}}^{-1}$ in the equation below line 203.

Good catch. We will fix it.

Response to Reviewer jQuD

It’s unclear why naive MatrixNet and MatrixNet-MC cannot extrapolate to longer word length.

While MatrixNet and MatrixNet-MC underperform compared to our other two variants, it is overstated to say they cannot extrapolate to longer word lengths. Despite their high MSE, both approaches maintain relatively high accuracy compared to our baselines. That said, MatrixNet-LN and MatrixNet-tanh have better extrapolation. The reason for this performance discrepancy is that MatrixNet and MatrixNet-MC have higher relational error indicating they do not learn group representations as accurately. This error compounds for longer words. To help make this difference clearer we computed the relational errors of the different models on the Artin Braid group, shown in Table 1, which we will add to the paper.

Empirical evaluations are a bit limited. It might be helpful to evaluate groups with different properties.

Thank you for the feedback. The task we used for the braid group was a primary motivation for our approach. We are limited to $B_3$ since multiplicity counts for larger braid groups are not known. We chose the symmetric group for our initial experiment in section 5.1 as it is a well-studied group that has the unique property that all finite groups are isomorphic to a subgroup of a symmetric group. It also is closely connected to the braid group making it well suited for ablation tests. We believe that these two groups provide a strong foundation for evaluating MatrixNet but we will perform further evaluations on an abelian group as well.

Questions:

Q1: In the matrix block, is it possible to introduce commutative operations between different matrix representations of group generators if the input group element comes from an abelian group?

A1: Yes this is possible. For an abelian group commutativity can be enforced in two ways in MatrixNet. One is with a loss term $L = |M_1 M_2 - M_2 M_1|$ and the other is by choosing the learned matrix representation to be diagonal, i.e. a direct sum of one dimensional representations. Since irreps for an abelian group are one dimensional, this would not harm expressivity of the learned representation and would enforce exact commutativity. We did not use diagonal matrices for the non-abelian groups specifically for this reason. More concretely, MatrixNet-MC with scalar channels is an architecture that is constrained to learn commutative representations.