Incorporating Arbitrary Matrix Group Equivariance into KANs

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Claims And Evidence

However, the key methodological innovation—incorporating equivariance via equivariant linear layers—is a direct adaptation of the approach used in EMLP (Finzi et al., 2021). The primary difference is that this method is now applied to KANs rather than MLPs. The authors claim novelty by applying equivariance to spline-based architectures, but they do not introduce a fundamentally new method for enforcing equivariance. This aspect is not sufficiently acknowledged in the paper.

Due to the complexity of B-spline function formulations, embedding strict equivariance into KANs is challenging. Hierarchically employing gating mechanisms is a relatively straightforward approach, although we acknowledge this does have expressive limitations. Our claim is that we aim to introduce equivariance into KANs to address their poor performance on symmetric tasks and non-symbolic representation tasks, while preserving their advantages over MLPs. Thus, as an improvement to KANs, we strive to retain their original hierarchical structure and B-spline basis function form. From another perspective, if major structural modifications were made to propose an entirely new architecture, how would it remain related to KANs? Additionally, EKAN is not merely a simple fusion of KANs and traditional equivariant mechanisms—the design of the EKAN layer's post-activation space structure is also challenging (see Sections 4.1 and 4.2 for details).

Experimental Designs Or Analyses

However, the paper does not explore settings where equivariance may not be useful or could degrade performance. It is noted in the EMLP paper that equivariant linear layers and gated nonlinearities are not universal, and there exist simple equivariant functions for some groups that cannot be approximated by EMLP. Since EKAN directly adopts the EMLP technique, it is likely that these limitations still exist and should be discussed in the paper.

Indeed, the use of gating mechanisms can reduce the expressive power of the network. However, due to the inherently complex structure of KANs (which involve intricate B-spline formulations), introducing symmetry into KANs is challenging. Therefore, for the sake of simplicity and clarity, we opted to incorporate gated non-linearities in a hierarchical manner. We anticipate that future improvements could enhance the expressive power and flexibility of EKAN. We will include a discussion of this limitation and potential future work in the paper. Thank you for your suggestion!

Also, for the top tagging experiment, the authors have not included the comparison with GNN-based methods, e.g. LorentzNet (Gong et al., 2022), which generally have higher accuracies.

Thank you for your addition! LorentzNet [1] achieves good results in top quark tagging, but it specifically focuses on this particular task. EKAN aims to provide a general framework for all symmetric scientific problems, so methods targeted at downstream tasks are not our primary benchmark for comparison. For example, CGENN [2] can incorporate more symmetries beyond the Lorentz group and outperforms LorentzNet when observations are complete (with 200 components). Therefore, in our paper, we chose to compare with CGENN. As we claimed, our EKAN outperforms CGENN in both accuracy and parameter efficiency when observations are incomplete (with only 3 components and a smaller dataset). Combining EKAN with task-specific models remains a direction for future research.

Supplementary Material

There is no instruction on how to run the experiments, and I did not review the code in detail.

The instructions for running the experiments are provided in the README.md document within the supplementary materials zip file. We will also make the code publicly available upon acceptance.

Relation To Broader Scientific Literature

Overall, the paper positions itself as an application of existing equivariant principles to a new network architecture rather than a significant methodological advance. While the application to KANs may be useful, the extent of novelty is limited.

Refer to the "Claims And Evidence" section in the Rebuttal.

Reference

[1] Gong, Shiqi, et al. "An efficient Lorentz equivariant graph neural network for jet tagging." Journal of High Energy Physics 2022.7 (2022): 1-22.

[2] Ruhe, David, Johannes Brandstetter, and Patrick Forré. "Clifford group equivariant neural networks." Advances in Neural Information Processing Systems 36 (2023): 62922-62990.

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Other Strengths And Weaknesses

Weaknesses

(1) Indeed, the use of gating mechanisms can reduce the expressive power of the network. However, due to the inherently complex structure of KANs (which involve intricate B-spline formulations), introducing symmetry into KANs is challenging. Therefore, for the sake of simplicity and clarity, we opted to incorporate gated non-linearities in a hierarchical manner. We anticipate that future improvements could enhance the expressive power and flexibility of EKAN. We will include a discussion of this limitation and potential future work in the paper.

(2) First, let's intuitively understand the dual ($\*$), direct sum ($\oplus$), and tensor product ($\otimes$) operations. Consider two vector spaces $X=R^2,Y=R^3$, and vectors $x=(x_1,x_2)\in X,y=(y_1,y_2,y_3)\in Y$. Then, all $x\oplus y=(x_1,x_2,y_1,y_2,y_3)$ form the space $X\oplus Y=R^5$, all $x\otimes y=(x_1y,x_2y)=(x_1y_1,x_1y_2,x_1y_3,x_2y_1,x_2y_2,x_2y_3)$ form the space $X\otimes Y=R^6$, and all the coefficients $vec(W)$ of the linear maps $Wx=y$ form the space $Y\otimes X^*=R^6$.

Then, if we define how the group transformation acts on $X,Y$, the form of its action on these composite spaces can naturally be derived. Eqn (4) provides the derivation rule. The paper assumes the group to be a matrix group. If a group element $g\in G$ acts on a vector $x\in X$ in the form of its corresponding linear transformation $gx$, then we call $X$ the base vector space of $G$ (as described in Lines 102-105, Column 2).

We have defined the "addition" and "multiplication" between spaces. Thus, for the base vector space $V$ of a group $G$, any complex spatial structure can be organized into the form of a "polynomial" with respect to $V$, which is the origin of Eqn (3). Note that Eqn (3) simultaneously defines $T(p,q)=V^p\otimes (V^*)^q$. We abbreviate $T(p,0)$ as $T_p$.

In the vast majority of scenarios, the feature space of a dataset takes simple forms such as vector stacking $cT_1=cV$ or matrices $T_2=V\otimes V$, while complex spaces like $T(p,q)$ rarely appear. However, the latent spaces between equivariant layers can be highly intricate (e.g., they may have 384 dimensions), and their decomposition forms with respect to the base vector space $V$ often involve $T(p,q)$ (the decomposition is automatically handled by the software based on dimensionality, as described in Lines 317-320, Column 2). The practical implication is that, according to the rules of Eqn (4), we define how group transformations operate on the latent space.

We will add these intuitive explanations to the appendix to avoid confusion. Thank you for your suggestion!

Other Comments Or Suggestions

(1) Thank you for pointing that out! We will note this in the paper to be more rigorous.

(2) We aim to express that $p_{i,a}$ and $q_{i,a}$ are not simultaneously zero. Indeed, in the case where both are natural numbers, $p_{i,a}^2+q_{i,a}^2>0$ and $p_{i,a}+q_{i,a}>0$ are equivalent, with the latter being more concise. We will make the corresponding revision. Thank you!

Questions For Authors

(1) Refer to point (1) in the Other Strengths And Weaknesses section of the Rebuttal.

(2) When the dimension of the latent space between equivariant layers is too small, its parameters may degenerate to all zeros. However, this issue rarely occurs when the latent space is more complex.

(3) Overfitting often occurs with small datasets, so we reduced the number of parameters in EMLP and trained it on particle scattering with a relatively small training set size. The experimental results are shown below, which indicate that it does not perform as well as models with larger parameter sizes. In fact, such equivariant architectures are less prone to overfitting compared to non-equivariant models because they strictly respect the symmetries in the data.

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Claims And Evidence

For all trained models $f_\theta$ in the paper, we use $L_{equi}=E_{x,g}\\|\rho_o(g)f_\theta(x)-f_\theta(\rho_i(g)x)\\|^2$ to evaluate their equivariant loss. The experimental results are presented as follows, which indicate that our EKAN and EMLP can structurally guarantee strict equivariance, whereas non-equivariant models cannot.

Particle scattering (for this experiment, the results of EMLP and MLP are sourced from the original paper, so we do not present their equivariance errors here):

3-body problem:

Top tagging:

Other Strengths And Weaknesses

As mentioned in the first paragraph of Section 5 (Lines 269-272, Column 2), EKAN is composed of stacked EKAN layers. The EKAN layer constructed in Section 4 has input space $U_{gi}$ and output space $U_{go}$ as shown in Equation (9). However, the feature space $U_i$ and label space $U_o$ of real-world datasets may not conform to this structure (specifically, they lack gate scalars). Therefore, as discussed in the second paragraph of Section 5 (Lines 295-299, Column 1), alignment is required. To align $U_{go}$ with $U_o$, we simply discard the gate scalars in $U_{go}$. For aligning $U_{gi}$ with $U_i$, we prepend a lift layer before the first EKAN layer to introduce gate scalars into $U_i$.

The lift layer is essentially an equivariant linear layer between $U_i$ and $U_{gi}$ (as described in Lines 300-302, Column 1). Its specific construction is detailed in Section 4.3, ensuring that it inherently preserves equivariance. Additionally, in the Claims And Evidence section of the Rebuttal, we also experimentally validate that the entire EKAN architecture is strictly equivariant.

Other Comments Or Suggestions

Figure 1 provides a general comparison between the EKAN and KAN architectures, which we elaborate on in the third paragraph of Section 1 (Lines 45-54, Column 2, and Lines 55-66, Column 1). Figure 2 illustrates the structure of an individual EKAN layer, which we explain in detail in the last paragraph of Section 4.1 (Lines 214-219, Column 1, and Lines 180-182, Column 2). Figure 3 presents the overall architecture of EKAN, which we describe in the final paragraph of Section 5 (Lines 304-318, Column 1). We will make it clearer to avoid confusion—thank you for pointing this out!

Thank you for your careful reading and valuable feedback! Below we will address each of your concerns point by point.

Claims And Evidence

Our claim is: We propose a method to incorporate symmetry into KANs. As mentioned in Section 1 (Lines 25-29, Column 2), KANs struggle to respect symmetry, which is one of the reasons for their underperformance in non-symbolic representation tasks. By introducing equivariant layers, we aim to address this limitation. Experimental results show that EKAN outperforms KANs on symmetry-related tasks while preserving KANs' advantages over MLPs, which supports our claim. Furthermore, even in scenarios where KANs are weaker than MLPs, EKAN still surpasses EMLP—an insight we expect to provide valuable contributions to this emerging topic of KANs.

Experimental Designs Or Analyses

(1) The model architecture of EMLP is exactly the same as in the original paper [1]. To control the number of parameters, we adjust its depth and width (i.e., shape), with relevant details provided in Section 6 of the main text (specifically the second paragraphs of Sections 6.1, 6.2, and 6.3). For fairness, EKAN and all baseline models follow identical training settings, which we elaborate on in Appendix E (Implementation Details). We will emphasize this in the paper to avoid confusion. Thank you for pointing it out!

(2) Overfitting often occurs with small datasets, so we reduced the number of parameters in EMLP and trained it on particle scattering with a relatively small training set size. The experimental results are shown below, which indicate that it does not perform as well as models with larger parameter sizes. In fact, such equivariant architectures are less prone to overfitting compared to non-equivariant models because they strictly respect the symmetries in the data.

(3) Our dataset for the three-body problem originates from LieGAN [2] rather than CGENN [3] (note that LieGAN is a method for symmetry discovery rather than the design of equivariant networks, so it is not our comparison target), so there are differences in dataset generation details and experimental setups. We supplement a comparison between CGENN and EKAN on the three-body problem, with the results shown below, which demonstrate EKAN's higher accuracy compared to CGENN.

Other Strengths And Weaknesses

The Background section primarily helps readers unfamiliar with symmetry theory to understand the relevant knowledge and avoid confusion. An important purpose of Eqn (4) is to inform readers that we decompose the space $U$ in the form of Eqn (3), whose practical significance is to define how group transformations act on $U$. This is crucial when constructing the latent space of the EKAN Layer (especially at the code implementation level), because the form of the group representation on the latent space depends on the definition provided by Eqn (4). More details can be found in point (2) of Other Strengths And Weaknesses section in the Rebuttal to Reviewer nHH7. In the revised version, we will move relatively less important concepts to the supplementary material. Thank you for your suggestion!

References

[1] Finzi et al. "A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups."

[2] Yang et al. "Generative adversarial symmetry discovery."

[3] Ruhe et al. "Clifford group equivariant neural networks."