Equivariant Polynomial Functional Networks

Q1: If you refer to the hyperbolic tangent as "tanh" I suggest you refer to "sine" as you "sin"

Q3: I think that in lines 194 and 222 in the right hand column "with" should be "which"

Q4: The sentence on line 118 in the right hand column seems a bit garbled

Answer Q1+Q3+Q4. We are grateful to the Reviewer for highlighting these points and will incorporate the appropriate edits into the manuscript.

Q2: Overall I like it that the notions used in the paper are relatively compact. However, the "Wb" "bW" "WW" ... here are separate, specific matrices.

Answer Q2. We are pleased that the Reviewer appreciated the choice of notation used in the paper. For example, the notation $[WW]^{(s,t)}$ refers to the product

$[W]^{(s)} \cdot \ldots \cdot [W]^{(t+1)}$ (line 178).

After applying a group element $g$, this term becomes

$[gWgW]^{(s,t)} = [gW]^{(s)} \cdot \ldots \cdot [gW]^{(t+1)}$.

We did consider using $[W]^{(s,t)}$ instead of $[WW]^{(s,t)}$, but ultimately chose the latter for consistency and clarity—particularly in cases like $[Wb]$, where a one-character notation would be ambiguous or insufficient.

Terms after group action, such as $[gWgW]$, $[gWgb]$, etc., can indeed be expressed more concisely as $[gWW]$, $[gWb]$, and so on. Given the notational complexity—though necessary—we will continue refining the notation to improve the overall clarity of the paper

W1: The main question is how much ... but the difference seems a little incremental.

Q5: The main question in my mind, as mentioned above, ... are to a larger number of linear layers.

Answer W1+Q5. The motivation for introducing polynomial terms into the layers is to address limitations caused by parameter sharing in the construction of equivariant functionals. Specifically, enforcing equivariance through parameter sharing often forces many parameters to be zero or identical, which can significantly limit the model’s expressiveness.

Theoretically, the representational capacity of the proposed MAGEP-NFN is greater than that of the Monomial-NFN layers introduced in [Tran et al., 2024]. Empirically, MAGEP-NFN also shows slightly improved performance over Monomial-NFN. However, we have not yet explored whether using a small number of higher-order polynomial layers is more effective than using a larger number of linear layers. From the perspective of representational capacity, this remains an interesting theoretical question, and we plan to investigate it in future work.

Q6: How many of these layers is it reasonable to use an actual functional net?

Answer Q6. We believe the appropriate number of layers depends on the specific task being considered. For example, in the CNN generalization prediction task, we found that two layers of $E(U)$ followed by one layer of $I(U)$ are sufficient. In contrast, for the INR classification task, we use three layers of $E(U)$ followed by one layer of $I(U)$. Across all experiments, we ensure a fair comparison by keeping the number of parameters comparable across all baselines. Further details can be found in Appendix E.

Q7: When you mention that the framework is applicable to CNNs as well, do you mean that neglecting the x-y dimensions in the image plane, the way that the CNN mixes channels is actually just like a fully connected network and you can apply the same formalism to it for this reason?

Answer Q7. For CNNs, the weight space described in Eq. (1) includes distinct $w_i$'s and $b_i$'s, in contrast to MLPs where all $w_i$'s and $b_i$'s are set to 1. To normalize these differences, we first applied a Monomial-NFN model from [Tran et al., 2024]—which shares the same structure as our proposed layers but without the polynomial terms—to rescale the values. This was followed by applying the MAGEP-NFN layers. This is mentioned in the implementation details (lines 2684–2690).

Q8: [Tran et al., 2024] also talks about sign-flipping symmetry. Does that have to be sacrificed in the polynomial case?

Answer Q8. Our method is applicable to sign-flipping symmetries. In fact, the layers computed in the paper are themselves sign-flipping equivariant or invariant. However, to ensure that the overall functional—composed of these layers—remains equivariant or invariant under sign-flipping, it is necessary to use odd activation functions. This requirement is consistent with the approach in [Tran et al., 2024].

---

We thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.

W1: The authors present intermediate claims on stability ... Functional Neural Networks thereof.

Answer W1. We kindly refer the Reviewer to our response to W1+W3 in Reviewer FUc7’s review.

W2: Also, the notation is particularly heavy, ... to clarify this.

Answer W2. We kindly refer the Reviewer to our response to Q1, part d in Reviewer HSsC’s review.

W3: The supplementary material contains ... such as ICML.

Q1: I would strongly suggest the authors to include proof sketches as a section of the main text.

W4: The presentation of the problem ... at least 15 times in the article.

Answer W3+W4+Q1. While the proofs are extensive, we believe they are essential to ensure the rigor and validity of our method. We thank the Reviewer for the helpful suggestion and will include a proof sketch in the revised version.

In broad terms, the derivation of the equivariant and invariant layers follows a two-fold structure:

• Computational Derivation: Starting from the general formulation provided in Equation (10), the specific layer structures in Equation (13) are derived through explicit computation, as detailed in Appendices C and D.

• Theoretical Justification: During this computational process, we address several theoretical challenges in Appendices B.3 and B.4, focusing on key properties like linear independence and basis completeness essential for capturing all equivariant and invariant formulations.

In the revision, we'll include a summary of these steps in the main text to guide readers through the logical structure, and direct them to the appendices for the complete technical details. We'll also refine the writing and references for clarity and readability. We refer to [3] multiple times throughout the paper, as the authors address the same core problem: constructing linear-based functionals with the same equivariance/invariance properties. In contrast, other related works often focus on different types of symmetries or employ alternative methods, such as graph-based approaches. Additionally, their system of notation aligns well with our approach, making it particularly suitable for introducing polynomial terms in our framework.

Q2: The authors do not confirm experimentally ... equivariant.

Answer Q2. Our equivariant layer is theoretically guaranteed to be equivariant by design, as it is derived through a deterministic and principled framework presented in the paper. Therefore, experimental verification of this property is not strictly necessary.

However, for readers who prefer empirical validation over detailed proofs, we provide a simple method to do so. For the layer $E$, we randomly sample multiple input weights $U$ and group actions $g$, and check whether the equivariance condition $E(g(U))=g(E(U))$ holds. This condition will be generally satisfied, though minor deviations may occur due to rounding errors in computation. A verification code is provided in https://sites.google.com/view/polynomialfunctional-rebuttal.

Q3: Page 3 first column, ... page 1 second column.

Answer Q3. We appreciate the Reviewer’s observation and will revise the relevant part accordingly to address the issue.

Q4: In my opinion, for readers ... neural networks themselves.

Answer Q4. In the literature on neural functional networks—also known as hypernetworks or metanetworks—it is important to use precise terminology when describing models that operate on data consisting of neural network weights. Since these models are themselves neural network architectures, referring to them explicitly as neural functional networks (or hypernetworks, metanetworks) helps prevent confusion between the model and its input. We believe it is essential to distinguish these concepts through clear and consistent naming. This practice is also reflected in prior works, including but not limited to: [1], [2], [3], [4], [5], [6], [7].

Q5: How does one go from ... a neural network?

Answer Q5. The implementation details of the functional modules used for various tasks are provided in Appendix E. In summary, to construct equivariant functionals, we stack multiple equivariant functional layers with ReLU activations in between. For invariant functionals used in regression or classification tasks, we append an invariant functional layer to the end of the equivariant stack, followed by a final MLP.

Q6: Notations for Theorem 3.3 ... composed $s$ times?

Answer Q6. $g^{(s)}$ is the $s$-th component in the group element $g$. The group and its action on weight spaces are defined in Section 2 (see lines 142-164).

---

Due to space constraints, some responses refer to similar points addressed in other reviews, with references at the end of our response to Reviewer FUc7.

We appreciate the Reviewer’s feedback and hope our clarifications are satisfactory. If so, we kindly ask you to consider raising the score. We’re happy to address any further concerns in the next discussion phase.

W1: The paper could benefit from a more detailed discussion of how their use of stable polynomials ... the broader scientific context.

W3: In summary, the paper makes a meaningful contribution to the literature on equivariant neural networks ... would enhance the discussion of its relation to broader scientific advances.

Answer W1+W3. We refer to the proposed stable polynomial terms as "stable" because they are inherently equivariant under the considered group action and therefore remain unchanged during the equivariance-enforcing process—specifically, the parameter-sharing mechanism discussed in the paper (lines 160-167). The term "stable" does not pertain to empirical notions such as stability, robustness, or generalization.

W2: While the authors mention the potential applicability of their parameter-sharing ... highlight distinctions or potential synergies.

Answer W2. Existing frameworks that involve parameter sharing can be mentioned as follows:

• [1]: NFNs for MLPs or CNNs with permutation equivariant.

• [2]: NFNs for Transformer with permutation equivariant.

• [3]: NFNs for MLPs or CNNs with monomial matrix group equivariant.

• [4]: NFNs for Transformer with equivariance under the maximal symmetry group of Multihead Attention mechanism.

W4: However, one area for improvement is the theoretical grounding ... strengthen the contribution and enhance its impact.

Answer W4. The rigorous computations in Appendices C and D establish the equivariance and invariance of the proposed functional layers. Additionally, Theorem 3.5 provides an expressivity result, stating that any invariant layer computed via the general formulation (10) must take the form given in Eq. (13). An analogous theorem for equivariant layers can be formulated similarly, given that the computational process for these layers follows the same structure.

It is worth noting that the proof of this theorem is non-trivial. It is derived through an analysis of linear independence, as detailed in Appendices B.3 and B.4, which we found to be mathematically challenging.

W5: The abstract is informative and covers the key contributions ... improve readability and make the key messages more impactful.

Answer W5. We sincerely appreciate the Reviewer’s feedback on the abstract and will make the appropriate edits to improve its clarity and accuracy.

Q1: In Theorem 3.5, you prove the G-invariance of the proposed equivariant polynomial layer. Could you clarify where the main technical challenges lie in establishing this invariance? Understanding the core difficulties in the proof would help appreciate the contribution more fully.

Answer Q1. While it is relatively straightforward to write down a specific formulation of a layer that is equivariant or invariant under the considered group action, it is significantly more challenging to characterize all possible formulations of such layers. The main technical difficulties arise from two aspects: first, the computations presented in Appendices C and D; and second, the theoretical results in Appendices B.3 and B.4 that underpin these computations, which are essential to ensure that no valid layer type is overlooked.

Q2: In Table 3, the performance of MAGEP-NFN appears ... (e.g., efficiency, scalability) that distinguish your approach in practice?

Answer Q2. We believe the task in Table 3 is comparatively less challenging than other benchmarks, which explains why Monomial-NFN already performs well and our model offers only a slight improvement. This is reflected in the high Kendall’s $\tau$ values, ranging from $0.913$ to $0.940$ across all models. The strengths of MAGEP-NFN become more evident on more challenging tasks. For instance, in the Classify INR-MNIST task, where the accuracy of other models widely ranges from $10.62\\%$ to $69.82\\%$, MAGEP-NFN achieves the accuracy of $77.55\\%$, surpassing the second-best model by a notable margin of $7.73\\%$.

---

We thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.

---

References.

[1] Allan Zhou et al., Permutation Equivariant Neural Functionals, NeurIPS 2023.

[2] Allan Zhou et al., Neural functional transformers, NeurIPS 2023.

[3] Hoang Tran et al., Monomial matrix group equivariant neural functional networks, NeurIPS 2024.

[4] Hoang Tran et al., Equivariant Neural Functional Networks for Transformers, ICLR 2025.

[5] Derek Lim et al., Graph Metanetworks for Processing Diverse Neural Architectures, ICLR 2024.

[6] Ioannis Kalogeropoulos et al., Scale Equivariant Graph Metanetworks, NeurIPS 2024 Oral.

[7] Miltiadis Kofinas et al., Graph Neural Networks for Learning Equivariant Representations of Neural Networks, ICLR 2024 Oral.

Q1. I think the main issue is that this paper is written as a follow-up ...

Answer Q1. We appreciate the suggestion from the Reviewer and will include explanations of relevant concepts from prior work in the revised Appendix. Below, we address each part of the question.

a), b) In the case of MLPs, all $w_i$'s and $b_i$'s values are equal to 1, whereas in CNNs, the $w_i$'s and $b_i$'s values correspond to the sizes of the convolutional kernels.

We present the weight space using arbitrary $w_i$'s and $b_i$'s values for an additional reason: within a functional layer, $w_i$'s and $b_i$'s also represent the number of hidden functional units. This generalization allows the framework to accommodate a broader range of architectures and internal structures.

For MLP's, the way the model is written explicitly is:

$f(\mathbf{x} ~ ; ~ U, \sigma) = W^{(L)} \cdot \sigma \left( \ldots \sigma \left(W^{(2)} \cdot \sigma \left (W^{(1)} \cdot \mathbf{x}+b^{(1)}\right ) +b^{(2)}\right) \ldots \right ) + b^{(L)}.$

c) The $\Phi$ and $\Psi$ are trainable parameters. We mentioned in the appendices that, in constructed functional layers, $\Phi_{-}$'s and $\Psi_{-}$'s are trainable parameters (line 1573 for equivariant layers and line 2372 for invariant layers).

d) We provide the following examples to clarify the rationale behind our choice of notation in the paper:

$[W], [b]$: Square brackets are used to distinguish whether a term belongs to the weight space or represents learnable weights in functional models.

$\Psi$: In the expression $[bW]^{(s)(L,t)} = [b]^{(s)} \cdot \Psi^{(s)(L,t)} \cdot [W]^{(L,t)}$, the index $(s)(L,t)$ for $\Psi$ is chosen to reflect that it connects two indexed components: $(s)$ and $(L,t)$. This way, simply by inspecting the indices of $\Psi$, one can infer the terms it links—namely, $[b]^{(s)}$ and $[W]^{(L,t)}$.

$\Phi$: In equivariant layers, the notation $\Phi_{(s,t):pq}^{(i):jk}$ represents a scalar used to compute the output component $[E(W)]^{(i):jk}$ (indicated by the top index), corresponding to the input component $[W]\_{(s,t):pq}$ (indicated by the bottom index). In invariant layers, the output does not lie in the weight space, and thus no top index is used—for example, $\Phi_{(s,t):pq}$.

While this indexing system might initially seem verbose, we find it brings clarity and consistency to the paper, particularly when describing the computation of functional layers. We appreciate the Reviewer’s question and will include a paragraph explaining the notation system more thoroughly.

e) The dimension $d'$ refers to the output dimension of the invariant layer. It can be chosen arbitrarily, and we use $d'$ to distinguish it from $d$, which denotes dimensions associated with the input.

Q2. Another major issue is ... or other simple models.

Q3. Is there some motivation for why choosing the \Psi ... they were chosen to be in this form?

Answer Q2+Q3. In our functionals, all the trainable parameters are $\Phi$'s and $\Psi$'s.

The main motivation for introducing the stable polynomial term with parameters $\Psi$ in our work—compared to the equivariant functionals in Tran et al. (2024)—is to address limitations arising from parameter sharing during the construction of equivariant functionals. Specifically, enforcing equivariance through parameter sharing often results in many parameters being forced to zero or constrained to be equal, thereby reducing expressiveness. In contrast, the parameters $\Psi$ in our stable polynomial term are inherently equivariant under the considered group action, and thus remain unaffected during the equivariance-enforcing process. As a result, our functionals gain greater representational capacity.

It is important to note that $\Psi$ cannot be placed arbitrarily within the polynomial expression, as doing so could break equivariance. Instead, we carefully position $\Psi$ between terms in a way that ensures the group actions on both sides cancel out, preserving the equivariance of the overall construction.

---

Methods And Evaluation Criteria. Is that previous work (e.g. Zhou et al. 2023) ... on MNIST and CIFAR-10.

Answer. We do not include experiments on 3D datasets such as ShapeNet or ScanNet because the dataset of pretrained weights used in Zhou et al. 2023, originally from De Luigi et al. 2023, has not been publicly released. Additionally, the implementation code for Zhou et al. 2023 on these datasets is also not publicly available. Given these unfortunate constraints, it is not feasible to include this baseline within the one-week rebuttal period.

---

We thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.