Symmetry-Preserving Conformer Ensemble Networks for Molecular Representation Learning

We sincerely thank you for your detailed and constructive review. We appreciate your recognition of our well-written paper, our work’s ability to handle different GNN backbones, and our comprehensive evaluation.

Q1. Units of different quantities in Table 1

Thank you for raising this important clarification. The units for each property are:

• Drugs-7.5K: IP, EA, χ in eV (electron volts)
• Kraken: B5, L, BurB5, BurL in Å (Angstroms)
• CoV2, CoV2-3CL: ROC scores are dimensionless (0-1 scale)

We will add these units clearly to Table 1 and define all property abbreviations in the main text. We also note the formatting error you identified in Table 1 and Page 2 and will correct these inconsistencies.

Q2. Training loss increase around 100 epochs in Fig. 2b

The loss spike corresponds to the expert upcycling transition phase (Section 3.2.2). During upcycling, additional experts are activated and routing weights are reinitialized, causing temporary optimization instability. This is expected behavior that typically resolves within 20-30 epochs as the routing mechanism adapts to the expanded expert capacity. We will clarify this in the main text.

Q3. Computational cost comparison

We provide time comparison of our model with baselines as follows.

Training Time (5000 epochs using PaiNN backbone)

SPiCE requires 3.4× more GPU-hours for training (configuration: 8 experts, 2 active), primarily due to the mixture-of-experts forward passes, cross-conformer attention mechanisms, and 2D topology processing. However, inference costs remain highly similar to baseline methods, showing only minimal overhead (0.03s vs 0.02s per prediction on A100). We also note that training time can be shortened when using early stopping strategies. Despite the increased training cost, the model delivers consistent performance improvements across all tasks.

Q4. Early stopping strategy

Our conservative 400-epoch patience was chosen to ensure complete convergence across all configurations. Preliminary analysis suggests 200-250 epochs would suffice for most cases, potentially reducing computational cost by ~40%. We will explore adaptive early stopping strategies based on dataset size and model complexity in future work.

Q5. SPICE naming confusion

Thank you for flagging this potential confusion with the SPICE dataset from Eastman et al. (2023). We will rename our method to SPECE to avoid ambiguity.

Thank you for your detailed and constructive review. Please see our responses below.

Q1. Dimension clarification for $H^s$ vs $r^s$

• $H^s$: Type-0 features with shape $\mathbb{R}^{n \times |V| \times 1 \times d}$ (features for all nodes across conformers)
• $r^s$: Scalar routing scores with shape $\mathbb{R}^{N_I}$ per node ($N_I$ = number of invariant experts)

The seemingly mismatch occurs because $r^s$ operates per-node, while $H^s$ represents the full feature tensor. For node $i$, we have $h^s_i \in \mathbb{R}^{1 \times d} \rightarrow r^s_i \in \mathbb{R}^{N_I}$, and we collectively denote $H^s = [h^s_i]_{i=1}^{|V|}$.

We will include a comprehensive notation table in the camera-ready version.

Q2. Conformer dataset source

The conformers ${C_i}$ are supplied by the datasets, not generated by the authors. The paper closely follows the established protocols [30,51] for conformer preprocessing. For the specific datasets: Drugs-7.5K uses refined conformers from MARCEL [51], Kraken uses DFT-computed conformer ensembles [52], and CoV2 uses conformers from GEOM-Drugs [53]. We will make this clearer in the revision by explicitly stating the conformer generation protocols for each dataset.

Q3. R = SO(3) or O(3)?

The paper uses R to denote the rotation component of the geometric symmetry group G (as in lines 99-100). The choice of R depends on which geometric group is selected and which GNN backbone is used:

• When G = SE(3) (special Euclidean group), then R = SO(3) (rotations only)
• When G = E(3) (full Euclidean group), then R = O(3) (rotations + reflections)

Our model supports both options (lines 89-90) and we note that different GNN backbones correspond to different symmetry groups, e.g., ClofNet is SE(3)-equivariant (R=SO(3)). We will clearly state which geometric group and corresponding rotation group are used for each experiment.

Q4. Why $r^s$ is linear in h while $r^v$ is quadratic?

This design choice is driven by symmetry preservation requirements. From Eqs. (2-3):

• $r^s$ uses GCN aggregation: $r^\text{s}(h_i^s) = softmax(\sum_{j \in \mathcal{N}(i)} h_j^s W^s / \sqrt{d_id_j} + b)$, which is linear in $h^s$
• $r^v$ uses inner product: $r^\text{v}(h_i^v) = softmax((h_i^v)^T h_i^v W^v)$, which is quadratic in $h^v$

The quadratic form for vector features is necessary to maintain rotational invariance (Lemma 1, part 3). The operation $(h^v_i)^T h^v_i$ performs inner products over the equivariant dimension, ensuring the routing scores remain rotationally invariant while the linear form for scalar features naturally preserves invariance.

Q5. Meaning of $R^n$

From the context in line 196, $R^n$ refers to the product group of $n$ independent rotational transformations, where each conformer can undergo independent geometric transformations. This is clarified in line 91: "We allow individual transformations on each conformer individually and independently, hence the overall geometric transformation $G^n$." So $R^n$ represents $n$ copies of the rotation group $R$ (either SO(3) or O(3)), not the matrix group SO($n$) or O($n$). Each conformer has its own geometric transformation capability.

Q6. Line 180 dimension reduction: $H^s$ is $N\times d$, yet $r^s$ is $N_I$

$H^s$ represents node features across all conformers ($N = n\times |V|$ nodes total), while $r^s_{e,i}$ in Eq. (4) represents routing scores for the $e$-th expert and $i$-th node, with $N_I$ being the number of invariant experts. The dimensions $N$ and $N_I$ have very different meanings, and this switch occurs through the routing computation in Eqs. (2-4), where each node gets assigned scores for each of the $N_I$ invariant experts. We will clarify the dimensional flow in our revision.

Q7. Expert-routing bottleneck alleviation

The routing mechanism addressed three limitations of prior uniform ensemble methods:

1. Uniform processing: Traditional ensemble methods treat all conformers uniformly through simple pooling, but different properties may require different conformational features (Section 3.2.2) For instance, binding affinity depends on pocket conformations while solubility depends on surface accessibility.
2. Symmetry preservation: The type-separated routing maintains distinct symmetry properties. A unified routing would violate physical laws by mixing R-invariant (scalar) and R-equivariant (vector) features. Separate routers ensure scalar features remain invariant while vector features preserve equivariance under geometric transformations.
3. Computational efficiency: Sparse activation (top-k selection) enables fine-grained expert specialization without proportional computational cost increases.

Q8. $\bar{H}$ $S_n$-invariant vs $S_n$-equivariant

As in lines 223-224, $\bar{H}$ is defined as the mean-pooled representation: $\bar{H} = (1/n)\sum^n_{c=1} H^s_c$. Mean pooling across conformers breaks the equivariance to conformer permutations and instead produces an invariant representation. If conformers are permuted, the mean remains the same, making $\bar{H}$ $S_n$-invariant rather than $S_n$-equivariant.

Q9. Parameter counts and matching

We ensured fair comparison by matching backbone parameters across all methods. All backbone architectures used identical configurations (hidden dimensions, number of layers, cutoff distances, RBF functions, etc.), ensuring backbone parameters remain constant with only ensemble strategies varying.

Note: SPiCE values represent activated parameters (8 experts, 2 activated in GMoE)

Despite parameter increase, SPiCE demonstrates strong parameter efficiency through sparse activation where only a small number of experts are active during inference, significantly reducing computational cost. The consistent performance gains across all backbones and datasets justify the overhead.

We acknowledge in Appendix J that “the hierarchical architecture and dependence on equivariant GNN backbones introduce additional computational overhead compared to simpler pooling methods, potentially limiting scalability in resource-constrained environments or applications requiring real-time inference”. Note that several established techniques can address these scalability concerns in future work. For example, linear attention models like Performer [1] can reduce complexity from $O(n^2)$ to $O(n)$.

Additionally, expert pruning [2] could further reduce computational requirements while maintaining strong performance. We will include this detailed analysis in the camera-ready version.

[1] Rethinking Attention with Performers, ICLR 2021

[2] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models, ACL 2023

Q10. Performance drops for both too few and too many experts

The U-shaped curve reflects the tradeoff between expert specialization and training stability.

• Too few experts (2-4): Insufficient specialization capacity for diverse molecular patterns. Conformer ensembles contain varied geometric features (linear chains, cycles, branches, functional groups) requiring distinct computational pathways that cannot be captured with too few experts.
• Optimal range (8-16): Provides adequate specialization while ensuring sufficient training examples per expert.
• Too many experts (32+): Over-specialization leads to overfitting as each expert sees too few examples. The routing mechanism also becomes unstable with excessive choices, causing inconsistent expert selection.

Q11. Computational cost comparison

We provide time comparison of our model with baselines as follows.

Training Time (5000 epochs using PaiNN backbone)

SPiCE requires 3.4× more GPU-hours for training (configuration: 8 experts, 2 active), primarily due to the mixture-of-experts forward passes, cross-conformer attention mechanisms, and 2D topology processing. However, inference costs remain highly similar to baseline methods, showing only minimal overhead (0.03s vs 0.02s per prediction on A100). We also note that training time can be shortened when using early stopping strategies. Despite the increased training cost, the model delivers consistent performance improvements across all tasks.

Q12. $\tilde{r}_i$ dimensionality

The dimensionality $\hat{r}_i \in \mathbb{R}^{N_I + N_E}$ is correct as written, since $\hat{r}_i = (\hat{r}_i^\text{s}, \hat{r}_i^\text{v})$ denotes the concatenation of routing scores for both scalar experts ($N_I$) and vector experts ($N_E$), rather than stacking. Thanks to your inquiry, we will clearly state the concatenation operation in our updated manuscript.

Q13. Loss spike near epoch 100

Thank you for noting this. The loss spike corresponds to the expert upcycling transition phase (Section 3.2.2). During upcycling, additional experts are activated and routing weights are reinitialized, causing temporary optimization instability. This is expected behavior that typically resolves within 20-30 epochs as the routing mechanism adapts to the expanded expert capacity. We will clarify this in the main text.

Q14. Domain-specific shortcuts

Thank you for your valuable suggestion. We will add more comprehensive background information in the appendix, including Group Theory, Equivariance and Tensor Products, to benefit readers outside the field.

Q15. Font size

We will enlarge the labels and text in Figure 3 for improved readability in the revision.

---

Thank you again for taking the time to review our paper. Please let us know if you have any further questions or suggestions.

We thank you for your detailed and constructive review. Your recognition of our method’s novelty, comprehensive experimental validation, and interesting technical contributions is very encouraging. We address your specific concerns below:

Q1. Ambiguity in scalar and vector feature definitions

Thank you for raising this clarity concern. Our terminology follows established geometric deep learning conventions [1] where each tensor has three distinct axes: channel ($d$), tensor-type ($l$), and tensor-component ($m$). We recognize this notation will benefit from clearer explanation. For tensor type $l$, there are $(2l+1)$ tensor components:

• $h^s_i \in \mathbb{R}^{1 \times d}$: scalar (type-0, rotationally invariant) features with 1 tensor component and $d$ channels
• $h^v_i \in \mathbb{R}^{3 \times d}$: vector (type-1, rotationally equivariant) features with 3 tensor components and $d$ channels

The tensor-type axis ($l$) is handled by separating $h^s$ and $h^v$, which are then concatenated to form $\mathbf{X} = (\mathbf{X}^s, \mathbf{X}^v) \in \mathbb{R}^{n \times |\mathcal{V}| \times 4 \times d}$, where the 4 comes from concatenating tensor components (1+3=4). As for the definitions of $\mathbf{X}^s$ and $\mathbf{X}^v$:

• For the first interaction block ($l=1$):
  • $\mathbf{X}^s$ (Type-0) : Initial molecular features including embeddings of atomic numbers $z$, distance-based radial basis function $\phi_k(r)$, and topological descriptors $\mathcal{E}$
  • $\mathbf{X}^v$ (Type-1) : Initial geometric features including embeddings of relative position vectors $\boldsymbol{\rho}$ and directional bond features
• For subsequent interaction blocks ($l>1$), they include both the initial features above plus the processed outputs $\mathbf{X}^s$ and $\tilde{\mathbf{H}}^v$ from the previous interaction block.

We will add clearer explanations of this tensor notation early in the paper and provide concrete examples to aid reader comprehension.

[1] A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems, arXiv 2312.07511

Q2. Section 3.2.2 clarity and readability

Thank you for your feedback! We will improve readability by adding relevant subfigures from Figure 1(d) directly into the text, providing step-by-step walkthroughs of Equations (2)-(8) with intuitive explanations, and introducing the routing mechanism in a separate subsection of expert network design.

Q3. Attention mechanism clarity (Figure 1c)

Thank you for raising this question. We use only 3D type-0 features as queries because type-0 features are rotationally invariant. This ensures attention weights remain orientation-independent. Using type-1 features as queries would violate this invariance property since rotating the molecule would change attention scores, leading to inconsistent ensemble representations. In our model, type-1 features are processed separately through dedicated equivariant experts in the GMoE module to preserve their directional properties and geometric information. We will add this justification to Section 3.2.4 and revise Figure 1c with clearer annotations.

Q4. Redundant equivariance discussion

Thank you for this organizational suggestion. We will restructure Section 3 by creating a single theorem early that establishes all symmetry preservation properties, then reference it throughout rather than re-explaining concepts.

Q5. Missing limitations discussion

Thank you for raising this point. Due to space limitations, we included our limitations discussion in Appendix J. The key limitations include: “While SPiCE achieves strong performance across a variety of molecular property prediction tasks, the hierarchical architecture and dependence on equivariant GNN backbones introduce additional computational overhead compared to simpler pooling methods, potentially limiting scalability in resource-constrained environments or applications requiring real-time inference. Furthermore, although SPiCE facilitates cross-conformer interactions through attention-based integration, the framework does not explicitly incorporate thermodynamic priors or statistical mechanical principles that govern conformational ensembles.”

We will move limitations to the main paper in the final version.

---

Thank you again for taking the time to review our manuscript. Please let us know if you have any additional questions.

Dear Reviewer CzgV,

This is a reminder that the author-reviewer discussion period is ending soon on Aug. 6 (AOE), and you have not yet responded to authors' rebuttal. Please read authors' rebuttal as soon as possible, and engage in any necessary discussions, and consider if you would like to update your review and score. Please at least submit the Mandatory Acknowledgement as a sign that you have completed this task.

Thank you for your service in the review process.

AC

We sincerely thank you for your positive assessment and recognition of our work’s technical rigor, clarity, generalizability to future research, and improvements to molecular property prediction.

Q1. Dataset size impact on model performance

Thank you for raising this important consideration about data scale. We agree that evaluation on large-scale datasets will strengthen our claims. However, we would like to note that many molecular property prediction tasks are of similar or smaller scales due to experimental costs, making our evaluation practically meaningful.

Additionally, our evaluation demonstrates consistent improvements across diverse dataset sizes (1.5K to 75K molecules), and we studied the scaling of training data to model performance. As shown in Section 4.3 and Figure 2(a), SPiCE exhibits approximately linear scaling from 7.5K to 75K molecules, reducing MAE from 0.4929 to 0.4386. This suggests that SPiCE maintains stable learning characteristics regardless of data scale.

Q2. SPICE naming confusion

Thank you for flagging this potential confusion with the SPICE dataset from Eastman et al. (2023). We will rename our method to SPECE to avoid ambiguity.

Dear Reviewer LtiX,

This is a reminder that the author-reviewer discussion period is ending soon on Aug. 6 (AOE), and you have not yet responded to authors' rebuttal. Please read authors' rebuttal as soon as possible, and engage in any necessary discussions, and consider if you would like to update your review and score. Please at least submit the Mandatory Acknowledgement as a sign that you have completed this task.

Thank you for your service in the review process.

AC