First of all, thank you for your careful review and comments! We deeply appreciate them and hope to address your main points below:

• Comparison with State-of-the-Art Methods: We appreciate the references to foundational work in neural operators and have made sure to cite them in the paper. While these methods demonstrate impressive capabilities in data-driven PDE solving, our work deliberately focuses on the physics-informed learning paradigm where solutions are learned without access to training data. This choice is motivated by several factors:

  1. The unique challenges of physics-informed learning, where models must learn to satisfy differential operators through residual minimization rather than direct supervision.
  2. The fundamental differences in problem formulation between operator learning (learning solution maps across PDE parameters) and our focus on high-accuracy solutions for specific PDEs.
  3. The opportunity to establish ActNet's capabilities in a well-defined setting before extending to hybrid frameworks.

  To ensure rigorous evaluation within the physics-informed framework, we have expanded our comparisons to include PirateNet, currently considered state-of-the-art for PINNs due to its sophisticated inductive biases and demonstrated performance across numerous benchmarks. Our results show that ActNet achieves superior performance on challenging test cases including the Advection equation and Kuramoto-Sivashinsky system. Notably, ActNet accomplishes this without employing PirateNet's advanced components like physics-informed initialization or adaptive residual connections.

  Regarding extension to neural operators, we are actively investigating principled approaches to incorporate ActNet into operator learning frameworks. Rather than simply replacing MLPs with ActNet blocks, we believe the key challenge lies in designing appropriate conditioning and modulation mechanisms that preserve ActNet's theoretical foundations from Kolmogorov theory while enabling parameter-dependent solution maps. This represents an exciting direction for future work that could potentially combine the strengths of both physics-informed and data-driven paradigms.

• Theoretical Contributions:

  • Relevance of Universal Approximation: While we agree that it is not necessarily surprising that ActNet has universality properties, this serves as a good “sanity check” when designing new neural network architectures. It is true that a network with universal approximation might not perform well, but a network without this guarantee is most likely doomed to failure. As such, our universality statement gives a basic theoretical guarantee, in addition to presenting alternative perspectives into the way in which a neural network can approximate multivariate functions.
  • Relevance of Theorem 3.4: This theorem tells us that the activations of an ActNet will not vanish or blow up to large values at initialization. It differs from LayerNorm in the sense that it guarantees that for a single neuron, activations will be roughly distributed as a standard normal across different inputs to the network, while LayerNorm enforces that for a single input, activations are normalized across different neurons in the same layer. In this sense, our approach is in fact closer to a BatchNorm layer. However, the stochastic nature of BatchNorm, where the normalization factor changes depending on unrelated inputs from the same batch, effectively prevents its integration with methods that hope to achieve very high accuracy for PDEs, as the added noise perturbs the inner layers of the network. Being able to compute this normalization in closed-form as we have done for ActNet, without depending on given inputs in a mini-batch, yields further stability, which allows for better training, as indicated in our experiments. We hope this helps clarify your question!
  • Distribution of Inputs: The assumption that inputs to a neural network are normalized is common in many deep learning papers, and in fact most practitioners recommend normalizing datasets so that each entry has mean 0 and variance 1, which is also what we have done in our experiments. Additionally, since we are composing several ActLayers sequentially in an ActNet and we know that outputs of an ActLayer are roughly normally distributed, this construction becomes useful not only for the initial ActLayer, but also for every subsequent one.

First of all, we thank you for your careful review and comments! We deeply appreciate them and hope to address your main points below:

• Comparison against PirateNet: That is an excellent point, thank you for bringing it up! The reason we didn’t originally include comparisons against PirateNet in the original manuscript is because the paper did not run these specific benchmarks. Having said that, you are absolutely right that PirateNet is another important competitive method to compare against, and we have since ran the advection and KS equations using a PirateNet and have confirmed that ActNet still generally outperforms PirateNet in these benchmarks. More specifically, ActNet significantly outperformed PirateNet both in the advection equation and the full time solution of the KS equation, but very slightly underperformed PirateNet (1.3e-5 vs 1.2e-5) when considering only the first time window, which is fairly smooth and easier to get good results on. ActNet particularly shines in the KS equation for later times, where the chaotic nature of the system is more apparent.

  Here we should also note that, additionally, PirateNet utilizes a physics-informed initialization and an adaptive residual connective structure that is also compatible with ActNet. Although we leave this integration as future work, we believe that using this strategy would further improve the performance of ActNet, making it even more competitive against PirateNet.

• Ablation on ActNet hyperparameters: You are correct that we have overlooked including detailed information about ablations with hyperparameter tuning in the manuscript. This is something we have done but did not report in the original manuscript. Here we take the opportunity to revise our Appendix and include the test error for all hyperparameter combinations considered for ablation experiments. In short, we find that the basis size $N$ does not need to be large for accurate results. In fact, except for small parameter counts, we don’t see accuracy gains for increasing the value of $N$ beyond 4 or so, while the computational complexity of the model increases for larger $N$ (even when network size is fixed). Additionally, as is often the case for deep learning, we find that composing more layers and increasing network width generally improves performance by increasing network capacity. We also have all over 7,000 runs logged on WeightsAndBiases, and plan on making them publicly available, including loss curves, if the paper is accepted (we currently cannot do this, as there is no way of anonymously sharing data using the WeightsAndBiases platform).

• Dense Theoretical Sections: While we did our best to present the mathematical details in a clear and understandable manner, we understand some derivations might be dense for some readers. However, we believe these mathematical insights are a key contribution of our paper, and the 10 page maximum limits our ability to further extend some explanations. Having said that, we plan to produce other material later to accompany this paper, including detailed blog posts and video explanations. While we regret there is not enough space for more detailed exposition in the paper, we hope this helps readers parse the material in the future.

• Comparison with State-of-the-Art Methods: As discussed above, we have conducted comprehensive comparisons against PirateNet, demonstrating ActNet's strong performance on challenging benchmarks. Our original comparisons utilized the JaxPi framework with all optimizations from Wang et al. (2023) "An Expert's Guide to Training Physics-Informed Neural Networks." ActNet's implementation incorporates several advanced techniques from this work, including causal learning strategies for all time-dependent PDEs (Allen-Cahn, advection, and Kuramoto–Sivashinsky equations). These methodological choices ensure fair comparison against current state-of-the-art approaches while highlighting ActNet's inherent architectural advantages. The fact that ActNet achieves superior performance even without some of PirateNet's sophisticated components (such as physics-informed initialization) further validates our theoretical framework.

• ActNet on Chaotic Systems: That is a great question! The Kuramoto-Sivashinsky (KS) experiment considered in section 4 is considered a chaotic problem for the choice of PDE parameters we used, and we observe performance above what can currently be found in the literature. We have not thoroughly ran comparisons using the Navier-Stokes equation, but believe ActNet can perform well under this setting as well. We will do our best to include an additional benchmark using the Navier-Stokes equation before the rebuttal period is over, but we are currently limited by implementation/computational time constraints.

First of all, we thank you for your careful review and comments! We deeply appreciate them and hope to address your main points below:

• Sinusoidal Basis: While it is true that for the 2D Poisson and Helmholtz problems used as ablation the sinusoidal forcing terms make Fourrier-like architectures better suited, the Allen-Cahn problem with Dirichlet boundary, as well the Kuramoto-Sivashinsky PDEs do not, displaying very sharp transitions in the ground truth solution for later times. These sharp transitions often present a challenge for Fourier-like methods, due to Gibbs phenomena, but are remarkably not an issue for the ActNet architecture, as can be seen in the solutions plotted in Figures 3, 4 and 12. Additionally, we also compare against Siren specifically to consider an MLP baseline that is well-suited for this type of problem. Overall, we find that ActNet generally outperforms other methods, both for sinusoidal-like problems and not.
• Comparison against MLP baselines: Initially, we only used Siren as a baseline for the MLP for fairness, since, as you mentioned, some of the forcing terms/initial conditions we used are sinusoidal, so we felt that Siren would present the most realistic/best performing baseline for MLP. Having said that, you are absolutely right that it is also worthwhile to include a non-sinusoidal MLP, so we have now ran tests using more conventional MLPs (with tanh, gelu and sigmoid activations) and included them in the paper. Overall, the performance trend resulted in roughly what we originally expected: ActNet > Siren > Traditional MLPs > KAN, although surprisingly we found that traditional MLPs sometimes overperformed Siren (see the updated table 2). Overall, we still see ActNet outperform alternative methods, especially in the more challenging problems with high frequency content. You can see these results in our experiments section, as well as in appendix G.
• Comparison against classical methods: While established numerical methods (e.g., FEM/FVM) excel at solving forward PDE problems, the deep learning approach offered by PINNs presents unique advantages that warrant continued research and development. First, PINNs are particularly well-suited for inverse problems and parameter identification tasks, where traditional methods often struggle. Second, once trained, PINNs can enable rapid inference across different parameter configurations without requiring additional solving - a capability especially valuable for real-time applications and parameter studies. Third, PINNs' meshless nature allows them to handle irregular geometries and adaptive resolution naturally. Our work with ActNet advances the PINN framework by introducing a more theoretically-grounded architecture that demonstrates improved accuracy and stability. While we acknowledge classical methods' current superiority for standard forward problems, we believe innovations like ActNet are crucial steps toward realizing the full potential of physics-informed deep learning for a broader class of scientific computing challenges.
• High dimensional inputs: The theoretical foundations of ActNet through the Kolmogorov Superposition Theorem (KST) extend naturally to arbitrary dimensions, potentially making our architecture well-suited for high-dimensional problems. In fact, our formulation based on Laczkovich's theorem scales linearly with input dimension ($\mathcal{O}(d)$), unlike the original KST's quadratic scaling ($\mathcal{O}(d^2)$) used in KANs. We strategically focused our initial investigation on low-dimensional domains (2-4D) for several reasons: (1) these dimensions encompass many important scientific computing applications including PDEs, neural fields, and physics simulations; (2) they allow rigorous empirical validation of our theoretical guarantees; and (3) they provide a clear benchmark against existing methods. The success of ActNet in this regime, particularly for challenging problems like the Kuramoto-Sivashinsky equation, suggests promising scalability to higher dimensions – a direction we are actively exploring for future work.
• Exact vs approximate representation: In the case of exact representation, inner functions are continuous, but may present an infinite set of discontinuities. By requiring only approximation as opposed to exact representation, however, it might be possible to use smoother inner functions. Of course, in the limit as we require the approximation to be infinitely accurate, we might still run into inner functions that are differentiable, but have very high modulus of continuity. However, we also conjecture that the added compositional structure of ActNet, where several layers are computed in sequence may further alleviate some of these theoretical constraints, allowing for efficient representation using smooth inner functions. We believe our experiment section provides some evidence towards this conjecture.

First of all, we thank you for your careful review and comments! We deeply appreciate your positive review and hope to address your main points below:

• While we agree that evaluating ActNet in data-driven problems and neural operator learning is an important research direction, we decided to focus this work on establishing ActNet's capabilities in the physics-informed setting first. We are actively investigating principled ways to extend ActNet to neural operator architectures by designing appropriate conditioning and modulation mechanisms that maintain the theoretical foundations from Kolmogorov theory. Rather than simply replacing MLPs with ActNet blocks in existing architectures, we believe a more systematic approach is needed to fully leverage ActNet's unique properties in the operator learning setting. This represents an exciting direction for future work that could potentially combine the strengths of both frameworks.
• During initial exploratory phases of our work, we also explored using polynomial basis functions in the implementation of ActNet. Overall, this approach also worked, but slightly less robustly than using the sinusoidal basis. For the sake of simplicity, we decided to stick to a single type of basis function, but as future work we may include other basis possibilities as well.
• We have added a section to the appendix detailing the results of some hyperparameter ablations. In short, we find that the basis size $N$ does not need to be large for accurate results. In fact, except for small parameter counts, we don’t see accuracy gains for increasing the value of $N$ beyond 4 or so, while the computational complexity of the model increases for larger $N$ (even when network size is fixed). Additionally, as is often the case for deep learning, we find that composing more layers and increasing network width generally improves performance by increasing network capacity.