We would like to thank the reviewer for their careful reading and constructive suggestions. We have conducted additional experiments to showcase KANs’ ability to perform more complicated tasks and compare them with stronger baselines (Appendix A, B, C), and study hyperparameter dependence of KANs & MLPs (Appendix D). We respond to each point below.

Q1:The paper contains very strong claims, but the evidence isn’t enough to support the strong claims. As the saying goes, extraordinary claims require extraordinary evidence, which unfortunately is missing here. The authors should either weaken their claims or provide strong empirical evidence.

A1: We totally agree! We attempt to work on both directions here: (1) We added more empirical evidence to demonstrate KAN’s ability to tackle more complex problems (including image fitting, complex PDE solving, MNIST classification in appendices A, B, C); (2) we have toned down our contributions and highlighted the limitations. Specifically regarding the neural scaling law, we have added a caveat “We also remark that: since the assumption in the theorem is a strong one, the neural scaling law should not be expected to be universally applicable to all machine learning applications.”. We also acknowledge the slow training problem in the abstract and the PDE section.

Q2: The name “Kolmogorov-Arnold network” overstates the difference to MLPs.

A2: Thanks for bringing up the concern. We have downplayed our novelty claim in conclusions by acknowledging the possibility that MLPs and KANs might be equivalent in some aspects but different in others, which will need more formal study in the future. We have added this paragraph in the conclusion part: “We also acknowledge that the similarities and differences between MLPs and KANs require more study, both theoretically and empirically. For example, a reasonable criticism of KANs is that they can be rewritten as MLPs or the other way around since the notion of "edge" vs "node" is somewhat dual. Future work should aim to better clarify similarities and differences from the perspective of optimization, generalization, etc. For example, a recent preprint Wang et al. (2024b) shows that although KANs and MLPs are both universal approximators, KANs have fewer spectral biases than MLPs.”

Q3: The scale of the problems is relatively small. The paper lacks evaluation on larger, more complex datasets. The reviewer suggests adding the MNIST experiment.

A3: Thanks for the great suggestion! We have now added MNIST experiments in Appendix C. KANs and MLPs have comparable performance in terms of loss and accuracy. This is probably because the MNIST dataset is too simple. KANs consume much more training time than MLPs. We expect these slowdown factors to have straightforward solutions, and combining KANs with Convolutional neural networks is a promising direction to incorporate symmetry inductive biases into architectures. Another interesting observation: the shape [784,100,10] (1 hidden layer of size 100) is optimal both for MLP and for KAN, which seems to imply there is something universal across different architectures. This might be a good starting point to formally study Q2 in the future.

Besides the MNIST dataset, we also added image-fitting experiments (Appendix A) and complex PDE experiments (Appendix B), demonstrating KANs’ applicability to handle more complex datasets.

We would like to thank the reviewer for their careful reading and constructive suggestions. We have conducted additional experiments to showcase KANs’ ability to perform more complicated tasks and compare them with stronger baselines (Appendix A, B, C), and study hyperparameter dependence of KANs & MLPs (Appendix D). The new experiments use standard training protocols to avoid the possibility of favoring KANs. We have also downplayed our claims and acknowledge the limitations of our method. We respond to each point below.

Q1: comparison to state-of-the-art methods.

A1: In our added image fitting task (Appendix A), we have used stronger baselines: including SIREN (MLP with sine activation functions), and MLP with random Feature features. We show that KANs can outperform all baselines in terms of PSNR. However, we also want to mention that KANs have 10x - 20x higher training wall time. Our chosen images include “Camera Man” (a natural image), “Turbulence” (from PDEBench), and “The Starry Night” (Van Gohn’s painting), to show KANs’ consistently high performance across different styles. In our new PDE-solving tasks (Appendix B), we also use the MLP with random Fourier features as a stronger baseline, and KANs appear to obtain comparable accuracy with baselines.

Q2: Overstated claims and conclusions

A2: On the one hand, we added more experiments in Appendix A - D (please refer to the global rebuttal for more details), to stress test KANs’ ability to perform more challenging tasks, as well as conduct fairer comparisons between KANs and baselines. On the other hand, we have downplayed our claims and acknowledged our limitations. We acknowledge the slow training problem in the abstract and the PDE section.

Q3: interpretability claims and symbolic regression protocol

A3: Thanks for the great question! We have deleted the discussion about symbolic regression in the Appendix but added a fairer discussion at the end of Section 3. As a network-based method, KANs’ strong capability (in fitting even non-symbolic functions) makes it unfavorable for standard symbolic regression benchmarks. For example, KAN ranks second-to-last in GEOBENCH (https://openreview.net/forum?id=TqzNI4v9DT), whereas the last-ranked one EQL is also a network-based model although EQL has been shown to be useful in many scientific applications despite its inability to recover exact symbolic formulas. On the one hand, we would like to explore ways to restrict KANs' hypothesis space so that KANs can achieve good performance on standard symbolic regression benchmarks. On the other hand, we want to point out that KANs have good features that are not reflected by existing benchmarks: (1) interactivity. It is very hard to ``debug'' evolutionary-based symbolic regression methods since the evolution process is long and random. However, it is relatively easier to visualize the training dynamics of KANs, which gives human users intuition on what could go wrong. (2) The ability to “discover” new functions. Since most symbolic regression methods require the input of the symbolic library, they cannot discover things they are not given. For example, if the ground truth formula contains a special function but is not given in the symbolic library, all SR methods will fail definitely. However, KANs are able to discover the need for a new function whose numerical behavior suggests maybe it is a Bessel function; see Figure 23 (d) for an example. In fact, Figure 4 (e) shows an example where we are very close to discovering a new equation, but after our mathematician collaborator dug into the literature for a week, they realized that this is a known fact in the literature. In the future, we hope to use KANs to make truly novel discoveries in science.

Q4: Unconventional training protocols for training MLPs, e..g, widths are too small, using LBFGS instead of Adam.

A4: We agree that it is a standard protocol to use Adam to train reasonably wide MLPs. In our new experiments (Appendix A-C) for more challenging tasks, we use the Adam optimizer for training and choose MLP to have a width of 128. We find that KANs are comparable to or outperform baselines (for image fitting in Appendix A) in all tasks in terms of accuracy. However, we also show that KANs are significantly slower to train by reporting their wall time. In Appendix D, we also conduct to grid scan of hyperparameters to show the effect of different hyperparameter choices. We observe that KANs consistently achieve lower training losses than MLPs, although KANs might be overfitting when the training data size is small or they become too large (large depth or grid size). We are aware of the papers that conduct fair comparisons between MLPs and KANs. They are not contradictory to our paper, but indeed bring up KANs’ limitations: (1) slow training; (2) non-conventional protocols (LBFGS optimizer, grid update, grid extension) to achieve the best results; (3) applicability to other domains beyond function fitting and PDE solving.

Q5: Comparison against plain MLPs in PINNs.

A5: We have included stronger baselines (MLPs with random Fourier features) in Appendix B, for solving the Poisson equation, Allen-Cahn equation, and the Darcy equation. KANs are comparable to the stronger baseline in terms of accuracy but have longer training. We understand that solving PDEs with neural networks is a big and well-developed field itself, so we do not expect to achieve SOTA results in this paper. Instead, we have a paragraph at the end of Section 4 to acknowledge the limitations of KANs. Said that, we do feel combining KANs with advanced techniques in PINN will give rise to fruitful directions that will we would like to explore in the future. For example, we have attempted to combine KANs with causal training in the Allen-Cahn example (Appendix B.2), and it will be interesting to see how the gating mechanism in PirateNets can be incorporated into KANs as well.

Q6: Baselines for symbolic regression.

A6: Please refer to A3.

Q7: Baselines for function regression

A7: In Appendix A-C, we now included MLPs with random Fourier features as a stronger baseline than MLPs. For image fitting specifically, we include SIREN as an even stronger baseline. In Appendix D, we conduct to grid scan of hyperparameters to show the effect of different hyperparameter choices (including optimization method, learning rate, width, and depth). We observe that KANs consistently achieve lower training losses than MLPs, although KANs might be overfitting when the training data size is small or KANs become too large (large depth or grid size).

Q8: Baseline for PDE solving.

A8: We included the random Fourier features (inspired by the PirateNet) to obtain a stronger baseline (MLP_RFF, i.e., MLP with random Fourier features). We find that KANs are comparable to MLP_RFF and MLP in the examples we tried in Appendix B, when KANs are trained using conventional protocols. As we wrote in A5, we understand that solving PDEs with neural networks is a big and well-developed field itself, so we do not expect to achieve SOTA results in this paper. Instead, we have a paragraph at the end of Section 4 to acknowledge the limitations of KANs. Said that we do feel combining KANs with advanced techniques in PINN will give rise to fruitful directions that we would like to explore in the future. For example, we have attempted to combine KANs with causal training in the Allen-Cahn example (Appendix B.2), and it will be interesting to see how the gating mechanism in PirateNets can be incorporated into KANs as well.

Q9: Continual learning.

A9: We want to provide updated results about deeper KANs in this continual learning example. We find that the continual learning feature is partially lost for a [1,5,5,1] KAN, especially when the grid size gets smaller. This is somewhat expected because the locality gets lost when the grid size becomes smaller or when KANs get deeper. We agree that this “continual learning” feature is not specific to KANs, but specific to the locality of activation functions. To add a caveat in Appendix Q “We want to mention that this toy example is somewhat trivial and is attributed to local activation functions rather than the KAN architecture. We simply feel this is a cute example to share in case anyone is inspired by it. However, this should not be interpreted as solving the continual learning problem. Indeed, when we try a deeper KAN [1,5,5,1], the continual learning feature is partially lost, depending on grid sizes.”

As a general comment, I would like to give a brief response to the following comment by the reviewer:

While I believe that this paper puts forth a creative idea that is worthwhile publishing, the current presentation makes it very difficult to assess whether KANs are indeed a good and competitive method to employ for all the different tasks presented in this work: symbolic regression, function regression, PDE solving, etc.

Please see the following manuscripts and journal papers. They show the applications of KAN for complex geometries and PDEs (e.g., the full Navier-Stokes equations):

1. A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov–Arnold Networks

--> https://www.sciencedirect.com/science/article/pii/S0045782524007722

2. A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks

--> https://www.sciencedirect.com/science/article/pii/S0045782524005462

3. Kolmogorov-Arnold PointNet: Deep learning for prediction of fluid fields on irregular geometries

--> https://arxiv.org/abs/2408.02950

I hope that it helps.

We would like to thank the reviewer for their careful reading and constructive suggestions. We have conducted additional experiments to showcase KANs’ ability to perform more complicated tasks and compare them with stronger baselines (Appendix A, B, C), and study hyperparameter dependence of KANs & MLPs (Appendix D). The new experiments use standard training protocols to avoid the possibility of favoring KANs. We have also downplayed our claims and acknowledged the limitations of our method. We respond to each point below.

Q1: There are three major problems regarding experiments: (a) Regression tasks are very simple. Reviewers suggest regressing against complex PDE solutions, images and do weather modeling. (b) Training protocols are not standard and may disfavor MLPs. (c) Missing details for reproducibility.

A1: Thanks for the very constructive suggestions!

To address (a), we conducted three families of experiments:

(i) Image Fitting (Appendix A), given that images usually contain sharp local features or high-frequency features. We show that KANs can outperform all baselines (including more advanced baselines, i.e., SIREN, and MLP with random Fourier features) in terms of PSNR. However, we also want to mention that KANs have 10x - 20x higher training wall time. Our chosen images include “Camera Man” (a natural image), “Turbulence” (from PDEBench), and “The Starry Night” (Van Gohn’s painting), to show KANs’ consistently high performance across different styles.

(ii) PDE solving (Appendix B). In B.1, we use the Poisson equation with high-frequency solutions to test KANs’ ability to learn high-frequency modes. In B.2, we use the Allen-Cahn equation to test KAN’s ability to learn temporal PDEs and sharp boundaries. In B.3, we use Darcy flow to test KAN’s ability to learn unstructured solutions. We find that KANs can achieve comparable accuracy with MLPs and MLPs with random Fourier features (under “fair” comparison, i.e., using the Adam optimizer instead of LBFGS and without grid extension). However, we again note that KANs are significantly slow to train, as reflected in the report of wall time.

(iii) MNIST classification (Appendix C). We test KANs’ applicability to high-dimensional data. We find that KANs and MLPs have similar performance (arguably KANs are slightly better but not significantly), again with significantly more training time. We find an interesting phenomenon that the optimal MLP shape (i.e., depth & width) is the same as the optimal KAN shape, which might suggest something universal about MLPs and KANs.

To address (b), we do a thorough scan of hyperparameters (Appendix D), including depths, widths, optimization methods, and learning rates. We find that KANs consistently outperform MLPs across these hyperparameter choices in terms of training losses, however, KANs overfit more significantly than MLPs when KANs are too large (either too deep, or fine-grained) and training data size is small. We also report wall time to show that training wall time is a big limitation of the current KAN implementation.

To address (c), We have uploaded our codes as a zip file to facilitate reproducibility.

Q2: Theoretical assumptions. A main assumption of theorem 2.1 of the paper is that the functions ϕl,i,j are (k+1)-times continuously differentiable. However, the KAT only guarantees that these functions are continuous, as and the authors themselves mention, it has been shown in the literature that these functions can provably be very chaotic, with infinite set of non-differentiability. The authors argue that by using wider/deeper representations, these irregularities may disappear, but this would require further theoretical evidence. I believe there are more reasonable ways of proving universal approximation guarantees for KANs, albeit without the strong scaling laws implied by the k-order spline functions. This main theorem therefore can likely be stated differently, in order to obtain a weaker but more theoretically reasonable statement.

A2: Thanks a lot for the comments. We completely agree. We demonstrated using examples and showing empirically that by using wider/deeper representations, these irregularities may disappear, in figure 2. See also the comment that a deeper KAN is at least as good as shallower ones by setting the subsequent layers to identity. See a version of universal approximation established in a recent preprint by Wang et al. (2024b).

Although proving the advantage of deeper KANs in terms of smoothness can be challenging, we present a concrete example in Figure 2 and Section 2.6. For this function, a three-layer KAN can smoothly represent it, while a two-layer KAN learns highly oscillatory activation functions. And the RMSE loss of the 2-layer KAN is 200x worse than the three-layer KAN.

We would like to thank the reviewer for their careful reading and constructive suggestions. We respond to each point below.

Q1: 1. About the theoretical guarantee of the approximation ability in Theorem 2.1.

A1: Yes your understanding is absolutely correct. This also motivates partially our choice of making KANs deeper. We have added the discussion in the article.

Q2: How to select the number of layers in experiments?

A2: Thanks for the question and apologize that we haven’t made this point clear. In the accuracy section (section 4), we employ a hyperparameter scan both for MLPs and for KANs and select the best-performing models. In the interpretability section (section 3), we start with a KAN as small as possible (e.g., we surprisingly find that one hidden layer & one hidden neuron are enough to solve the knot invariant prediction), and then gradually expand the network if performance is bad. We first expand the width. If expanding the width doesn’t help, we expand the depth. Grid size G does not appear to have a large impact as long as it is set to typical values, e.g., G = 3,5,10.

Q2(a)(b): What if the KAN has more layers than needed? Does increasing depth degrate interpretability?

A2(a)(b): Thanks, this is a very great question! We have performed a simple study in Appendix E and Figure 14. The conjecture is correct that the rest layers will (roughly) become an identity map. There are some subtleties though: (1) they might not be exactly identify maps, due to the fact that $f\circ g = (f\circ h)\circ(h^{-1}\circ g)$ for any invertible h. (2) The identity map can appear in early layers or later layers, depending on random seeds. (3) In some cases, a deep network (4-layer in Figure 14) may fail to learn a sparse graph as expected, which we would like to look into in the future.

Q2(c): If we use a KAN that is too deep, would there be difficulties in optimization? If so, how many layers would you recommend starting with?

A2(c): KAN with 5 layers can optimize just fine on regression problems with no sparsity loss. However, deep KANs can become bad in training in certain cases: (1) when the sparsity penalty is non-zero. (2) When high-order derivatives are involved, as in PDE solving. So we suggest trying KANs with 2-layer (or even 1-layer, if applicable, equivalent to additive models), and gradually increasing the depth if needed. We find that shallow KANs of depth 2 & 3 are already quite powerful in many of our test cases.

Q2(d): Are there strategies (maybe heuristics) to select hyperparameters?

A2(d): We would like to suggest these heuristics (they should be tested more formally in the future, but they are useful at least in our cases): (1) When one cares about performance. If one doesn’t know what is reasonable shape for MLPs, a hyperparameter scan is unavoidable. However, if one has experience playing with MLPs for the task, using the same shape that the optimal MLP has for KAN is a safe choice (and G~3,5,10) because KANs are more expressive than MLPs given the same shape. (2) When one cares about interpretability and doesn’t care that much about accuracy (e.g, 85% accuracy is enough), we suggest starting from a KAN as simple as possible (without sparsity penalty), and then gradually increasing width and depth (width first, depth second) until a reasonable performance is achieved. Finally, crank up the sparsify penalty to explore the sparsity vs performance trade-off. Hopefully, the performance is not lost (much) as the network becomes sparser, but this is probably task-dependent.

Q3: How to select the number of nodes in experiments.

A3: When we care about performance, we usually choose the width that this comparable to or smaller than a typical MLP. When we care about interpretability, we start from a KAN network as simple as possible. The knot invariant example lies in the second category where we start from the minimalist shape (1 hidden layer, 1 hidden neuron) and surprisingly find it to be working already.

We would like to thank the reviewer for their careful reading and constructive suggestions. We respond to each point below.

Q1: The paper should focus on establishing whether the problem of non-smoothness is in fact avoided in deeper versions.

A1: Thanks for the great suggestion! We have added a new Section 2.6 to expand on this point. We admit that providing theoretical proof can be challenging, so we instead use a concrete example to illustrate this point. We cooked up a four-dimensional smooth function. As shown in Figure 2, a three-layer KAN is able to represent this function smoothly, while a two-layer KAN learns highly oscillatory representations and has 200x higher RMSE than the three-layer KAN.

Q2: The paper presents Theorem 2.1 which gives a bound on the supremum of the error in the derivative. However, it is not clear whether the bound is actually useful since it depends on f and the network in an unknown way through the constant C. If a general theoretical argument demonstrating smoothness is too difficult, perhaps the paper could investigate the smoothness of the learned functions (or the gradients during training) to provide empirical evidence that the model learns smooth functions.

A2: Thanks for the question. It is indeed a valid concern to verify the smoothness in practice. Thm 2.1 is stated in terms of best approximation, not necessarily implying what will happen in practice during the training dynamics. In the experiments in section 4, the loss curves indeed match the theoretical scaling law, thus providing empirical evidence that the model learns smooth functions.

Q3: Regarding the proof of Theorem 2.1 in Appendix B. It is not clear to me how the bound on the residual is obtained in line 894. If I understand correctly, the bound on B-spline approximation in line 887 is for a single univariate function, phi_{l,i,j}, whose inputs are exact. However when the inputs from the layer below are also approximated and then summed, one would expect the bound of sup of error in the derivatives to also be summed? Whereas the error in line 887 for exact inputs and that in line 894 for approximate inputs is exactly the same. Could you clarify how the error for the residual is obtained?

A3: Thanks for the question. Your understanding is indeed correct. The constant C was indeed summed which varies from line to line, rendering the constant C to depend on the architecture and number of compositions. C is however independent of G, which highlights how we avoid the curse of dimensionality.

Q4: How are units automatically converted into symbolic functions? What are the metrics used for symbolic regression?

A4: We apologize that we missed this detail in the main paper. We have added this sentence in Section 2.4 “The idea is to match learned spline functions with candidates in a symbolic function library specified by human users and replace the spline functions with the best-fitting ones.” Since both examples (knot theory and adnerson localization) are classification tasks, we used test accuracy as the metric to evaluate discovered formulas.

Q5: The motivation that “we care intrepretability rather than smoothness” should be rephrased.

A5: Totally agree! We have deleted the sentence, but instead discussed the concrete example we have in A1 and the new Section 2.6.

Q6: Typos.

A6: Thanks for reading so carefully! We have corrected the typos.

Q7: Are the constants C in the proof in appendix B and the statement of Theorem 2.1 the same? If not, different notation would be better.

A7: See answer to Q3. We have adopted a different notation thanks to the suggestion.