On the expressiveness and spectral bias of KANs

1. Elaborate on statement 3.4:

Thanks for the question. As we mentioned after the statement of the theorem, this is the same rate in terms of approximation as MLPs in the Sobolev class. See also remark 3.5 for how this compares with the superior rate in Thm 3.1.

2. Comparison between KANs with kth order B-splines with ReLU-k MLP for spectral bias:

Thanks for the valuable comment. We agree with the suggestion and have incorporated the comparisons. Basically theoretically by the works of Zhang et al. (2023) and via experiments, ReLU-k MLP suffers more from spectral bias, on top of being harder to train.

3. In Figure 3 and 4 the learning of KANs display instabilities in contrary to MLP which are attributed to grid extension during the undersampled regime, I don't really understand what is meant by this, is it that the grid size of KAN is progressively extended during learning:

Thank you for the question. We have specified the grid sizes there and given a recap of grid extension in Sec 3.1. The instabilities at the grid extension points were caused by the suboptimal way that grid extension were implemented in pykan package. On the other hand, they were caused by the fact that there was not enough training data; see the instabilities even when training on the coarsest grid.

4. The experimental results presented could be more concise and more informative:

Thanks. Figure 1 and 2 were both included to draw analogy to what was observed in the original spectral bias paper Rahaman et al. (2019). Figure 3 and 4 were used to demonstrate training and test losses, and discuss overfitting so that one can see clearly the two losses. We have magnified the legends and put Figure 2 in the appendix. We have also added additional discussion about the conclusions we draw from the experiments, and added additional experiments to support our claims to the appendix.

1. Analysis of the spectral bias and PDE example very similar to [1]:

Thanks for the comment and sorry that we did not notice earlier. We agree that [1] used NTK to empirically analyze the spectral bias and perform an example very similar to our PDE example. However, notice that they only used that example for function fitting, not for PDE solving. Such an example is very natural and this paper proposed it unaware of [1]. NTK is only valid in the infinite width or lazy regime where the parameters don't change much, so the eigenvalues of NTK might not be the most indicative in the active learning regime during training. In contrast, our spectral bias analysis is focused on the finite width regime. Nonetheless, it was our oversight that we did not recognize their contribution in this aspect and we have modified it in the Prior Works section. We also believe that the NTK analysis will be helpful in future work analyzing deeper KANs.

2. Expand on the consequences of Corollary 3.4:

This result on general Sobolev spaces complements Thm 3.1 in the more restrictive class, in terms of approximation theory. It shows that for the general class, KANs are at least as good as MLPs (ReLU-k) for approximation.

3. Understand the consequence of thr.3.2 and th.3.3 for more "mainstream" architecture like ReLU and SiLU:

Thanks. In terms of exact representation, it might be hard since the nonlinearities are different from that of B-splines. However, we invoke Thm 3.2 mainly to derive Cor 3.4, where we achieve a very nice approximation rate in the general Sobolev spaces. This complements Thm 3.1 in the more restrictive class, in terms of approximation theory.

4. Expand the Hessian analysis and connect with the NTK one:

Thanks for the suggestion. As explained in point 1, NTK is only valid in the infinite width or lazy regime where the parameters don't change much, so the eigenvalues of NTK might not be the most indicative in the active learning regime during training. We are concerned with a finite number of neurons, where NTK regime is not valid even in the shallow case. The Hessian analysis drives the exact dynamics of training, while it is only valid in the shallow case. Notice that the trace of the Hessian, when output dimension $d'=1$, equals the trace of NTK as the number of points goes to infinity. We have added additional remarks comparing these two regimes.

5. Perform the spectral analysis for section 4.4, instead of the error trend, if not, what is the test error:

Thanks. We are in the regime of solving PDEs instead of function fitting, and we report the $L^2$ and $H^1$ losses rather than the training loss, which is the equation residue.

1. All the theorems 3.1, 3.2 and 3.3 are extremely mathematical, and their assumptions are not agreeable with practical cases; and the proof of mathematical analysis is oversimplified by assumpting extremely shallow KAN network:

Thanks a lot for the comment. The assumption of Thm 3.1 is mathematical but also relevant in lots of tasks including symbolic regression; see the original KAN paper Liu et al. (2024e). For Thm 3.2 and 3.3, we are comparing how one uses MLPs to represent KANs and vice versa. You probably also meant Thm 4.1 on the spectral bias; for expressiveness we don't assume shallowness. In terms of spectral bias, we have already acknowledged the restriction of our analysis and used the subsequent examples in various tasks to demonstrate the empirical observation for deep cases that KANs perform better for high-frequency tasks.

2. The experiments are not sufficient; For the examples of PDE, the equation is very simple, and the functions are of sufficient smoothness. If the functions concerned becomes chirp function and so on, how about the performance:

Thanks. We agree with this comment. We use these examples mainly to demonstrate the spectral bias, and agree that these experimental results are preliminary. We have added some more challenging examples in the appendix, and leave a more extensive empirical testing of this phenomenon to future work.

3. In Fig. 4, the performance of approximation does not always tend to better:

Yes indeed. For smooth tasks the MLPs behave better, but their performance degrades significantly for rough cases, which is exactly what we would expect based upon our analysis of the spectral bias. Also in the middle row, since we don't have enough training samples, KANs overfit, which is another point we are making showing that KANs don't suffer much from spectral biases as they fit the higher-frequency noises as well. We agree that our initial presentation was not clear and have added remarks clarifying these issues.

1. The parameter count in KANs and MLPs are not directly comparable:

We acknowledge that the parameters of KANs and MLPs are not directly comparable since they are different architectures. We have also added a remark that the number of parameters is not necessarily the most useful measure of complexity of a model. However, the total number of parameters is generally indicative of the efficiency of a model and as a first step in our analysis, we have taken this measure of complexity. We remark that future work would entail comparing KANs and MLPs using more fine-grained measures of complexity. On the other hand, we note that if one focuses on the running time; in the $3d$ GRF examples using $50000$ samples, KANs are almost ($\approx$74mins) as expensive as MLPs ($\approx$56mins) on a gpu, while in practice one can also make KANs even more efficient by optimizing the implementation. These additional remarks have been added in the appendix.

2. Implication of Thm 4.1 to spectral bias or high-frequency performance:

Thanks for the suggestion. We have added more discussion about how the eigenvalues of the Hessian relate to the spectral bias, or performance of the model on high-frequency components. Specifically, Thm 4.1, compared with its MLP counterpart established in Hong et al. (2022) and Zhang et al. (2023), implies that for shallow models, KANs are able to fit high-frequency components of the target function more efficiently than MLPs when trained using gradient descent.

3. Restriction of the analysis to shallow KANs:

We acknowledge that our analysis of the spectral bias is fairly restrictive (this is in contrast to our analysis of the expressiveness and approximation theory, which holds for general deep MLPs and KANs). We have added more clear remarks on the limitations of our analysis before section 4.2. Nonetheless, we argue that despite its limitations, our analysis provides a theoretical explanation for observation that KANs fit high-frequency data better than MLPs. Finally, we remark that in addition to the theoretical analysis, we also provide empirical evidence supporting this claim. We leave a detailed analysis of the spectral bias of deep KANs, which we expect to be quite challenging, to future work.

4. Statistical significance of the experiment:

Thank you for pointing this out. We agree that in order to conclusively demonstrate a difference in the spectral bias of KANs and MLPs, one would require a large body of experiments covering a diverse set of practical applications., which is beyond the scope of a single paper. In our work, we have done a preliminary set of experiments that indicate a difference in the spectral bias of KANs vs MLPs. To the best of our knowledge, we have followed best practices and performed the relevant ablation studies. For instance, in the first example, we average over numerous random seeds. In the latter two experiments, we used a fixed random seed to make the experiments reproducible, but we have also observed a similar pattern using other random seeds. We have also added additional experiments to the appendix to strengthen our conclusions. In conclusion, we argue that our preliminary experiments are significant, and agree that future empirical work is necessary to conclusively validate our findings.

5. Theoretical Expansion: How does the depth of Kolmogorov-Arnold Networks (KANs) influence their spectral bias and approximation capabilities:

Thanks for this question, which we believe is too complicated to include in this paper, and which we intend to leave for future work. For example, one could compute the eigenvalues of the NTK matrices of deeper KANs empirically to investigate their spectral bias from a theoretical perspective, see for example Wang et al. (2024). Deeper KANs have better approximation capabilities and rigorously studying their spectral bias properties is an important future research direction.

6. Limitation of KANs, including computational cost, overfitting, scalability, practical implementation challenges, and real-world applications and comparison with SOTA models:

This is a very good point, and in the paper, we remark on this limitation of our analysis. Indeed, we only take into account the approximation, representation power (in terms of the number of parameters), and spectral bias of KANs. Our analysis of the spectral bias of KANs is relevant to overfitting, and indeed our results indicate that KANs may be more prone to overfitting since they are more sensitive to higher-frequency noise. Other important practical considerations including those mentioned by the reviewer are not discussed in this work. We remark that some of these aspects are discussed in the contemporary literature, including some that we cite in the Prior Works section. Right now our observation is that KANs are able to address large-scale problems very well once we scale up the sizes of KANs, made possible by the more efficient implementations of KANs now.

Thank you! I have updated the score.