Generalization Bounds for Kolmogorov-Arnold Networks (KANs) and Enhanced KANs with Lower Lipschitz Complexity

We would like to thank the reviewer for their careful reading and constructive suggestions. We respond to each point below.

Q1: Role of Activation Family Cardinality.

Yes, we thank the reviewer for the thoughtful question.

The activation family size $G$ reflects the cardinality of the basis function family, treated as a constant that captures overall family size, while its internal structure (e.g., smoothness) remains tunable. For instance, KANs can employ different basis families—B-splines (original KAN), Fourier (Fourier-KAN), and rational functions (Rational-KAN)—each with adjustable basis affect generalization without altering $G$.

From a practical perspective, we conducted sensitivity analyses on B-spline basis functions by varying degree and grid size, which together determine the shape and cardinality of the activation family. The key findings are:

1. Internal structure matters beyond size: Even when $G = \text{Grid size} + \text{Spline order}$ is fixed, generalization can vary significantly. On MNIST, both (3,4) and (5,2) yield the same $G$, yet the Final Gen Gap differs by over 7×—highlighting that basis structure, not just size, plays a crucial role.
2. Opposing effects of grid size vs. spline order: With fixed grid size, increasing spline order improves generalization; conversely, with fixed spline order, increasing grid size degrades generalization. This aligns with our view that smoother basis functions (higher order) help regulate model complexity.
3. Efficiency trade-offs: Larger $G$ increases computational cost, with spline order contributing more to time and memory overhead than grid size.

These results reinforce our theoretical claims and demonstrate the framework’s robustness across different architectural settings. We will include this analysis in the revised appendix.

Q2: Symbolic Interpretability.

Yes, we appreciate the reviewer’s attention to symbolic interpretability, which is an important strength of KANs. We confirm that LipKANs preserve the symbolic interpretability of standard KANs.

Specifically, the core architecture of KANs remains unchanged: each layer computes explicit linear combinations of parameterized basis functions (e.g., B-spline, Fourier, Chebyshev), which can be symbolically extracted and visualized. The Lip layer is applied outside the basis compositions and acts as a global scaling operator—it does not interfere with the symbolic structure of the basis expansions. Likewise, the $L_{1.5}$-regularization promotes generalization via weight norm control, without altering the functional form or symbolic interpretability of the model.

To empirically verify this, we follow the symbolic regression setup from the original KANs paper and train LipKANs to fit the same target function: $f(x, y) = \exp(\sin(\pi x) + y^2)$, which combines periodic and polynomial components. We use soft pruning (threshold $10^{-4}$ from epoch 50). The results show that LipKANs achieve a 96.66% sparsity ratio, indicating that only a small subset of basis functions is retained, which is sufficient to represent the symbolic structure. Besides, training and test MSE are identical, confirming that the learned function captures the true symbolic form.

Therefore, LipKANs enhance generalization while preserving symbolic transparency. In addition, Appendix D.1 presents further function fitting results on more complex symbolic targets, reinforcing LipKAN’s interpretability. We will clarify this in the final version.

W: Limited Experimental Breadth. & OOD Generalization.

We sincerely thank the reviewer for the valuable suggestions. While our current focus was on establishing the theoretical foundation and demonstrating initial empirical benefits, we fully intend to pursue these extensions in future work, and we will mention them as part of our roadmap in the final version.

Suggestions: Visualization of Complexity Metrics. & Empirical Study of Covering Bounds.

Thanks for the constructive suggestions. We leave these directions for future work and view them as promising avenues to further bridge our theoretical analysis with practical model behavior.

We thank the reviewer for raising important points, and we will incorporate the corresponding revisions accordingly.

W1: Loose Generalization Bound: While the proposed generalization bound successfully captures key factors influencing KANs’ generalization, it does not exploit dependencies or relationships among different weight matrices. As a result, the bound can be quite loose. For instance, for two KAN models where each weight matrix has the same norm, the calculated complexity—and thus the bound—will be identical, regardless of the relative directions of the matrices. However, the actual Lipschitz constant, and thus the true generalization capability, can be significantly affected by these directional relationships. As a consequence, the proposed bound may fail to distinguish between models with genuinely different complexities, potentially affecting model selection.

Q2: Is it possible that we take the relation among weights into consideration so as to refine Lipschitz complexity?

Thanks for highlighting this limitation. While our current generalization bound does not make inter-layer dependencies explicit, it already captures layer-wise alignment through the operator norm products $\prod_{i=1}^{L}\|\mathbf W_i\|_\sigma$, which originate from the covering number bound of the entire network in Lemma 4.2. This implicitly reflects structural relationships across layers. We agree that making these dependencies more explicit could tighten the bound, and we consider this a valuable direction for future refinement.

W2: Insufficient Discussion of Bias-Variance Trade-Off: The paper mentions the bias-variance trade-off, but does not clearly explain how it relates to their results or the broader context of estimation versus generalization. Clarifying this relationship could help readers understand how LipKANs balance model expressiveness and generalization, and why the trade-off is enhanced compared to the conventional method.

Thanks for raising this insightful point. To clarify the bias-variance trade-off in our setting, we revisit the generalization bounds in Theorem 4.5. The Lipschitz complexity term reveals that the model variance—its sensitivity to training data perturbations—is closely related to the structured $(2,2,1)$-norm of the weight tensors, which controls the overall magnitude of the spline coefficients. A larger norm enables greater flexibility in fitting the data but also increases the risk of overfitting, thereby contributing to higher variance. In contrast, the bias stems from the approximation capacity of the basis functions (e.g., B-spline, Fourier, and Chebyshev basis). These basis functions provide strong approximation capacity for smooth target functions, enabling the learned model to closely align with the underlying data distribution when appropriately regularized. To this end, the proposed $L_{1.5}$-regularization penalizes the structured norm, reducing variance while preserving expressivity—thereby encouraging a favorable bias-variance trade-off and enhancing generalization.

Furthermore, this trade-off aligns closely with the decomposition of uncertainty. The complexity-dependent term reflects epistemic uncertainty, which encompasses both bias and variance as it arises from limited data and model capacity, and can be reduced by regularization or more training data. In contrast, the second term, $\sqrt{{1}/{n\epsilon}}$, derived from statistical concentration inequalities, quantifies aleatoric uncertainty, reflecting intrinsic randomness in the data that persists regardless of the model's capacity. In this way, our generalization bounds—interpreted via uncertainty decomposition—offer a complementary perspective on the bias-variance trade-off: epistemic uncertainty encompasses both bias and variance and can be effectively reduced via $L_{1.5}$-regularization, while aleatoric uncertainty sets the irreducible limit. Our method improves generalization by explicitly targeting this trade-off in a principled manner.

Empirically, we validate this in the following experiments, where LipKANs consistently show lower overfitting ratios and generalization gaps than KANs with standard $L_{1}$-regularization. This confirms that our design not only aligns with the theory but also improves generalization in practice by facilitating a better bias-variance trade-off.

We will clarify this connection in the revised version to improve the presentation.

W3: It is better to move definition 4.2 before assumption 2 as the definition is mentioned in assumption.

Sincerely thanks for the thoughtful suggestion. We will make this revision in the camera-ready version to enhance readability.

Q1: Is it common that assumption 2 holds?

Yes, as discussed in Appendix C.3, we provide concrete examples to support Assumption 2. For instance:

• B-spline basis (degree 3, grid size 5): $|\Sigma|_{\text{Lip}} \leq 10\sqrt{2}$;
• Fourier basis (frequency 4): $|\Sigma|_{\text{Lip}} \leq 2\sqrt{15}$;
• Chebyshev polynomial basis (order 4): $|\Sigma|_{\text{Lip}} \leq \sqrt{14}$.

These values confirm that the Lipschitz constants remain bounded in practical basis families, validating the assumption.

Q3: The theory is based on sub-Gaussian distribution. Can it extend to other distribution?

Yes, and thanks for the thoughtful question. Our analysis currently uses the sub-Gaussian assumption to control input tail behavior and derive generalization bounds via standard concentration tools. However, the framework is not inherently restricted to sub-Gaussian inputs. In principle, it can extend to broader distribution classes—such as sub-exponential or heavy-tailed distributions—by leveraging tools from empirical process theory, including Bernstein-type inequalities, truncation methods, or Orlicz norm bounds. We view this as a natural direction for future theoretical extensions.

We thank the reviewer for the constructive suggestions and will further revise the manuscript accordingly.

W1: While experiments cover multiple domains, the gains on simpler datasets (e.g., MNIST, CIFAR-10, AVMNIST) are marginal (<1% in Table 1), suggesting limited utility for low-complexity tasks.

Q1: From the ablation study results in Figure 1, the main performance improvement comes from the L_{1.5}-regularization. Does this mean that it is not important to keep low Lipschitz complexity using Lip layer in practical applications?

Thanks for the insightful question. We confirm that the Lip Layer remains an important and complementary component for controlling generalization. Both mechanisms stem from our unified Lipschitz complexity definition (Definition 4.3), but target different terms: the Lip Layer directly constrains the smoothness of the basis functions $\|\Sigma\|_{\mathrm{Lip}}$, while the $L_{1.5}$-regularization controls the structured (2,2,1) norm of the weight tensors $\mathbf W$. These two components regulate orthogonal axes of the Lipschitz complexity, thereby acting in a complementary fashion to improve generalization.

Empirically, to provide a more comprehensive analysis of generalization performance, we report three additional generalization metrics:

• Final Generalization Gap: Difference between training and validation accuracy at convergence
• Average Generalization Gap: Mean gap across the entire training trajectory
• Overfitting Ratio: Final gap normalized by final training accuracy

The results are summarized below. We observe that the Lip Layer consistently reduces the generalization gap and overfitting ratio across multiple datasets, particularly on MNIST and CIFAR-10, where the basis smoothness plays a more critical role than weight sparsity. This supports the practical value of maintaining low Lipschitz complexity via basis smoothing. Overall, while both Lip Layers and $L_{1.5}$-regularization are designed to reduce variance, they operate via distinct mechanisms—basis-level smoothing versus structured weight penalties—resulting in different sensitivities to sample noise.

Moreover, as shown in our full ablation table, combining both components (i.e., LipKANs) consistently yields the lowest generalization gap and best validation accuracy, indicating their complementary effects. Interestingly, this advantage does not always translate into significantly higher validation accuracy—some models achieve comparable validation performance yet differ markedly in their generalization gaps. This phenomenon reflects differences in the underlying bias–variance trade-off: models with high training variance may overfit the training data, resulting in deceptively strong validation accuracy but poor generalization. In contrast, methods like those in LipKANs introduce mild bias through regularization while substantially reducing variance, yielding more stable and reliable generalization across datasets.

We will clarify this intuition and add further discussion in the camera-ready version.

W2: The cost of Lipschitz layers and L1.5 regularization is not analyzed, leaving practical efficiency unclear.

Thanks for the constructive suggestion. We would like to clarify that both the Lip layer and the $L_{1.5}$-regularization are lightweight mechanisms that introduce minimal computational burden. The Lip layer imposes Lipschitz norm constraints to control the smoothness of basis functions, without modifying the network architecture or introducing auxiliary parameters. The $L_{1.5}$-regularization is a direct substitute for the commonly used $L_1$ penalty—it operates on the same weight tensors already present in training and does not incur additional memory overhead.

Empirically, we quantify efficiency using two widely adopted metrics: training time (in seconds) and peak GPU memory usage (in MB). Our measurements show that both the Lipschitz layers and the $L_{1.5}$-regularization incur only moderate overhead relative to baseline KANs, while delivering notable improvements in generalization performance.

Q2: This paper starts with the lack of KAN's generalization, but only verifies the performance on several small-scaled datasets. Discussing the transfer learning performance of KAN (for example, evaluating the performance of a model trained on the MNIST dataset on CIFAR-10) should better verify the generalization ability of the model.

Thanks for the good suggestion! While our current framework does not address out-of-distribution (OOD) scenarios, investigating LipKANs’ behavior under distribution shifts is an interesting direction for further work. We would like to clarify that our theoretical and empirical analyses in the current draft focus on in-distribution generalization under the standard i.i.d. setting (as formalized in Section 3), which is commonly adopted in statistical learning theory. Specifically, our generalization bounds (e.g., Theorem 4.5 and Corollary 5.1) quantify the expected risk gap between the training and test distributions drawn from the same underlying data distribution, with controlled model complexity via Lipschitz complexity. This notion of generalization differs from OOD generalization or transfer learning, which instead assesses the model’s ability to adapt or perform on a different target domain. As a preliminary exploration, we conducted transfer experiments from MNIST to CIFAR-10, but both KANs and LipKANs achieved relatively low target-domain accuracies (19.37% and 23.28%, respectively). Returning to the in-distribution setting, we have evaluated our models across multiple datasets as reported in Table 1, where LipKANs consistently yield higher validation accuracy than the original KANs, indicating enhanced generalization performance under standard i.i.d. settings.

We would like to thank the reviewer for the careful reading and constructive suggestions. We respond to each point below.

Q1: Is the F1/accuracy score shown in Table 1 on a held-out test set? If you haven’t, can you specifically mention this somewhere? Also, an explicit quantification of the generalization gap would be useful.

Thanks for your valuable suggestion, and we apologize for missing the details! We confirm that all reported accuracy and F1 scores in Table 1 are evaluated on held-out test sets using the officially predefined training/test splits for each dataset, consistent with the evaluation protocols established in prior work [1,2]. The test sets are strictly excluded from the training process, preventing any form of data leakage. To make this explicit, we have updated the caption of Table 1 and added a clarifying note in the Experiments section.

In addition, to provide a more comprehensive analysis of generalization performance, we follow the prior work [3,4] and quantify generalization gaps using the following three metrics:

• Final Generalization Gap: Difference between training and validation accuracy at convergence.
• Average Generalization Gap: Mean gap across the entire training trajectory.
• Overfitting Ratio: Final gap normalized by final training accuracy.

The results are summarized below.

Across all datasets, the LipKANs consistently achieve the lowest overfitting ratio and final generalization gap, suggesting stronger generalization. Interestingly, this advantage does not always manifest as significantly higher validation accuracy—some models achieve similar validation accuracy, yet differ notably in their generalization gaps. This discrepancy reflects a more favorable bias–variance trade-off. Specifically, models with higher variance may fit the training data well yet generalize poorly, resulting in larger generalization gaps despite acceptable validation accuracy. In contrast, regularization methods such as those employed in LipKANs may introduce slight bias but effectively reduce variance, leading to more reliable generalization behavior.

We will include these clarifications and additional generalization metrics in the updated camera-ready version to improve transparency and completeness.

Q2: Could you provide confidence intervals for the output metrics shown in Table 1?

We appreciate your question and apologize for missing these important details. We have conducted five independent runs with different random seeds and now report the corresponding 95% confidence intervals for each validation accuracy. These intervals quantify the variability due to optimization randomness and demonstrate the robustness and reproducibility of our results. We have included these confidence intervals in Table 1 in the revised submission.

Q3: From the ablation study, it seems that the L1.5 generalization is the biggest driver of improvements in generalization, and the Lip layers only help for MNIST. I wonder if this effect is just result of optimization/training splits? Can you show that LipKANs consistently show an improvement for MNIST? This for me is the biggest weakness in the paper, and any additional experiments to improve confidence in this would be useful.

W1: The results of the ablation study make me question the value of the Lip layers in improving generalization. Improvements seem to be mainly driven by the L1.5 regularization. I would want to see more experiments showing that Lip layers reliably contribute to generalization for MNIST and other datasets to be convinced that the proposed practical architecture genuinely improve generalization in line with theoretical results.

Thanks for the great question. We would like to clarify that both components are principled and arise naturally from our formulation of Lipschitz complexity (Definition 4.3). Specifically, the Lip Layer constrains the smoothness of the basis family through the Lipschitz norm of $\Sigma$, while the $L_{1.5}$-regularization penalizes the structured (2,2,1) norm of the linear weight tensors $\mathbf W$. These two components regulate orthogonal axes of the Lipschitz complexity and hence play complementary roles in promoting generalization.

To rigorously assess generalization beyond final validation accuracy, we report three additional metrics in the Table of Q1: Final Gap, Average Gap, and Overfitting Ratio. Notably, on multiple datasets such as CIFAR-10 and MNIST, we observe that KANs + Lip Layers outperform KANs + $L_{1.5}$-regularization on most metrics, suggesting that basis smoothness control is particularly effective in low-noise, low-complexity regimes where architectural inductive bias is more beneficial than weight norm constraints. Conversely, on other datasets (e.g., CIFAR-100, STL-10), the $L_{1.5}$-regularization yields greater improvements, indicating its strength in high-variance regimes where structured norm constraints are better suited to mitigating overfitting. Overall, while both Lip Layers and $L_{1.5}$-regularization aim to control model complexity, they mitigate training variance through different mechanisms, leading to distinct sensitivities to sample-level noise.

Finally, to ensure that the reported gains are not due to specific training splits or optimization variance, all experiments are averaged over 5-fold cross-validation and multiple random seeds, ensuring statistical reliability and reproducibility. We will include these clarification tables and extended results in the camera-ready version to fully address the reviewer’s concern.

References

[1] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov–arnold networks. In ICLR, 2025.

[2] Liangwewi Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, and Weitong Chen. Free-knots kolmogorov-arnold network: On the analysis of spline knots and advancing stability. arXiv:2501.09283, 2025.

[3] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In NeurIPS, 2017.

[4] Jingwen Fu, Zhizheng Zhang, Dacheng Yin, Yan Lu, and Nanning Zheng. Learning trajectories are generalization indicators. In NeurIPS, 2023.