Improving Memory Efficiency for Training KANs via Meta Learning

C1: Some algorithmic choices seem arbitrary

A1: We appreciate the reviewer's point regarding the clarity of our algorithmic choices. We acknowledge that the rationale behind certain design choices, such as the specific architecture of the meta-learner (e.g., a two-layer MLP) and the initial dimension of the learnable prompts (e.g., d=1), may not have been sufficiently stated in the original manuscript. In the revised version, we have added explanations for these choices in Section 3.2.2. We selected a two-layer MLP as the meta-learner because it balances expressive power and parameter efficiency, which is a common choice in many meta-learning and hypernetwork studies [1-3]. For the prompt dimension d=1, we chose it as the most parameter-minimal for experimentation. We also explored the impact of different prompt dimensions in the ablation study in Appendix C.4. The results show that increasing the dimension can improve performance but also comes with additional parameters and potential training instability. This confirms the rationale behind our initial choice of d=1 as the baseline setting, while allowing for flexibility to adjust the dimension when needed. We strive to ensure that all design choices are grounded in empirical observations or inspired by related work, and we will clarify these considerations more explicitly in the revised manuscript.

[1] HyperNetworks, ICLR 2017.

[2] Meta-weight-net, NeurIPS, 2019.

[3] Continual learning with hypernetworks, ICLR, 2020.

C2：Why MetaKANs have more parameters than KANs in Table 2.

A2: We sincerely appreciate the reviewer's careful attention to these important details. You're absolutely correct to note that MetaKANs can have more parameters in certain configurations. Please refer the details about the analysis of the situation to the A3.2 for reviewer tj4z. Besides, the potential benifits of MetaKANs are scaling up to even larger KAN models (the results of large KANs in Table 2; please refer to response for Reviewer A7wa).

And we will reformat all tables to highlight only cases with both MSE improvement and parameter reduction.

C3: Why does depth matter?

A3:

The depth of MetaKAN motivates our design choice of using multiple meta-learners (hypernetworks) in deeper architectures. In deep networks, features become more abstract with depth—early layers capture low-level patterns, while deeper layers model higher-level semantics. Accordingly, the optimal form of activation functions may vary across layers.

Using a single global hypernetwork to generate all activation functions may be overly restrictive, as it must capture diverse functional shape through input embeddings. Instead, grouping layers and assigning a separate meta-learner to each group allows better specialization, aligning with the hierarchical structure of deep features.

Our ablation study (Appendix C.4) supports this: increasing the prompt dimension from $d=1$ to $d=4$ improves accuracy but also increases variance. Using multiple meta-learners per layer group reduces this variance, especially in deeper networks, suggesting enhanced stability. We plan to further explore the theoretical basis of this effect in future work.

C4: Visualize how B-spline functions vary with prompt dimension

A4: We appreciate the reviewer’s suggestion regarding visualization. In response, we conducted an experiment by varying the promp dimension from 1 to 4 and visualized the corresponding learned B-spline functions trained on the dataset $f=\exp(\sin(x_1^2+x_2^2)+\sin(x_3^2+x_4^2))$ . These visualizations reveal how increasing the embedding dimension enriches the expressivity of each spline function.

With a lower embedding dimension, the splines exhibit relatively simple, smooth transformations. As we increase the embedding dimension to 3 and 4, the spline structures become more complex, reflecting the model’s enhanced capacity to capture more complex functions. For the detailed visualization, please refer to the link https://github.com/icmlrebuttal25/Append

C5: Combining with network pruning

A5: Thank you for your interest in combining our meta-learning approach with network sparsification/pruning. Since MetaKAN focuses on generating activation parameters without altering network structure, pruning can be applied independently. For example, in symbolic regression tasks (Appendix B), we applied sparsification by removing low-activation edges and using symbolic simplification. These results show MetaKAN works well with sparsification techniques.

C1: Complexity comparison and Training time between KAN and MetaKAN.

A1: We thank the reviewer for the insightful question. Below is a summary of theoretical complexity

1. Complexity Analysis notations see https://github.com/icmlrebuttal25/Append

2. Advantage condition

MetaKAN becomes cheaper when $N_{edges} > 4d_{hidden}$, typically when $N_{edges} \sim 10^4$ or higher. This is due to trading forward-pass overhead for lower backward complexity. In practice, with >4 layers and >500 neurons per layer ($N_{edges} \sim 10^6$), MetaKAN training time becomes comparable to KAN. For deeper networks (8+ layers, width >1000), MetaKAN can be faster due to backward efficiency gains.

3. Empirical Comparison

4. Potential

These results align well with our theoretical FLOPs analysis. Although MetaKAN introduces a modest overhead in the forward pass due to the hypernetwork, it significantly reduces the cost of the backward pass—a major bottleneck when training large models. Furthermore, MetaKAN achieves this with drastically fewer trainable parameters and significantly lower memory usage. These advantages indicate that MetaKAN is not only efficient in current setups but also holds strong potential for scaling up to even larger KAN models—scenarios where traditional KANs may become impractical due to memory and gradient computation overhead.

C2: Performance and training time comparison between MLP and MetaKAN A2:

1. For deeper structure, the theoretical parameter count of MetaKAN scales mainly with the number of edges and the number of meta-learners leads to a few parameter count increasing.

2. We compare the two models on two tasks: function fitting and image classification.

On the function fitting task $f_1(x)=\exp\left(\sum_n(1/n)\sin^2(\pi x/2)\right)$ with $n=1000$, MLP trains faster, but its accuracy is significantly lower. MetaKAN retains the inductive bias of spline-based models, achieving both compactness and high accuracy, making it ideal for tasks requiring interpretable and function representations.

For image classification (MNIST), MLPs again train faster and achieve slightly better accuracy. However, MetaKAN still maintains strong performance with substantially fewer parameters.

3. In summary, while MetaKAN trains slower than MLPs, it retains the core strengths of KANs—strong inductive bias and interpretability—while dramatically reducing parameter count and memory usage compared to KAN. This allows MetaKAN to close the training cost gap with MLPs, offering a practical and scalable alternative that balances efficiency with symbolic inductive biase.

C1:Why use the MLP as the meta-learner?

A1: We understand the reviewer's interest in the theoretical motivation and deeper implications of employing an MLP as the hypernetwork, particularly in comparison to works like Low-Rank PINNs where architectural choices may have stronger connections to the underlying problem structure (e.g., conservation laws). We acknowledge that our paper could expand more on this aspect. Our choice of an MLP was primarily driven by several considerations: Firstly, as a universal function approximator, the MLP has been widely adopted in the meta-learning field to learn the common task distribution, such as [1-3] . Secondly, the MLP architecture is relatively simple, computationally efficient, and straightforward to implement and train.

[1] HyperNetworks, ICLR 2017.

[2] Meta-weight-net, NeurIPS, 2019.

[3] Continual learning with hypernetworks, ICLR, 2020.

C2:Clarify Figure 3

A2: We apologize for any lack of clarity in Figure 3. To clarify: the LHS does show the actual shapes of activation functions learned by MetaKAN, visualized from the weights generated by the trained hypernetwork. The RHS displays the cosine similarity between these generated spline coefficient vectors to reveal structural similarities in the learned function shapes. We could utilize the different color (positive sign or negative sign) to group the activation functions into different classes and see the compact structure of the learned function family with only a few function classes.

C3:How small can you make the hypernetwork? How does the architecture constrain the weights? What is the extent it can be reduced?

A3: We thank the reviewer for these insightful questions regarding the hypernetwork's design and its impact. These three aspects – minimum size, architectural constraints, and the extent of reduction – are closely related and central to the MetaKAN approach.

The smallest effective hypernetwork is task-dependent. On simpler tasks, small hidden dimensions (e.g., 4–16) suffice. More complex tasks typically need larger dimensions (e.g., 32–128) to maintain expressivity. The choice reflects finding an point on the efficiency-performance trade-off for the specific application.

The hypernetwork architecture constrains the generated weights by acting as a shared meta-learner for all activation function coefficients. Instead of learning separate parameters, MetaKAN learns this shared rule and a unique, low-dimensional embedding Z for each function, as mentioned in our approach. The hypernetwork takes this small embedding Z as input and uses the learned rule to generate the much larger set of spline coefficients (W). Because all coefficients originate from the common function family. The hypernetwork's hidden dimension restricting the variety and complexity of coefficients that can be produced from the input embedding Z.

The extent of parameter reduction achievable by MetaKAN primarily scales with the ratio of the chosen embedding dimension to the original spline coefficient dimension, as the total parameters become dominated by the embeddings in larger networks. As illustrated in table, for small networks (e.g., [4,2,2,1]), the hypernetwork's own size , related to hidden dimension significantly impacts the total count, making a compact hypernetwork essential for savings. However, for larger networks, increasing hidden dimension(from 32 to 64) adds negligible parameters (0.1% increase) compared to the vast overall reduction (89%) achieved relative to KAN. Thus, while hypernetwork size matters for small models, the principal reduction factor in practical, larger-scale scenarios is determined by the embedding dimension.

C4:Improvement with KAN coefficient reduction.

A4: We strongly agree that combining MetaKAN with other KAN coefficient reduction techniques holds significant potential. Our generative approach (reducing learned parameters) is largely orthogonal to methods like pruning, quantization, or coefficient sharing (reducing final coefficient complexity or count). Integrating these strategies could lead to further efficiency gains and is a promising avenue for future research.

C5:Physical meaning of latent parameters Z.

A5: In MetaKAN, the Z embeddings identify the specific activation functions needed per edge. These embeddings organize meaningfully in latent space: nearby Z values produce similar activation shapes. This was verified by comparing similarity matrices of Z and the generated coefficients, which showed strong alignment (as in Figure 3). Thus, Z effectively encodes functional similarity.

C1: Show a task solvable by MetaKAN but not by KAN

A1:We thank the reviewer for this important suggestion. In fact, our experiments in Section C.1 (High dimensional function) and Table 4 (function fitting task) already demonstrate a key scenario where standard KANs fail while MetaKAN succeeds:

For function $f₂(x) = ∑x² + x³$ at dimension $d=50$, standard KANs completely fail to converge (marked as "NA" in Table 4), while MetaKAN achieves stable performance (MSE=1.15×10⁻³). At $d=1000$, KANs show severe error amplification (MSE=$1.68×10²$), whereas MetaKAN maintains reasonable accuracy (MSE=$1.43×10⁻¹)$ with 7× fewer parameters.

From the theoretical FLOPs analysis in A1 (response to Reviewer A7wa), as network size increases, the computational complexity of KAN grows rapidly with the number of edges. In contrast, MetaKAN benefits from reduced backward-pass complexity, which can result in faster training times on large-scale tasks.

For instance, in the empirical comparison table provided in A1, MetaKAN not only consumes just 30% of the peak memory used by KAN, but also achieves faster training time in the largest network stucture. These results highlight MetaKAN’s strong potential for scaling to even larger models—scenarios where traditional KANs may become impractical due to memory limitations and the high cost of gradient computations.

C2: Line 197: functions -> function Line 215: formalizes -> formalize

A2:We have corrected the typos.

C3: Table 2: There are some apparent inconsistencies. On Line 362, the text says that, at G = 5, MetaKAN achieves a lower MSE in 14 out of 16 functions with parameter reductions. None of these numbers match the table: There are 17 functions, 10 of them correspond to the parameter reduction (by the way, why is that the case?), and all of them are highlighted by the bold font implying the improvement in the MSE. Could you please explain this inconsistency?

A3:We apologize for the confusion in Table 2. The inconsistencies arose because:

1. The text mistakenly stated "16 functions" (now corrected to 17).
2. For detailed comparasion of the parameter count, We take $G=5,k=3$, and the network struncture is $[n_0,n_1,...,n_L]$, the parameter count of KAN is $\sum_{l=0}^{L-1}(n_l\times n_{l+1})\times (G+k+1) =9\sum_{l=0}^{L-1}(n_l\times n_{l+1})$, and the parameter count of MetaKAN is $\sum_{l=0}^{L-1}(n_l\times n_{l+1})+(d_{hidden}+1)\times (G+k+1) = \sum_{l=0}^{L-1}(n_l\times n_{l+1})+9(d_{hidden}+1)$.The condition for MetaKAN to have fewer parameters than KAN is: $9\sum_{l=0}^{L-1}(n_l\times n_{l+1})>\sum_{l=0}^{L-1}(n_l\times n_{l+1})+9(d_{hidden}+1)$, which implies the $d_{hidden} < \frac{8}{9}\sum_{l=0}^{L-1}(n_l\times n_{l+1})-1$. So as for some simple network structure (where $\sum_{l=0}^{L-1}(n_l\times n_{l+1})$is small), it does not have the parameter reduction cases.
3. Bold font was incorrectly applied to all MSE improvements, even without parameter reduction. We have now highlighted only cases with both MSE improvement and parameter reduction.