KAC: Kolmogorov-Arnold Classifier for Continual Learning

Major weaknesses

• Various claims in this paper lack substantial proof. For instance:
  • l.502 "leverages KAN’s inherent locality capability to reduce feature shifts during the learning process of new tasks." or l.12 "11. Most existing methods utilize linear classifiers, which struggle to maintain a stable classification space while learning new tasks". While this might be true, I do not see any evidence in the paper for both claims.
  • l.82 "We find that the conventional KAN with B-spline functions struggles with high-dimensional data, leading to inadequate model plasticity, which may weaken the models’ plasticity when directly introduced as a classifier." Same remark here, the drop in average accuracy displayed in Figure 2 can also be the consequence of a lack of stability. Plasticity-focused metrics should be used. This also applies to lines 223 and 227.
  • Figure 1 caption: "Conventional linear classifiers activate each weight equally across all tasks, whereas our Kolmogorov-Arnold Classifier learns class-specific learnable activations for each channel across all categories, minimizing forgetting caused by irrelevant weight changes". There is no proof of such behavior here.
• The experiments are quite limited in terms of reported metrics and number of runs. In CL especially, the seed used can have a significant impact, thus I strongly encourage the authors to run several experiments and report the average and standard deviations.
• Reporting only the last task accuracy on Table 1 and Table 2 is very biased toward plasticity. However, as mentioned, this paper claims a huge gain in stability. Similarly, the last task's accuracy is very dependent on the seed used.
• The analysis of why the methods perform especially well on long sequences is lacking. The authors claim l. 411 "It demonstrates that KAC effectively helps model mitigate forgetting at each step". However, as pointed out above, it is only the final task accuracy, so this narrative is not proven.

Minor weaknesses

• In equations 5, 6, 7, 8, and 10, if my understanding is correct, $x_p$ is a scalar and $c_i$ a vector. I think you should clarify this operation.
• In Figure 3, you first use a stack of 1d Gaussians and then 3d Gaussians after weighting. Since it represents almost the same thing (just with different weights), in the same diagram, I personally find it a bit confusing.
• l.229 "This is because splines cannot effectively model the compositional structure present in the data, while linear classifiers benefit from their fully connected structure, allowing them to learn this structure effectively (He & Xu, 2023)." If my understanding is correct, the KAN architecture is also fully connected (every output depends on every input). So how is that a beneficial property specific to the linear classifier?
• Figure 4 is unclear. Is this the activation of randomly selected inputs of each class? The description l.460 could be improved. A visualization of the learned activation functions for some features across various tasks could be interesting.

typos

• $x_p$ font not consistent in equation 6. Same in equation 8, and 9 and 11 (for $x$).
• figure 2 and figure 5, legends would be better on the figure rather than caption.

The originality of this work is low. The KAN has been used for continual learning in ‘KAN: Kolmogorov–Arnold Networks’. Also, the RBF has been employed for the KAN in ‘Kolmogorov-Arnold Networks are Radial Basis Function Networks’.

The experiments are insufficient. For example, no Monte Carlo experiments measure the mean and deviation of the prediction accuracy. Also, there is no average prediction accuracy over all tasks in the CL experiments.

• Time Complexity and Computational Cost: The paper does not provide a detailed analysis of the time complexity and computational cost associated with the Gaussian kernel-based KAN method compared to traditional MLPs, like linear classifiers. This is a significant weakness as the practicality of the approach could be limited by increased resource requirements.
• Lack of Comparison with Traditional Continual Learning Methods: The paper primarily compares the KAC with prompt-based approaches. There is a lack of comparison with more traditional continual learning methods, such as EWC [1], which could provide a more comprehensive understanding of the KAC's performance relative to the state-of-the-art.

[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.

（2024/11/14: The parts marked in bold and italics are some newly added specific examples for comparison, which can be used as a reference for authors.）