Diversity-aware Continual Learning with Latent Knowledge Hypergraph

1. Some parts of the explanation are not clear enough, e.g., the third challenge in the last paragraph in section 2, that c is a non-differentiable function. I am confused about how it is solved and what is the explicit definition of c. Maybe it was mentioned in the paper, but I didn't see it. Hopefully, the author will explain it in a more prominent place.
2. As mentioned in section 3.2, the novel knowledge discovery and hypergraph growth are based on the k given by the evolutionary search. So I wonder if this step introduces too much time consumption and reduces training efficiency compared to other comparison methods.

I am not convinced that capacity saturation (under parameterization) is a prevalent issue in continual learning (CL), given that most modern deep neural networks (DNNs) are sufficiently large or over parameterized to accommodate knowledge from a large sequence of tasks. Many CL methods based on a single DNN (5-10M parameters) achieve competitive performance in large scale memory constrained CL problems e.g., ImageNet classification task. Rather capacity saturation might be a limitation of specific design choices for CL models.

Lack of experiments on large scale problem settings with large datasets e.g., ImageNet (1000 classes, 1.28M images, higher number of tasks) which may reveal if capacity saturation assumption holds. Given the capacity saturation motivation of this work, chosen experimental settings based on small datasets (small number of tasks) may not exhibit capacity saturation and thus may not need dynamic expansion mechanisms to solve these small tasks. Many CL methods work only on these toy datasets and fail for larger datasets. Should test on multiple distributions rather than only ones that are as non-iid as possible. Many baselines are old and extremely weak, e.g., EWC fails to perform well for incremental class learning and for large-scale datasets.

Also efficacy of CL models on small datasets is not predictive of their efficacy on large datasets. So more challenging settings may be required to verify HyperGKL’s effectiveness.

It is unclear how task sequence is constructed and how data ordering (with or without class boundaries) intensifies catastrophic forgetting. It is unclear how HyperGKL performs in different data orderings e.g., class incremental learning (with class boundaries, severe forgetting) and IID (classes revisit, less forgetting).

Comparisons do not include rehearsal-based or stronger dynamic expansion SOTA CL baselines. It is unclear if HyperGKL provides performance competitive to recent SOTA methods. Given all the complexities in designing HyperGKL, it is worth checking the advantages and disadvantages of this method compared to other simple CL methods with a single DNN.

Computational overhead (number of SGD steps or iterations) and/or training time have not been reported. Besides memory, compute is also critical for real world applicability. Continual learning needs to demonstrate some benefit for real-world applications to really have impact.

Performance improvements over comparison methods are negligible even on these toy datasets.

There is no justification for not testing on largescale datasets. All experiments were done on an A100, and people have been doing continual learning on larger datasets like ImageNet on far weaker GPUs over the past 7 year or so.

• The setting in which capacity saturation and parameter efficiency really matters is, as mentioned in the manuscript, when CL methods are deployed on edge devices with little memory. This is the main reason why replay-based CL methods are seldom compared to in the submission but I disagree. If using a replay buffer is too costly, what's to say that expanding the model size (however efficiently that might be) is not too costly? Replay based CL methods can be understood as having the best parameter efficiency (as the model size does not grow) and tackling the capacity saturation issue with past task samples in the replay buffer. I think more comparisons both in terms of concepts and experiments should be done against replay based CL methods to really justify the proposed trade-off in the paper.

Content: The results on accuracy are marginally better, but the results on parameter size do not seem to support the claim that the method achieves better balancing between generalisation power and parameter compactness.

Form: Overall, I found the presentation of the paper pretty poor. The formatting lacks consistency and restrain, needlessly convoluted language sometimes hides the point, and the work isn't contextualised enough as the related works section is put in the supplementary material, which makes the baselines section seem very ad hoc. Putting the results on half the datasets Here are examples of specific concerns that highlight my impression on the presentation:

• Too much bold, italic, bold+italic and even underlined formatting that isn't always necessary and actually distracts from reading, I understand this might be intended to help the reader skimming over the paper but it achieves the opposite
• References section not cleaned up as it contains duplicates
• The results on half the datasets are put in the supplementary material
• Explanation of eq 6 lacks clarity
• Unclear syntax and typos (I didn't list all of them but give some examples):
  • after eq1: without the access of -> without access to?
  • to summary -> to summarise
  • "til" isn't a proper word
  • "without the access of T1, T2..." -> without access to
  • typos are normal, english isn't my first language so I understand, but a publication should be proofread to avoid these