Persistence Homology Distillation for Semi-supervised Continual Learning

A1. We will make substantial revisions to enhance the overall clarity and readability:

(1) Provide detailed results to verify the description of our advantages. As shown in Table 1. of attached PDF file, we provide additional comparison of the forgetting rate (BWT) among strong baselines and our method to demonstrate its effectiveness in alliviate catastrophic forgetting. Our method achieves 3.4% forgetting mitigation with 0.9% accuracy increasement on average compared with DSGD, and 18% forgetting mitigation with 6.15% accuracy increasement on average compared to NNCSL. Moreover, the effeciency comparison is also provided in Table 2, and the stability contrast to noise interference is supplemented in Table 4.

(2) Redesigned Figure 4 to make it more informative and readable. As shown in Figure 1 of the attached PDF file, we conducted additional ablation experiments on larger distillation loss weight $\lambda$, specifically {1.2, 1.5, 2} to further verify the effect of proposed homology distillation loss in overcoming catastrophic forgetting. We selecte the optimal weights $\lambda=1$ for the three dataset that balance the forgetting (BWT) and average incremental accuracy (Avg) best. Additionally, we expand the ablation study of persistence homology dimension to determine appropriate homology featuresfor distillation, using 0-simplex for the simple dataset CIFAR10, and 0,1-simplex for complex data, such as CIFAR100 and ImageNet-100. Lastly, we added legends to each diagram for better readability.

A2. Selecting an optimal ratio of labeled to unlabeled data in a limited memory buffer to represent an entire dataset is an open problem in semi-supervised continual learning with different degrees of supervision. We firstly offer experimentally valid allocation ratios for several benchmarks. Regarding the limited improvements of Table 1, we enhance the results by providing the degree of forgetting (BWT). We further provide evidence of the efficiency and stabilization of PsHD compared to newly strong baselines.

(1) First consideration of ideal allocation labeled vs. unlabeled ratio in SSCL. Our experiments indicate that indiscriminate utilization of unlabeled data can diminish accuracy. As shown in Table 2 of the PDF file, the introduction of logits distillation on unlabeled data (iCarL_U) leads to 4.97% decrease on average compared to distillation only on labeled data (iCaRL). These diminishes are different across supervision ratios, since higher supervision can ensure confident representation of unlabeled data. Therefore, the ideal allocation ratio remains unresolved in existing SSCL methods. We provide experimentally valid allocation ratios for labeled data allocation ratios between 0.6 and 0.9 for a general reproducibility and transferability.

(2) Enhanced forgetting rate in Table 1 and better knowledge distillation on unlabeled data. As shown in Table 1 of the attached PDF file, our method surpasses the NNCSL by 6.15% with 18% forgetting on average, outperforms DER_Fix by 1.63% with 6.52% forgetting on average, and exceeds DSGD by 0.89% with 3.35% forgetting on average. Moreover, our methods exhibit better adaption in distillation knowledge in unlabeled data, achieving a 35.2% reduction in training time, a 60.9% reduction in examples buffer, and a 28.4% memory size reduction compared to the best indexes in CIFAR100_5%.

(3) More stable distillation for SSCL. The training stability of our method can be verified through comprehensive ablation experiments. As shown in Table 4 of the attached PDF file, we add Gaussian noise with weight {0.2,1,1.2} to five typical distillation-based continual learning methods: iCaRL, Foster, Podnet, LUCIR, and R-DFCIL. Our method demonstrates stability in both the degree of forgetting and accuracy, even under higher noise levels. These results are consistent with our Theorem 1.

To sum up, our method PsHD effectively mitigates catastrophic forgetting of unlabeled data and achieves a better trade-off between adapting old tasks and preserving new tasks in SSCL.

A3. We include more detailed experimental results to emphasize the strengths of our method and validate our conclusions. Additionally, to ensure rigor, we will also improve the modifier of experiment results and illustrate the limitations at the same time. Such as use 'comparable' or 'competitive' to illustrate the accuracy improvement less than 1%, and clarify the limited improvement on 500 replayed samples.

A4. We supplement more experiments to validate the effect of distillation weight lambda value and persistence dimension H0 and present the results in Figure 1 of the attached PDF file. Higher distillation weight results in lower forgetting (BWT) and higher average incremental accuracy (Avg). The optimal persistence homology distillation weights are also highlighted in each benchmark. Additionally, we improve the arrangement of Figure 4 for better readability.

A1. Traditional CL methods are not always effective for unlabeled data, as they assume the knowledge of previous models is accurate. This specific challenge of SSCL has been explored in previous SSCL methods (NNCSL, DSGD, etc). As suggested by the reviewer, our ablation experiments provide additional verification of this motivation for overcoming catastrophic forgetting of unlabeled data.

(1) Negative influence with Traditional KD [1] on SSCL. The performance decreases when employing traditional KD [1] directly on SSCL, which is the iCaRL method in our comparison method. As shown in Table 2 of PDF file, the introduction of logits distillation on unlabeled data (iCarL_U) leads to 4.97% decrease on average compared to distillation only on labeled data (iCaRL). We also conduct pseudo-label distillation on unlabeled data based on DER, resulting in 5.23% accuracy decrease (DER_U) on average. This indicates that inaccurate representation preservation has a negative effect on SSCL.

(2) Diminished effectiveness of KD variants from [2] for SSCL. Table 3 of PDF file provides comprehensive ablation experiments of traditional KD [1] and its more recent variants from [2] on our SSCL problem. The effectiveness of logits and feature distillation diminishes, especially as tasks become harder and more numerous, such as in CIFAR100_5%.
Although relation and topology distillation are relatively effective for SSCL, their stability, computation costs, and accuracy require better balancing.

Given the limitations of existing KD methods in SSCL, an adaptive knowledge distillation strategy for SSCL should be further proposed.

A2. We acknowledge we didn't seen the reference [3] TopKD, so the citation and comapration are missed even our method is better. The idea of topology feature distillation is similar while ours requires less computation and is more adaptive to SSCL.

(1) Lower Computation Costs. Heavy computation usually accumulates during the whole learning process. Our lower superior computation cost is mainly because of smaller number of persistence homology points in the PH, which is $u$ in Equation (5). The number of $u$ is proportional to simplices number $n$ since HP complexity is $\mathcal{O}(n^{2.5})$. $n$ is hard to approximate, while we can compute the its upper bound in our PsHD $\sum_{t=0}^{M/b}\binom{M/b}{t}$ and TopKD $\sum_{t=0}^{M}\binom{M}{t}$, where $M$ is the replayed samples in each batch and $b$ is approximate to class number. The ratio is $2^{M-M/B}$, indicating a significant reduction of distillation computation.

(2) Better Adaptability to Unlabeled Data. As shown in Table 3 of PDF file, applying TopKD to CIFAR10_5% based on iCaRL results in a decrease in accuracy from 79.2% to 78.7%, indicating TopKD's reliance on accurate representations. In contrast, our method improves iCaRL and outperforms TopKD by an average of 2.28% across four benchmarks. Furthermore, our training time on CIFAR10_5% is 7.1 hours, compared to TopKD's 16.5 hours, reducing training time by 56.9%.

A3. $L_{Cl}$ is the continual learning loss, which corresponds to the persistence homology distillation loss in our method. It varies among different types of CL methods. We will change $L\_{Cl}$ to $L\_{hd}$ to avoid confusion.

A4. According to the loss of Equation (5), PsHD is back propagated along with the persistence point $u$ in the persistence diagram. So the gradient backpropagation flow is: $L_{hd}\rightarrow u=[u_s,u_e]\rightarrow[dia(\sigma_s), dia(\sigma_e)]\rightarrow [dia(v_1,v_2), dia(v_1,v_2,v_3)]$.

(1) Illustration of differentiating PsHD. $u=[u_s,u_e]$ represents a lifespan of one simplex $\sigma_s=\\{v_1,v_2\\}$ exists in $u_s=dia(\sigma_s)$ and disappear or fused to higher dimension simplex $\sigma_e=\\{v_1,v_2,v_3\\}$ in $u_e=dia(\sigma_e)$. $dia(\sigma)$ is the diameter of simplex $\sigma$, which is the maximum distance of any two samples in $\sigma$, and $v_i$ represents the sample. The loss is finally optimized to update the representation.

(2) End-to-End training of our PsHD. The origin Gudhi hasn’t been developed on pytorch, so the end-to-end training is unprocurable directly. Our method achieves this progress by making the differentiation on PsHD along the chosen samples.

A5. We supplement further ablation experiments on noise interference. As shown in Table 4 of attached PDF file, we add Gaussian noise with standard deriviation {0.2,1,1.2} on 5 distillation-based continual learning methods iCaRL, Foster, Podnet, LUCIR, and R-DFCIL. It can be seen that our method is stable on the forgetting degree and accuracy despite a higher noisy interpretation, which is consistent with the conclusion of Theorem 1.

A6. The unlabeled data is part of the memory buffer, with the remaining data being labeled. PsHD loss is applied only to the unlabeled data, while labeled data is distilled through the traditional KD. Thus, our method is compatible with the traditional KD.

Necessity of replaying both labeled and unlabeled data. Labeled data provides accurate representations for discrimination, while unlabeled data has the potential to represent the entire dataset. Traditional knowledge distillation strategies are usually efficient in labeled data. Our PsHD loss is applied only to the unlabeled data, reserving intrinsic information and reducing computation costs, as illustrated in Answer 2.

A7. The semi-supervised loss (SSL) used in our method follows the approach from DSGD, which is based on the SSL method Fixmatch. The SSL loss is not the primary motivation for our PsHD, as its effects in SSCL have already been explored in DSGD. We will clarify this point in the Section 3.2.

A8-10. The simplicial complex corresponding to the diagrams in Figure 2 is simplified, which leads to persistence feature inconsistency. We will update this figure to ensure greater consistency. We will correct the two typo issues.

A1. The limited replied 500 samples restrict the improvement space, while our method reduces the degree of forgetting (BWT) by a substantial and balanced margin compared to the newly strong methods. Furthermore, our methods demonstrate effective utilization of replayed unlabeled samples, as evidenced by the substantial improvements over strong baselines when the memory buffer size is increased.

(1) Lower forgetting degree with comparable incremental accuracy. As shown in Table 1 of the attached PDF file, our method surpasses the NNCSL by 6.15% with a decreased average BWT of 18%, outperforms DER_Fix by 1.63% with a decreased average BWT of 6.52% forgetting on average, and exceeds DSGD by 0.89% with a decreased average BWT rate 3.35% in the four benchmarks.

(2) More balanced forgetting degree. The BWT our method is under 27 in these benchmarks, avoiding the high forgetting rate observed with NNCSL. The overall BWT is lower than state-of-the-art with even higher incremental accuracy. These results show that our method PsHD achieves a better trade-off between adapting old tasks and preserving new tasks.

(3) Sufficient utilization of replayed unlabeled samples. As shown in Table 1, when the number of replayed sampled increases to 5120, our method leverages the topology structure information of the unlabeled data more effectively, surpassing NNCSL and DSGD by 3.28% and 2.07% on average across the four benchmarks, respectively.

Table 1. Comparison on CIFAR10 and CIFAR100 with 5120 examples replayed

A2. Our method exhibits better performance, fewer memory budgets, and lower computation costs compared to these strong baselines. We are sorry for not fully expressing these advantages. The detailed verification is illustrated as follows.

(1) Highest accuracy with aligned memory size. To verify the effectiveness and efficiency of our PsHD, we provide a comprehensive comparison of the overall resource overheads in Table 2, including the computation time and storage costs. The experiments are conducted on the same server for a fair comparison. Referring to the storage costs with aligned example 5120, our method achieves the highest accuracy at 47.5% with the aligned memory size of 30.8 MB. It also surpasses the NNCSL by 1.5% while using 45% less memory, due to our smaller model buffer.

(2) Optimal solution in terms of both effectiveness and efficiency. When reducing the example size to 2000, our method still achieves the best performance 46.0% while reducing training time by 35.2%, examples buffer size by 60.9%, and memory size by 28.4% compared to the best indices, which do not belong to the same method. Therefore, while existing strong baselines excel in different aspects —NNCSL in accuracy, DER_Fix in training time, and DSGD is relatively balanced computation costs and accuracy, our method achieves the overall state-of-the-art performance with the least resource overheads and the highest accuracy.

Table 2. Effectiveness and efficiency comparison on CIFAR100_5%. The memory size is the sum of the storage of model parameters and replayed examples.

We thank the reviewer for the insightful suggestion to highlight the efficiency comparison, and we have reorganized our expression to highlight the superiority of the proposed method.

A3. One limitation of the proposed method is that the partial computation of persistence homology is conducted on the CPU, which increases training time. We have include this limitation in the Conclusion.

Further improvement plan on limitation. Although our acceleration algorithm mitigates this issue without disrupting end-to-end training, there is still room for improvement. The primary reason for this limitation is that the Python package Gudhi has not been developed for GPU use. To address this problem, we plan to implement persistence homology computation on the GPU, which is expected to further reduce training time.