L3A: Label-Augmented Analytic Adaptation for Multi-Label Class Incremental Learning

Thank you for your constructive feedback and insightful suggestions. Below are our point-to-point responses:

---

W1: Comparison between WAC and learnable parameter-based methods.

---

To clarify, the Weighted Analytic Classifier (WAC) is a recursive analytical learning method, where we address class imbalance using a weighting mechanism. Since analytical learning relies on least-squares solutions, it operates without loss functions or gradient computations. This differs from many parameter-based weighted methods using gradient-based optimization, for example: KRT, which adds an Asymmetric Loss [1] in the loss function to handle class imbalance. CSC, which incorporates maximum entropy regularization [2] in the loss function.

In contrast, L3A directly computes the closed-form solution for network parameters (without backpropagation), requiring less computational time and lower memory than most parameter-based methods. L3A outperforms the existing learnable parameter-based methods in terms of performance as shown in the manuscript and we further validate the efficiency here. L3A can also improve over existing methods in terms of efficiency.

Table1: Comparison of methods in terms of runtime and memory usage in COCO B40C10 benchmark.

---

W2: Lack of comparison with class imbalance methods.

---

We have selected several class imbalance methods for comparison in the manuscript, including BiC and PRS, which are originally designed for single-label long-tailed continual learning. However, these methods perform poorly in multi-label scenarios. Another compared method KRT is designed for the class imbalance issue in MLCIL which L3A also outperforms. Detailed experimental results can be found in Table 1 of our paper. If you believe we need to include more relevant techniques, could you please provide some methods we need to include? We are open to include the experiments accordingly.

---

W3: Unclear pretrained parameters for comparison methods.

---

In this paper, all the backbones including TResNetM and ViT-B/16 are pretrained on ImageNet-21k. We will clarify this in the manuscript.

---

W4: Large performance gap between baseline and mislabeling in Table-4.

---

1. Performance Gap, the baseline method in Table-4 of paper is a continuous learning method that does not employ any anti-forgetting method. Only the classifier is trained with a frozen backbone in the baseline and the baseline could suffer from catastrophic forgetting, resulting in a very low Last mAP. With the WAC and PL, L3A can achieve high performance with less forgetting.

2. Label Correction, thank you for pointing the mislabeling problem. We will correct "WAC" to "L3A" in the manuscript.

---

S1: The motivation behind each module.

---

Our paper proposes two main modules, and their motivations are mentioned in the Introduction part. We provide additional clarification:

1. To address the challenges of class imbalance and privacy protection, we introduce a replay-free recursive analytic continual learning method. However, since this approach struggles with class imbalance, we further propose the Weighted Analytic Classifier (WAC), which employs a sample-specific weighting mechanism that adaptively adjusts the importance of samples weights based on class frequency. This correction ensures balanced least-squares optimization

2. To tackle the label absence problem, we adopt a pseudo-labeling strategy (PL module) to generate labels for past tasks. This allows WAC to utilize complete label information during training.

---

S2: Lack of experiments with different backbones.

---

We add experiments with L3A on pre-trained ViT-B/16. The performance of L3A on ViT-B/16 is slightly worse. The reason could be that vanilla ViTs achieve the less local attention than CNN networks when dealing with complex multi-label images (e.g., scenes with many small objects) [3].

Table2: Performance results on different backbones

[1] T. Ridnik, et al, "Asymmetric loss for multi-label classification," in ICCV 2021.

[2] K. Du, et al, "Confidence self-calibration for multi-label class-incremental learning," in ECCV 2024.

[3] Z. Liu, et al, "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows," in ICCV 2021.

Thank you for your insightful feedback and constructive suggestions. Below, we provide a point-by-point response to your questions and concerns.

---

Q1: Ambiguous confidence scores and refinement mechanisms in the PL module.

---

1. Ambiguous confidence scores, To clarify this issue, we validate our L3A with different thresholds. The results are shown below and we find that the change of results from 0.65 to 0.80 is not significant. This demonstrates that PL could be robust to ambiguous confidence scores since varying thresholds do not have large effect.

Table1: The threshold studies on confidence threshold ($\eta$) in COCO dataset.

2. Self-Correction, During the learning, pseudo-labels are not treated as static annotations. Instead, they are continuously re-evaluated as new data is introduced. This iterative correction mechanism helps mitigate the impact of early-stage pseudo-labeling errors.

---

Q2: The choice of different weighting mechanism in WAC module.

---

In this paper, we use the 1/sqrt(f^(k)) to compute the weight. To validate its superiority, we compare different weighting mechanisms and weighting strategy using 1/sqrt(f^(k)) achieved the best performance. Due to smoothing effect of 1/sqrt(f^(k)), WAC can maintain a good balance against extreme class imbalances. We will include this study in the manuscript.

Table2: Different weighting mechanism in WAC module.

---

Q3: Analyze the generalization of the L3A method in more diverse multi-label settings.

---

Regarding dataset selection, we followed previous multi-label class incremental learning studies [1,2] by using COCO and VOC datasets as benchmarks. The data for multi-label classification needs to contain multi-classes within a single sample, and "multiple objects in every single image" is easy to meet this requirements. We acknowledge that real-world multi-label data can extend beyond object tags to include attributes like time, mood, action, or fine-grained details, but we do not find a dataset having such features while meeting the requirement of multi-classes in a sample (ImageNet-C performs different noises on an image to generate several samples).

For the scenario that multi-labels from different aspects, our L3A can also work in theory. When denoting the fine-grained tags as distinct labels, there is less difference between the labels coming from different objects in the training of WAC. Moreover, the buffer layer could enhance separability of features [3] then help distinguish originally similar features during fine-grained classification.

We are willing to include the validation of L3A on such a dataset, but unfortunately, we did not find a suitable dataset which has different aspects and meets the requirement of multi-classes in a single sample with limited time. Since we may overlook any relevant datasets, could you please designate a suitable dataset? We are open to include the validation accordingly.

---

Q4: Comparison with open-vocabulary detectors and taggers methods.

---

In this paper, we focus on the class-incremental continual learning, where the goal is for the model to learn new classes while retaining previously learned knowledge. We need to achieve good performances on both old and new datasets. In contrast, open-vocabulary detectors and taggers typically evaluate on data outside their training distribution (unseen data) to test generalization ability. The motivation is quite different from the continual learning and it may be not appropriate to directly transfer L3A to this task.

[1] S. Dong, et al, "Knowledge restore and transfer for multi-label class-incremental learning," in CVPR 2023.

[2] K. Du, et al, "Confidence self-calibration for multi-label class-incremental learning," in ECCV 2024.

[3] K.-A. Toh, et al, "Between classification-error approximation and weighted least-squares learning," TPAMI, 2008.

Thank you for your insightful feedback and constructive suggestions. Below we provide point-by-point responses to your comments:

---

W1 & Q1: Uniqueness of challenges in MLCIL problem.

---

Following prior research [1,2], we consider label absence as a unique challenge in MLCIL. Unlike single-label class-incremental learning (SLCIL), MLCIL inherently suffers from missing labels of previous and future labels during training. This increases the difficulty of learning new classes while preserving knowledge of previous ones since learning with only current labels could erase the previous knowledge, making MLCIL fundamentally different from SLCIL in handling label distribution shifts.

For class imbalance, while it is a common issue across machine learning tasks, it has a more severe impact in MLCIL due to catastrophic forgetting. In MLCIL, class distributions are imbalanced not only within each learning phase but also across different incremental stages, where multiple labels coexist per sample, making the imbalance problem even more complex. This leads to the severe under-representation of certain classes as training progresses. As a result, simple class weighting methods often fail to mitigate forgetting effectively in MLCIL, necessitating additional analysis of the class distribution shifts across different incremental learning phases.

For privacy protection, it is a challenge across CIL and existing MLCIL approaches rely on data replaying [1,2], which are incompatible with real-world scenarios when data storage is restricted. This makes exploring exemplar-free methods particularly crucial in MLCIL.

We will further clarify these challenges in the manuscript.

---

W2 & Q2: Parameter sensitivity analyses of $\gamma$ and buffer layer size.

---

We include the parameter analysis as below. For $\gamma$, as it increases, the performance of L3A first improves and then declines in Table 1. A value of $\gamma$ (1000) achieves consistently good results.

Table1: The ablation study on regularization term $\gamma$.

For the buffer size, in Table 2, the accuracy of L3A improves with an increase in buffer layer size, but the improvement becomes negligible once the size reaches 4096. The size of 4096 is sufficient and a larger size entails more computation on matrix inversion.

Table2: The ablation study on buffer layer size.

---

W3 & Q3: PL module effectiveness analysis.

---

We include the analysis as follow. "The PL module to address the label absence problem. However, notably, we found that the improvement of using PL upon WAC is less significant in our experiments, while directly using PL upon baseline introduces large improvement. The reason could be that, during the training of WAC, it retains information of previous classes via $R_t$. Then adding the PL is less effective. Nonetheless, the use of the PL module could be still beneficial when using upon both baseline and WAC."

---

S1: Refine the threshold strategy and analyze limitations.

1. Threshold strategy. To refine this threshold strategy, we explore and validate a dynamic threshold approach, where the threshold is adjusted based on the number of pseudo-labels to be generated in each phase. The result of this approach is denoted as "Dynamic" and is not better than current results. It can add additional hyperparameters of target number of pseudo-labels, which may be unavailable in practice. We will further explore to improve the threshold strategy in future.

Table3: Studies on $\eta$

2. Limitations analysis. L3A utilizes a frozen backbone and updates only the WAC. This could limit the improvement of feature extraction. We will explore to strengthen the backbone's representation to further improve L3A.

[1] Dong, S., et al. Knowledge restore and transfer for multi-label classincremental learning. in ICCV, 2023. [2] K. Du, et al, "Confidence self-calibration for multi-label class-incremental learning," in ECCV 2024.

Thanks for your review and valuable comments. Here we address your concerns individually as follows:

---

W1: Recursive updates in Eq. 13 may be computationally intensive.

---

The computational cost of our method primarily comes from the matrix inversion in Eq. 13 with the complexity of $\mathcal{O}(d^2N_t + N_t^2d + N_td|C^{1:t}|)$ where N_t is the number of samples at stage t and |C^{1:t}| is the total number of classes up to stage t. The complexity scales up linearly with the dataset size and L3A provides training with one-epoch closed-form solution. This makes L3A more efficient than existing iterative BP optimization. As shown below, L3A is 10x faster than KRT.

Table1: Comparison of methods in terms of runtime and memory usage in COCO B40C10 benchmark.

---

W2: Fixed confidence threshold in PL.

---

To address this weakness, we explore and validate a dynamic threshold approach, where the threshold is adjusted based on the number of pseudo-labels to be generated in each phase. The results of this approach is denoted as "Dynamic" and is not better. It can add additional hyperparameters of target number of pseudo-labels, which may be unavailable in practice. We will further explore to address the fixed threshold in future.

Table2: The threshold studies on confidence threshold ($\eta$) in COCO dataset.

---

S1: Test on long-tailed datasets.

---

We follow [1] to construct a long-tailed dataset (LT-COCO) by applying a power-law decay sampling method on the COCO dataset then validate on it. Although L3A's performance slightly declined under the long-tailed distribution, it still outperformed other methods, demonstrating its robustness in multi-label continual learning scenarios.

Table 3: Comparative results on the LT-COCO dataset (classes are randomly shuffled).

---

S2: Clarify the buffer layer.

---

The buffer layer is used to project the model's output to a higher-dimensional space. According to Cover's theorem [2], projecting features into a higher-dimensional space via nonlinear projection functions can improve linear separability. Following ACIL [3], we use a randomly initialized linear projection in this paper and it is simple and effective. We will further clarify this in the text and explore other type of buffer in future.

---

Q1: Choosing weighting strategy.

---

In this paper, we use the 1/sqrt(f^(k)) to compute the weight. To validate its superiority, we compare different weighting mechanisms and weighting strategy using 1/sqrt(f^(k)) achieved the best performance. Due to smoothing effct of 1/sqrt(f^(k)), WAC can maintain a good balance against extreme class imbalances. We will include this study in the manuscript.

Table4: Different weighting mechanism in WAC module.

---

Q2: How Theorem 3.1 helps MLCIL.

---

Our work focuses on the MLCIL where we cannot directly apply standard ridge regression (Eq. 10) to optimize all data at once. Here, we extend standard ridge regression into the recursive analytical learning method presented in Theorem 3.1. This allows us to update the classifier through analytical solutions at each incremental learning phase, effectively addressing the MLICL problem.

Also, Theorem 3.1 incorporates two key components of our approach to address issues in MLCIL:

1. PL (Eq. 3) generates pseudo-labels for past phases, solving the label absence issue.
2. WAC uses weighted iterative least squares to update classifier, addressing both class imbalance and privacy protection concerns. Theorem 3.1 provides theoretical support for L3A by proving optimal closed-form solution sample-specific weighting for MLCIL.

[1] W. Tong, et al, "Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets," in EECV 2020.

[2] K.-A. Toh, et al, "Between classification-error approximation and weighted least-squares learning," TPAMI, 2008.

[3] H. Zhuang, et al, "ACIL: Analytic class-incremental learning with absolute memorization and privacy protection," in NeurIPS 2022.