Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

1.The contributions are slightly limited. Integrating the models to balance different stages of tasks has been applied in incremental learning, such as [1]. 2.Many descriptions and notations are confusing. For example, in line 143 of page 3, a lack of explanation of “b” in “b|Y|”. In lines 179-185, the author seems to have used different words (features, representation, and embedding) to convey the same meaning, and I don't quite understand why the author did so. In Eq.(4), there lack of interpretation of , maxima value, and initial values in the summation notation. In line 257 of page 5, the description of “at most two backbones in memory” is confusing since I find there are at least three models (,, ) in memory according to lines 256. Finally, the authors are conflicted on which way to integrate, as shown in lines 256 and Algorithm 1. [1] Zheng Z, Ma M, Wang K, et al. Preventing zero-shot transfer degradation in continual learning of vision-language models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19125-19136.

1. The major concern is that, as shown in the ablation study in table 2, the main reason for the accuracy boost in DUCT could be attributed to task vector, which is an existing technique for addressing multiple tasks simultaneously. In reviewer's opinion, despite that the authors make certain modification on the weighting strategy, the paper fails to provide new insights to this technique on why it is effective in addressing DIL problem. One possible aspect the reviewer can think of is to explain why applying DUCT 'places the same class of different domains together', as suggested in the visualization in fig. 5.

2. The notation in the paper needs improvement. First, in equation 5, $\phi^m_i$ should not use $i$ as subscript as it indicates the index of the summation. Second, $\beta$ and $\gamma$ should be explained once they appear in line 300.

(1) Many notations are not introduced before hand. This makes the math hard to follow and the method ambiguous. For example, what is $\phi_i^m$ and $\alpha_\phi$ in equation 4? Furthermore, equations are not introduced in the correct order. For example, Eq. 5 depends on a value that is not defined until Eq. 7.

(2) It is unclear why the proposed method is better than model merging (intuitively).

1. Equations 4 and 5 attempt to build a unified embedding through weighted summation of model weights, raising questions about feasibility. Given the complexity and lack of interpretability in deep network weights, is this combination effective, or could it intensify conflicts within the feature space? More comprehensive theoretical analysis is required.
2. While the authors suggest that DIL could benefit applications like autonomous vehicles and face recognition, their experiments focus on classification tasks. Testing on more realistic applications could be more convincing.
3. The proposed DUCT method relies on model merging. However, as domains accumulate, previously merged models may become overly complex, containing information from multiple domains, while models from newer domains include only the latest domain data. This could lead to an imbalance between older and newer domains, creating potential confusion and forgetting.
4. The authors tested DUCT on ViT-B/16, but other methods, like S-Prompts, report results on the more powerful CLIP backbone. Does DUCT maintain its effectiveness on a stronger backbone?

1. There are many vision language models like CLIP that can perform zero shot. Can the author report the results of CLIP and CLIP related fine-tuning methods such as Coop, CoCoop, etc., to demonstrate the advantages of the article's method compared to these general models.
2. I wonder what data is used to calculate the class center of the pretrained model? If using pretrained data such as ImageNet, the first issue is how to ensure consistency with downstream task categories to calculate class center similarity? The second question is how to reduce the overhead caused by large number of categories and data size?
3. The author should conduct experiments on more backbones to demonstrate the effectiveness of the method, such as convolutional neural networks like Resnet.
4. Task similarity is calculated based on all categories, may it lead to the influence of some categories being overly magnified while the influence of others is ignored.