Relaxing Representation Alignment with Knowledge Preservation for Multi-Modal Continual Learning

• This work lacks novelty by using the relaxed contrastive loss which in fact is quite similar to a common multimodal contrastive loss. Take an example of the well-known Contrastive Language-Image Pre-Training (CLIP) model, the multimodal contrastive loss is already presented in this work, though the modalities are different. Also, the contrastive loss is not connected directly to the continual learning problem. The contrastive loss is more intended to align between representations to boost up overall performance rather than retaining past knowledge.
• The other components and losses have been adopted from past works in continual learning e.g., Li et al., “Learning without forgetting” and Hou et al. “Learning a Unified Classifier Incrementally via Rebalancing”. The weights update allows two separate entities of fast and slow learners. The novelty of this approach is also limited as the previous work has already presented this approach (see Simon et al., “On Generalizing Beyond Domains in Cross-Domain Continual Learning”).
• The main contribution of this work is multimodal continual learning. However, except the relaxed contrastive loss, the other components do not directly contribute to multimodal continual learning problems. For instance, the paper can discuss the impact of distillation in the mixed feature from two modalities compared to only one. In the current state, the focus of this paper is not quite clear rather than obtaining better results in continual learning.
• In experiments, we need to also observe forgetting factors caused by acquiring new information. However, there is no such discussion and performance analysis in the experimental section. Assessing the model's effectiveness only through accuracy might be misleading, as the observed performance gains could be attributed to the introduced contrastive loss, while the forgetting score (i.e., the performance drop between tasks) might remain unaffected.

• The investigation into representation learning is inadequate. The clustering analysis provide little information about how the proposed method improve the learned representations.
  • as this work is not built upon large-scale experiments (say on 10 or 15 datasets), I would expect to see more concrete and in-depth analysis from the representation learning aspect to support the key ideas of their modeling.
• The ablation study on loss functions only addresses $L_{\text{relax}}$ and $L_{\text{preserve}}$, omitting other potentially significant aspects such as $L_{\text{distill}}$, which could be crucial for aligning currently learned representations with previous experience.
• The current method only constructs the memory dataset $\mathcal{M}_t$ naively, while the quality of $\mathcal{M}_t$ should be essential for the proposed method, potentially limiting its application to more general scenarios.
• The presentation needs improvement; for instance, Fig. 3 does not clarify which representation is visualized, and the details of the network parameters are ambiguous: are there two hidden layers with 2048 neurons each or 2048 neurons in total?
• The evaluation of model performance may be biased in favor of the proposed method, as evidenced by the ESMER method, which appears less impressive primarily due to its low accuracy on the latest task. For further details, refer to the Questions section.

1. This paper lacks a clear research motivation, and some claims in the paper appear contradictory. For instance, the abstract suggests that multi-modal continual learning is more challenging than single-modal continual learning, which is more commonly explored. Conversely, in the introduction, the authors claim that multi-modal data offer significant advantages for continual learning. Critically, both the challenges and benefits of multi-modal continual learning are not sufficiently clarified, raising questions about the necessity and significance of investigating multi-modal continual learning.
2. In the experimental section, the authors only compare their method with AV-CIL on the AVE dataset under class-incremental setting. While AV-CIL is specifically designed for multi-modal continual learning, the other methods compared are solely for single-modal continual learning. This is especially notable given that the authors have already mentioned several recent proposed methods in the related work.
3. Although the authors provide a detailed description of the datasets, it is unclear how these datasets were divided into class-incremental tasks for most experiments, except for the ablation study shown in Fig. 5.
4. The ablation study on loss weights is insufficient. The authors only conducted ablation experiments on the preserve loss and relax loss, and provided an explanation for determining the weight of the former. However, there is no ablation study for the distill loss, and it is entirely unclear how the authors set the weights for the relax loss and distill loss.
5. There has been extensive prior work attempting to develop dual-learner frameworks in continual learning. Could the authors explain how this work differs from previous efforts and specify the improvements and innovations it introduces?

1. In continual learning, forgetting is a crucial evaluation metric that should be compared. A commonly used metric for evaluating forgetting is backward transfer, as mentioned in [1]. It is recommended to compare multiple metrics in the comparative experiments, i.e., in Table 1.
2. The dual framework and contrastive learning have been applied in many works like MoCo[2], Co2l[3], and AV-CIL[4]. It appears that this paper primarily extends previous work incrementally and may not present a sufficiently novel contribution.

[1] Lopez-Paz D, Ranzato M A. Gradient episodic memory for continual learning[J]. Advances in neural information processing systems, 2017, 30.

[2] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9729-9738.

[3] Cha H, Lee J, Shin J. Co2l: Contrastive continual learning[C]//Proceedings of the IEEE/CVF International conference on computer vision. 2021: 9516-9525.

[4] Pian W, Mo S, Guo Y, et al. Audio-visual class-incremental learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7799-7811.