Dual-Modality Guided Prompt for Continual Learning of Large Multimodal Models

1. Although the experiment improves the performance and inference speed, the proposed method involves modality-specific prompts for each task, which is too simple compared to existing work that devises advanced prompt strategies in visual scenarios. Simultaneously, they lack of comparison with the amount of prompt-based methods. Such as: DualPrompt [1], L2P[2], CODA-Prompt[3].
2. There exist some typos in the paper:
   1. in line 100, `prpredominant'.
   2. in line 128, ... set `prompt of prompts' ...
3. The author proposes the setting of refrains from computation expansion in proportion to the number of tasks. Whether means we can continuously learn the sequential data in one model and the performance will continuously improve. In other words, how many tasks can the proposed method effectively handle within one model?
4. In the experiment, there is a lack of results that compare one task in the continuous process, i.e. compare the performance at the time axes, which directly reflects the transfer capability when more previous knowledge is learned.
5. There is no difference between the two items in equation 12 with the add operation.
6. How does the proposed method assess forgetting? Does it require saving a lightweight projection layer for each task, or should the projection layer from a previous task be re-tunned after learning a new one?
7. In Line 203, why does the encoder of visual E_I and textual E_T in CLIP realize the mapping of E_I(·) : R^{n_v ×d_v} →R^{d_v} ,E_T(·) : R^{n_t ×d_t} →R^{d^t}, which should exist error description?

[1]. DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

[2]. Learning to Prompt for Continual Learning

[3]. CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

1. The baseline lacks a comparison with other prompt learning methods.
2. Complexity in configuring prompt numbers and selection features may limit broader accessibility without further simplification or automation.
3. ModalPrompt needs to convincingly differentiate itself from prior work in prompt-based continual learning, likely through robust comparative experiments and ablations.

The paper is difficult to read, as it presents simple ideas in an abstract and complex manner. It requires a substantial revision before one can properly evaluate its soundness and contribution.Thus, I do not believe it is ready for publication at ICLR. Here are some areas of confusion I encountered:

• In line 161, it states “The characteristic of LMM continual learning includes: …” It is unclear whether the authors refer to a general consensus on LMM continual learning or their specific proposals.
• The summation in Eq.(3) lacks a dummy variable. Are you summing over individual prompts within a set for a specific task $t$?
  • Consider using $\bar{x}$ for the average of prompts, as bold symbols could be confusing since they typically represent vectors.
• In line 201, the projection should be defined as $\text{Proj}_v(\cdot):\mathbb{R}^{d_v}\rightarrow\mathbb{R}^{d_t}$.
• In Eq.(7), What is $X_p$? Is it the collection of all prompts? It's unclear how prompts are selected in your process.
  • One possible understanding: You have $N$ prompts for each of the $T$ tasks, so $T\times N$ in total. The selection is performed over all the $T\times N$ and produce $k$ most relevant ones.
• Line 242 states, “To enhance knowledge transfer, the dual-modality features could serve as guiding cues for prompts to accurately get close to multimodal distributions of current task in feature space.” What are the dual-modality features? Are they the features of the current task? What do you mean by “multimodal distributions”? I don't think those terminologies are self-explanatory and commonly used in the field. Why is the closeness to the distribution helpful in enhancing knowledge transfer?
• Eq.(9) abuses the symbol $\mathbf{x}^t_p$ for prototype features, the same term is used for the “prompt features” in Eq.(3).
• In Eq.(10) what are the definitions of $\alpha^{\le t}$ and $\beta^{\le t}$? What is the shape of $\tilde{X}^t_p$?
• In line 265, where do you define the parameters $\theta_p^t$ of prototype prompts?
• In Table 1, what is the metric of the first two methods?
• In Table 2, what do $B_i$ and $M_i$ represent in the second row?
• Previous text implies that “number of selection prompts $k$” refers to selecting the top-k most similar prompts. However, by line 448-455, it seems $k$ refers to the number of sets of prototype prompts. Which is the correct understanding?
• Line 456 is confusing when it mentions “choosing three sets of prototype prompts.” Based on subsection 3.2 (line 237, “we term set of prompt for each task as prototype prompts”), shouldn’t the number of prototype prompt sets match the number of tasks, which is eight?
• In Fig.5, it is not clear what quantity is plotted. Is it the average similarity between the prototype features and task features across all in task samples and targeting prototypes?

In addition, the visualization subsection at P.10 provides little information. Cherry-picking examples do not represent the overall behavior of your model. and I don't understand how these examples support the claim that your model retains previously learned knowledge.

1. The proposed method lacks substantial novelty, as prompt learning has already been widely used in fine-tuning pre-trained vision-language models in the continual learning setting.
2. The baseline is too weak, thus the effectiveness of the method is not very convincing. For example, the baseline accuracy of zero-shot on the REC task is 0.00.