Vision and Language Synergy for Rehearsal Free Continual Learning

Q1/W1. The comparison mentioned in lines 183-184 between "the photo of car" and "the photo of cat," stating they have 15/16 similarity but 1/16 dissimilarity, is confusing. The authors compared the similarity between these two prompts on a letter-by-letter basis and concluded that they are similar. However, the similarity between two prompts should not be evaluated in such a manner. After being processed by the tokenizer, the embeddings of the two prompts are not similar, as their meanings differ significantly.

Our Response: Thank you for your concern.

• About the writing error in lines 183-184. We apologize for our writing error. We have revised the sentences in our revised manuscript. The sentences are revised into: "LGCL produces class prototype $L_c^n$ by encoding string ”the photo of class name”. However, the prototypes could be misleading due to high similarity between different classes, e.g. the prototype of class ”Great White Shark” has 0.9 cosine similarity to the prototypes of class ”Tree Frog” and ”Iguana”, please see Appendix D.2. "

• Our statement is supported by our empirical study on the high similarity of language embedding encoded by CLIP as presented in Appendix D.2.

Q2/W2. The proposed method relies on using ChatGPT to generate descriptive terms for each class and requires an additional Sentence Transformer to obtain embeddings. The introduction of these extra resources creates an unfair comparison with other methods and limits the practical applicability of the approach in real-world scenarios.

Q3/W3. Although the proposed method achieves state-of-the-art performance on multiple datasets, the improvement is minor on some, such as the comparison with HiDe-Prompt on the CUB200 dataset. This is particularly relevant given the substantial additional resources, like ChatGPT, required by the method, which other approaches do not utilize. Moreover, this suggests that the performance gains on fine-grained datasets may not be significant, as the generated descriptive terms for each class are quite similar.

Our Response: Thank you for your concern.

• We added a new experiment and analysis on the "class name" as descriptors where we have no class descriptors generated by GPT or other LLMs. Thus we utilize equal resources as language-guided prompting such as LGCL and even less resource than ConvPrompt and GMM. Our experiment results as presented in Appendix D4 shows that our method outperforms the existing SOTAs by a significant margin in 3 datasets despite not using the descriptors by LLMs. It proves that our method can substitute class descriptors with the class name to generate prompts.

• Regarding comparison on CUB dataset, It may look small difference between our method and HiDe-prompt in CUB dataset if we see from final task measurements i.e. 1.24% FAA and 0.6% FFM. However, in terms of cumulative measurements, our method outperforms HiDe-Prompt with a significant margin i.e. 3.3% CAA and 1.36% CFM. The table A19 in Appendix G shows the complete numerical result of the CUB dataset. The table shows that except in the last 2 tasks, LEAPGen outperforms HiDe-Prompt by a significant margin i.e. 2.8-5.5%. In addition, HiDE-Prompt suffers from a significant drop after the first task i.e. 6% accuracy drop, while LEAPGen only experiences a 3% drop. Thus, in earlier tasks, LEAPGen outperforms HiDePrompt with more than a 5% margin. In terms of forgetting measures, LEAPGen consistently experiences lower average forgetting than HiDe-Prompt in every task with a 1-2.5% margin. In the earlier tasks, It is even close to zero. Those numerical result shows that LEAPGen outperforms HiDE-Prompt in CUB dataset.

• Please see the running time analysis in Table A9 in Appendix D9 that shows HiDe-Prompt requires 12x higher running time than our proposed method (3.94h vs 0.32h) for the CUB dataset. This drawback is important to be considered along with the comparison of the performance.

• As for the fine-grained dataset, we confirm that ImageNet-R also has fine-grained classes i.e.:

a. Subtypes of Dog: ['west highland white terrier', 'golden retriever', 'labrador retriever', 'cocker spaniels', 'collie', 'border collie', 'rottweiler', 'german shepherd dog', 'boxer', 'french bulldog' 'saint bernard', 'husky', 'dalmatian', 'pug', 'pomeranian', 'Chow Chow', 'pembroke welsh corgi', 'toy poodle', 'standard poodle']

b. Subtypes of fish: ['goldfish', 'great white shark', 'hammerhead','stingray', 'clown fish', 'puffer fish', 'grey whale', 'killer whale']

c. Subtypes of bird: ['hen', 'ostrich', 'goldfinch', 'junco', 'bald eagle', 'vulture,'peacock', 'lorikeet', 'hummingbird', 'toucan', 'duck', 'goose', 'black swan','flamingo', 'american egret', 'pelican']

The result in ImageNet-R as shown in Table 2 shows that our method works well in spite of the presence of fine-grained classes. Our method outperforms existing SOTA with a significant margin i.e. 12-30% FAA and 8-24% CAA in 20-task settings, 7-22% FAA and 4-18% CAA in 10-task settings, and 3-18% FAA and 2-17% CAA in 5-task settings. These numerical results given the fact that Imagenet-R contains fine-grained classes prove that fine-grained classes don't decrease the significance of our method.

Q4/W4. The authors have used excessive line spacing adjustments, which negatively impact the visual presentation of the paper, such as in lines 437-438 and 446-447. They should revise the layout of the entire paper to provide a better reading experience for the reader.

Our Response: Thank you for your suggestion. We have ensured the the clear spacing for the mentioned parts and other components to improve the visualization of our paper. Please kindly see our latest manuscript.

We would like to follow up on our response and revised manuscript. We would appreciate it if Reviewer bACr could look at our revised manuscript, and offer additional comments.

Thank you for your suggestion.

1. We have added the advised metrics i.e. inference time only, along with total time, including additional time for descriptors generation as utilized by LEAPGen and ConvPrompt. Our method has the lowest total running time, and moderate inference time i.e. lower than ConvPrompt and higher than HiDe-Prompt. We have added the required storage, please see Appendix D.3.

2. We would like to emphasize the following points:

• Following ConvPrompt, the LLM such as GPT is one of the alternatives to generate descriptors before the training phase. Thus the GPT is not part of the methods. In the case of no classes' descriptors, Our method still works excellently i.e. utilizing class names as descriptors (Appendix D.4).

• GPT is utilized via online query, thus we don't need extra storage to save it.

• Descriptors generation by GPT indeed consumes a fair amount of time. Even though spending additional time for descriptors generations, our method still has a lower total running time than HiDe-Prompt and ConvPrompt.

Dear Reviewer bACr,

Thank you for your feedback on our paper. Could you please kindly review our latest manuscript regarding your concern? Here are the pointers to your points on the use of ChatGPT:

• Section 5.2.d Tables 2,3,4, and Figure 3: Analysis of LEAPGen-lite (lightweight generators and without descriptors) performance that proves the significant performance of our ideas is not bounded by LLM-generated descriptors or the complexity (parameters and size) of the generators.

• Section 5.2.g: Analysis of parameters, running time, and storage

• Appendix D.3: Detailed Analysis of Performance and Cost TradeOffs.

Thank you.

Best Regards,

Author of Submission 6684

W2. Increase of Parameters:

• Section 5.2.d Tables 2,3,4, and Figure 3: Analysis of LEAPGen-lite (lightweight generators and without descriptors) performance.

5.2.d) LEAPGen-lite’s Performance: As shown in Table 2-4 and Figure 3 Despite utilizing far smaller (2.67% params) generators and without class descriptors generated by LLM, LEAPGen-lite still outperforms the existing methods significantly i.e. 4.7-25% FAA and 3-11% CAA in CIFAR100, and 3-30% FAA and 2-26% CAA in ImageNet-R dataset. LEAPGen-lite also achieves a low for- getting rate in these 2 datasets for all task-settings. In the CUB dataset, LEAPGen-lite archives a comparable performance to HiDe-Prompt and outperforms other SOTA with a significant margin i.e. 6-20% FAA and 3.3-14% CAA. This evidence proves our ideas i.e. language embedding as input for prompt generation, task-wise generators, soft task-id predictor, and learning with auxiliary data are not bounded by the generated descriptors and the size of generators.

Here is the snapshot of the tables, please see our revised paper for the full tables.

Please Note: The analysis of LEAGen-lite empirically proves that our method doesn't rely on GPT/LLM or complex generator networks (high number of parameters)

Thank you.

Best Regards,

Author of Submis.6684

Dear Reviewer bACr,

Thank you for your feedback on our paper. Could you please kindly review our latest manuscript regarding your concern?

Here are the pointers to your key points:

We also copied the updates (or snapshots) here, thus you can review them directly.

• Section 5.2.g: Analysis of parameters, running time, and storage

T.Desc, Tr, and Inf denote time for generating descriptors, training, and inference respectively, detailed in Appendix D3.

5.2.g) Parameters, Running Time, and Storage: The table above compares the number of parameters, running time (ImageNet-R), and storage of our methods and existing SOTAs. Despite having a higher number of parameters and storage, LEAPGen consumes less running time than existing SOTAs both training+inference and total running time. LEAPGen-lite consumes the least costs in total running time and storage and requires relatively low parameters and inference time. LEAPGen and ConvPrompt require additional time to generate descriptors that increase their total simulation time. Despite having the least parameters, Hide-Prompt requires the longest training and total times since it needs extra operations to generate uninstructed class representations.

• Appendix D.3: Detailed Analysis of Performance and Cost TradeOffs.

(a) Running Time for All Datasets

(a) Performance vs Storage.

HiDe-Prompt-Large and ConvPrompt-Large is the large version of HiDe-Prompt and ConvPrompt that have 12.2M and 13.3M parameters respectively.

Again, we want to emphasize these points:

• Following ConvPrompt, the LLM such as GPT is one of the alternatives to generate descriptors before the training phase (Please see figure 2 and section 4.2-4.3). Thus the GPT is not part of the methods. In the case of no classes' descriptors, Our method works excellently i.e. utilizing class names as descriptors as demonstrated by LEAPGen-lite and LEAPGen-CN(Appendix D.4).

• GPT is utilized via online query, thus we don't need extra storage to save it.

Thank you for your response.

Please kindly note that:

• LEAPGen utilizes the same setting and resources (text encoder, GPT-generated descriptors, etc) as ConvPrompt.

• LEAPGen-lite utilizes even less resources (without GPT-generated descriptors, fewer parameters, less running time, and less storage) that are comparable to the other SOTAs.

Considering such comparable resources (LEAPGen vs ConvPrompt, and LEAPGen-lite vs other SOTAs), we believe we have satisfied the fairness aspect.

Also please kindly note that:

• Descriptors are optional in our method, our method works excellently and significantly outperforms existing SOTAs by utilizing class names as language modality. Thus, our method doesn't rely on GPT or LLMs.
• GPT is utilized via online query so we don't save the GPT model. It adds fair additional running time, but not storage. Again, LEAPGen and ConvPrompt are evaluated with the same setting. Online querying GPT utilizing API and Python script is a common practice. We don't need to download and save the GPT model. This is applicable in real-application as in our simulation.

Please go thoughtfully on our revised manuscripts (mainly sec. 5.2.g. and Appendix D.3) and our previous responses (comments).

In summary, once again we emphasize that the impact of our method is not bounded by GPT, parameters, running time, and storage.

Please let us know which part you are not sure about, Thank you.

Best Regards,

Author of Submission 6684

Dear Reviewer bACr,

We would like to follow up on our responses. Could you please kindly review our latest response that detailed in our updated manuscript? Could you please point out which part you are not sure regarding the fairness aspect and GPT-generated descriptors?

We believe we have addressed all your concerns in our updated paper and previous responses (comments). Thus, we kindly request Reviewer bACr to reevaluate the score for our paper.

Thank you.

Best Regards,

Author of Submission 6684

Dear Reviewer bACr,

First, we thank reviewer bACr for your effort and time to review our work, rebuttal, and revised paper. We believe we have addressed all your concerns in our revised paper and previous responses (comments) Second, since the discussion phase has ended, We would like to summarize our discussion and emphasize the following points:

1. Fairness Setting : LEAPGen utilizes the same resources as ConvPrompt and LEAPGen-lite(no descriptors with lightweight generators) utilizes less resources than ConvPrompt and comparable resources to existing SOTAs (sec. 5.1, sec. 5.2.g, Appendix E). All methods are evaluated in the same setting and run with respective best (recommended by official paper/code) hyperparameters. Thus we believe we have satisfied the fairness aspect in our study.

2. The presence of GPT :

• GPT is not part of our method (sec 4.2.a, and fig. 2), It is an optional part to generate descriptors following ConvPrompt. Even without GPT-generated descriptors. our method works excellently and significantly outperforms the existing SOTAs, as demonstrated by LEAPGen-lite (sec. 5.2.d) and LEAPGen-CN (Appendix D.4).

• GPT is utilized in an online way (through the Internet), by querying it using API and Python script. It is a common practice nowadays as in our simulation (please see https://platform.openai.com/docs/guides/). Thus we never download and save the GPT model in our storage. LEAPGen indeed saves the descriptors embedding (numerical vector), but not the GPT model. The embedding adds to the whole model's storage size but is still in a reasonable amount, and we have included it in our storage analysis.

• Descriptor generation by GPT (before training) indeed takes a fair amount of time. However, despite spending extra time on it, our method still has a lower total running time than ConvPrompt and HidePrompt.

• LEAPGen-lite proves the significant impacts of our ideas i.e. (1) language as input for prompt generation, (2) task-wise generators, (3) soft task-id predictor, and (4)learning with auxiliary despite without GPT-generated descriptors, and utilize least parameters, running time and storage.

We believe, these highlights clarify your uncertainties about the fairness setting and (again) prove the significant impacts of our ideas, without being bound to the presence of GPT.

Best Regards,

Author of Submission 6684

W1/Q1. The writing of this paper needs significant improvement. Additionally, the structure is overly compact, affecting readability. For instance, the theorem presented in the paper has weak relevance to the proposed method, and I believe it is unnecessary. I suggest moving it to the supplementary materials. Given the complexity and sophistication of the method, I encourage the authors to provide an overview initially.

Our Response: Thank you for your suggestion. We have improved the readability of our paper by adding an overview of our method as presented in section 4.1, moving theorems 2 and 3 into the Appendix section, and improving the general layout of our paper

W2/Q2. The provided ablation study is not sufficiently convincing. I recommend that the authors validate the effectiveness of the proposed modules step-by-step starting from a standard baseline.

Our Response: Thank you for your suggestion. We have revised our ablation study with a step-by-step scenario from fine-tuning (FT) baseline, emphasizing the clarity of each component contribution, Please see section 5.2.d.

W3/Q3. The inclusion of the ChatGPT model raises the question of how to ensure the fairness of experimental comparisons under such settings, and to what extent the performance gains are due to more accurate descriptions.

Our Responses:

• We added a new experiment and analysis on the "class name" as descriptors where we have no class descriptors generated by GPT or other LLMs. Our experiment results as presented in Appendix D4. shows that our method outperforms the existing SOTAs by a significant margin in 3 datasets despite substituting class descriptors with the class name.

• In our main experimental setting, We follow ConvPrompt which utilizes GPT to generate class descriptors before the training process. Thus we emphasize that our method and ConvPrompt utilize the same external resources that are considered as fair. Please note that in our method, the GPT (LLM) is solely utilized to generate descriptors, It is different from GMM which utilizes an LLM decoder in its learning process.

• For the other methods, we can't enforce ChatGPT in them as they were designed to learn without language descriptors. The most feasible setting is evaluating the methods following their official (optimal) setting and hyperparameters that we have done in our study. Thus we believe that we maintain fairness in our evaluation.

• As for the relation between descriptors and performance, we have extended our analysis about the performance of our method w.r.t types of descriptors and LLM (GPT, Gemini, and Llama) in comparison to ConvPrompt. The table below shows the performance of our method compared to ConvPrompt in those settings. Please kindly see Appendix D.5 for a detailed sample of descriptors, results, and analysis. In summary, we conclude that long descriptors are preferable over short and narrative descriptors as generally they carry richer visual descriptions of an object without being contaminated by unrelated words. However, our methods can utilize three of the descriptors types, and gain better performance than ConvPrompt.

W4/Q4. Considering the authors (are about to) open-source their code, I suggest making the descriptions obtained via ChatGPT publicly available for deeper analysis. I believe this will enhance the impact of the paper.

Our Response: Thank you for your suggestion, We have published all the descriptors including short, long, and narrative descriptors generated by ChatGPT, LLama, and Gemini. As for now, the descriptors can be accessed via https://anonymous.4open.science/r/xt124j05/descriptors/.

We would like to follow up on our response and revised manuscript. We would appreciate it if Reviewer UGi2 could look at our revised manuscript, and we offer additional comments.

Dear Reviewer UGi2,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission6684

Dear Reviewer UGi2,

We would like to follow up on our response. Could you please kindly review our updated manuscript regarding your concerns?

Here are the pointers to your key points:

W1. Proposed Method Overview: Section 4.1

W1. Presentation Improvement: Please check our latest manuscript.

W2. Step-by-step Ablation: Section 5.2.e.

W3. The use of GPT:

• LEAPGen-lite (lightweight generators and without descriptors) performance analysis in Section 5.2.d, Tables 2,3,4, and Figure 3 that proves the significant performance of our ideas is not bounded by LLM-generated descriptors or the complexity (parameters and size) of the generators.

• Running time analysis in 5.2.g.

• Detailed setting in Appendix E, LEAPGen and ConvPrompt use the same setting

W4. Open the descriptors: Please check https://anonymous.4open.science/r/xt124j05/descriptors/.

Thank you.

Best Regards,

Author of Submission 6684

W2. Step-by-step Ablation: Section 5.2.e.

e) Ablation Study: yable above shows the impacts of the proposed method's components on its performance. ** (1) Descriptor Embedding $E$** as one of LEAPGen's main components, elevates fine tuning (FT) baseline significantly i.e. $>11\%$ FAA and $>13\%$ CAA. It shows the promising impact of our idea . (2) Task Key $K^t$ and Loss $\mathcal{L}_t$ transform the method into task-wise prompting based on input to $K^t$ and $\mathcal{L}_t$ similarity. They produce a better model with 1-2% FAA and CAA improvement and reduce FFM and CFM by $>2\%$. It shows the effectiveness of task-wise decomposition as applied in the task-wise prompt approach. (3) Task-wise generator $G^t()$ enhances the model recognition capability by $2\%$ and $1\%$ for FAA and CAA respectively. It also decreases its forgetting rate by 1%. It means the trainable generator produces a more discriminative prompt than the descriptor embedding only. (4) $L^t_c$ Class-wise Key $L^t_c$ and Loss $\mathcal{L}_c$ for top-k embedding selection i.e. $E_i$ for $i \in [1...k]$ improves the model performance with 1% margin. It also reduces the model forgetting rate with up to 1% margin. Thus, measuring the input similarity to $L^t_c$ offers a better embedding selection than to$E^t_c$ directly. (5) Auxiliary} embedding improves the LEAPGen's performance with a more than $1\%$ margin. It shows the secondary contribution of descriptors embedding as an auxiliary modality along with its primary role in prompt generation. 6) Inter-tasks loss $\mathcal{L}$_inter improves LEAPGen accuracy significantly i.e. $5.4\%$ and $4.0\%$ for FAA and CAA respectively even though it increases FFM and CFM by $0.5-2\%$. It contributes to balancing the knowledge of previously learned classes (stability) and currently learned classes (plasticity), and emphasizes the risk of forgetting previously learned classes. Soft task-id predictor} substitution to conventional task-id (as in DualPrompt) improves the accuracy with up to $1\%$ and reduces FFM and CFM by $1.5\%$ and $0.5\%$ respectively. It implies that our designed task-id predictor outperforms the existing task-id predictor both in accuracy and forgetting.

Thank you.

Best Regards,

Author of Submission 6684

• Running time analysis in 5.2.g.

T.Desc, Tr, and Inf denote time for generating descriptors, training, and inference respectively, detailed in Appendix D3.

5.2.g) Parameters, Running Time, and Storage: The table above compares the number of parameters, running time (ImageNet-R), and storage of our methods and existing SOTAs. Despite having a higher number of parameters and storage, LEAPGen consumes less running time than existing SOTAs both training+inference and total running time. LEAPGen-lite consumes the least costs in total running time and storage and requires relatively low parameters and inference time. LEAPGen and ConvPrompt require additional time to generate descriptors that increase their total simulation time. Despite having the least parameters, Hide-Prompt requires the longest training and total times since it needs extra operations to generate uninstructed class representations.

• Detailed setting in Appendix E, LEAPGen and ConvPrompt use the same setting

E DETAILED E XPERIMENTAL S ETTING

Existing Methods: The existing SOTAs are run by executing the official implementation (code) of the respective methods. The hyper-parameters setting is chosen based on the official setting. HiDe-Prompt utilizes S-Prompt(Wang et al., 2022a) i.e. similar to DualPrompt but without the global (task-shared) prompt. LGCL and PGP don’t propose a specific prompt structure but rather utilize L2P and DualPrompt. The reported results for PGP and LGCL are obtained with DualPrompt structure which is the best from their result. The other methods i.e. L2P, DualPrompt, CODA-P, EvoPrompt, CPrompt, and ConvPrompt utilize their proposed structure. All the evaluated methods utilize ViT B/16 pre-trained on ImageNet 21K as the backbone model. LGCL utilizes pre-trained CLIP text encoder, while ConvPrompt utilizes SentenceTransformer pre-trained on BERT as its text encoder.

LEAPGen: Our proposed method is implemented on top of the ViT backbone pre-trained on ImageNet-21K. The prompt structure is as defined in section 4. The prompt length is set to 30, and the prefix tuning layers is set to 7 i.e. [0,1,2,3,4,5,6] for all main experiments. We utilize Adam optimizer with a cosine learning rate scheduler. For CIFAR100 dataset, We set 0.01 initial learning rate and 3, 10, and 10 epochs for 5-task, 10-task, and 20-task settings respectively. For ImageNet- R dataset, We choose 5, 10, and 20 epochs for 5-task, 10-task, and 20-task settings respectively. The initial learning rate is chosen from the best of 0.04,0.05,0.06. For CUB dataset, We choose 20 epochs and 0.005 initial learning rate. Similar to ConvPrompt, we utilize SentenceTransformer as a text encoder. All the pre-trained models i.e. ViT, SentenceTransformer, and CLIP(LGCL) are kept frozen (not fine-tuned).

W4. Open the descriptors: Please check https://anonymous.4open.science/r/xt124j05/descriptors/.

• LEAPGen-lite (lightweight generators and without descriptors) performance analysis in Section 5.2.d, Tables 2,3,4, and Figure 3.

Compared to other approaches

5.2.d) LEAPGen-lite’s Performance: As shown in Table 2-4 and Figure 3 Despite utilizing far smaller (2.67% params) generators and without class descriptors generated by LLM, LEAPGen-lite still outperforms the existing methods significantly i.e. 4.7-25% FAA and 3-11% CAA in CIFAR100, and 3-30% FAA and 2-26% CAA in ImageNet-R dataset. LEAPGen-lite also achieves a low for- getting rate in these 2 datasets for all task-settings. In the CUB dataset, LEAPGen-lite archives a comparable performance to HiDe-Prompt and outperforms other SOTA with a significant margin i.e. 6-20% FAA and 3.3-14% CAA. This evidence proves our ideas i.e. language embedding as input for prompt generation, task-wise generators, soft task-id predictor, and learning with auxiliary data are not bounded by the generated descriptors and the size of generators.

Please Note: The analysis of LEAGen-lite empirically proves that our method doesn't rely on GPT/LLM or complex generator networks (high number of parameters)

• LEAPGen-lite (lightweight generators and without descriptors) performance analysis in Section 5.2.d, Tables 2,3,4, and Figure 3.

Dear Reviewer UGi2,

We would like to follow up on our response. Could you please kindly review our updated manuscript regarding your concerns? Thank you.

Here are the pointers to your key points: We also copied the updates here, thus you can review them directly.

W1. Proposed Method Overview: Section 4.1'

In this study, we propose a novel LanguagE As Prompt Generator (LEAPGen) accommodating our main principles that are emphasized in the introduction section. The structure and flow of LEAPGen are visualized in figure 2. LEAPGen generators produce prompts from top-k selected embedding as input. LEAPGen also produces auxiliary (aux) data from the top-k embedding. The prompts are prepended into ViT MSAs while the aux is appended into input patches, thus producing feature and final prediction by ViT layers and MLP head respectively. The top-k embedding is selected based on the cosine similarities between an input and the class-wise keys. LEAPGen limits the search space into task 1 to predicted task t, by performing soft task-id prediction. In each task of the training phase, LEAPGen updates task-associated learnable parameters i.e. generator, task-wise key, and class-wise keys. In the inference phase, LEAPGen selects the generators based on the predicted task t. LEAPGen utilizes cross-entropy loss and cosine similarity loss to optimize its parameters. Task- wise generators, language embedding for prompt generation, and soft task-id prediction are unique to recent SOTAs of evolving generator methods e.g. (Roy et al., 2024) and (Kurniawan et al., 2024), task-wise fixed prompt methods such as (Wang et al., 2024a) and (Gao et al., 2024), and pool-based method (Wang et al., 2022c) in terms of prompt generation/selection, task-prediction mechanism, and modality for prompt generation. The detailed architecture, flow, and learning mechanism are presented in sub-section 4.2 and 4.3.

W3. The use of GPT:

• LEAPGen-lite (lightweight generators and without descriptors) performance analysis in Section 5.2.d, Tables 2,3,4, and Figure 3.

Dear Reviewer UGi2,

We would like to follow up on our responses. Could you please kindly review our updated manuscript or previous comments regarding your concerns?

We believe we have addressed all your concerns in our updated paper and previous responses (comments). Thus, we kindly request Reviewer UGi2 to reevaluate the score for our paper.

Thank you.

Best Regards,

Author of Submission 6684

Dear Reviewer UGi2,

First, we thank reviewer UGi2 for your effort and time to review our work. We believe we have addressed all your concerns in our revised paper and previous responses (comments) Second, since the discussion phase has ended, We would like to summarize our revision and responses, and emphasize the following points:

1. Revision of Ablation: We have revised our ablation study in a step-by-step as you advised. It explains the contribution of each component in a more detailed way (sec. 5.2.e).

2. Method Overview and Writing Improvement: We have added an overview explaining the general ideas of our proposed method in sec 4.1. We have improved the writing of our paper in terms of layout, presentation and clarity.

3. Fairness Setting : LEAPGen utilizes the same resources as ConvPrompt and LEAPGen-lite(no descriptors with lightweight generators) utilizes less resources than ConvPrompt and comparable resources to existing SOTAs (sec. 5.1, sec. 5.2.g, Appendix E). All methods are evaluated in the same setting and run with respective best (recommended by official paper/code) hyperparameters. Thus we believe we have satisfied the fairness aspect in our study.

4. The presence of GPT :

• GPT is not part of our method (sec 4.2.a, and fig. 2), It is an optional part to generate descriptors following ConvPrompt. Even without GPT-generated descriptors. our method works excellently and significantly outperforms the existing SOTAs, as demonstrated by LEAPGen-lite (sec. 5.2.d) and LEAPGen-CN (Appendix D.4).

• Descriptor impact to the model performance has been demonstrated by the performance difference between LEAPGen and LEAPGen-CN (Appendix D.4). We also extend our analysis of the performance of LEAPGen vs ConvPrompt in various types of descriptors (Appendix D.5).

• LEAPGen-lite proves the significant impacts of our ideas i.e. (1) language as input for prompt generation, (2) task-wise generators, (3) soft task-id predictor, and (4)learning with auxiliary despite without GPT-generated descriptors, and utilize least parameters, running time and storage.

We believe, these highlights resolve your concerns and prove the significant impacts of our ideas, without being bound to the presence of GPT.

Best Regards,

Author of Submission 6684

Q6. Is the MLP head (classifier) also trained for each task? How can it produce list of softmax values of all classes for equation 7?

Our Response: Thank you for your question. Yes, the MLP head is trained in every task. Please note that following standard practice in continual learning, the MLP head gradually adds output neurons when entering a new task. For example, in the case of class incremental learning with 10 classes per task. In the first task, the MLP head has 10 outputs, when entering task 2, the output neurons are added by 10 (now 20 outputs), and so on. Therefore It can produce logits for all and ongoing learned classes. In equation 7, we compare all the logits to the labels, while in equation 6 (intra-task loss) we only consider the logists from currently learned classes, excluding previously learned classes.

Q7. Please explain why inter-task loss help improve FAA and CAA? What does it tell us about the learned

Our Response: Thank you for your question. Inter-task loss improves the discriminative ability between currently learned classes and previously learned classes (from previous tasks). The newly added parameters/weights are driven to produce different representations between the currently learned classes and previously learned classes. Thus, the inter-task loss contributes to improving CAA and FAA. However, at the same time, inter-task loss increases the risk of forgetting since it changes (adjusts) model's parameters that optimal for previous learned classes.

Q8. Has the importance of inter-task loss already explored in previous continual learning papers?

Our Response: Thank you for your question. Our literature study shows that inter-task loss is the default cross-entropy loss that is widely utilized for continual learning. Following HidePrompt, we emphasize the importance of intra-task and inter-task loss to improve within-task prediction and inter-task prediction. However, our method and HiDe-Prompt have different mechanisms for intra-task loss. The new joint loss function including intra-task loss, inter-task loss, cosine similarity to $K^t$ ($\mathcal{L}_t$), and cosine similarity to $L_c^t$ ($\mathcal{L}_c$) as presented in eq.10 is unique and it is one of our contributions.

Sa. In Figure 1, reposition the labels “fixed”, “trainable”, and “generated” since these labels affect entire Figure, not just the green block.

Our Response: Thank you for the suggestion. We have revised Figure 1, we moved the labels outside the green block.

Sb. In section 3.b, I suggest to use emphasized text style for “pool-based approach”, “task-specific prompt approach”, and “growing component approach”.

Our Response: Thank you for the suggestion. We have bolded the mentioned texts in our revised paper.

Sc. In Line 242, correct the typo “tem” -> “them”

Our Response: Thank you for the correction. We have corrected the typo in our revised manuscript.

Sd. Line 267 mentions t_hat, but eq. 3 uses notation as t, but not t_hat.

Our Response: Thank you for the correction. We have revised the symbol into $t$ (without hat).

Q1. Between L155-160, under the context of previous task-specific approaches, it is mentioned that two different tasks could produce similar key vectors. It would make a stronger point to justify this statement with some empirical

Our Response: Thank you for your question. We have added extra analysis on the similarity between task keys as presented in Appendix D1. Our analysis shows the high cosine similarity of task keys after completing all task (10 tasks) learning .

Q2. What is the value of k for top-k descriptors and how many Generators are chosen for each task? Can we chose a single generator for each task? How {E_i}^t of size |C^t|gets chosen for k set of generators?

Our Response: Thank you for your question. The value of k is the same as the number of generators, and it is assigned as a user parameter. We choose k=3 for our experiment. Our sensitivity analysis (sec 5.2 point e.3) shows the performance of our method in various k. Yes, we can use a single generator as well. In our method, using a single generator means we choose k=1 which also means we choose only a single descriptor for prompt generation. How to select top-k $E_i$? Given a predicted task $t$, we choose top-k $E_i$ from the predicted (current task) task i.e. $E_c^t$, and from the previous task $E_c^1$, $E_c^2$,... $E_c^{t-1}$. The top-k selected descriptor embedding i.e. $E_1$, $E_2$... $E_k$ will be inputted into k generators of task $t$ i.e. $G_1^t$, $G_2^t$... $G_k^t$. We explain the complete components and procedures in sec. 4 and Figure 2.

Q3. From the Figure 2 or its caption, it is not clear what is the component that is compared against input x to get the cosine similarity. The sentence “search space for top-k” in the figure gives some hint, but the arrows for entire blue block rising the confusion. Please clarify it in the figure or caption.

Our Response: Thank you for your question. We compare x to $L_c^i$ that is associated with $E_c^i, i \in [1,2,...t]$. We have adjusted the font in Figure 2 so that the string "search space for top-k" and symbol $L_c^i$ have the same size. We set both of them in italics for better cohesion between them.

Q4. Why it is interesting to solve equation 3, if the task is to search for E_i in all tasks [1, 2, ….t]? Similarly, why consider K^t and why not just use {L_i}^t ?

Our Response: Thank you for your question.

• We search for top k $E_i$ as the input for prompt generators. Along with the $E_i$ from the expected class, we look for other k-1 descriptors that are closest to the expected class which may be located in previous tasks, not in the same task. For example, descriptors of class "pear" are closer to the descriptors of class "apple" than to the descriptors of class "palm tree". Class "apple" is located in the previous task, while class "palm tree" is in the same task as class "pear". Thus, it is better to select other k-1 descriptors from previous tasks. Why not search from all tasks? In that case, our method will always use the same (latest) generators that might be incompatible (un-optimal) for all tasks. Besides, It will increase the chance of wrong-selected descriptors.

• The use of ${L_i}^t$ only has the risk of misprediction due to similar keys from different classes from different tasks. For example, images of class 'Scottish terrier' have high similarity with images of class 'West Highland white terrier' from a different task. The use of $K^t$ only has the risk of similar key vectors from different tasks as we mentioned earlier. Thus, task prediction using ${L_i}^t$ and $K^t$ together has less misprediction risk than using ${L_i}^t$ and $K^t$ only.

Q5. Under the ablation study, what does absence of generator mean? How do you get prompts in this approach without generator?

Our Response: Thank you for your question. The absence of a generator means the prompt components are produced directly from selected (the top-k) descriptors i.e. $PC_i = E_i$, and the prompt is produced as $P = \Sigma_i^k PC_i = \Sigma_i^k PC_i = \Sigma_i^k E_i$.

Q1/W1. The paper does not explicitly specify some key details about the pre-trained models used. Which Sentence Transformer does the paper employ? Whether this is a pre-trained language model and if any fine-tuning of these models occurs during the continual learning process.

Our Response: Thank you for your concern.

• Following common practice in prompt-based methods, all the evaluated methods in tables 2 and 3 utilize the same model i.e. ViT-B/16 that is pre-trained on imagenet21K. Following ConvPrompt, we utilized SentenceTransformer pre-trained on Phase-BERT as our text encoder. Other language-guided methods i.e. LGCL and GMM utilize CLIP as a text encoder. There is no fine-tuning process for SentenceTransformer in our method. The model is solely employed to encode the descriptors into embedding before starting a training process in each task.

• We explain the detailed setting for all method in Appendix E.

Q2/W2. The paper does not provide a clear comparison of the increase in parameters due to the addition of generators for new tasks. Based on the paper, the generator is not a lightweight module. As the proposed method requires adding new generators for some new tasks, even if not all tasks, this could lead to a significant growth in the number of parameters as the number of tasks increases. And also an unfair comparison with related works.

Our Response: Thank you for your concern.

• We added a new experiment and analysis on the trade-offs between cost and performance that comprehensively evaluate the proposed methods and existing SOTAs in terms of performance (accuracy and forgetting), number of parameters, and running time. We also compare our proposed method with the large version of SOTAs that executes a higher number of parameters. Please see Appendix D3. Our analysis shows in comparison to the standard versions and large versions of SOTAs, our method achieves better performance (average accuracy and average forgetting), and requires a shorter running time.

• Even in comparison with the SOTAs in a smaller number of parameters, our proposed method still achieves a smaller running time. The Smallest number of parameters i.e. HiDe-Prompt doesn't imply shorter running time as It requires mode additional operations i.e. drawing (augmenting) prototypes and constructing uninformed representations that run on many epochs.

• To ensure fairness, we evaluated all the methods by applying the best setting i.e. as mentioned in the references and official code manual, including the preferred hyper-parameters. Please note that increasing the number of parameters i.e. by increasing the number of generators, or prompt length doesn't always improve the model performance. As shown in the table above, our sensitivity analysis in sec 5.2. point e and figure 4, after some points, the increase in prompt length or number of generators decreases the model performance. Thus, we believe, we have evaluated the consolidated methods in a fair way i.e. by applying their respective official setting and parameters.

We would like to follow up on our response and revised manuscript. We would appreciate it if Reviewer j5sH could look at our revised manuscript, and offer additional comments.

Dear Reviewer J5sH,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission6684

Dear Reviewer J5sH,

We would like to follow up on our response to your comments Could you please kindly check our revised manuscript?

Here are the pointers to your key points:

W1. Details of ViT and Text Encoder: Appendix E. Detailed Experimental Setting

W2. Increase of Parameters:

• Section 5.2.g. Analysis of Parameters, Running Time and Storage

• Appendix D.3. TradeOffs Between Cost and Performance including detailed growing parameters on each task

• Section 5.2.d., Tables 2,3,4, and Figure 3 i.e. Analysis of LEAPGen-lite (lightweight generators and without descriptors) performance that proves the significant performance of our ideas is not bounded by LLM-generated descriptors or the complexity (parameters and size) of the generators.

Thank you.

Best Regards,

Authors of Submission 6684

Thank you for these detailed experiments. From the revisited paper, I now clearly understand that the increase in parameters is reasonable, and comparable with other research works. These results answer my questions, and I decide to increase the rate accordingly.

W2. Increase of Parameters:

• Section 5.2.d., Tables 2,3,4, and Figure 3 i.e. Analysis of LEAPGen-lite (lightweight generators and without descriptors) performance

5.2.d) LEAPGen-lite’s Performance: As shown in Table 2-4 and Figure 3 Despite utilizing far smaller (2.67% params) generators and without class descriptors generated by LLM, LEAPGen-lite still outperforms the existing methods significantly i.e. 4.7-25% FAA and 3-11% CAA in CIFAR100, and 3-30% FAA and 2-26% CAA in ImageNet-R dataset. LEAPGen-lite also achieves a low for- getting rate in these 2 datasets for all task-settings. In the CUB dataset, LEAPGen-lite archives a comparable performance to HiDe-Prompt and outperforms other SOTA with a significant margin i.e. 6-20% FAA and 3.3-14% CAA. This evidence proves our ideas i.e. language embedding as input for prompt generation, task-wise generators, soft task-id predictor, and learning with auxiliary data are not bounded by the generated descriptors and the size of generators.

Here is the snapshot of the tables, please see our revised paper for the full tables.

Please Note: The analysis of LEAGen-lite empirically proves that our method doesn't rely on GPT/LLM or complex generator networks (high number of parameters)

W2. Increase of Parameters:

• Appendix D.3. TradeOffs Between Cost and Performance including detailed growing parameters on each task

(1). Performance vs #Parameters and Running Time:

(2). Detailed Running Time: T.Desc, Tr, and Inf denote time for generating descriptors, training, and inference respectively.

(3). Performance vs Storage:

Dear Reviewer J5sH,

We would like to follow up on our response. Could you please kindly review our latest manuscript regarding your concern? Thank you.

Here are the pointers to your key points:

We also copied the updates (or snapshots) here, thus you can review them directly.

W1. Details of ViT and Text Encoder: Appendix E. Detailed Experimental Setting

All the evaluated methods utilize ViT B/16 pre-trained on ImageNet 21K as the backbone model. LGCL utilizes pre-trained CLIP text encoder, while ConvPrompt utilizes SentenceTransformer pre-trained on BERT as its text encoder. LEAPGen: Our proposed method is implemented on top of the ViT backbone pre-trained on ImageNet-21K. The prompt structure is as defined in section 4. The prompt length is set to 30, and the prefix tuning layers is set to 7 i.e. [0,1,2,3,4,5,6] for all main experiments. We utilize Adam optimizer with a cosine learning rate scheduler. For CIFAR100 dataset, We set 0.01 initial learning rate and 3, 10, and 10 epochs for 5-task, 10-task, and 20-task settings respectively. For ImageNet- R dataset, We choose 5, 10, and 20 epochs for 5-task, 10-task, and 20-task settings respectively. The initial learning rate is chosen from the best of 0.04,0.05,0.06. For CUB dataset, We choose 20 epochs and 0.005 initial learning rate. Similar to ConvPrompt, we utilize SentenceTransformer as text encoder. All the pre-trained models i.e. ViT, SentenceTransformer and CLIP(LGCL) are kept frozen (not fine-tuned).

W2. Increase of Parameters:

• Section 5.2.g: Analysis of parameters, running time, and storage

T.Desc, Tr, and Inf denote time for generating descriptors, training, and inference respectively, detailed in Appendix D3.

5.2.g) Parameters, Running Time, and Storage: The table above compares the number of parameters, running time (ImageNet-R), and storage of our methods and existing SOTAs. Despite having a higher number of parameters and storage, LEAPGen consumes less running time than existing SOTAs both training+inference and total running time. LEAPGen-lite consumes the least costs in total running time and storage and requires relatively low parameters and inference time. LEAPGen and ConvPrompt require additional time to generate descriptors that increase their total simulation time. Despite having the least parameters, Hide-Prompt requires the longest training and total times since it needs extra operations to generate uninstructed class representations.