Text as Any-Modality for Zero-shot Classification by Consistent Prompt Tuning

Thanks for your valuable suggestions and we will try to address your concerns as follows. We are eager to engage in a more detailed discussion with you.

Weakness 1: More discussions about the novelty and contributions.

• Thank you for your valuable feedback during the review process. We have noted your concerns regarding the potential similarities between our TaAM-CPT method and existing techniques. Here is a further elaboration on the novelty of TaAM-CPT:
  • The core innovation of our work, TaAM-CPT, lies in its minimalist training process, which relies solely on text data without any modality-specific annotated data, further enhancing the applicability of the multimodal pre-trained model in data-scarce scenarios.
  • Compared to methods such as TAI-DPT and PVP, TaAM-CPT simplifies the prompt design by representing each category as a randomly initialized vector. This design not only reduces the complexity of the model but also allows for more flexibility in adding new categories and modalities without disrupting the learned category prompts.
  • TaAM-CPT adopts a unidirectional contrastive learning strategy, using modalities with stronger representational capabilities to guide those with weaker learning abilities. This approach not only improves the performance of weaker modalities but also further enhances the representational power of stronger modalities.
  • TaAM-CPT is entirely dependent on easily collectible text data and does not require annotated data for specific modalities. This gives the method an advantage in situations where annotated data is scarce or prohibitively expensive to obtain.

Weakness 1: Demonstrate robustness and flexibility, such as training cost or parameters.

• We appreciate your concerns regarding the applicability. We believe TaAM-CPT is still a recommended choice when pursuing performance on specific task. For clarity, we adopt Bert-base model with 110MB parameters as reference. TaAM-CPT initializes each prompt with a 512-d vector, meaning one class prompt occupies 512 parameters. Therefore, for n modalities, TaAM-CPT can accommodate approximately 110,000,000512n class prompts. For example, for 10 modalities, the prompt pool size can reach 21,484, which is sufficient to cover the common categories.

• Additionally, we conduct new experiments on K400/MSCOCO/ESC50 datasets, as shown in the below table. Different from randomly initializing the prompt in the paper, we use the output embeddings by ~CLIPs text encoder to initialize class prompt and remove inter-modal learning. Therefore, each class prompt encompasses class-specific textual prior knowledge, allowing TaAM-CPT to converge fastly with less training data (50 texts for one class). Although without inter-modal learning, TaAM-CPT achieves higher performance compared to CLIP/ViCLIP/CLAP.

Weakness 1: Certain sentences were perceived as verbose and challenging to comprehend.

• Thank you for your valuable feedback on the clarity and conciseness of the TaAM-CPT paper. We will reorganize lines 51-53 and rephrase Equation (6). Additionally, we will add more explanatory content to the paper, such as definitions of key concepts, detailed descriptions of algorithmic steps, and in-depth analysis of experimental results, to help readers better understand the TaAM-CPT method.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Weakness 5: Experiments about improving the inference speed.

• We conduct an experimental analysis of inference cost with TAI-DPT [1] and PVP [2] on the MSCOCO2014 dataset, as shown in the table below:

• TAI-DPT [1] designs multi-granularity prompts (global prompt and local prompt), while PVP [2] designs pseudo-visual prompt and text prompt, and both require prompt encoding processes. TaAM-CPT represents each category with a class-specific prompt, avoiding the prompt encoding processes, and presenting significant advantages in terms of inference time.

Weakness 6: Compare the performance difference between TaAM-CPT and MLLM.

• The main difference between TaAM-CPT and MLLMs is that our work only uses LLMs to generate text training data, LLMs are not employed in the training process.

• MLLMs are generative models, whose aim is to generate high-quality images, videos, or audio given a text prompt, rather than performing classification or regression. The performance advantage of TaAM-CPT stems from its class-specific prompt design and training with text data.

Weakness 7: Exhaustive permutation of many labels is time-consuming and may degrade model quality.

• In our experiments, we do not traverse all possible permutations of labels, instead, we randomly sample a portion of the text data for training.

• For any category of any modality, generating 500~1000 text descriptions is sufficient. As shown in Figure 7 in the manuscript, additional training data does not further improve the performance.

Weakness 8: The burden of training intra- and inter-learning with more modalities and labels.

• We conduct theoretical analysis in Sec 4.4 in the manuscript and take the bert-base model as an example, presenting the feasibility of TaAM-CPT for extending unlimited modalities and categories. We believe TaAM-CPT is still a recommended choice when pursuing unlimited modalities or categories.

• Adopt the Bert-base model with 110MB parameters as a reference. TaAM-CPT initializes each prompt with a 512-d vector, meaning one class prompt occupies 512 parameters. Therefore, for n modalities, TaAM-CPT can accommodate approximately 110,000,000512n class prompts. For example, for 10 modalities, the prompt pool size can reach 21,484, which is sufficient to cover the common categories. We can see that the training cost of training 10 modalities with 21,484 labels is comparable with the finetuning Bert-Base model. Moreover, TaAM-CPT directly updates the prompt without continual backpropagation paths.

[1]. Texts as Images in Prompt Tuning for Multi-Label Image Recognition. CVPR 2023

[2]. TAI++:Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt. IJCAI 2024

Thanks for your valuable suggestions and we will try to address your concerns as follows. We are eager to engage in a more detailed discussion with you.

Weakness 1: Whether different prompts need to be designed for different datasets.

• During the text data generation, the construction of the input prompt template for LLMs is only dependent on the modality and categories. For example:
• Prompt template for the video labels {playing the piano, driving the car}: "Making several English sentences to describe a $\textbf {video}$. Requirements: Generate 5 English sentences! Each sentence should be less than 25 words and includes: $\textbf {playing the piano, driving the car}$".
• Prompt template for the image labels {dog, cat, person}: "Making several English sentences to describe a $\textbf {image}$. Requirements: Generate 5 English sentences! Each sentence should be less than 25 words and includes: $\textbf {dog, cat, person}$".

Weakness 2: Can TaAM-CPT check prompt quality during prompt generation?

• For the text data generation strategy in PVP [1], after obtaining a text description, PVP [1] would again query LLMs whether the description is reasonable (that is, whether it is a reasonable description in the real world) to filter out potentially unreasonable content.

• However, we removed the rationality judgment step as we found that it does not significantly affect the training performance of the model. After using powerful LLMs to generate sufficient text data, the impact of noise in text data is negligible, which is also demonstrated by the related experiments in the PVP [1] (PVP [1] compares the performance of publicly annotated text data with those generated by LLMs, and the performance difference is small). Moreover, rationality judgment increases the cost of LLMs generation. We conduct the following experiment, further filtering out 300,000 high-quality text description data through rationality judgment:

• We believe this may be because the large-scale pretraining data of pretrained models may also contain too much noise, hallucinations, etc., which desensitizes the model to noisy text.

Weakness 3: is there an information leak as inter-modal Learning based on validation performance?

• During the training of TaAM-CPT, relying on the validation performance is just a simple and direct strategy that we employ. Actually, besides the validation performance, other factors such as training performance, training loss, convergence speed, etc, can all serve as criteria for determining weak modalities. For example, the training loss for each modality is as follows:

• We can see that the training loss for the video modality is the highest all the time, which may be due to the strong correlations among categories. Therefore, in the experimental benchmarks, the video modality is always considered as the weak modality.
• Furthermore, as shown in Table 7 in the manuscript, if image or audio modalities are treated as weak modalities, it will decrease the performance of all modalities.

Weakness 4: Should TaAM-CPT be trained individually based on the validation performance of each dataset?

• Determining weak modalities is based on the overall performance across all datasets within each modality. For instance, the average performance of the video modality on the k400, k600, and k700 datasets. Hence, there is no need to train TaAM-CPT separately for each dataset. Moreover, we can also determine the weak modality by using the average training performance, and average training loss, rather than relying on the validation performance.

[1]. TAI++:Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt. IJCAI 2024

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Thanks for your valuable suggestions and we will try to address your concerns as follows. We are eager to engage in a more detailed discussion with you.

Weakness 1: The contrastive and ranking losses are not new.

• We understand your concerns about the novelty. The innovation of TaAM-CPT does not lie in the choice of loss function, both contrastive and ranking losses are merely optimization objectives for TaAM-CPT. The key novelty of TaAM-CPT is its minimalist training process, which relies solely on text data without any modality-specific annotated data. This enhances the applicability of multimodal pre-training models in scenarios with scarce data.

• TaAM-CPT simplifies prompt design by using randomly initialized vectors to represent all categories, thus avoiding complex prompt design and encoding processes, and significantly reducing training costs. Moreover, as a versatile representation model, TaAM-CPT can be extended to accommodate unlimited modalities and categories. This flexibility and scalability are not present in existing methods.

Weakness 2: Include additional tasks, such as conditional image or audio generation using the novel class.

• We agree with your suggestions regarding the practical application. TaAM-CPT primarily focuses on the classification task in zero-shot scenarios, using only textual data instead of annotated modality-specific data. It achieves significant performance across various modalities and datasets through simplified class-specific prompts and multimodal pre-training models.

• However, we also fully recognize the limitations of classification tasks in real-world applications. In scenarios with intricate generation tasks, the model needs to be able to generate modalities such as video, images, or audio. Therefore, we will continue exploring generation abilities in our future research to enable TaAM-CPT to also generate modalities corresponding to novel categories, thereby expanding the scope and utility of TaAM-CPT in various applications.

Weakness 3: Provide empirical results on the time complexity of adding new concepts in comparison to initial training time.

• The trainable parameters of TaAM-CPT are the class-specific prompts. Therefore, for simplicity, we random samples K400 with 30 video labels and MSCOCO with 30 image labels as the initial label set. The novel concepts consist of additional labels (K400 with 20 video labels, MSCOCO with 20 image labels), and the novel audio modality (ESC50 with 20 labels), with each label containing 500 text descriptions using LLMs. The comparison of training time results is shown below:

• Among them, (K400_Novel_20, MSCOCO_Novel_20, ESC50_Novel_20) represent the added new labels and new modality. We can see that for the normal labels (K400_Normal_30, MSCOCO_Normal_30), whether they continue training (Continue training) or are frozen (Frozen), neither affects the learning of the new modality labels and maintain the performance that matches the initial joint training (Initial training). For the novel modalities and labels, their Novel training performances are still comparable with Initial training.

Dear Reviewer, we are looking forward to your professional suggestions and hope to receive your guidance to further discuss and refine the contents of the work. We are eager for your response. Thank you.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Thanks for your valuable suggestions and we will try to address your concerns as follows. We are eager to engage in a more detailed discussion with you.

Question 1: How to add a new modality or label? Is it possible to train only new prompts while freezing the old ones? Will the performance on new prompts be as good as training all prompts from scratch? How to introduce a new modality encoder with a different output dimension (e.g., 768)?

• TaAM-CPT learns a class-specific prompt for each label, when a new label appears, only requires initializing a class-specific prompt for it. The pre-trained models selected in our experiments all share uniform embedding dimensions, which avoid other trainable parameters besides the prompt as much as possible. The more trainable parameters there are, the more likely it distorts the original knowledge in the pre-trained model. Therefore, when a new modality encoder has a 768-dimensional space, we can add a simple linear layer to map the dimension of this modality to the unified dimension, and also consider this linear layer as trainable parameters. We will continue to explore related experiments in future work.
• For simplicity, we use K400 with 30 video labels and MSCOCO with 30 image labels as the initial label sets. Novel concepts are other labels (K400 with 20 video labels, MSCOCO with 20 image labels), and the novel audio modality (ESC50 with 20 audio labels), and each label generates 500 text data by LLMs. The experimental results are shown below:

• Among them, (K400_Novel_20, MSCOCO_Novel_20, ESC50_Novel_20) represent the added new labels and new modality. We can see that for the normal labels (K400_Normal_30, MSCOCO_Normal_30), whether they continue training (Continue training) or are frozen (Frozen), neither affects the learning of the new modality labels and maintain the performance that matches the initial joint training (Initial training).

Question 2: Which criteria claim the video modality as "weak"?

• During the training of TaAM-CPT, we directly employ validation performance as a simple strategy. In fact, besides validation performance, metrics such as training performance, training loss, convergence speed, etc., can all serve as criteria for determining weak modalities. We assume that these criteria can assess the hard degree of learning a modality, considering modalities that are difficult to learn as weak modalities and expect strong modalities can further guide the learning of weak modalities.

• We can see that the training loss for the video modality is consistently the highest, which is due to the strong correlation between some video categories. Therefore, in the benchmarks, the video modality is always considered a weak modality. Moreover, as shown in Table 7 in the manuscript, if image or audio are regarded as weak modalities, it will adversely affect the performance of all modalities.

Question 3: Number of labels per query: How are these specific values determined? Why not use 3 for video and 4 for image? What effect does using only one label per query have? Clarify the number of labels per query.

• We apologize for any confusion caused. In our experiments, the setting is as follows: the video modality has a maximum of 2 labels, while the image and audio modalities have a maximum of 3 labels. This is the result we determined through experimentation:

• We believe that for the video modality, each video sample typically represents a complex action, such as playing a guitar or riding a motorcycle. When the number of {labels} is too large, it can result in generating longer text data, making it difficult to accurately express the semantics of each action category. For the image and audio modalities, the categories usually represent a specific object. Moreover, an image or audio often contains several different object categories, so a larger number of {labels} can reflect the interdependencies between categories.

Question 4: If there are common labels across modalities, can it confirm that they are not merged?

• If common labels appear, each modality needs to have a customized, class-specific prompt. For example, for the "crying" label, TaAM-CPT defines separate prompts, aligning them with the video modality, image modality, and audio modality, respectively. This way, in downstream tasks, it will be possible to perform video classification, image classification, and audio classification tasks separately.

Question 5: What is the value of N (v+a+w=N) in the paper? How many categories for video, image, and audio? Is it a combination of all benchmark dataset labels?

• (v+a+w=N) represents the concatenation of all labels from all modality datasets, as shown in Figure 3 of Section 3.4 in the manuscript. Each modality includes all categories from all modalities.

Question 6: Unclear ablation study for Prompt Design. What does FC stand for? Is it 512-d without transformation?

• The ablation studies we conducted primarily investigated the impact of different prompt design methods on the performance of TaAM-CPT, including:
  • Shared internal dimension: We initialized the prompts as 1024-dimensional or 512-dimensional vectors and projected them into a 512-dimensional space through a fully connected layer (FC).
  • Shared internal dimension (512): We initialized the prompts as 512-dimensional vectors and used 512 dimensions directly as the final dimension.
  • Shared internal dimension (512) + Shared external dimension: We shared a fully connected layer for all modality prompts and projected them into a 512-dimensional space.
• We found that when the prompt dimension is high and projection is necessary, the model's performance significantly decreases. This may be due to an excessive number of parameters, which makes it difficult for the model to learn effective feature representations.
• FC stands for the fully connected layer, which is used to map vectors from a 1024-dimensional space to a unified 512-dimensional space. In TaAM-CPT, we use 512 dimensions as the final dimension for the prompt pool and do not require projection.

Question 7: Minor presentation notes and Minor typos.

• Thank you to the reviewer for pointing this out. We will use $N$ consistently throughout to represent the number of categories and use $M$ to indicate the number of modalities. We will correct all spelling and grammatical errors in the future version to ensure that the format adheres to the standards.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?