Multi-Vision Multi-Prompt for Few-Shot Learning in Vision-Language Model

Overall, the contribution of this paper is quite limited. The two basic components, i.e., multi-view augmentation and multiple prompts, of the newly introduced method have actually been explicitly explored in the community, yet this paper does not yield significant improvements with its simple and straightforward design. The weaknesses are detailed below:

1. The design of feeding multiple prompts into text encoder has long been a conventionally accepted method for zero-shot vision-language inference. For example, the original CLIP paper leverages 80 different text prompts for ImageNet inference. This technique is generally termed as prompt engineering but this paper lacks related discussions about that.
2. The mechanism of introducing text prompts for multiple transformer layers is also confusing. I understand multi-layer prompting might facilitate extracting multi-scale visual features for the image encoder, but why also applying it to text encoders? The paper does not give an explanation or discussion of this design, and it seems that the performance gain is merely because of introducing additional parameters.
3. Some important details are also missing. How are the prompt tokens processed in the transformers? Are prompts introduced in the i-th layer simply pruned after this layer, or are they retained until the last layer? These two protocols do make difference.
4. The Mixed Self-Augmentation looks more like an additional trick to enhance performance. This augmentation does not exhibit too much relation to your prompting framework, i.e., even without this augmentation, your method can achieve comparable performance, and when equipped with it, your baseline models also seem to be able to obtain the same level of improvements.
5. Some baseline performances might be underestimated. For example, why CLIP's test accuracy on ImageNet is only 32.1%? In CLIP's original paper it obtains 63.2% and 68.6% zero-shot accuracy with ViT-Base/32 and ViT-Base/16, respectively.
6. Comparisons are unfair. You method has both data augmentation and prompt learning but every baseline only has one of them. You should at least give ablation results of your method without augmentation/prompting or baselines with both sophisticated augmentation and prompting.
7. Minor comments: the paper is not well-written and hard to understand; many grammatical errors and imprecise expressions in the paper; In figure 6, the table should not be screenshotted to an image, but there are many ways to place images and tables side by side in Latex.

There's ambiguity regarding the benchmarking of the proposed methods. It's essential to ensure that each method has been equitably compared to provide a clear understanding of its efficacy.

Figure 6 indicates that a score of 73.8 is achieved by integrating multi-prompt, multi-vision, and text diversity. However, Table 2 seems to compare this with CLIP combined with other mix-up methods along, which would be more appropriate to compare it with just CLIP + multi-vision. According to Figure 6, multi-vision alone offers a mere 0.5% improvement. Given the renowned effectiveness of mix-up methods as data augmentation, it's not evident if the inclusion of multi-vision is truly indispensable. Moreover, the approach bears notable resemblance to mix-up, with distinctions primarily in implementation specifics.

Regarding multi-prompt, it might be more insightful if it was contrasted with a baseline where diverse prompts are introduced simultaneously across training stages. Introducing different prompts at separate stages complicates the algorithm, necessitating elements like a prompt bank. At present, it's challenging to discern the imperative of such staged or "curriculum" learning.

The paper's primary arguments seem to combine various methods from distinct directions. However, there's a lack of explicit justification underscoring the necessity and efficiency of each technique, particularly when benchmarked against its direct counterparts. Comparing with prior state-of-the-art methods doesn't necessarily bolster the paper's credibility, especially given the significant differences in implementation and methodologies.

• The paper writing and structuring can be substantially improved.
• All argument referencing Figure 2 is weak as they are not a direct reflection of the proposed method in this paper.
• Figure 4 is confusing, readers are unable to tell frozen parameters versus learnable parameters.
• The proposed image augmentation, i.e. mixup regions of image cross samples, is not well motivated and does not make much sense. The ablation result in Figure 6 also does not justify the necessity.
• Gaussian weighting for high-layer prompts seems like a fixed heuristic with no technical novelty.
• The results for cross-dataset benchmark evaluation are underwhelming, with the average accuracy higher than MaPLe by merely 0.2%.

• There remain many unclear points regarding the proposed method.

  • What do $P_{fixed}$ and $P_t$ refer to? In other words, where do you obtain $P_{fixed}$ and $P_t$? Are $P_t$ and $P_{w}^{(e)}$ the same in the context? These notations are very confusing without clear explanations.
  • What does evaluation parameter $\epsilon_e$ mean? What is the formula to calculate it?
  • What does $H(x)$ mean in equation (8)? I assume you are referring to image features; however, it is somewhat challenging to interpret this notation if it's being introduced for the first time.
  • Is there any explanation why enforcing the consistency loss between original images and mixed images benefits few-shot learning?
  • Similarly for the text distillation loss? Please do not repeat the sentences mentioned in your paper in your rebuttal, since they do not provide any insights towards understanding your method.

• The conclusion in Figure 5 is that leveraging 5 prompts works the best. Here comes a question: how do you select the 5 prompts given so many combinations among hand-designed prompts?

• The paper claims that their approach doesn't introduce additional parameters or escalate computational costs. The question that arises is whether employing more prompts results in an increase in computational expenses. Nevertheless, the authors do not provide any comparison of computational costs between their method and other approaches.

• There are some other state-of-the-art few-shot approaches [1] missing for comparison, where the results are much better.

• Poor writing.

  • While the performance of few-shot learning can improve through meta-learning -> While the performance of few-shot learning can be improved through meta-learning
  • While the performance of few-shot learning can improve through meta-learning or image augmentation strategies, these approaches may increase computational cost and affect accuracy. This sentence is self-contradictory. You mentioned that the few-shot performance could be improved through those strategies, however, you later state that they affect accuracy.
  • improving accuracy by 4.6% and 2%. What do you mean?
  • our method improves overall accuracy by 2% to 4.6% -> our approach improves the overall accuracy by a range of 2% to 4.6%.

[1] Exploiting Category Names for Few-Shot Classification with Vision-Language Models