Text Descriptions are Compressive and Invariant Representations for Visual Learning

We would like to thank the reviewer for their thoughtful comments. Our responses are below.

Weaknesses

1 It is actually a favorable result that $I(H_{avd}, A)$ is almost identical to $I(H_{cp}, A)$. One can think of $H_{cp}$ as a postprocessing of $H_{avd}$ (since the former is a subset of the latter). This result suggests that even if $H_{avd}$ has more features, these extra features do not harm invariance. The difference between $I(H_{avd}, Y)$ and $I(H_{cp}, Y)$ is reflected in the newly revised Figure 3. There the L1 class prompts perform far worse than AVD.

2.1 Even though SLR’s features are derived from text encoders, its coefficients are not regularized to be close to ZS-AVD. Instead, we only enforce sparsity. Therefore, the model can still be wrong in arbitrary ways. The results suggest that the discrete text space has a stronger implicit bias than the image embedding space. 2.2 the average ensemble corresponds to ZS-VD or ZS-AVD. Notice that without WISE-FT, SLR doesn’t really incorporate the strong language prior. Since only sparsity is enforced, the coefficients can still overfit the few-shot data in almost arbitrary ways. One can of course overcome this issue by regularizing the difference between the SLR coefficients and the zeroshot weights, which is almost what WISE-FT does. Compared to regularization during SLR training, WISE-FT is post hoc. So we only need to train the model once. We choose to train this way because it helps us understand the implicit bias of each space (image embedding space vs the discrete text space that is congruent to natural language), rather than the effect of the zeroshot prior.

We add WaffleCLIP zeroshot performance in Table 12, and we notice that its performance tends to decrease as you add more random tokens. On the other hand, our descriptors have length around 8-10 tokens. This suggests that the most important factor is to perturb your basis around the class prompt in a controlled way, which is more easily achieved by using language congruent descriptions. We also compare WaffleCLIP+WISE-FT to SLR+WISE-FT. The result in figure 15 also suggests that SLR has a better performance than WaffleCLIP.

Questions

In Table 12, we also tried to average the templates for ZS-VD, the results are listed in column ZS-VD$^2$. We see a slight improvement.

We also want to remark that the gap between WaffleCLIP and our ZS baseline is because we use the hand-selected 7 templates later released in the OpenAI CLIP github repo, and the original WaffleCLIP paper randomly selects 30 out of the original 80 templates for ensembling.

We would like to thank the reviewer for their positive review of the paper and we hope that the following comments address their outstanding concerns.

Weaknesses

W1. We have added comparisons to several of the methods you suggested.

W1.1 We have added L2-regularized AVD (i.e. L2 regularized logistic regression replacing L1/sparse logistic regression) to show that L1 regularization is very important.

W1.2 For random projection, we initialize $U\in R^{512 \times 300}$, where 512 is the image embedding dimension. We pick the projected dimension to be $300$ since we found that the number of parameters picked by L1 is around 100-300 per class. This bottleneck underperforms our method. See Figure 3 for a detailed comparison.

W1.3 We have also added a comparison to MLP in Table 13 in the appendix. The MLP has 3 layers of sizes 512-4500-1000, which equals the raw number of parameters in SLR. It consistently underperforms linear models because the CLIP embeddings are trained to be linearly classified, and MLP can easily overfit.

W2. We added the comparison in Table 13 in the appendix for k=4, since this is the k that we have used for every model.

W3. We have added the comparison to CoOp on 32 shots, see figure 15 in the appendix. For Full FT, we are limited by our resources.

W4. See Fgure 16 in the appendix for larger k. We tried k=64, 256, 1024. L1 AVD still outperforms linear probing on most datasets.

W5. The reason behind this statement is that CoOp optimizes a continuous prompt like “X X X X {classname}” for each class, and only the “X X X X” part is optimized in the continuous space; the “{classname}” is always fixed. On the other hand, the coefficients of SLR-AVD are not regularized until it incorporates WISE-FT. To prevent confusion, we change Table 5 so that CoOp also incorporates WISE-FT, and SLR-AVD still outperforms CoOp, especially when the amount of data increases.

W5.2. CoOp comparison now is done on ViT-B/16.

W6. This is one drawback that we inherit from WISE-FT. The nice property of WISE-FT is that it’s fully post hoc. So in reality, even if the $\alpha$ is not optimally picked initially, it can always be easily updated. The original WISE-FT method suggests a heuristic of choosing $\alpha=0.5$. However, their original setting has abundant ID data, and 0.5 fails in our case. Our heuristic suggests that picking $\alpha=0.05k$ seems to perform pretty well.

W7. WISE-FT+SLR-AVD only finetunes the last layer. WISE-SLR is the method that first finds a sparsity pattern using SLR-AVD. Then we fixed the pattern and finetuned the whole model. WISE-FT+LP is linear probing with image embeddings +WISE-FT on only the last linear layer.

W8. We added more details in the appendix, and also Table 6 for a better visualization.

W8.2 We updated the figures. As for optimal checkpoint, it is a little ambiguous since sometimes the optimal ID acc and OOD acc are achieved by slightly different $\alpha$, so we decide not to include it.

Questions

Q1. The total learnable parameters in SLR-AVD are 6804*1000. The average number of non-zero entries for each class after learning is $447, 248, 182, 177, 173$, and $135$ for $k=1,2,4,8,16,32$. The numbers are rounded to the nearest integers. So the total number of “used” parameters is the aforementioned numbers * 1000 (for each class). Linear probing optimizes 512*1000 parameters. So SLR does in the end lead to much less parameters. Also, as the training data increases, SLR-AVD can eliminate more irrelevant descriptors.

We thank the reviewer for their comments and we hope that the following points address the concerns they brought up.

Weaknesses

1. The main motivation is that we want to use multiple features to learn a classifier in the presence of few shot data. If we only have one prompt for each class, then there is no candidate to learn from (except using the vanilla image embeddings). An important finding is that the visual descriptions perturb the original class prompt in a specific way such that the perturbed features have nice information theoretic properties. See the newly added Table 12 in the appendix for a comparison to WaffleCLIP, which appends random words to the class prompts. Our visual descriptors have length around 8-10 tokens, and WaffleCLIP’s performance decreases as the random token length increases. WaffleCLIP only does well when a couple words are added. This small perturbation is beneficial even if the words are irrelevant. However, when more words (8-10) are added, WaffleCLIP performance decreases, while our method can pick useful descriptors of this length that constructively enhance the original class prompt.
   We also have a small scale experiments on CIFAR10 in the appendix, paragraph "Choosing $\gamma$ and LLM prompting". There we found the most important factor is to generate more diverse descriptors by setting the frequency penalty.

2. See revised figure 3. L1 consistently leads to better performance than L2. Sparsely chosen class prompts perform worse than AVD on most datasets. These results still hold when combining with WISE-FT, see figure 15 in the appendix. All results indicate that choosing features with L1 is important.

Minor comments:

-"1st sentence of Introduction: "Self-supervised vision-language models (VLMs) like CLIP..." is there any reference claiming CLIP to be self-supervised?” We changed this to “natural language supervised.”

-"2nd sentence of Section-2: "WLOG..." what is WLOG?” This stands for “without loss of generality.” We modified the text to clarify this term.

-"Missing closing paranthesis in Figure-1” Noted - we fixed this in the paper.

-"4th paragraph of Section-3.1 ("Denote M ...") is very confusing. It would be nice to explain all $U$ and $W$ in a diagram/visualization. " We added a visualization of $U, W$ at the end of the appendix

-"Figure-3 caption "the x-axis represents..." (not y)” and “what is the label for y-axis?” The y-axis refers to test accuracy; we have modified the text accordingly.

We would like to thank the reviewer for their thoughtful comments. Please see our responses below.

Weaknesses

1.1 The model is trained by the normal multiclass cross entropy loss + L1 regularization.

1.2 For any zeroshot matrix $W\in R^{|C|\times k}$, the inference is done by taking $\arg\max_{i\in[C]} (WUx)_i$, note that $WUx\in R^{|C|}$ for each $W, U$ with subscripts cp, vd, and avd, respectively.

1.3 The particular algorithm we use to perform cross-entropy loss minimization with L1 regularization is a mini-batch invariant of SAGA, which is a first-order method, detailed in [1]. The appendix paragraph “hyperparameter” gives a brief introduction to the method used.

2 While the improvement observed is relatively modest, we believe the exploration of learning with descriptive features is still noteworthy. The application of information theoretic results in this context is itself valuable, as it provides a useful framework for analyzing these features. Furthermore, the MI-based analysis offers a convincing explanation for the strong performance and robustness of both AVD and the zeroshot baseline, which is an interesting aspect on its own.

3 The results are added, see the updated figure 3. VD performs worse than AVD but better than class prompts, suggesting the descriptors are useful.

4 The OOD datasets are ImageNet-A (an adversarial dataset), ImageNet-R (only contains arts and paintings etc), ImageNet-Sketch (only contains sketch), ImageNet-V2 (natural distribution shift), and ObjectNet.

Questions

Figure 4 has been updated according to the comments.

[1]: Optimal mini-batch and step sizes for SAGA

We appreciate the reviewer's willingness to update their score and would be happy to provide any more information as needed to inform this decision.