Embracing Diversity: Zero-shot Classification Beyond a Single Vector per Class

tldr: Per your suggestion (thx!), we conducted new experiments on 9 finer-grained datasets, including ImageNet and 4 of its variants. We observe gains for our method consistent with those seen in our original submission, suggesting that our method is not tuned to the original dataset suite and the prompts we used can be effective on new datasets with no tuning required. We hope this provides ample evidence demonstrating our method's performance gains over existing baselines.

We sincerely thank the reviewer for taking the time to really engage with our paper and offer so much feedback. We now address weaknesses and concerns one by one.

(W1) First, we note that despite WaffleCLIP’s simplicity, it is a recent and strong baseline. In all settings we study (hierarchical, states, real-world geographic diversity), we improve accuracy over WaffleCLIP. In fact, aside from GeoDe where there are fewer errors to correct, we improve accuracy by at least 0.9% for each setting (tables 1,2). These consistent gains also hold when using a different VLM (BLIP-2), where we improve by at least 0.8% for every setting aside from GeoDe (tables 7,8). These gains are consistent on each of the 8 datasets as well, as shown in table 9. Averaging over all datasets and both VLMs, we improve accuracy compared to WaffleCLIP by 1.4%, with a larger gain of 2.6% and 2.4% for the worst 20% of classes and subpopulations respectively (see table 6). We stress that these latter metrics (worst class/subpopulation) more closely reflect the core problem we seek to tackle, and see the larger gains there as encouraging evidence that our method improves upon the limitation it was designed to address (i.e. underperformance on atypically appearing instances).

Moreover, WaffleCLIP offers no interpretability, while we offer faithful and fine-grained explanations for each inference, for free (Sec. 5.2). Similarly, overall, the reason why WaffleCLIP is effective is not well understood. In contrast, we make principled arguments as to why our method should work, and we support these arguments with extensive empirical evidence, such as notable gains for the least performant classes and subpopulations, and precise ablations (Sec. 5.3, 5.4).

In summary, while WaffleCLIP is surprisingly strong, our method is (i) consistently stronger, (ii) unambiguously more interpretable, and (iii) better understood with respect to why it works.

(W2) Respectfully, we disagree that diversity within classes is an artificial problem. Even something as simple and well defined as a ‘pear’ can actually manifest in many visually distinct ways. Moreover, this is a problem with precedent, as there are numerous existing efforts within the AI community to ensure models work beyond typical instances. For example, the field of out-of-distribution (OOD) generalization focuses on improving performance where some aspects of test inputs, such as spurious correlations [1,2,3] or domains [4], are shifted compared to the training data. Many works have also focused on algorithmic fairness towards mitigating biases, which often consist of model’s underperforming on instances that are less common / atypical (e.g. due to belonging to a demographic group underrepresented in the training data) [5]. In fact, our worst subpopulation/class metrics are inspired by the OOD community (e.g. ‘worst group accuracy’ is standard), and DollarStreet and GeoDe come from the fairness community, as they show diversity in images (resulting in harmful discrepancies in performance) is a naturally arising real-world problem. Notably, these datasets contain everyday objects whose labels are intuitive and unambiguous (that is, the class names are in no way designed to artificially capture many sub-categories). Many other benchmarks also exist to measure how classifiers will perform when encountering atypical data [6,7], suggesting this is a problem the scientific community deems important.

Nonetheless, we agree that it's important to verify that our method works generally, and not just for one set of attributed datasets curated to inspect accuracy along various axes of diversity. We explore this below.

[1] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, https://arxiv.org/abs/1911.08731 [2] WILDS: A Benchmark of in-the-Wild Distribution Shifts, https://arxiv.org/abs/2012.07421 [3] Invariant Risk Minimization, https://arxiv.org/abs/1907.02893 [4] In Search of Lost Domain Generalization, https://arxiv.org/abs/2007.01434 [5] Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, https://proceedings.mlr.press/v81/buolamwini18a.html [6] The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization, https://arxiv.org/abs/2006.16241 [7] ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models, https://openreview.net/forum?id=SkgnRNHgIS

We thank the reviewer for their feedback. We now address the raised concerns.

Strong and consistent gains compared to using a single vector for zero-shot inference: We note that we did indeed consider using the zero-shot capabilities of VLMs directly without attributes. This corresponds to our ‘vanilla’ baseline, where a single vector (i.e. the classname embedding) is used per class. Our method significantly outperforms this baseline, with average (over 8 datasets and two VLMs) improvements of +1.5% in accuracy, +2.6% for the worst 20% of classes, and +2.7% for the worst 20% of subpopulations (see table 6). Notably, we achieve higher accuracy than this vanilla baseline for every setting. In additional experiments over 9 new datasets conducted during the rebuttal period, we see similar gains over this baseline: +1.8% in accuracy and +1.5% for the worst 20% of classes* (see table A in the response to Reviewer KwVV). In fact, we see similar gains to stronger, more recent baselines that also incorporate additional information beyond classnames: compared to each baseline, on average, our method yields gains of 1.3% and 1.7% in accuracy overall and for the worst 20% of classes respectively.

The reason why we infer attributes is to enrich the model’s coverage of all instances within a class, as the classname embedding often inadequately covers atypical subpopulations. The larger gains for worst class and subpopulation metrics suggest our method indeed improves on the baseline by better covering challenging (usually atypical; see sec 3.1) instances, showing that including attributes is well-motivated and effective. Our method also provides clear interpretability advantages; both the standard method and most baselines have no or little interpretability, while we offer fine-grained and faithful interpretations for each inference.

*The new datasets do not have attribute annotations, so we cannot report accuracy on the worst 20% of subpopulations, as a subpopulation consists of all instances within a class that share an attribute.

On computational cost: Crucially, we only infer attributes and embed subpopulation vectors once for any classification task. The other extra computations are all very simple (e.g. computing the similarity to more vectors only involves the dot product), and importantly, they take far less time than encoding the test image, which is a necessary step for any zero-shot method. Thus, aside from a constant cost done once and a small number of very quick extra computations, our method is nearly just as fast as the standard method per test image. Moreover, our method improves accuracy, performance on atypical samples, and interpretability. Finally, we note that our method does indeed improve as more attributes are incorporated, even though the cost of adding attributes is minimal. Further, our method is unique in this regard: Sec. 5.4 shows how baselines saturate or deteriorate as more attributes are added, while ours continues to improve. Thus, (i) the added computational cost is very small (and zero asymptotically), and (ii) our method continues to improve with more attributes, so we believe the benefits of our method certainly outweigh the very small extra cost.

To summarize, we’ve studied the requested baseline, in the original submission and once more in a new set of evaluations during the rebuttal. In both cases, we see significant gains from our method. Also, we’ve explained how the added computational cost of our method is minimal. We hope these address your concerns, an if you feel as though they have, we’d greatly appreciate it if you could raise your score to an accept. We’d also be more than happy to answer any follow up questions if concerns linger. Thank you!

First, I don't agree with R1 that there is no value to be able to have fine-grained zero-shot classification. In real-world, fine-grained classification is important, and it can be crucial to policy making if a model can correctly identify "arctic fox" from "wolves". This is especially true in conservation areas. We can not always rely on supervised learning approaches as it is not scalable. Therefore, I think it is important for people to keep looking for solutions for zero-shot fine-grained recognition in large vlm era. I think the comments from the author answered most of my concerns, and the method proposed is simple but still effective. However, like R1 mentioned, the performance is relatively limited, so I will keep my original rating. Thanks and sorry for the late response.

Thank you very much for your feedback. We especially appreciate that you note how our method takes advantage of the complementary nature of LLMs and VLMs to tackle the problem of improving classification in the face of high intra-class diversity. We now address each of your concerns. We highlight that we conduct two additional experiments: (1) a large evaluation of 9 new datasets, (2) a time analysis of the added cost of our method.

1. Your intuition is very close to correct. One added detail, as shown in figure 4, is that our consolidation method allows for an atypical subpopulation to be classified correctly even when it is far from most subpopulation embeddings for that class. This property is unique to our consolidation method: using a single vector (either by not including attributes or by averaging over attributes), a sample is penalized if it is far from most subpopulations, even if the subpopulation it belongs to happens to be very different from all other subpopulations (like the Arctic fox or Red wolf). In figure 4, we use $k=1$, which means that a sample only needs to be close(st) to one subpopulation from its true class in order to be correctly classified. We use $k=1$ for simplicity in Figure 4, and discuss the choice of $k$ in depth in Sec 5.4.

2. Thank you for pointing out our experiments warrant further investigation of finer-grained datasets. Based on your suggestion we ran extra experiments on 9 new datasets, which are nearly all fine-grained. Fortunately we find that our method still improves against all baselines consistently over datasets, with an average improvement of +1.3% over each baseline when using CLIP (see table A). Nonetheless, we will add a limitations section to discuss when our method may not be best suited (e.g. super fine-grained tasks with precise class names), at your suggestion.

3. In practice, hierarchy or attribute labels are not available. Thus, one must infer them. Doing so manually is costly, and even prohibitively so when the number of classes grows, or when you consider as many axes of variation as we do. In contrast, our method is fully automatic, allowing for the attribution of dozens of classes along many axes of variation to be done in just tens of minutes. Specifically, we timed how long it took to generate LLM responses to all queries in our method over multiple datasets, and found that on one GPU (no parallelism), our attribution takes roughly 18.5 seconds per class. Thus, attributing datasets with 30 classes takes only about 10 minutes, and this number can be sped up by parallelizing. Crucially, this is done only once per classification task. Therefore, the computational cost per test image to classify is nearly equivalent to standard zero-shot classification.

Feel free to comment if there are any lingering questions. If you feel that we have addressed all your concerns (namely that our method is too costly computationally or that it may not work for finer grained datasets), we would be very grateful if you could increase your score. Thank you!