Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition

Thank you for your thoughtful review and encouraging feedback. We are glad that you considered our work “enjoyed reading, impressive theory, effective”. We are glad to address your questions.

Q1: Why do CLIP models still suffer from co-occurrence bias despite their strong zero-shot capabilities?

A1: Thank you for raising this important point. While CLIP performs well on familiar object–context pairs, it also encodes object–context shortcuts from its pretraining data (e.g., LAION-2B), which limits its generalization to novel combinations. Our benchmark are designed to expose such biases, with worst-group accuracy dropping as low as 34.5% (Tab. 8). We believe addressing this issue is critical for the following reasons:

-Context bias is inherent in pre-trained VLMs. As shown in our PMI analysis (Fig. 2), LAION-2B shares gender–scene co-occurrence patterns with COCO and contains low-level artifacts like watermark–carton correlations [1]. These biases stem from the data itself and cannot be removed by simply scaling it, but can be addressed through targeted counterfactual intervention.

-Larger models amplify shortcut learning. Scaling to ViT-L/14 or ViT-H/14 does not resolve the issue. On the contrary, larger models tend to memorize and exploit spurious correlations more aggressively in ambiguous or occluded cases. This matches findings from Kaiming He et al. [2], which show stronger models are more sensitive to dataset-specific shortcuts, making bias mitigation even more crucial as models scale.

-Biases affect downstream systems. Many LVLMs, such as MiniGPT-4, LLaVA, use CLIP as their image encoder and inherit these hallucination patterns [3], leading to potential failures in real-world applications. Therefore, we believe CLIP remains a relevant and important testbed for studying and mitigating object–context bias.

We will include these discussions in the revised version.

$$

Q2: Whether predefined context pools are limited in open-ended or zero-shot settings?

A2: Thank you for raising this insightful concern. We agree that context pool diversity is essential to our method. Our approach simulates interventions by imagining an object in alternative contexts at representation-level, which is conceptually similar to randomized trials. For these to be effective, we only require a diverse set of dissimilar contexts, which discourages reliance on pretraining shortcuts (see Fig. 11).

In practice, the candidate pool is not fixed or dataset-specific. For structured datasets like Waterbirds, we can design targeted pools to match domain-specific scenes via a priori. For open-domain datasets like COCO, it makes sense to sample from Places365. Our method also supports more larger open-world datasets (e.g., Open Images, SUN397, ADE20K), which cover diverse real-world scenes without extra labeling costs. With our sampling strategy (lines 220–223), selecting from such pools remains efficient and lightweight.

Moreover, the virtual variant enables text-guided counterfactual construction using LLMs like GPT-4, allowing for rich and diverse prompts beyond any limited scenes dataset. This makes our method task-agnostic and extensible: once a diverse virtual context pool is built, it can be reused across tasks or deployed in zero-shot settings.

We will clarify this in the revision and emphasize that our method does not rely on a fixed context pool, but rather supports open-ended generalization through flexible sampling strategies.

$$

Q3: Is the high performance of the WB dataset a coincidence that TDE solves overfitting?

A3: Thank you for raising this question. We clarify that the large improvement on the Waterbirds dataset (over 44%) is not due to overfitting, but rather from effectively reducing the strong object–context bias that CLIP inherits from pretraining on LAION-2B. As shown in Tab. 8, CLIP performs well on common object–background pairs but fails on rare ones, which is precisely the issue our method targets.

While some gain may come from using Places365 as an external pool which aligns well with WB backgrounds, we also observe strong performance with internal and virtual variants, confirming the method’s generalization across context sources. Moreover, Our method generalizes well across different CLIP backbones (ViT-B/32 to H/14) and datasets (UrbanCars, COCO-GB, NICO), consistently improving worst-group accuracy. This suggests that it reliably mitigates hallucinations caused by contextual bias.

Besides, the effectiveness is not due to a simple subtraction. It is derived from a counterfactual optimal estimation framework (Sec. 3.1), where we enable to compute the Total Direct Effect (TDE), using $\hat \lambda$ to suppress the confounding context-only pathway. This formulation, along with multiple $C(z)$ variants simulating interventions (Eq. 17), helps prevent context-driven misclassification, such as predicting “albatross” based only on the ocean in Fig. 1(b).

We will revise the paper to clarify these distinctions and reinforce the broad applicability of our method beyond any single dataset.

$$

Q4: Clarification of the analytical experiments

A4: Thank you for pointing this out. We fully agree that in-depth analysis is crucial to help readers better understand our method. Due to space limitations in the main paper, many of our extended analyses were placed in the appendix.

In particular, we provide additional experiments across four datasets in Appendix D, and introduce detailed pseudocode in Appendix C.3 to facilitate reproducibility. Moreover, Appendix E.2 offers a comprehensive analysis of hyperparameters, and Appendix E.1 compares our method with 4–5 classical augmentation techniques by adapting them into counterfactual variants.

Additionally, we plan to include more comparison baselines and provide FLOPs analysis (access to relevant tables in the responses to reviewers XSqm and 7obT), to further strengthen the evaluation. Finally, we will improve the main paper by explicitly pointing to relevant appendix sections, so readers can more easily access the extended analysis.

$$

Q5: Discussion on the use of term "hallucination"

A5: Thank you for raising this point regarding term. We clarify that, at the mechanism level, object–context bias refers to the learned co-occurrence shortcuts during training, while hallucination describes the inference-time outcome where the model makes overconfident predictions based on misleading contextual cues [4, 5].

In our paper, “hallucination” refers to cases where the model predicts a label even when the target object is absent. For example, in Fig. 1(b), the model predicts “albatross” based solely on the ocean background. Alternatively, one can view it as CLIP mistakenly retrieving or generating a caption that strongly matches the background, but does not reflect the true content of the image.

To avoid ambiguity, we will clarify it in the revised draft, avoiding the use of “hallucination” or replacing it with “context-induced misprediction”.

$$

Q6: Discussion on semantic-token direct effect

A6: Thank you for the insightful suggestion. We agree that Section 3.1 can be improved, and we will revise it to better explain how the semantic-token is captured. Specifically, we will clarify how the direct-effect decomposition helps assign distinct semantic roles to different tokens.

MSA dominates semantics. Unlike MLPs which process each token independently, multi-head self-attention (MSA) allows tokens to exchange information, enabling the model to gather semantic cues across spatial regions. Prior works support this: Caron et al. [6] show that attention heads can recover segmentation structures even without supervision, and Naseer et al. [7] find that ViTs use long-range attention to model global semantics. Recent study [8] also shows that attention-guided token grouping transfers well across datasets and tasks. Our mean-ablation study on ImageNet (Tab. 5) further confirms that removing MSA significantly weakens semantic attribution.

Direct effect makes sense. Although Transformers contain nonlinear components, our decomposition remains interpretable in CLIP, as the final prediction is computed by a linear inner product between image and text embeddings. This allows us to estimate each token’s semantic contribution to the final representation (see Eq. 6). Furthermore, ViTs include residual connections after each MSA and MLP block, allowing us to compare representations with and without specific modules (see Eq. 33), making direct-effect estimation possible even in nonlinear models.

We will clarify these points in the revision and cite relevant work to support our method.

Reference:

[1] Li, Zhiheng, et al. "A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others." CVPR2023.

[2] Zhuang Liu and Kaiming He. "A decade’s battle on dataset bias: Are we there yet?" ICLR2025.

[3] Chen, Junzhe, et al. "Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models." CVPR 2025.

[4] Liu, Yufang, et al. "Investigating and mitigating object hallucinations in pretrained vision-language (clip) models." EMNLP 2024.

[5] Chen, Xuweiyi, et al. "Multi-object hallucination in vision language models." NIPS2024.

[6] Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV 2021.

[7] Naseer et al., “Intriguing Properties of Vision Transformers,” NeurIPS 2021.

[8] Yossi Gandelsman, et al. "Interpreting CLIP’s image representation via text-based decomposition." ICML2024.

Thanks for your appreciation of our paper. We are glad that you considered our work “sound, effective, easy to follow”. We are glad to answer all your questions.

Q1: Discussion on the impact of context diversity

A1: Thank you for the insightful comment on the diversity of the background pool for synthesizing counterfactual embeddings. We fully agree that the quality, especially the diversity, of counterfactual embeddings is more critical than their quantity. As discussed in lines 94–97 and analyzed in Fig. 11, a diverse candidate pool better approximates ideal interventions and more effectively mitigates biases embedded during training.

Our sampling strategy (lines 220–223) is designed to achieve scene diversity and semantic plausibility. The external and internal variants sample from Places365 and in-batch sources, respectively, helping align the synthesized embeddings with real-world object–context co-occurrences. Additionally, we introduce a virtual variant, where counterfactual embeddings are constructed directly in the shared vision–language space via text. Leveraging recent advances in LLMs such as GPT-4, we can efficiently generate rich and diverse scene prompts at scale, enabling low-cost counterfactual construction.

This approach also offers a practical benefit: it enables task-agnostic deployment. Once a diverse virtual pool is built, the same intervention strategy can be reused across tasks without additional manual effort.

We will clarify these points in the revised draft, and we thank the reviewer again for emphasizing this important aspect.

$$

Q2: Clarifying table and figure representation

A2: Thank you for pointing this out. We will revise the layout and captions to make the results easier to read. Specifically, we will explicitly indicate the corresponding dataset in the captions of Tables 1, 4, 11, 12, and Fig. 11, and place each table or figure closer to its first mention in the text. We will also include brief descriptions of the metrics and datasets in each caption to avoid confusion and reduce back-and-forth reading. These changes will be reflected in the revised version to enhance readability and accessibility.

$$

Q3: Explanation of typos on lines 121-124

A3: Thank you for pointing out this confusion. You are right, and there is indeed a typo in lines 121–124. Our intention was to illustrate how a novel background $z'$ can reverse the prediction for the same object $x$, especially considering the co-occurrence pairs $(x,z)$ and $(x',z')$ observed during training. Concretely, consider a novel pair $(x,z')$ that rarely (or never) co-occurred during training, and thus has a low interaction term $r_i$. Define the decision margin as:

$$\delta = S(\mathbf i_{x,z'},T_{c_x})-S(\mathbf i_{x,z'},T_{c_{x'}})$$

where $c_x$ is the true class of $x$ and $c_{x'}$ is a confusable alternative.

Then, using the decomposition in Eq. 4, this margin can be rewritten as:

$$\delta =(\langle e_i(x), f_i(T_{c_x}) \rangle - \langle e_i(x), f_i(T_{c_{x'}}) \rangle) \;-\; (\langle e_i(z'), f_i(T_{c_{x'}}) \rangle - \langle e_i(z'), f_i(T_{c_x}) \rangle)$$

This expression shows that if the contextual interaction effect (inherited from the co-occurrence pairs $(x',z')$) dominates the object evidence (the first term), then $\delta<0$ and the model incorrectly favors $c_{x'}$. This explains the failure case shown in Fig. 1(a), and is the underlying mechanism for the hallucinated scores visualized in Fig. 1(b).

We will correct the notation and clarify the intent in the final version.

$$

Q4: Clarification on whether the embeddings are centered

A4: Thank you for raising this point. Yes, we explicitly apply mean-centering to the embeddings. Before downstream operations, such as CLIP inner product scoring or TDE estimation, we subtract the mean embedding (computed over the full dataset or mini-batch) from each vector. This ensures that the theoretical assumption in Appendix B (lines 456–457) is faithfully reflected in our actual implementation. We will publish the algorithm code in the future.

$$

Q5: Suggestions on clarity, presentation, and typos

A5: Thank you for the detailed and constructive suggestions. We agree that Algorithm 1 provides a clear summary of our method, and we will explicitly reference it in Section 3 to help readers navigate to the full procedure in Appendix C.3. We also appreciate the recommendation to introduce the hyperparameter $\hat{\lambda}$ earlier, and will include a brief explanation alongside Eq. 14 in the main text.

Additionally, we are grateful for the careful review of typos, inconsistent notations, and formatting issues (e.g., notation for $f_t$, attention weights, expectation symbols). We will thoroughly proofread the paper and correct all potential issues to improve overall clarity and precision in the final version.

Thanks for your constructive comments. We are glad that you considered our work “novel, well-structured theoretical, interesting” and we will answer all your questions.

Q1: Motivation and reason for selecting CLIP

A1: Thank you for raising these valuable concerns. We are glad to clarify our motivation in the revised version.

-Stronger encoders do not eliminate co-occurrence bias: While recent backbones (e.g., DINO, PE) improve average, they still rely on object-context shortcuts in challenging or occluded scenarios [1]. Dataset classification studies [2] interestingly show that larger pre-trained models amplifies such easy inter‑domain cues and passes them to downstream tasks [1]. PMI in our paper confirms that LAION‑2B (CLIP’s pretraining dataset) shares the same gender‑scene patterns as COCO, meanwhile Li et al. [3] reveal watermark–carton co‑occurrences that skew predictions. Thus, simply switching to larger encoders or datasets does not remove co‑occurrence bias and it continues to threaten both safety and fairness.

-Lightweight way to address shortcut reliance, not encoder competition: Our goal is to provide a lightweight, inference-time plugin to mitigate hallucinations, without the high cost of generative editing. We adopt CLIP-style VLMs as our base models because CLIP serves as a strong and widely used baseline, and its contrastive pretraining reveals the object–context shortcuts analyzed in Section 2.

Moreover, our method is architecture‑agnostic: built on attention‑based direct‑effect decomposition and TDE, it can be transferred to stronger encoders. We provide initial evidence across four CLIP backbones with increasing capacity, demonstrating its compatibility (Tab.1, 8-12). Starting with the CLIP allows a clean theoretical exposition and avoids interference from extra modules in newer models.

$$

Q2: Additional benchmark

A2: Thank you for this suggestion. We deliberately chose Waterbirds, UrbanCars, COCO‑GB, and NICO because each dataset explicitly encodes object‑context shortcuts (e.g. backgrounds, gender cues, co‑occurring objects, etc.) that let us quantify worst‑group accuracy, the primary metric for our debiasing method. As shown in Fig. 1 and Tab. 8, worst‑group accuracy reveals a model’s true bias sensitivity: even strong zero‑shot CLIP variants score only ~40 % on this metric across all four architectures.

Datasets such as Winoground are designed for fine‑grained text–image reasoning. They focus on text-generated hallucinations or caption mismatches and also do not report group‑wise accuracy. Hence they are better suited to evaluating LVLMs hallucinations when CLIP is used merely as an image encoder  component [1]. While that setting is not our current focus, our method can be applied to MiniGPT‑4, LLaVA, etc. methods on these datasets. In future work we plan a broader evaluation on benchmarks like Winoground to complement the object‑context bias tests.

$$

Q3: Algorithm implementation clarity

A3: We are glad to explain any ambiguity. The complete implementation can already be reconstructed from three parts of the paper:

The end-to-end inference pipeline is detailed in Algorithm 1 (Appendix C.3, p. 29):

(i) extracting main-effect embeddings (Eq. 6 and 8) → (ii) constructing counterfactual samples(Eq. 15 and 16)→ (iii) fusing scores (Eq. 14, 17, and 18).

Additionally, we provide ready-to-use YAML configs for each dataset (Waterbirds, UrbanCars, COCO-GB, NICO) in the original supplementary README.md, including the sampling script and full hyperparameter settings. In the revised version, we will reference Algorithm 1 more clearly in the main paper, expand more hyperparameter details, and release a public repo for reproducibility.

$$

Q4: Quantitative Analysis of Efficiency

A4: We appreciate the reviewer’s concern. While our method introduces additional inference steps, we emphasize that all operations are performed in the embedding space, with no image re-encoding or pixel-level augmentation required.

We agree that quantitative metrics are essential for evaluating overhead. In the revised version, we will add some experiments, including comparing to ALIA [4], a language-guided generative baseline. By default, FLOPs measures the total number of floating-point operations, assuming pre-computed text embeddings. Values in table are reported as FLOPs increments relative to CLIP baseline.

Table Note: Efficiency on Waterbirds across four different backbones during inference-time. $M$ is the number of counterfactual samples per image in Eq. 15.

Table Note: Efficiency on ViT-B/16 varies from the number of counterfactual samples $M$ on Waterbirds.

As shown above, the computational introduced by our method is minimal, especially compared with generative approaches. CLIP performs a single forward pass through a frozen encoder, and our method preserves this efficiency: it constructs counterfactual embeddings using closed-form computations (Eq. 9, 17) over precomputed token features, followed by lightweight TDE fusion (Eq. 18). All operations involve only matrix multiplications and element-wise computations, without any backpropagation or additional network training. Moreover, since the context candidate pool can be prepared offline, our method remains highly efficient in large-scale data deployment..

$$

Q5: Discussion on scaling law in our work

A5: Thank you for this insightful question. Our results on Waterbirds (Fig. 1(a) and Appendix Tab. 8) closely mirror those reported by Kaiming He et al. [2]. Across four CLIP backbones' results, we draw two key conclusions:

-Small models (B/32, B/16): Their limited ability under-represents the object itself, so worst-group errors stem primarily from insufficient object representation. Increasing capacity or resolution directly boosts both average and worst-group accuracy.

-Large models (L/14, H/14): For “easy” high-resolution images, object cues quickly saturate. Extra capacity then tends to amplify memorized object–context shortcuts in ambiguous or occluded cases, leading the model to “guess” from background more aggressively, precisely the hallucination our method mentions.

This pattern mirrors tests in [2]: even a tiny 7K-parameter ConvNeXt can achieve 72.4% on dataset-ID classification, rising to 85% when scaled to 87M parameters. Intuitively, classifying dataset splits should be randomly, but larger models systematically learn dataset-specific shortcuts, producing stronger hallucinations downstream [1].

Hence, increasing backbone scale without bias control actually reinforce shortcut learning. In contrast, our representation-level counterfactual plugin pairs integrates with any model size to preserve object recognition while suppressing context-driven hallucinations,enabling safer and more reliable deployment. We plan to clarify these scaling effects in the revised version.

$$

Q6: Discussion on training paradigms

A6: This is a very insightful question. Although current design is inference-only, the counterfactual mechanism is compatible with training paradigms. The key idea which modifies object–context combinations via counterfactual synthesis, can be naturally integrated as a data augmentation or regularization strategy during training.

More broadly, we agree that incorporating more data often enhances generalization. However, data scaling alone is insufficient—large-scale datasets tend to contain spurious correlations that models exploit as shortcuts. Our counterfactual approach provides a targeted and controllable intervention to address it directly.

In particular, the virtual variant constructs semantically grounded counterfactual embeddings using text prompts in the vision–language space. This enables the generation of lightweight synthetic supervision signals without retraining or external annotation cost. It also makes our approach suitable for scalable training-time integration in the future, offering a promising path to debias models.

Reference:

[1] Chen, Junzhe, et al. "Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models." CVPR 2025.

[2] Zhuang Liu and Kaiming He. "A decade’s battle on dataset bias: Are we there yet?" ICLR2025.

[3] Li, Zhiheng, et al. "A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others." CVPR2023.

[4] Dunlap et al., “Diversify your vision datasets with automatic diffusion-based augmentation,” NeurIPS2023.

[5] Yossi Gandelsman, et al. "Interpreting CLIP’s image representation via text-based decomposition." ICML2024.

Thanks for your appreciation of our paper. We are glad that you considered our work “well-written, effective, interesting”. We are glad to answer all your questions.

Q1: Comparison to [1] and why focus mainly on contextual bias

A1: Thanks for the insightful comment. While Sauer et al. [1] utilize generative models to disentangle different components, our method focuses specifically on addressing contextual bias considering in three aspects:

- Empirical evidence that context drives most failures: Well-trained models easily capture low-order, “easy-to-learn” semantics, while high-order cues are harder to generalize [2]. In our error analysis on Waterbirds and COCO-GB, we found that most misclassifications arise from small, ambiguous, or obstructed object, which leads the model tend to over-rely on background. After all, we wouldn’t want a system that sees a kitchen and always predicts woman, or sees shrubbery and always predicts landbirds.

- No external generative networks or fine-tuning: Disentangling texture and shape requires training or fine-tuning separate encoder-decoder networks. In contrast, we operate directly at inference time on a frozen CLIP model, using a closed-form solution (Eq. (9), Appendix B.7) to estimate object and background direct effects. This avoids the need for additional parameters and training.

- Better zero-shot generalization: Introducing dedicated texture and shape models may anchors the model to its specific data domains (e.g., bird textures in Waterbirds), which limits transfer to novel categories or instances (e.g., humans appearances in COCO-GB). Since context is more readily decoupled from object identity than low-level appearance, our strategy better supports zero-shot recognition performance.

We will include these discussions in the revised version.

$$

Q2: Supplemental citation [3]

A2: Thank you for suggesting the Memo [3]. We have added this citation in Section A and clarified that, differing from pixel-level TTA methods (which adjusts model more robust under augmentations), our framework operates in the semantic space of frozen CLIP. It decomposes and subtracts background effects and synthesizes counterfactual embeddings. This causal, representation-level approach directly addresses the issue of object-context confusion and controls hallucination effects caused by context.

$$

Q3: Experimental details on sample size for constructing counterfactual embeddings

A3: Thank you for your interest in experimental details. The full parameter settings, including sample size selection, are provided in the README.md of our original supplementary materials. We also promise to release the code in the future.

Regarding the number of counterfactual samples generated, we conducted a parameter analysis in Fig. 11. For instance, the ViT backbones with increasing parameter sizes in Waterbirds , the virtual variant method sampled 270, 270, 190, and 90 from pool, as shown in Tab. 1 (sampling rules can be found in lines 220-223). The scale of the candidate pool (400) can be adjusted by the experiment designer offline. Larger and more diverse candidate pools may result in fewer size.

$$

Q4: Additional baseline information and clear baseline comparisons

A4: Thank you for the helpful suggestion. While the information citation "[2]" may have been unintentionally omitted by reviewer, we agree with the intention of introducing more augmentation-based baselines. Beyond Mixup, CutMix, AugMix, and Cutout shown in Appendix E.1 (Tab. 13, 14), we further include ALIA [4], a diffusion-based method that performs language-guided image editing. We adapt ALIA into a counterfactual variant (see lines 773–777) and report its results. However, it incurs much higher computation cost compared to our lightweight representation-level approach.

Table Note: Performance on Waterbirds on ViT-B/16 backbones. Efficiency are reported as FLOPs increments relative to CLIP baseline. Avg. and Worst. indicates average accuracy and the worst group accuracy.

We acknowledge that Appendix E.1 did not clearly reference and we will update the draft to clarify table 13-14 indices and improve formatting for better readability.

$$

Q5: Justification for COCO-GB performance

A5: Thank you for raising this point. On COCO-GB v1 we report two key metrics: average accuracy which measures overall recognition performance and utility; gender gap, which indicates fairness. In Tab.8, our virtual variant achieves the lowest gap of 0.8% (compared to CLIP’s 6.2%) while still reaching 91.6 % average accuracy, outperforming all other methods on both fairness and utility simultaneously.

A further advantage of our framework is its tunable suppression factor $\hat{\lambda}$ . Users who prefer higher accuracy over strict parity (or vice-versa) can adjust $\hat{\lambda}$ to smoothly trade off between average accuracy and gap, making the method adaptable to different deployment requirements. We will clarify these points in the revised draft.

$$

Q6: Discussion on multiple models approach

A6: Thank you for thoughtful suggestion. Ensemble methods that aggregate predictions from multiple models or strategies are indeed effective in bias mitigation. For instance, Li .et al [5] combines classifiers trained on various augmentation models (e.g., watermark, texture, background) and aggregates their outputs to improve robustness. However, this approach requires retraining multiple specialized models, making it unsuitable for zero-shot settings. To provide a fairer comparison, we included the counterfactual variant of AugMix, which uses various image transformation strategies similar to [5].

Additionally, we implemented the zero-shot ensemble strategy proposed in [6]. This approach aggregates multiple pre-trained VLMs (e.g., CLIP variants) without fine-tuning, improving over any single model by leveraging architectural diversity. We evaluate these on benchmarks and present results below.

Table Note: Performance on ViT-B/16 backbones.

Reference:

[1] Sauer A, Geiger A. "Counterfactual generative networks". arXiv 2021.

[2] Nam J, Cha H, Ahn S. "Learning from failure: De-biasing classifier from biased classifier". NIPS 2020.

[3] Zhang, Marvin, Sergey Levine, and Chelsea Finn. "Memo: Test time robustness via adaptation and augmentation." NeurIPS 2022.

[4] Dunlap et al., “Diversify your vision datasets with automatic diffusion-based augmentation,” NeurIPS 2023.

[5] Li, Zhiheng, et al. "A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others." CVPR2023.

[6] Lu, Zhihe, et al. "Beyond sole strength: Customized ensembles for generalized vision-language models." ICML2024.