Debias your VLM with Counterfactuals: A Unified Approach

Thank you for your insightful review and suggestions. Please refer to the general response for issues not discussed below.

---

Novelty in terms of the main framework.

Please see point “Novelty” in general comments. We appreciate the additional reference of NIFTY (Agarwal et al., 2021) which is conceptually similar and will acknowledge the work in the updated version, although we argue that the scope of the two works differ substantially: while NIFTY aims to fair and robust GNNs with a known underlying causal graph, our work involves vision-language transformer models on raw images and text as inputs, with the underlying graph unknown and simulated by image-text editing models. The primary contribution of CVLD is that it demonstrates the effectiveness of image-text editing models in debiasing VLMs across multiple downstream tasks with no architectural changes, which has not been demonstrated in prior work to our knowledge.

Counterfactual image seems noisy.

Refer to “Quality of counterfactual editing” in general response. More examples of intervened images are provided in Fig. 7 and 8, highlighting the capabilities of editing models. Quantitative measures of edit quality are also included. We agree that the noise and imperfections in the counterfactual edits are unavoidable, but would like to emphasize that CVLD has shown its effectiveness for improving model fairness on standard metrics across classification, retrieval and captioning. Such improvements cannot be achieved simply by adding random noise to training images.

Data-extensive approach.

See point “Computational cost/overhead of CVLD” in general response. It should also be noted that the experiments are deliberately conducted in a low-resource regime where we considered few-shot classification and fine-tuned retrieval/captioning on as low as 1% of COCO, which greatly reduces the cost from adapting to the full training set. CVLD is found to be effective across different scales of training set size.

Thank you for your insightful review and suggestions. Please refer to the general response for issues not discussed below.

---

Unified approach.

We acknowledge that the framing of CVLD as “universal” and “task-agnostic” may have been misleading, since they refer to the data debiasing process, while the subsequent model adaptation still involves “a simple regularized training objective and minimal modifications to the adaptation pipeline” for each task, as mentioned in the Intro section. However, in defense of the title of the paper, we argue that CVLD, as a data-centric approach, has the distinctive advantage over model-centric mitigation methods in that it greatly reduces the effort to tailor the training pipeline to downstream tasks. The adaptation procedures in Sec. 4 can be distilled into two modes: For classification and captioning, the training is merely augmented by counterfactual examples with a weight parameter ($\lambda$), with no change to model computation at all. For retrieval, the contrastive loss itself is modified to account for counterfactual candidates in each batch, which reduces to simply modifying the optimization targets during training. The internals of the VLM is again unchanged, as illustrated in Fig. 3, meaning no overhead in deployment over a vanilla VLM (more discussion in “Computational cost/overhead of CVLD” in general comments). In contrast, model debiasing approaches tend to utilize additional modules, multiple training stages, and/or sophisticated loss functions, making them less trivial to adapt to a wide range of tasks. While there was not enough time to conduct the experiments, we believe that edited datasets produced by CVLD may also benefit tasks like detection/segmentation on COCO, as we find the edits to preserve the scene layout and background objects with high probability (Fig. 7 in appendix). In general, we believe that while not truly universal, CVLD is a significant step forward in minimizing the efforts needed to debias foundation VLMs on different downstream tasks.

Evaluation of the data generation pipeline & relation to debiasing performance.

Refer “Quality of counterfactual editing” in general comments. We appreciate the suggestion to include additional evaluation metrics for the counterfactual intervention. It is difficult to conclude how the edit quality metrics relate to the eventual debiasing performance, as the metrics rely on pre-trained CLIP which itself may be biased. However, the current version of CVLD performs sample selection based on visual similarity to the input image, and we observed in preliminary experiments that this strategy produces slightly better debiasing results than uniformly sampling edits at random, suggesting that poor quality images deviating from the input data may be detrimental.

In general, we believe that the ability of CVLD is bounded by the capabilities of the editing models, and that future study may be needed to establish a definitive link between edit quality and debiasing performance. However, as demonstrated in “Evaluation beyond gender bias” in general comments, we observe positive sign that the existing editing models are powerful enough for many commonly studied forms of bias in vision-language.

Experimental results.

We would like to clarify that all evaluation datasets and metrics in this work are well-established standards in the literature. These include DEO for classification on CelebA (Ramaswamy et al., 2021), Bias and MaxSkew for retrieval on FairFace (Berg et al., 2022), BiasAmp and LIC for captioning on COCO (Hirota et al., 2022). For all fairness metrics lower is better, as is now indicated in the main tables of the paper. The fact that CVLD not only reduces model bias by large margins without sacrificing in-distribution performance, but also achieves so through simple image-text editing and no task-specific model debiasing, is particularly striking to us.

In Tab. 1, we showed results of CVLD in both 16- and 256-shot settings, and compared them to naive BLIP using the same number of shots. In the first section of the table, BLIP-PT is not adapted to CelebA dataset while BLIP and RN-50 are fine-tuned on the full dataset, making them not directly comparable to the few-shot results below. Best results of each section of the table are highlighted in bold.

Computational cost of data generation.

See “Computational cost/overhead of CVLD” in general response.

Thank you for your insightful review and suggestions. Please refer to the general response for issues not discussed below.

---

Only covers gender biases; unquantifiable biases & multiple biases.

Please see “Evaluation beyond gender bias” in general response. Additional results on background bias on Waterbirds and racial bias on COCO are included, demonstrating that CVLD is generally applicable to various forms of biases. Our understanding is that CVLD is potentially useful for all settings where 1) bias can be described in natural language, and 2) corresponding causal interventions (e.g., change gender/race of subjects, or replace the background) can be reliably performed by the image-text editing models.

Simultaneous mitigation of multiple biases is a great suggestion. One simple way is to extend CVLD with multiple edits, one for each protected attribute, which presents a challenge to the compositionality of editing models. However, as we intend for CVLD to become a baseline in data-centric VLM debiasing, we believe that the study of multiple bias attributes lies beyond the scope of this work and should be left for follow-up research.

Lack of novelty; comparison to other methods.

Please see point “Novelty” in general comments. CVLD showcases the effectiveness of image-text editing models in debiasing VLMs across multiple downstream tasks with no architectural changes, which has not been demonstrated in prior work. We consider the simplicity of the proposed framework a strength, as it does not require tweaking the internal model architecture or sophisticated training procedures, making the method more practical for tuning and deploying models at scale. It also makes the method complementary to model debiasing approaches, which may further benefit from the availability of debiased data.

Qualitative results.

As mentioned in “Quality of counterfactual editing” in general response, we have added qualitative results of generated images to the appendix of the paper, as well as a set of quantitative evaluation for edit quality.

Generating diverse examples.

This is a great point. While the editing models are not deterministic in their output, text-to-image generation certainly may improve the diversity of the images, albeit at a cost of lower accuracy. Using the CelebA dataset we studied the potential of generating examples of target attributes (e.g., “a person with blond hair”) with Stable Diffusion, instead of editing existing images. This has the benefit of scaling easily to larger numbers of examples in theory, yet in our experiments, the trained models on generated samples (BLIP-SD) underperform those trained with real/edited data substantially, as shown in the table. The gap only widens after increasing the number of shots from 16 to 256. This can be explained by a larger domain gap between generated images and real images, compared to a much smaller gap with editing models.

Anecdotally, we also found the diffusion models to struggle on certain attribute combinations of CelebA, such as men wearing earrings. Editing models on the other hand produce more accurate results by visual inspection. Further study may be needed to understand the trade-off between accuracy and diversity of synthetic images.

Thank you for your insightful review and suggestions. Please refer to the general response for issues not discussed below.

---

Limited empirical evaluation.

Please see “Evaluation beyond gender bias” in general response. In short, while we experimented primarily with gender bias, CVLD is by no means exclusive to gender editing, and is as capable as the ability of the image-text editing models to alter protected attributes under text prompt. We show additional results on 1) background bias on Waterbirds, and 2) racial bias on COCO, to demonstrate the potential of the method.

Bias from editing models.

Thanks for the insightful suggestion. As discussed in “Quality of counterfactual editing” in general comments, it is true that the external editing models may be biased themselves, although very few studies have attempted to study this bias in a quantitative manner. Therefore, we resort to standard fairness benchmarks in classification, retrieval and captioning in the main paper for evaluating the debiasing quality. The fact that CVLD reduces model bias across dataset and tasks can be seen as strong evidence that the potential bias of external models is not enough to offset their effectiveness to remove spurious bias in the training data.

Although it is difficult to directly study the robustness of CVLD to the bias of LLMs/diffusion models, there are steps taken to reduce the potential impact. For example, by sampling multiple edits per image and keeping those most similar to the original, we minimize the chance of inadvertently altering image regions not causally related to the protected attribute (e.g., if some edits cause the color of the clothes or background objects to be changed while others preserve the original colors, the latter would be accepted).

Improve clarity of figures.

Thanks for the suggestion. We have revised Fig. 1 and 3 to improve the clarity of modules and editing operations. We will continue to improve the readability of the paper.

Quality of counterfactual editing (R2, R3, R4)

We have revised the paper to include qualitative examples of image-text editing on COCO and CelebA datasets in the appendix (Fig. 7 and 8). On both datasets, the models produced reasonable gender interventions while preserving the overall visual context.

We also include below a quantitative evaluation of the intervention quality:

• CLIP score [3] measures the overall consistency between synthetic images and target captions.
• Gender accuracy specifically evaluates the success rate of attribute intervention through a CLIP gender classifier (using “a photo of a man/woman” as prompts).
• Directional CLIP similarity [4] measures the affinity between shifts in edited images and text captions in the CLIP embedding space.
• Visual similarity captures the ability to retain visual features from the input images apart from the bias attribute.

Evaluation on COCO reveals that InstructPix2Pix produces more visually consistent images to the original image (higher visual similarity), while null-text inversion alters the image more substantially leading to higher directional similarity and gender accuracy. Notably both the CLIP score and gender accuracy are lower for edited images than original ones. This could be due to the counterfactual edits producing scenes underrepresented in the CLIP training data, in addition to visual artifacts in synthetic images.

It should be noted that the evaluation itself relies on pretrained models like CLIP which can themselves be biased: higher edit quality by the above metrics do not guarantee lower bias in the edited data. Therefore, we resort to standard fairness benchmarks in classification, retrieval and captioning in the main paper for evaluating the debiasing quality. An extended version of the analysis between edit quality and debiasing performance will be added to the paper upon acceptance.

[3] Hessel et al. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. https://arxiv.org/abs/2104.08718

[4] Gal et al. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. https://arxiv.org/abs/2108.00946

Novelty (R2, R4)

We concede that the proposed solution is straightforward, with minimal modification to the models other than a few task-specific losses. However, we argue that the greater contribution of the work is that it demonstrates the effectiveness of image-text editing diffusion models in debiasing VLMs across multiple downstream tasks with no architectural changes, which has not been demonstrated in prior work to our knowledge. This makes possible for CVLD to be applied to novel datasets and tasks as long as the intervention on the protected attribute can be performed, as exemplified in the background and racial bias experiments above. Furthermore, as CVLD performs debiasing on the dataset level, it can be considered a complementary approach to the vast majority of existing mitigation methods for model debiasing, instead of a competing one. We will continue to explore the possibility of combining data and model debiasing in future work.

Computational cost/overhead of CVLD (R3, R4)

The complexity of CVLD depends on the type of the image editing models. Of the two used in this work, InstructPix2Pix is more efficient as it uses a standard generation pipeline of Stable Diffusion, unlike null-text inversion which involves hundreds of iterations to optimize. In practice, using a batch size of 8 images, we found the editing time for Instruct/Null-text editing to be around 10/30 seconds on a single RTX3090 card. While this may less efficient than model-centric approaches for a single training episode, we note that data debiasing does not need to be repeated for retraining models (e.g., for hyperparameter search), which is typically the most time consuming phase of a research or product development cycle. Once the counterfactual data is generated, VLM debiasing reduces to training on an augmented dataset, with no additional overhead to both training and inference.