Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association

Q: The whole paper lacks many details of method design choices, data curation, method, training, and inference details, thus hindering the readers to fully understand the effectiveness of the proposed method. Only empirical results are not enough.

A: Thanks for the valuable comment. We have reorganized and refined the paper from the following perspectives:

• Update Figure 2 and the methodology with clearer presentation and more implementation details for the readers to better understand our approach.
• Please refer to Section 5.1 for the training details.
• Please refer to Section 4.1 for the implementation details including the data source and data processing.
• We add human evaluation for each phases in this framework to better demonstrate the proposed method beyond the empirical results.

Q: Prompt: The font color is hard to read.

A: We have updated the prompt in Appendix C with carefully designed style for better readability.

Q: No bias analysis, which is the most important point in the introduction.

A: We want to clarify that reporting bias is the motivation of our method but not the exact problem we're studying in this paper.

The proposed method is motivated by the phenomenon that object-attribute are usually associated (which is caused by reporting bias). This association could hinder the model to understand the attributes for an object, e.g. VL model could think lemon is always yellow because it sees that association frequently. In the proposed method, a richer array of object-attribute pairs are added into the datasets explictly, thus it can help the VL model's ability to understand the object-attribute, instead of just memorizing the frequent patterns.

Q: Lacking experiments using different key tools, e.g., LLM, detector/grounder, image generator, etc, which may affect the data generation a lot.

A: Thank you for your suggestion! Reporting bias (as its property suggested) is often overlooked in existing models and datasets. As a result, it is non-trivel to reveal and fix it, hence requires multiple language/visual tools, making it hard to conduct large-scale experiments on a mass of combination of different foundation models. Hence, we use state-of-the-art foundation models to construct our framework, so that the generation quality could be as good as possible at the moment. We will leave the exploration of the effect of different foundation models in the future work to find the best practice for deployment.

Any user study of the image generation quality, prompt quality, and rationality of the V-l relations?

A: Thank you for the suggestion! We add a subsection for user study in Section 4.4 in the updated paper. Please refer to the paper for details.

In summary, we did human evaluation for the foundation models outputs. We design five questions for the intermediate output:

1. Correct object: ''In step 1 and 2, does the framework generate reasonable objects?''
2. Implicit object: ''In step 1 and 2, do the valid objects contain the ones that are not explicitly mentioned in the original caption?''
3. Correct attribute: ''In step 3, does the framework add description for the correct given attributes?''
4. Valid negative: ''In step 4, can the pair of attributive objects serve as hard negative example to each other?''
5. Valid inpainting: ''In step 7, does the model inpatint the required object in the masked area?''

It shows that we got >90% correctness for correct object and valid negative, around 60% for correct attribute and valid inpainting. And around 60% augmented examples contains objects that are not explicitly mentioned in the original caption

Q: More solid analyses should be given to probe the relation between the data and the bias and performance of different tasks.
Q: I suggest the authors also discuss the problems in the causal view, like counterfactual samples.

Thank you for your thoughtful feedback. As we mentioned above, our experiments involves extensive data processing and model training, which can be quite time-consuming. Due to the time limit, it may not be feasible for us to add more experimental analysis by the end of the discussion period. However, we will continue to enhance the analysis and discussion sections in the future.

Q: CLIP-ft: fine-tuning CLIP on the augmented data? Missing details.

A: CLIP-ft refers to fine-tuning on the source dataset sampled from CC3M but not on the augmented data. CLIP-ft is the baseline variant we compare with the proposed method BiAug. Specifically, we first sample dataset with different sizes from CC3M, i.e. the source dataset, then we apply the proposed method on the source dataset to obtain the augmented dataset. CLIP-ft is CLIP trained on the source one and BiAug is CLIP trained on the augmented one. They are comparable because the datasets are from the same source.

We have modified the description in Section 5.1 to avoid misunderstanding.

We would like to express our sincere gratitude for the time and effort you have invested in reviewing our work. Your insights and comments have been invaluable in refining our manuscript.

We are eager to learn your perspective on our responses to your comments. If there are any points or concerns that you believe have not been adequately addressed, please let us know and we are willing to provide further clarification and responses.

Thanks for the time for reviewing our paper. Your comments and suggestions are very insightful! We're addressing your concerns as follows,

Q: It would be beneficial if the author could elaborate on whether they have considered modifying multiple attributes and multiple objects simultaneously.

A: This is a good point, especially we're processing a large scale of data. In fact, one attribute of one object at a time is just the example for presentation. We actually modify the all the four attributes for all valid objects to augment the captions, i.e. four attributes X multiple objects a time, please refer to Appendix C, prompt 2 for details.

However, for the image inpaining, we do one attribute-object pair at a time, because modifying multiple objects could corrupt the image, and more importantly, we would not be able to construct the hard negative pairs to highlight the attributive difference for one object.

Q: In addition to conventional attributes like color, shape, and material, has the author explored more detailed attributes

A: This is also a very good point, we did explore different attributes but found it would be hard to cover every attribute in the large-scale experiments. To address the diversity of attributes, we add an other attribute to leverage the LLM’s generalization capabilities to infer relevant attributes from provided captions, highlighting the most relevant commonsense knowledge besides color, shape and material. Please refer to footnote 2 in the 4th page, and the prompt 2 in Appendix C. By doing this, LLM is able to generate attributive description dynamically based on the given caption, beyond the preset conventional attributes.

Q: Add ARG experiments.

A: Thanks for the suggestions! We're trying to add broader experiments by the end of discussion period if possible as the time limit. Could you also provide the reference or link to the ARG benchmark you mentioned? We'd also like to kindly note that we're focusing on the reporting bias issue (which is overlooked previously) in VL dataset, ARO is the most relevant benchmark to verify our ideas.

Q: The meaning of the abscissa in Fig. 5 is not immediately clear. Further clarification or labeling may be needed for readers to grasp the content effectively. Furthermore, Fig. 5's subcaption would benefit from further adjustment.

A: Thanks very much for the suggestion! The x-axis refers to the size of the source dataset used for training the model, e.g., 40K, 100K, 200K and 300K, for demonstrating the trend of model's performance with increasing data used for fine-tuning. We have added description for the x-axis in the caption of Figure 4 (Figure 5 in the original paper), and in Section 4.1 for better readability. As for the formatting in the figure, we'll carefully adjust them in the final version.

Q: The paper would benefit from a more extensive discussion of its limitations and potential directions for future research, offering a more comprehensive evaluation of the current method's scope and boundaries.

A: Thank you for the suggestion! We agree that discussion would be very helpful. We cannot add another section due to the page limit. Please refer to Appendix A for our discussion about limitation and future directions.

Thanks for your insightful comments! We're addressing your concerns as follows,

Measurement of the quality of foundation models outputs

That's a very good suggestion. So we add a subsection for that in Section 4.4 in the updated paper. Please refer to the paper for details.

In summary, we did human evaluation for the foundation models outputs, as there is no benchmarks that are exactly designed for our case. To check whether these foundation models follow our instruction to work, we design five questions for the intermediate output:

1. Correct object: ''In step 1 and 2, does the framework generate reasonable objects?''
2. Implicit object: ''In step 1 and 2, do the valid objects contain the ones that are not explictly mentioned in the original caption?''
3. Correct attribute: ''In step 3, does the framework add description for the correct given attributes?''
4. Valid negative: ''In step 4, can the pair of attributive objects serve as hard negative example to each other?''
5. Valid inpainting: ''In step 7, does the model inpatint the required object in the masked area?''

It shows that we got >90% correctness for correct object and valid negative, around 60% for correct attribute and valid inpainting. And around 60% augmented examples contains objects that are not explicitly mentioned in the original caption

As for your questions,

is there a numerical evaluation of the box quality generated in step 2 in Figure 2? 

A: Grounding DINO achieves an impressive 60.7 mAP (refer to Table 2, [1]) in zero-shot object detection on the MSCOCO benchmark, without any prior training on this specific benchmark. This performance underscores its robust capability to reliably identify valid objects, which is essential for the effectiveness of our method.

In step 1 how is the quality of the "guessed" potential objects?

A: As the results shown Table 2 in the updated paper, the quality of the guessed potential objects are very good with a correctness >90%.

[1] Liu, Shilong et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.” ArXiv abs/2303.05499 (2023): n. pag.

Concern about the improvement on general VL retrieval task

The Filcker30K-order and COCO-order in Figure 4(Figure 5 in older verion) are not the results for general text-image retrieval task, instead it's the results on a very challenging ARO benchmark[2]. We have modified the paper to avoid misunderstanding. In ARO, the original Filcker and COCO captions are modified by changing partial word order, which are very similar to the original caption. This benchmark requires the VL model to pick the correct caption from a group of hard negative examples. Our results show that BiAug has a more stable trend in distinguish hard negative examples.

The claim that BiAug also acheives improvement on general image-text retrieval task is supported by the results in Table 3. Please refer to the paper for details.

[2] Yuksekgonul, Mert et al. “When and why vision-language models behave like bags-of-words, and what to do about it?” ArXiv abs/2210.01936 (2022): n. pag.

Comparison with NegCLIP

We're still reproducing the NegCLIP to make the comparison. We're afraid that it cannot be finished by the end of the discussion period as it requires a lot of time to preprocess the 300K data and train the models. We will update the results as soon as possible.

However, we want to note that BiAug and NegCLIP are not actually comparable because our focus is to mitigate the reporting bias issue by decoupling the object-attribute association for both visual and langauge modalities, while NegCLIP just simply shuffle the caption, without making any changes to the objects or attributes in the caption and the image.

We would like to express our sincere gratitude for the time and effort you have invested in reviewing our work. Your insights and comments have been invaluable in refining our manuscript.

We are eager to learn your perspective on our responses to your comments. If there are any points or concerns that you believe have not been adequately addressed, please let us know and we are willing to provide further clarification and responses.

We have refined the presentation in the updated version

Q: However, I think the authors need to spend more time thinking about the structure and also definitely re-writing the abstract.

Thank you for your insightful questions and suggestions. For your main concern about the presentation of this paper, we have carefully rewritten the abstract and method, updated the figure2, and updated the demonstration of prompt for LLM, for better readability. Please see the updated paper. Please also see our clarification below to your questions about the detail of the method.

Q: Many important concepts used in the paper are not clearly explained. For example "grounding object" need clear definitions.

A: Thank you for the suggestion. I went through the paper and add necessary background information for better readability.

Concern about the relative complicated framework

Reporting bias (as its property suggested) is often overlooked in existing models and datasets. As a result, it is non-trivel to reveal and fix it, such that our method consists of different important steps. For your specific concerns, we clarify them point by point as follows,

Q: “why can we assume that the grounding object detector can find objects that a different color or shape of it is still a valid object?”

A: It is important to emphasize that our system utilizes the state-of-the-art grounding detector, Grounding DINO [1], for detecting valid objects. This advanced detector is particularly efficient in recognizing objects based on textual input alone, even in a zero-shot context. Notably, Grounding DINO achieves an impressive 60.7 mAP (refer to Table 2, [1]) in zero-shot object detection on the MSCOCO benchmark, without any prior training on this specific benchmark. This performance underscores its robust capability to reliably identify valid objects, which is essential for the effectiveness of our method. Though GroundingDINO is generalizable, we agree that it is not perfect and could give mistakes. However, in our qualitative/human inspections in Section4.4, we found that GroundingDINO is generally reliable. And we expect that more advanced grounding models will be developed in the community and can be easily integrated to BiAug.

Q: “How can we make sure that the detected grounding object is big enough for the in-painting model to process this?”

A: We remove the detected boxes that are too large or too small to ensure they are valid for the following procedure. Please refer to the filtering strategies introduced in Section 4.1.

Q: “How the framework described in Figure 2 could be reliable for large scale experiments.”

A: We apply multiple filtering strategies, as introduced in Section 4.1, to remove low-quality grounding objects/augmented captions/generated images. Filtering strategies makes the examples we finally used for training are reliable as possible. We agree that we still cannot avoid noisy examples, even BiAug is utilizing state-of-the-art language and visual foundation models. The experimental results on general visual-language retrieval task shown in Table 3, also demonstrate that the dataset augmented by BiAug is reliable enough for training a VL model, as the performance keeps increasing with finetuning on the augmented dataset.

[1] Liu, Shilong et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.” ArXiv abs/2303.05499 (2023): n. pag.

Q1: Where do the samples in Figure 1 come from? Are they from a real dataset? Are they generated?

Q2: In Figure 2, in step 3, why "tennis ball" is dropped? It was part of the original text. Why "dog" is selected as the grounding object?

Q3: In section 3.2 how do you select from the list of predefined categories? For example, in figure 2, how come the "shape" is not selected for "grass"? Of course it does not make sense to modify the shape for grass, but how do you account for that and select "color"?

Q4: In 3.3 how do you mask the detected object?

Q5: In the example given in figure 3, what happens if non of the predicted objects are inside the image?

Q6: In figure 4, the "blue boat" and "red boat" samples, what was the missing reference object here according to your framework? What happened in the extraction phase? boat was already part of the original caption.

Q7: What is "other" in the preset attribute list?