FlexCap: Describe Anything in Images in Controllable Detail

We thank the reviewer for their time and effort in reviewing our work.

We address the concerns raised below:

About the dataset building:1) It seems that there is no human involvement in the data construction process. I worry about the correctness and diversity of the generated sentences.

We were also curious if a dataset generated in a fully automated manner would be useful. For correctness, we were interested to know if the produced captions are useful for downstream tasks and found them useful for a variety of VQA tasks and region classification. Diversity-wise we choose the n-grams from alt-text of the captions instead of pre-selecting the words. The vocabulary is diverse because of the phrases found in the alt-text of images. Please see examples of the dataset in Figure 11.

One of the key contributions of the data is the different lengths of descriptions for a region proposal. There should be at least a statistic about the length distributions.

These statistics are given in Figure 10 in the Appendix. We agree that this is important.

Based on Table 1(b) and the cases presented in Figure 5, is the max length 8 and is it enough for a complex region/object?

We have empirically found the length to be sufficient for tasks like region classification and VQA tasks across different datasets where our model outperforms baselines with a max length of 8. We also found that the model pre-trained with max length of 8 words can be fine-tuned to produce longer descriptions for regions like those present in the Visual Genome dataset. In the Supplementary material (flexcap-spatial.html), we show that the model produces a mix of short and long captions in the style of Visual Genome. There are 50 captions in the 40 images with length longer than 8. Finally, it is possible to use lengths longer than 8 words for pre-training.

About the baselines: I think it is important to list the backbone types and parameter scaling for each baseline in the result tables. For example, the OWL-ViT has ViT-H/14, ViT-L/14, ViT-B/32, etc. The CLIPM could be applied to different backbones. The parameter scaling (especially the LLMs) will significantly affect performance.

Yes, we agree. We will add this information to the final version of the paper.

About the evaluation: 1) Using the CLIP similarity to map captions and objects is plausible but just one-sided. Have you considered adopting some rule-based verbalization methods to map captions and class names?

It is possible to build a more complicated method than using CLIP similarity. But this is an evaluation technique that allows us to show the captions generated by our model are correct. So we use text-image embedding matching used commonly for evaluating zero-shot recognition in papers (CLIP). Even with this simplistic approach, we find that we significantly outperform the baselines. We will add an ablation using descriptive classnames in the final version of the paper.

2. Back to the motivation, I wonder how to evaluate the information density for each generated caption? Do the authors have any insights?

We are not directly addressing information density in captions. Instead, we are using caption length as a proxy for information content. We observe that captions incrementally add more information as we increase length and add this ability to the model for users to control. If a length 1 caption just mentioned object class name, length 4 would contain information about attributes, and length 8 would contain information about context and attributes. This can be seen in the examples shown in Figure 5 and the Supplementary material (flexcap-length.html). It is possible to parse the produced captions to show length 1 captions are mostly nouns, length 2-4 are adjectives and nouns, but for longer captions image context we need a more complex method of measuring information density than just parsing.

Overall, the proposed framework seems conventional. Can the author discuss what makes the proposed model outperform current similar architectures?

Our contributions are: endowing vision-language models with a new capability of producing length controlled localized captions and producing a large-scale dataset of image-box-caption triplets that can be used to train the model. We are not proposing architectural changes to achieve length-controlled localized captions.

How and to what extent does the prefix token influence the performance?

The prefix token enables a new capability for the captioning models. The same model can be used for producing short and long captions in a controllable manner. We further show our dataset creation technique allows us to generate length-controlled captions for both small and large regions in an image. Also, in the Length Conditioning subsection in Section 2, we describe how the length token helps when there are many captions for the same box.

Table 1a is not cross-references.

Thanks for catching this. We will fix this.

Line 137, for each region, do the authors generate 20 captions for each of the (1,2,3,4) lengths? So that equates to 20x4 = 40 captions per bounding box? Are you then averaging over all the 40 captions in CLIP text encoder?

We report both non-averaged accuracy using the highest scoring caption and accuracy using the average of top-20 captions (out of the 80 total captions) in the CLIP text encoder when presenting the results in Table 1. The top-k captions are chosen on the basis of mean log-likelihood of the generated caption. We find using top-20 leads to significant boost in performance without any additional training.

Line 160, what language queries are used here? Is the process similar to how the dataset is built, using N-gram processing?

We do not use language queries for this step. We use the top-128 boxes based on objectness score from OWL-ViTv2 which does not require any text queries to produce objectness score for a box. Please refer to Figure 15 for details.

Line 168, Isn't the model natively built to describe regions/boxes rather than localize them? How is the model adapted to localization?

The FlexCap model does not do localization. In this work, we are exploring a paradigm different from prior work which has usually followed describe-then-localize. We are looking at localize-then-describe. The localization can be done using object proposal methods. Then FlexCap takes each object of interest and describes them. In particular, in Line 168 we use the detection class name as the ground truth caption for fine-tuning.

We express our gratitude to the reviewer for their insightful feedback and thorough review.

However, how does the proposed method compare to [R1]?

In R1, the training dataset is generated by querying GPT4V with regions of interest found by SAM. While they do train OSPREY with this dataset, the main ability of describing regions was already there in GPT4V. We do not know publicly what the dataset and architecture used for GPT4V are. In our work, we are proposing a ground up way of creating a large-scale region description dataset from alt-text data with a publicly available object detector (OWL-ViT). Furthermore, R1 shows a language model (VICUNA) can be trained further with their dataset to learn a visual instruction tuned model. We show that for describing objects we do not need a full-fledged language model but training a reasonably sized text decoder from scratch is sufficient. The other difference is length conditioning but that was not the focus of R1.

Regarding [W1], there are some works [R2, R3, R4] published in ICLR 2024.

Thank you for sharing these references. We will add them to related work as they all are relevant.

[W2] How should one choose the length of the region descriptions?... What is the point of having a length-controlled method of generating region descriptions, if there still needs to be a manual selection of the length?

The objective is to have one model that can produce long and short descriptions as needed. For example, if someone is interested in recovering only object names they can request for length1 captions from the model. However if one is interested in a longer description of an object then they can ask for length8 captions. In a way, our model merges the label spaces of object detection datasets and dense captioning datasets by using length conditioning.

Although there is a common-sense based length prior that most people use for describing a region, people can also describe the content with a few or a lot more words when requested. Our model has this flexibility to be queried at a desired level of detail. But by fine-tuning the model with common-sense human captions (e.g. Visual Genome), our model adjusts itself to the average human biases in information density. We show examples of this in flexcap-spatial.html in supplementary material. Although some heuristics can be introduced, we believe fine-tuning with a dataset that captures the desired information density is a more configurable approach to the output information density problem.

[W3] Controllable image captioning itself is not a new concept. ... How does the method of the authors compare to this work?

There are several differences with R6. First, R6 only handles image captioning. While we handle object, regions, and full images by using the flexibility of conditioning with a bounding box. Second, their conditioning approach is a specific way of producing different captions by changing the order of detected objects as conditioning. They detect N objects in an image. The captioner is shown different orders of N objects to produce different captions. While this produces grammatically correct captions in different ways, it does not necessitate new information being generated in the captions. The objective in our work is to use length conditioning as a proxy for information content. Both are valid but different objectives of conditional image captioning. Finally, our approach allows for prefix conditioned captioning which can be used to extract attributes of interest: color, material, action, function and text. This capability emerges due to pre-training the model on a large-scale dataset which is missing from R6.

[W6] L178, regarding the zero-shot setting, the authors finetune the model on COCO (according to L168)... Therefore, this method cannot be seen as zero-shot.

In our study, we used the term "zero-shot" to indicate that the training process did not involve any VQA annotations of question-answers. The evaluation was conducted on non-train splits of COCO, ensuring that the model has not encountered the evaluation image was not seen in any phase of training. Hence, we maintain that the setting can still be considered zero-shot. To provide greater clarity, we will explicitly mention this in the final version of the paper.

[W7] For VQA, the amount of captions generated seem to be huge (128 regions × number of prefixes used), and therefore is this a fair comparision with other works?...

The VQA baselines systems are trained to produce answers using different approaches (mostly end-to-end). They do not have a similar system to ours where we generate a large chunk of text and then deduce a single answer from that. We will add an ablation on how the number of generated descriptions affects performance.

[W5] The dataset that the authors generate, as well as the performance of FlexCap, is highly limited by the performance of OWL-ViT ...

Yes we mostly agree with this assessment and have mentioned this in the limitations section. However, just to clarify, OWL-ViT does not define what to caption in images. This information comes from alt-text associated with the images in the form of n-grams. OWL-ViT mainly localizes provided n-grams on the images using the box proposals which can arguably cover almost all the interesting regions in the image. Moreover this automated approach allowed us to scale both in size and diversity which often is the main influencer of the performance, as demonstrated with many results in our experiments section.

[W4] InstructBLIP [R7] is not compared to ... Most works report on VQAv2.

To clarify any concerns, we will test our method on VQAv2 and provide the results on this dataset in the final version of the paper. We will also add InstructBLIP to the list of baselines.

We thank the reviewer for their feedback and meticulous attention to detail in reviewing our paper.

We address the concerns raised by them below:

Referring to FlexCap as a versatile flexible-captioning vision-language model might be an overstatement.

We used this description to highlight the fact FlexCap can be conditioned using bounding box, desired caption length, and caption prefix to produce diverse captions of objects and regions in images. We show many examples of the flexibility in Fig.5, Fig. 8 and Fig. 13 in paper and in the supplementary webpage. That said, we agree that, when implemented effectively, there could be many more spatial and textual controls for captioning and with FlexCap we take a significant step towards properly exploring this space. We will also soften the claims on versatility in the abstract by focusing more on the specific controls that we introduce (i.e. bounding boxes, length, and caption prefixes) while providing the larger context on potential spatial and textual controls.

Also, while theoretically the CaptionAnything system can produce outputs of different lengths, we tested it for length conditioning and it does not consistently return outputs of the desired length.

Regarding weakness about “statistical probabilities of "<e>" and "playing" following "dog" to support the authors' statement.”, can the authors provide some statistics to support their statement?

We agree that whether this occurrence will be a problem depends on the dataset statistics. To quantify the prevalence of this problem, we compute a statistic: for each image box, we consider all pairs of captions and measure the fraction sharing prefix words. For instance, one box has three captions: "dog <e>" "dog playing <e>", "dog playing with a frisbee <e>" which share a prefix "dog". While another caption for the same box: "brown dog <e>" does not share a prefix with captions beginning with "dog". After averaging this metric across all images in the dataset, we found that 30.8% of caption pairs share a prefix. The length conditioning token assists in distinguishing between captions with the same prefix while also providing the model with a novel capability during inference. When length conditioning is applied, the probability of prefix similarity decreases from 30.8% to 11.1%. We appreciate the encouragement to explore this issue quantitatively and plan to include this analysis in our paper.

We answer the other questions asked by them below:

Q1. The abstract is divided into two paragraphs (Line #6 and Line #7).

We will merge the two paragraphs into one.

Q2. Section 4.1 on Correctness lacks a reference of Table 1(a). Q3. Line 249 lacks a reference of Figure 8.

Thanks for catching this. We will add a reference to Table 1(a) in section 4.1 and to Figure. 8 in Line 249.