Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts

Q1: Speed and parameter.

As we discussed in the limitation, the speed and parameter problem can be gradually overcome by recent lightweight models. Besides, the main purpose of this paper is to provide a feasible solution to address the open-ended perception challenge; thus we do not consider the speed and parameters right now.

Additionally, we analyze the effect of SAM‘s parameters on VL-SAM, as shown in the following table. We can find that a larger model achieves better performance for VL-SAM. Besides, when parameters are reduced from 636 M to 91 M of SAM backbone, the performance of VL-SAM only drops 2.2 mAR. This demonstrates the effectiveness of the proposed framework once again.

Q2: Details and analysis of prompt generation.

We directly ask VLM to generate question prompts by itself without given images. Specifically, we use the question:

If we want you to list all possible objects in the given image, what questions should we ask? Please give 10 questions you prefer.

Here is the answer from CogVLM:

(1) Please analyze the image and list all the objects present.
(2) Identify and provide a comprehensive list of objects in the image.
(3) Using the image, generate a detailed inventory of all objects visible.
(4) Analyze the image and extract information about any objects present.
(5) Please describe all the objects you can identify in the image.
(6) From the image, generate a report listing all objects detected.
(7) Utilizing the image, identify and report all objects visible.
(8) Analyze the image and provide a comprehensive breakdown of all objects seen.
(9) Please process the image and generate a detailed list of all objects present.
(10) Using the image, identify and describe all objects observed.

Moreover, as Reviewer 6VtQ mentioned, we provide the average recall (AR) of VLM for object name generation in the following table. We find that prompt ensemble significantly alleviates the object missing problem in VLM. Thus, it can improve overall performance.

We will add the discussion to the paper.

Q3: AR of SAM's segment-anything mode (uniform sampling).

We use SAM's segment-anything mode to obtain box proposals and calculate the class-agnostic AR. The AR is 29.7. However, the boxes predicted by SAM do not contain category results. In contrast, VL-SAM can predict boxes with categories.

Q1: The performance of OWLv2.

Thanks for reminding. We will add it to the paper.

Q2: Results for non-rare classes in LVIS.

The following table provides results for common ($AP_c$) and frequent ($AP_f$) classes of LVIS. We can find that, though Close-set and Open-set methods achieve better results on $AP_c$ and $AP_f$, they require 1203 object names from LVIS as input. Besides, our method is training-free and achieves competitive results with GenerateU.

Q3: Average recall (AR) of VLM for object name generation.

The following table provides the AR of VLM for object name generation. $AR_{x}$ presents the similarity score calculated by CLIP larger than x is viewed as positive.

Q4: Cannot recover missing objects.

As shown in the table of Q3, we are currently trying to reduce the number of missing objects with the prompt ensemble. In the future, we will adopt techniques like Chain-of-thought to recover missing objects.

Q5: Clarifying the concepts of open-world, open-set, and open-ended perception.

As the reviewer mentioned, these three terms are used inconsistently in the ML/CV literature. In our opinion, open-world perception is a broad concept. It tries to give precise results in dynamic and unpredictable environments, which contain novel objects and involve scene domain shifting. Open-set and open-ended perception are subtasks of open-world perception, and try to address the novel objects problem. Specifically, open-set perception, like the grounding task, predicts object locations when given images and novel object names. In contrast, open-ended perception is similar to traditional perception that predicts novel object locations and their names simultaneously when only given images. We will add the discussion to the paper.

Q6: Argument of "predefined object categories are not available in real-world scenarios."

The reviewer mentioned that "autonomous vehicle companies have a list of a few hundred object categories - hence, it's predefined." Though they have predefined hundreds of categories, there are still categories they may not include, such as various rare animals. Besides, some objects cannot be presented by a simple category name, such as a human in an animal costume, which may look like an animal but is actually a human. As the reviewer said, generic obstacle detection can handle some unknown objects. However, many things do not have a significant 3D shape, like pits or grains on the ground. Thus, open-set methods cannot handle all situations. We will add the discussion to the paper.

Q7: Reason for attention maps collapse when aggregation.

The collapse in attention map aggregation is caused by the causal mask. For example, assuming there are uniformly distributed attention maps (3$\times$3 for simplicity) in all transformer layers (total 2 layers for simplicity):

\\begin{pmatrix} \frac{1}{3}&\frac{1}{3}&\frac{1}{3}, \\ \frac{1}{3}&\frac{1}{3}&\frac{1}{3}, \\ \frac{1}{3}&\frac{1}{3}&\frac{1}{3} \\end{pmatrix}

After using the attention rollout method with the causal mask, we can obtain the final attention map:

$$[I+\begin{pmatrix} \frac{1}{3} & 0 & 0, \\ \frac{1}{3} & \frac{1}{3} & 0, \\ \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \end{pmatrix}]\times[I+\begin{pmatrix} \frac{1}{3} & 0 & 0, \\ \frac{1}{3} & \frac{1}{3} & 0, \\ \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \end{pmatrix}]=I+\begin{pmatrix} \frac{7}{9} & 0 & 0, \\ \frac{8}{9} & \frac{7}{9} & 0, \\ 1 & \frac{8}{9} & \frac{7}{9} \end{pmatrix}$$

We can find that, for each row, the number in the front column is greater than in the back column, e.g., $\frac{8}{9}>\frac{7}{9}$ and $1>\frac{8}{9}>\frac{7}{9}$. Thus, simply adopting attention rollout with the causal mask will centralize the attention activation in the front patch, i.e., the top left corner of the image in order, as shown in Figure 5 in the paper.

Q8: Details of regularization.

As discussed in Q7, directly adopting attention rollout will make the number in the front column greater than in the back column. We introduce a simple regularization method that multiplies a small term to the front column and a bigger one to the back column. The value of the term uses a simple linear descent: $1-L_0/L$.

Q9: Details of prompt generation.

We directly ask VLM to generate question prompts by itself without giving images. Specifically, we use the question:

If we want you to list all possible objects in the given image, what questions should we ask? Please give 10 questions you prefer.

Here is the answer from CogVLM:

1) Please analyze the image and list all the objects present.
2) Identify and provide a comprehensive list of objects in the image.
3) Using the image, generate a detailed inventory of all objects visible.
4) Analyze the image and extract information about any objects present.
5) Please describe all the objects you can identify in the image.
6) From the image, generate a report listing all objects detected.
7) Utilizing the image, identify and report all objects visible.
8) Analyze the image and provide a comprehensive breakdown of all objects seen.
9) Please process the image and generate a detailed list of all objects present.
10) Using the image, identify and describe all objects observed.

We will add the details in the paper.

Q10: Typos and indexing inconsistency.

Thanks. We will revise them in the paper.

Q1: Evaluation is still on predefined object category names.

The inference of VL-SAM does not rely on predefined object category names; we only use them to calculate the evaluation metrics.

Specifically, as Lines 211-213 mentioned, we use VL-SAM to generate object categories by itself. However, the generated object categories from VL-SAM may not align with the category names specified in the LVIS dataset. For example, VL-SAM may generate "kids" or "adults," while the LVIS dataset provides a "person" label for these objects. To address this, we follow GenerateU to use the CLIP text encoder to map generated object categories from VL-SAM to the predefined class names in LVIS for mAP evaluation by calculating their similarities.

Q2: Performance is worse than open-set methods.

Our method addresses the open-ended detection and segmentation problem, which is different from the open-set problem. As mentioned in Lines 34-39, open-set methods require predefined object categories as inputs during inference. In contrast, open-ended methods do not need to input predefined categories and can predict the object categories themselves. Therefore, it is unfair to compare the performance of our method with open-set methods directly. They are two different tasks.

Moreover, our method is training-free, while open-set methods listed in the paper need additional training.

Q3: Vision language model selection.

As shown in Table 5 of our paper, empirical results demonstrate that VLMs with more powerful multi-modal chat and reasoning capabilities perform better when integrated into VL-SAM. Thus, we suggest using a stronger VLM. Besides, our framework is general. Any VLMs that generate object names from images and provide attention maps can be incorporated.

Q4: typos.

Thanks. We will revise them accordingly.

Dear reviewers,

Thank you for the comments on our paper. We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them by the end of the discussion period (Aug 13, 11:59 pm AoE). We hope that you can consider rasiing the score after we address all the issues.

Sincerely, Authors of submission 9204