OW-VAP: Visual Attribute Parsing for Open World Object Detection

Thank you very much for your valuable comments and suggestions on our manuscript. Below, we provide detailed responses to your feedback:

Q1: Similarity to Prior Work

While the referenced works also utilize textual attributes, our OW-VAP differs significantly from these methods in several key aspects:

1. Different Task Domain: The referenced works focus on open-set classification tasks, aiming to train a classifier capable of handling new classes. In contrast, our method addresses the open-world object detection (OWOD) task, which stems from object detection. Specifically, OWOD consists of two parts: novel class discovery and incremental learning. The former requires the detector to identify objects not seen in the training data, while the latter involves fine-tuning the detector to adapt to new tasks based on existing knowledge. For details on OWOD task settings, please refer to $\textcolor{red}{\text{Appendix B}}$.

2. Different Data Requirements: The referenced methods assume datasets with explicit attribute annotations for training, which is costly to obtain. In contrast, our approach does $\textcolor{red}{\text{not require any additional manual annotation efforts}}$. For example, the MOWODB benchmark is a combination of the VOC and COCO datasets, where the COCO dataset alone contains $\textcolor{red}{\text{over 1.5 million training instances}}$. It is practically impossible to annotate every object in such a large dataset. Instead, VAP training is seamlessly integrated into the detector training process.

3. Different Methodology: The referenced approaches employ statistical techniques such as Bayesian inference for unknown class prediction. In contrast, our method leverages deep learning to map visual embeddings of known classes into a general attribute space. In essence, our VAP functions more similarly to a clustering method.

Q2: Coarse Attributes

Our use of coarse textual attributes provides several advantages:

1. Coarse attributes utilize the same textual descriptions (only 10 in total) across all objects, making them more robust when transferring to new objects compared to fine-grained attributes , which heavily depend on the accuracy of LLMs. For example, in incremental learning, our OW-VAP can directly transfer previously learned knowledge without requiring additional training. In contrast, methods based on fine-grained attributes require generating new attributes followed by an additional round of selection and training ($\textcolor{red}{\text{Table 7 and 8}}$).

2. As the number of known classes increases, the number of fine-grained attributes also grows. For example, in Task 4 of the MOWODB benchmark, the number of fine-grained attributes reaches $\textcolor{red}{\text{over 8K}}$. Using such fine-grained attributes would be impractical for OW-VAP because it would significantly reduce real-time performance and hinder application in real-world scenarios.

3. We compared OW-VAP with fine-grained attribute methods (e.g., FOMO series) in Tables 7 and 8. OW-VAP achieves notable performance advantages, $\textcolor{red}{\text{with 4+ improvements in U-Recall and 8+ improvements in U-AP}}$. For further comparative analysis, please refer to $\textcolor{red}{\text{Appendix G}}$.

For insight into why VAP is effective, please see our response to Reviewer sWQp (Weak 1).

Q3: Hyperparameter Reproducibility

Our method involves only $\textcolor{red}{\text{three hyperparameters}}$, whose impact on detector performance is summarized in Table 6. For additional details, please refer to Appendix F. Moreover, the model's performance is not highly sensitive to these hyperparameter settings, meaning they can be shared across applications without modification.

Regarding reproducibility, as detailed $\textcolor{red}{\text{in the Code section of Appendix E}}$, all training logs—including weight files, training logs, and visualized results—will be released following code review for transparency and replication purposes.

We sincerely hope that our clarifications resolve remaining questions and provide further insight into our method. If you have additional feedback or need further assistance, please feel free to contact us.

We sincerely appreciate you taking the time to carefully review our manuscript. Your expert opinions and constructive suggestions are invaluable to our research. Below, we provide our responses to the concerns you raised:

Weak 1: Why is VAP effective?

Our OW-VAP is built upon the standard YOLO-World, which is a real-time open-vocabulary detector (OVD). YOLO-World is capable of open-vocabulary visual detection by leveraging training on large-scale datasets to align visual embeddings with textual attributes. This alignment not only supports known categories but also provides, in theory, a generalization capability for unknown categories. This is because the relationship between visual embeddings and textual attributes can be understood as a shared mapping in a latent space.

Specifically, OW-VAP utilizes visual embeddings generated by the trained OVD detector and designs a transformation mechanism to align these embeddings with predefined coarse attributes. This transformation mechanism acts as a mapping function from the visual embedding space to the attribute space. Leveraging the shared characteristics of this mapping function, OW-VAP is able to learn common cross-category representations from known categories, enabling semantic generalization to unknown categories. The theoretical basis for this generalization can be attributed to the following points:

1. The OVD detector gains strong visual-text alignment knowledge through pretraining on large-scale datasets.

2. The attribute space is intentionally designed to be coarse and abstract, reducing significant differences among object categories and thereby enhancing the generality of intermediate representations.

3. The transformation mechanism between visual embeddings and attributes prioritizes shared information instead of specific category features.

Additionally, our method enhances OW-VAP's capability to detect unknown categories by utilizing the pretrained knowledge of the OVD detector. This pretrained knowledge not only supports effective description of known categories but also establishes a robust foundation for aligning visual inputs with textual attributes for unknown categories. For instance, as demonstrated in Table 3, OW-VAP achieves a U-Recall of 45.6 and a U-AP of 10.4 by only using a combination of coarse textual attributes with out-of-distribution probabilities.

Weak 2: The annotations in Figure 3

In Figure 3, we illustrate the training process of VAP. In the figure, the blue circles represent samples already labeled in the training set. During training, these samples are treated as positive instances, and we train VAP to pull them closer to their nearest coarse descriptions (represented as yellow circles in the figure). The triangles, on the other hand, represent potentially unlabeled objects within the dataset. These unlabeled objects are further differentiated using the three selection rules outlined in Section 3.2. This process identifies potential objects from unlabeled samples (which correspond to background samples in a closed-set training setting). As such, we refer to this process as "background object mining." Additionally, you may refer to our explanation of this procedure in response to @Drok for further details.

Q1: Coarse Attributes

FOMO utilizes large language models (LLMs) to generate detailed attribute descriptions. Before creating the detailed descriptions, it categorizes the object attributes into 10 groups and then generates descriptions within each group. In our approach, we focus only on coarse descriptions. To achieve this, we use the 10 attribute categories and expand them to generate sentences indicating whether each attribute $\textcolor{red}{\text{is significant or not}}$ (as shown in Table 4).

Q2: Choice of the Base Probability Model

In fact, $\textcolor{red}{\text{this is a default choice in the OWOD field}}$. Since ORE introduced Weibull modeling for energy-based scenarios, the Weibull distribution has become the standard for similar modeling tasks (e.g., as used in MEPU). Of course, other distributions, such as the Beta distribution, could also be explored and used as a tunable hyperparameter to achieve better performance. However, introducing alternative distributions would increase the complexity of the method, which could hinder further research by the community. Therefore, we adopt the default setting in the OWOD field and use the Weibull distribution to demonstrate the effectiveness of PLSA. Moreover, using a Gaussian distribution is not feasible, as the shape of the distribution is not symmetric and does not match the characteristics of the data.

We hope these clarifications adequately address your concerns regarding our methodology. Should any questions remain, please feel free to contact us for further discussion. We sincerely appreciate your thorough review and valuable feedback.

We sincerely appreciate you taking the time to carefully review our manuscript. Your professional insights and constructive suggestions are invaluable to our research work. Before addressing the questions, we present a complete description of the overall OW-VAP process.

All Pipeline

OW-VAP adopts YOLO-World as its base model. YOLO-World is a real-time open-vocabulary detector. During inference, compared to YOLO-World, we additionally introduce a Visual Attribute Parser (VAP), which consists of only $\textcolor{red}{\text{a lightweight multi-layer perceptron (MLP) (e.g., with three layers)}}$.

Given an image $I$ a list of known class names $Name$, and coarse textual attributes $Att$, these inputs are fed into text and visual encoders to produce visual and textual embeddings: $E_{vis} \in \mathbb{R}^{w \times h \times d}, E_{c} \in \mathbb{R} ^ {c \times d}$ and $E_{att} \in \mathbb{R} ^ {n \times d}$. Here, $w$ and $h$, $c$, and $n$ represent the width and height of the feature map (downsampled compared to the original image), the number of known class names, and the number of coarse textual attributes, respectively. It is important to note that $E_{vis}$ is distributed across three feature levels, namely {C3, C4, C5}. For simplicity, we only provide the shape at one feature level. Subsequently, the visual $E_{vis}$ and known class names embeddings $E_{c}$ are passed into the VLPAN to enable interaction between visual and textual features.

$

X' _ {l} = X _ {l} \cdot \sigma(\underset{j \in {1, ..., c}}{max}(X _ {l}E _ {c}^\top))^{\top}.

$

Here, $X_{l}$ represents the visual embeddings at different feature levels {C3, C4, C5}, and $\sigma$ denotes the Sigmoid function. The resulting output $X_{l}'$ retains the same shape as the original inputs $X_{l}$. Each feature level is divided into a 3 $\times$ 3 grid, and max-pooling is performed within each grid cell. Across all levels, a total of 27 mbeddings are generated as $\widetilde{X}$. Finally, multi-head attention is applied to weight the generated embeddings and the original inputs: $W' = E_{c} + \text{MultiHead-Attention}(E_{c}, \widetilde{X}, \widetilde{X})$. The final weighted result retains the same shape as the original inputs $E_{c}$. Then, $W'$ will be used to replace $E_c$. In summary, VLFPN first enables interaction between visual and textual embeddings and then performs further fusion using multi-head attention to update $E_c$.

Then, the visual embeddings, along with the updated textual embeddings, are used to decode bounding boxes and generate confidence scores for known classes using cosine similarity. Our Visual Attribute Praser (VAP) operates on the visual embeddings, aiming to extract predefined textual attributes from them. In other words, VAP maps the visual embeddings into the attribute textual space. For details on training VAP, please refer to Section 3 and Appendix E. Afterward, as described in Section 3.4, we use in-distribution and out-of-distribution probabilities to assign confidence scores to unknown classes. An overview of the entire process is briefly described in Section 3.1.

Q1: On the use of LLMs

As described in the previous pipeline, $\textcolor{red}{\text{OW-VAP does not utilize any LLMs during the inference stage}}$. It is built upon the standard, real-time open-world object detection (OWD) model, YOLO-World, with only the addition of a lightweight multi-layer perceptron (MLP). In fact, for object detection tasks, the use of LLMs during inference is generally discouraged. This is because the goal of the community is to achieve a real-time, high-accuracy detection model. Thank you again for evaluating the practical applicability of OW-VAP; we hope these clarifications address your concerns.

Q2: On Comparisons

$\textcolor{red}{\text{In the experiments section and appendix}}$, we provide comparisons between OW-VAP and OWOD detectors utilizing foundation models. These include MEPU-FS based on Free-SOSO, SGROD, SKDF, and KTCN based on SAM, as well as ovow and FOMO, which are built on the $\textcolor{red}{\text{same OVD detector}}$ as ours. Overall, OW-VAP achieves comprehensive performance advantages. For example, in Task 1 of MOWODB, OW-VAP outperforms ovow by +13 in U-Recall and +8 in U-AP. For a detailed comparison between OW-VAP and these foundation model-based methods, please refer to Fig. 1.

We hope these responses resolve concerns regarding the methods we employed. Should you have any further questions, please feel free to reach out to us.

We greatly appreciate your careful and thoughtful review of our manuscript. Your meticulous approach to the review process contributes significantly to the advancement of the research community. Below, we provide our responses to the concerns raised in your comments:

$**Q1**$: Regarding the question on how the model's performance could reach or even surpass the theoretical upper bound in zero-shot generalization

In object detection tasks, performance evaluation must consider both recall capability (i.e., the ratio of correctly detected targets to total targets) and precision (i.e., the proportion of correct detections among all positive predictions). The mean Average Precision (mAP) metric has become the standard for balancing these competing objectives, as comprehensively explained in [1].

However, conventional mAP formulations predominantly focus on closed-set detection tasks. In the context of open-world object detection (OWOD), certain object categories may lack annotations in the training data, $\textcolor{red}{\text{including the class names}}$. As a result, object detectors typically demonstrate degraded performance in such settings. Under such conditions, direct application of AP metrics becomes inadequate to reflect the performance, as demonstrated by leading OWOD works including ORE, OW-DETR, and PROB. For example, KTCN and PROB achieve similar U-AP values, but in terms of U-Recall, KTCN achieves more than twice the performance of PROB.

Thus, benchmarks such as MOWODB and SOWODB prioritize U-Recall as a metric to evaluate unknown object detection performance. $\textcolor{red}{\text{We surpass the zero-shot performance of the detector only in U-Recall, but not in U-AP}}$. Notably, exceeding zero-shot testing performance in U-AP remains challenging without utilizing additional information about unknown classes. The relative comparison of all methods in U-AP and U-Recall is shown in $\textcolor{red}{\text{Figure 1}}$.

$**Q2**$: Comparison Between Vague Descriptions and Detailed Descriptions

As mentioned in the paper, detailed descriptions rely on large language models (LLMs) to describe certain aspects of known classes, with the expectation that unknown classes share similarities in these aspects (as demonstrated by FOMO). These similarities are then used to recall unknown objects. However, directly using detailed descriptions in our method is impractical because their quantity exceeds 8,000. In contrast, our method achieves detection of unknown classes using only coarse descriptions composed of 10 simple sentences. Furthermore, we compared our method with all FOMO variants (using detailed attributes) in $\textcolor{red}{\text{Tables 7 and 8}}$. Our method still demonstrated significant advantages, achieving improvements of +4 in U-Recall and +8.1 in U-AP. For more details, please refer to $\textcolor{red}{\text{Appendix G}}$.

$**Q3**$: Choice of the Base Probability Model

In line with OWOD conventions, we adopt the Weibull distribution for probability modeling, as introduced by ORE for energy-based methods. While exploring alternative distributions (e.g., Beta) could potentially enhance performance, such designs often increase methodological complexity, hindering replicability and adoption. By adhering to the established Weibull distribution, $\textcolor{red}{\text{we ensure both simplicity and effectiveness}}$. Therefore, we follow the default setting in the OWOD field by using the Weibull distribution for modeling, demonstrating the effectiveness of PLSA.

$**Q4**$: Explanation of Figure 3 Annotations

In Figure 3, we illustrate the training process of VAP. In the figure, the blue circles represent samples that have already been labeled in the training set. These samples are treated as positive samples during training, and we train VAP by pulling them closer to the nearest coarse description (represented by the yellow circles in the figure). The triangles, on the other hand, represent $\textcolor{red}{\text{potential objects}}$. These are further distinguished using the three selection rules described in Section 3.2. We hope this clarifies the visual annotations in Figure 3.

We sincerely thank the reviewer for the valuable feedback. We hope our responses address the raised concerns, and we remain available to clarify any remaining questions or suggestions.

[1] Measuring Object Detection Models - mAP - What is Mean Average Precision?