GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection

Thanks for the positive feedback on the effectiveness of modules in our method. We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:

The motivation is in contrast to previous methods. Thank you for raising this important point. Our motivation does not contradict, but rather builds upon and extends the principles of methods like OvarNet to address the unique challenges of FG-OVD. First, we concur with OvarNet's core idea that learning attributes of objects can be beneficial for object detection. Our AEF module is designed on this principle, aiming to fuse beneficial attribute information with object features. In this regard, our philosophy is aligned with prior work. Second, unlike conventional OVD (e.g., on MS-COCO) or attribute detection in OvarNet, FG-OVD involves class names that are dense with descriptive attributes (e.g., "blue striped shirt"). This co-occurrence can lead to an "attribute over-representation" problem in text embeddings, where attribute semantics overwhelm the core subject identity. OvarNet, while effective for its target tasks, does not explicitly address this semantic entanglement issue inherent to FG-OVD. In summary, our work proposes to leverage helpful attributes while mitigating their potential to obscure the subject's identity in the fine-grained text embeddings. We will refine our manuscript to make this distinction clearer.

Inference efficiency decreased by LLM. We acknowledge the reviewer's valid concern regarding the inference latency introduced by the LLM. This represents a trade-off between our method's enhanced semantic understanding and computational cost. Nevertheless, the LLM-based subject identification is performed once per class name, not per image. For any given dataset or application scenario, the set of fine-grained classes is fixed. Therefore, the parsing results can be pre-computed and cached offline, imposing no additional LLM-related latency. For scenarios requiring on-the-fly parsing of new class names, the latency can indeed be a factor. This can be alleviated by employing more lightweight LLMs(e.g. LLaMA-3.1-8B as shown in Table 1) or batch processing multiple subject identification tasks in one chat with LLMs. We will also provide the inference efficiency analysis in our main paper.

Table 1: The comparison of inference time (ms) between Baseline (LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP (%) of each method in FGOVD.

Performance of MLLMs. We evaluate a LLaVA-v1.6-Mistral-7B, one of the strongest open-source MLLMs available. For each class in an FG-OVD annotation, we queried the model with a RefCOCO-style prompt, where statement is replaced by the target class name:

Your task is to locate and identify all objects or regions in the image that match the provided textual description. Description: "{statement}" Please return a JSON list. Each element must contain a "bbox" in the form [x1, y1, x2, y2] and a "confidence" between 0.0 and 1.0.

Under this protocol, the model attains only 0.4~mAP on the FG-OVD benchmark, which is significantly below our GUIDED. This experiment reveals that direct MLLM inference is presently insufficient for practical or research use in FG-OVD.

It is not aligned with Figure 3(b) and Figure 2. Thanks for pointing this out. Figure 2 does not accurately depict the two query types. We will correct it in our revised paper.

The meaning of 'EDD'.
We apologize for the oversight. 'EDD' was a working name for our method from an earlier draft. We will correct this typo in the revised manuscript and perform a thorough check to ensure all terminology is consistent.

Thanks for the positive feedback on the reasonable rationale and the AEF module in our method. We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:

Key variables are not linked to the architecture block diagrams. We appreciate this observation. In the revised manuscript, we will explicitly annotate Figure 2 to map key variables to their corresponding components in the architecture diagram.

Unfair comparison with LaMI-DETR To address this concern, we clarify that in Table 1 of our main paper, we initially reported LaMI-DETR's performance trained on LVIS following the HA-FGOVD protocol, which matches the first-stage training in our method. Additionally, we report the performance of LaMI-DETR under the same training protocol in Table 4(see line 3 Baseline + Finetune) in our main paper. Under the same protocol, our GUIDED framework shows a 15.5% improvement, underscoring the effectiveness of our method. For clarity, we will also integrate the results trained on the same protocol into Table 1 in our main paper.

No evaluation on standard open-vocabulary benchmark. We present the performance on the LVIS benchmark in Table 1. The results show fine-tuning on the FG-OVD dataset degrades the performance on the LVIS benchmark. However, co-training on LVIS and FG-OVD during the second stage not only alleviates this issue but surpasses the original LaMI-DETR baseline while mainly preserving FG-OVD performance. This enhancement stems from our AEF module's attribute integration and expanded training data from FG-OVD. We will supplement our paper with these analyses.

Table 1: The comparison on the standard open-vocabulary benchmark LVIS dataset and the FGOVD dataset.

Whether subtraction should be meaningful or stable across diverse classes. The CLIP applies the contrastive learning loss to align the embeddings of images and their corresponding caption. The alignment of texts with fine-grained attributes enables CLIP to encode fine-grained attributes in the text embeddings. Nevertheless, we acknowledge that CLIP text embeddings may not always capture all the attributes. To address this issue, GUIDED strategically employs subtraction-based AEF to selectively exploit the knowledge in attributes for coarse-grained detection, which detects candidate regions for subjects of fine-grained classes without requiring exhaustive attribute knowledge. GUIDED leaves the discrimination of fine-grained attributes in FGAD. FGAD applies a refined CLIP with an additional linear projection trained on fine-grained data, which demonstrates superior fine-grained feature capture capability. Empirically, Table 1 of our supplementary material shows that the subtraction process enhances the MAP by 1.3%, demonstrating that applying the subtraction is more reasonable than directly applying the original attribute embeddings. Furthermore, employing the CLIP embeddings of the isolated attributes (e.g., "shiny red with tinted windows") declines the performance by 2.8%. It is because the embeddings of the isolated attributes do not capture the relation between subjects and attributes.

Whether subtraction introduces noise or ambiguity. While CLIP text embeddings may not capture all attributes perfectly, the subtraction process may output similar key embeddings for classes with similar or visually correlated attributes. Nevertheless, similar to the response for the last question, GUIDED inherently mitigates this issue by design. Specifically, AEF selectively integrates attribute knowledge for coarse-grained detection rather than exhaustive attribute discrimination, thus avoiding reliance on subtle inter-attribute distinctions. Crucially, when subtraction produces similar key embeddings, their corresponding value embeddings, which are directly obtained from attribute embeddings, exhibit analogous similarity. Consequently, AEF’s selective fusion integrates minimally divergent attribute signals into object queries, which naturally down-weighting noise or irrelevant information.

Comparing subtraction-based attention with simpler architecture. We compare the mAP of subtraction-based attention with additive fusion and concatenation in FGOVD in Table 2. For concatenation, we apply a linear projection layer to transform the dimensions back to the original dimensions. The results show that our AEF outperforms both additive fusion and concatenation by a clear margin, demonstrating the effectiveness of the subtraction-based attention on attribute integrations.

Table 2: The comparison of mAP(%) evaluation results on the FG-OVD benchmark between our proposed AEF and simpler architecture.

Thanks for the positive feedback on the well motivation, systematic approach and systematic approach in our method. We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:

Insufficient analysis of the attribute similarity score. We conducted an ablation study by setting $\alpha$ to extreme values (summarized in Table 1). Setting $\alpha$ to $0$ means relying solely on the detector's coarse confidence scores to estimate final scores and $L_{\text{contrast}}$, while setting $\alpha$ to $1$ means relying only on the attribute similarity scores. As shown, both extremes lead to a significant performance drop. This empirically confirms our hypothesis that a balanced fusion is crucial. Regarding alternative fusion strategies, we evaluate the weighted average in Table 2. It shows that weighted multiplication achieves a better performance. Compared with the weighted average, the multiplication enforces a high score on both the coarse confidence score and the attribute similarity simultaneously. This prevents candidates with a poor attribute similarity score from being promoted simply because their confidence scores are high (and vice versa).

Table 1: More ablation study on the hyperparameter $\alpha$.

Table 2: Ablation study with different fusion strategies.

Insufficient computational overhead analysis.
While our method utilizes GPT-4o for its reasoning capabilities, measuring its API inference time directly is problematic as network latency becomes a confounding factor. To provide a meaningful assessment of the computational overhead introduced by our LLM component, we instead evaluated inference times using locally deployable LLMs: LLaMA-3.1-8B and LLaMA-3.3-70B. Crucially, these models achieve comparable mAP performance to GPT-4o on the FG-OVD task (as validated in our experiments), making them valid proxies for analyzing inference speed. We measured the inference time in Table 3. As shown in the Table, the inference time increase in GUIDED primarily stems from LLM-based subject identification, while AEF and CLIP design in FGAD contribute minimally to latency. This time consumption gain can be alleviated by adopting a lightweight LLM or batch processing multiple subject identification tasks in one chat. For memory consumption in Table 4, the memory footprint of our core detection modules (AEF, FGAD) is negligible. The significant memory increase comes from loading the LLM. We will include these analyses in the revised paper.

Table 3: The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD.

Table 4: The Memory occupation(MB) of each module during inference in our GUIDED framework.

Failure cases. We identify a key limitation when fine-grained attributes describe spatial relationships extending beyond the detected subject's bounding box. For example, for the query "suitcase on the conveyor belt", the conveyor belt (contextual attribute) may not be captured within the candidate box of any suitcase. Consequently, GUIDED may incorrectly associate the attribute with a suitcase spatially adjacent to the conveyor belt. This occurs because our method relies on the detector's candidate boxes, which may not encompass all required contextual elements. Due to the NIPS policy, we can not provide the visualization here, and we will include visualizations of such cases in the revised paper to clarify this failure mode.

Reproducibility concerns and the choice of LLMs. We appreciate this important concern regarding reproducibility. To ensure transparency, we conducted experiments with open-source LLMs (LLaMA-3.1-8B and LLaMA-3.3-70B) as alternatives to GPT-4o. As quantified in Table 5, the mAP of GUIDED drops by merely 0.5% with LLaMA-3.1-8B and improves by 0.1% with LLaMA-3.3-70B. This demonstrates that GUIDED performance is not dependent on a specific proprietary model. We will provide more results with LLaMA-3.1-8B and LLaMA-3.3-70B forwith other OVD models and other datasets in our revised paper for reproducibility.

Table 5: The comparison of mAP(%) evaluation results on the FG-OVD benchmark with different LLMs in subject identification.

Robustness of LLMs on subject identification. To evaluate the robustness of LLMs on subject identification, we manually annotate the subjects from 300 fine-grained classes and assess the accuracy of subject identification with different LLMs. As summarized in Table~, failure cases are categorized into two types: (1) Hallucination toward in-context samples, where the LLM generates subjects irrelevant to the input text; (2) Other errors, such as identifying an attribute instead of an object. Overall, all three LLMs achieve high correctness rates, demonstrating robust performance across diverse architectures. The results confirm the high robustness of this stage. Crucially, they show that while the smaller LLaMA-3.1-8B model is prone to hallucination errors that always propagate (8/8), this critical failure mode is completely eliminated by larger open-source models. Both LLaMA-3.3-70B and GPT-4o exhibit near-perfect performance, with their rare errors being minor and not always affecting the final detection. This demonstrates that our method's success is not tied to a specific proprietary model and is robust when using state-of-the-art open-source alternatives. We will add this analysis to our revised paper.

Table 6: The number of failure cases in subject detection of 300 samples with different LLMs. The notation '$x/y$' denotes that there are $y$ failure cases, of which $x$ lead to detection errors.

Analysis of $m_{fine}$ and $m_{coarse}$. $m_{fine}$ and $m_{coarse}$are used as temperatures to perform softmax with CLIP embeddings, and 50 is a common choice for the temperatures, which is extensively applied in previous work. We also provide the result with different $m_{\text{fine}}$ and $m_{\text{coarse}}$ in Table 7, performance remains robust across variations of these values.

Table 7: Ablation study on $m_{fine}$ and $m_{coarse}$.

Thanks for the positive feedback on the extensive experiments and the superior performance of our method. We also appreciate the constructive comments to improve the paper. We would like to answer the proposed questions in the following:

The meaning of 'EDD'. We apologize for the oversight. 'EDD' was a working name for our method from an earlier draft. We will correct this typo in the revised manuscript and perform a thorough check to ensure all terminology is consistent.

Texts in some figures are difficult to read. We will resize and rearrange the text in these figures to improve clarity. Additionally, we will review all figures in the paper to ensure they are legible.

The meaning of the star next to HA-FGOVD. As noted in the caption of Table 1 in our main paper, the star indicates that the results for HA-FGOVD are from our reproduction because the original paper did not provide these results. We will make this explanation more prominent in the caption to avoid any confusion.

Full inference time of a sample. While GUIDED utilizes GPT-4o for its reasoning capabilities, measuring its API inference time directly is problematic as network latency becomes a confounding factor. To provide a meaningful assessment of the computational overhead introduced by our LLM component, we instead evaluated inference times using locally deployable LLMs: LLaMA-3.1-8B and LLaMA-3.3-70B. Crucially, these models achieve comparable mAP performance to GPT-4o on the FG-OVD task (as validated in Table 1), making them valid proxies for analyzing inference speed. We measured the inference time in Table 1 and will include these results in the revised paper.

Table 1: The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD.