CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detection

Thank you for the careful reading and kind words. We are pleased that you find our performance promising and our idea intuitive and reasonable. We address your concerns in the following.

Q1: Clearly formulate the process of Image-Guided Query Selection by equation of algorithm.

A1: Thank you for this valuable suggestion. Below, we provide the algorithmic descriptions (which we believe is more clear than equations) of the Image-Guided Query Selection process. This algorithm share enhance the clarity of our method in the revised paper.

Algorithm 1: Image-Guided Query Selection

---

Input:

• B: Batch size
• C: Number of category queries
• D: Embedding dimension
• C': Number of selected category queries
• H': Height of image features
• W': Width of image features
• $Q_{cat}$: Category queries ∈ $R^{B×C×D}$
• $F_{img}$: Image features ∈ $R^{B×D×H'×W'}$

Output:

• $Q_{cat}'$: Selected enhanced category queries ∈ $R^{B×C'×D}$
• $I$: Selection indices ∈ $R^{B×C'}$

---

1: $F_{flat}$ ← Reshape($F_{img}$, [B, D, H'×W'])

2: $Q_{enhanced}$ ← $Q_{cat}$     // Initialize enhanced queries

3: for l = 1 to 2 do     // Two cross-attention layers

4:     $Q_{attn}$ ← MultiHeadCrossAttention($Q_{enhanced}$, $F_{flat}$, $F_{flat}$)     // Cross-attention

5:     $Q_{enhanced}$ ← LayerNorm($Q_{attn}$ + $Q_{enhanced}$)     // Add & Norm

6:     $Q_{ffn}$ ← FFN($Q_{enhanced}$)     // Feed-forward network

7: end for

8: $L$ ← LinearProjection($Q_{enhanced}$)     // Compute selection logits ∈ $R^{B×C}$

9: for b = 1 to B do

10:     $I_{(b,:)}$ ← TopK($L_{(b,:)}$, C')     // Select top-C' indices

11:     $Q_{cat}'$[b,:,:] ← $Q_{enhanced}$[b, $I_{(b,:)}$, :]     // Select enhanced queries

12: end for

13: return $Q_{cat}'$, $I$

---

Q2: Compare the proposed selection methods with other solutions (e.g., using off-the-shelf classifiers)

A2: Following your suggestion, we conducted experiments comparing our end-to-end image-guided query selection with alternative selection strategies to demonstrate the necessity of joint optimization. We designed two alternative approaches for the query selection mechanism:

1.Frozen Selection: We pre-trained all encoder components (image encoder, category queries, and image-guided query selection module) for classification, then froze these components entirely during detection training.

2.Dual-Encoder Off-the-shelf: Based on the frozen selection method, we duplicated the image encoder architecture. One copy remains frozen together with category queries and selection modules to serve as an off-the-shelf classifier for category selection, while the second image encoder continues training for the detection task. This approach represents a compromise between using off-the-shelf components and task-specific optimization, separating classification and detection feature learning.

Table R-9: Comparison with alternative selection strategies.

Our end-to-end approach achieves the highest performance, outperforming the frozen selection by 1.8% AP and the dual-encoder approach by 0.4% AP. This demonstrates that joint optimization is important. Though the dual-encoder approach achieves similar results to our CQ-DINO (51.9% vs 52.3% AP), its main weakness lies in the significant computational overhead. It reduces inference speed to from 10.6 to 7.9 FPS while requiring 40% more memory (46.7GB vs 33.3GB). This brings more challenge for real-world deployment despite competitive accuracy.

Q3: Clearly formulate the process of Encoding Category Correlations

A3: Thank you for this important feedback. We provide a detailed algorithmic formulation of the Category Correlation Encoding process, which will be included in the revised paper.

2. Encoding Category Correlations

Algorithm 2: Explicit Hierarchical Tree Construction

---

Input:

• $Q_{cat}$: Category queries ∈ $R^{C×D}$
• T: Hierarchical tree structure with nodes V
• w: Base weight parameter (default: 0.3)

Output:

• $Q_{corr}$: Correlation-enhanced queries ∈ $R^{C×D}$

---

1: $Q_{corr}$ ← $Q_{cat}$     // Initialize with original queries

2: $N_{max}$ ← $max_{v∈V} |Children(v, T)|$    // Find maximum children count

3: L ← TopologicalSort(T)     // Get bottom-up ordering

4: for each node v in Reverse(L) do     // Bottom-up traversal

5:     if IsLeaf(v, T) then

6:        continue     // Keep original query for leaf nodes

7:     else

8:        C(v) ← GetChildren(v, T)     // Get children of node v

9:        $n_v$ ← |C(v)|     // Count children

10:        $α_v ← w + log(n_v + 1)/log(N_{max} + 1)$     // Adaptive weight (Eq. 7)

11:        $α_v ← min(α_v, 1.0)$     // Ensure $α_v$ ∈ [0, 1]

12:        $Q_{child}^{mean} ← (1/n_v) ∑_{c∈C(v)} Q_{corr}[c,:]$     // Mean pooling

13:        $Q_{corr}[v,:] ← (1-α_v)·Q_{cat}[v,:] + α_v·Q_{child}^{mean}$    // Update (Eq. 6)

14:     end if

15: end for

16: return $Q_{corr}$

---

Algorithm 3: Self-Attention for Implicit Category Relations

---

Input:

• $Q_{cat}$: Category queries ∈ $R^{C×D}$
• H: Number of attention heads

Output:

• $Q_{corr}$: Correlation-enhanced queries ∈ $R^{C×D}$

---

1: $Q_{attn}$ ← MultiHeadSelfAttention($Q_{cat}$)     // Multi-head self-attention

2: $Q_{corr}$ ← LayerNorm($Q_{attn}$ + $Q_{cat}$)     // Add & Norm

3: return $Q_{corr}$

---

Q4. More explicit gradient change analysis to show the effectiveness of rebalancing gradient and gradient dilution.

A4: We provide comprehensive gradient change analysis with extended training iterations to explicitly demonstrate the effectiveness of our gradient rebalancing mechanism in addressing both positive gradient dilution and hard negative gradient dilution.

1.Extended Positive-to-Negative Gradient Ratio Analysis

We extend the training iterations in Fig 2 as shown in Table R-6. Due to policy constraints that restrict the inclusion of images, we present the extended analysis through sampled key iterations across 200k training steps.

Results show that CQ-DINO maintains early rebalancing advantage. CQ-DINO maintains consistently higher ratios (>1.0) throughout the critical early training phase (less than 10k iterations), when gradient balance is most crucial for learning meaningful representations. In contrast, DINO with Cross-Entropy loss starts with critically low positive gradient ratios (0.01 at 0.5k iterations), demonstrating severe positive gradient dilution.

2.Hard Negative Gradient Dilution Analysis

To examine the hard negative gradient dilution problem discussed in Section 3.1, we define hard negatives as the top 10% of categories with the highest prediction scores among all negative categories, representing the most confusing negatives requiring focused learning.

Results in Table R-8 show that CQ-DINO consistently achieves the highest hard negative ratios throughout training, particularly during critical early stages (0.83 vs 0.23 for DINO w/CE at 0.5k iterations). The consistently high ratios confirm our theoretical analysis that image-guided query selection performs effective implicit hard negative mining by naturally selecting the most semantically challenging and relevant categories.

The results validate our theoretical analysis and demonstrate the practical effectiveness of image-guided query selection in vast vocabulary detection scenarios. The original plots will be included in the revised paper.

Table R-6: Positive-to-negative gradient ratio across extended training iterations.

Table R-8: Hard negatives-to-all negative gradient ratio across different training iterations

Thank you for your positive feedback and recognition of our work. We are deeply grateful for your positive assessment of our motivation and experimental results. Below, we address your concerns point-by-point.

Q1: The positive-to-negative gradient plot should be drawn in longer iterations.

A1: Thank you for this suggestion. We have extended our positive-to-negative gradient ratio analysis to 200k iterations to provide a more comprehensive view of the training dynamics. Due to policy constraints that restrict the inclusion of images, we present the results in Table R-6 below, sampling key iterations throughout the training process.

The extended analysis reveals several key insights: CQ-DINO maintains consistently higher positive-to-negative gradient ratios during the critical early training phase (less than 10k iterations), demonstrating effective gradient rebalancing when learning is most crucial. In contrast, DINO with Cross-Entropy loss exhibits severe positive gradient dilution in early iterations (starting at 0.01) before gradually recovering. While all methods eventually converge to similar gradient ratios (~1.1) in later training stages (>25k iterations), CQ-DINO achieves this balanced state much earlier, resulting in more efficient training.

This extended analysis strongly supports our hypothesis that CQ-DINO effectively addresses the positive gradient dilution problem identified in vast vocabulary detection, particularly during the crucial early training phase where proper gradient balance is essential for learning meaningful representations.

Table R-6: Positive-to-negative gradient ratio across extended training iterations.

Q2: Could the proposed CQ-DINO handle category correlations in datasets with incomplete or ambiguous hierarchies?

A2: Great question!  To evaluate CQ-DINO's robustness to incomplete hierarchical information, we conducted controlled experiments by artificially degrading the V3Det hierarchical structure in two ways:

1. Missing hierarchies: We randomly removed hierarchical connections for 1/6 of the categories, simulating datasets with incomplete structural information.

2. Ambiguous hierarchies: We introduced fuzzy/ambiguous hierarchical relationships for another 1/6 of the categories, representing scenarios where category boundaries are unclear.

When hierarchical information is incomplete, CQ-DINO achieves 51.7% AP and outperforms the baseline without category correlations (51.1% AP) by 0.6% AP. These results demonstrate that CQ-DINO can handle real-world scenarios where hierarchical structures are imperfect or incomplete. Besides, when hierarchical information is completely missing, CQ-DINO can seamlessly switch to implicit relation learning through the self-attention module (51.3% AP), which also outperforms the no-correlation baseline.

Table R-7: Performance under incomplete hierarchical structures.

Q3: Visualization and deeper analysis is missing.

A3: To provide deeper analysis beyond the positive-to-negative gradient ratio, we conducted additional experiments examining the hard negative gradient dilution problem discussed in Section 3.1. We define hard negatives as the top 10% of categories with the highest prediction scores among all negative categories, representing the most confusing negatives that require focused learning.

Results depicted in Table R-8 show that our CQ-DINO consistently achieves the highest hard negative ratios throughout training, particularly in the critical early stages (less than 10k iterations), confirming our theoretical analysis (in Section 3.1) that image-guided query selection performs effective implicit hard negative mining.

Due to policy constraints that restrict the inclusion of images in this response, we will include comprehensive visualizations in the revised paper showing: (1) positive-to-negative gradient ratio plot. (2) Hard negative-to-all negative gradient ratio plot. (3) Visualization of detection performance on challenging samples and failure case analysis.

Table R-8: Hard negative-to-all negative gradient ratio across training iterations

Thank you for thoroughly reading our paper. We are deeply grateful that you recognize the significance of our gradient dilution analysis. We are very thankful that you find our method to be a clever solution to the identified gradient dilution problems. We address your concerns below.

Q1: Complexity of our two-stage training approach.

A1: Indeed, the two-stage training introduces some additional complexity. In practice, the overhead is minimal. Stage 1 requires only ~1 hour compared to the 57 hours for full training. For deployment, only the final trained model is needed without two-stage training. For future work, we are actively exploring methods such as progressive query selection to develop stable end-to-end alternatives to keep high category recall throughout training.

Q2: Gradient dilution issue VS class imbalance problem.

A2: We appreciate the reviewer's insight connecting gradient dilution to class imbalance. However, our work identifies and addresses a distinct scaling challenge in vast vocabularies that differs from traditional class imbalance problems.

Traditional class imbalance mainly focuses on sample distribution disparities between rare and common classes. In contrast, gradient dilution in vast vocabularies affects even common classes due to the overwhelming negative space created by thousands of categories. Our theoretical analysis (Equations 2-4) demonstrates that the positive-to-negative gradient ratio scales as $\rho \propto \frac{1}{C\cdot ε}$, where $C$ is the vocabulary size and ε represents the average activation probability of negative classes. This means the problem severity increases proportionally with vocabulary size $C$, not just sample imbalance. Moreover, vast vocabularies cause informative hard negatives to be overwhelmed by numerous easy negatives.

To address the concern about whether CQ-DINO can alleviate the class imbalance problem in V3Det, we evaluate the results of rare classes (number of training samples less than 10) and common classes (otherwise) in Table R-3. The results show that even with carefully tuned hyperparameters, Focal Loss cannot fundamentally solve the 1/C scaling problem. Our dynamic category selection directly addresses the optimization challenges, achieving consistent improvements across both rare and common categories.

Table R-3: Performance of rare classes and common classes on V3Det dataset.

Q3: Is this just pre-filtering for existing Grounding DINO detector?

A3: No, CQ-DINO is fundamentally different from pre-filtering. It addresses the optimization challenges that Grounding DINO cannot solve through architectural modifications alone.

Grounding DINO faces scalability bottlenecks for vast vocabulary detection (Section 2.2, lines 115-119). With its 128 token limit per inference pass, detecting all 13,204 V3Det categories requires over 331 sequential inference passes per image. For the complete V3Det validation set (30k images), this requires approximately 768 GPU hours on H800 GPUs for inference alone, which is almost infeasible with our limited computational resources.

While we leverage Grounding DINO's proven feature enhancer and decoder (which excel at fusing image features and category queries), our baseline performance is nearly identical to the optimally Focal loss tuned DINO (47.3% Table3 row 1 vs 47.4% Table 9 row5). This further confirms that architectural components alone cannot solve vast vocabulary challenges.

Q4: What characterizes the 17% of missed objects causing the category recall bottleneck - rare categories, small objects, or visual similarity?

A4: We analyze the 3,527 categories with recall lower than 83.3% to understand the characteristics of missed objects. Our analysis reveals that both rare category performance and small object detection represent significant bottlenecks compared to our overall performance of 52.3% AP.

(1) Category frequency analysis (Table R-4): Among the low-recall categories, 374 are rare categories and 3,153 are common categories. On these challenging categories, CQ-DINO achieves 20.5% AP on rare categories, substantially lower than our overall 52.3% AP, indicating that rare categories face a significant performance gap.

(2)  Object scale analysis (Table R-5): The most significant performance gaps occur for small objects, where CQ-DINO achieves only 14.1% AP compared to our overall performance of 52.3% AP. This indicates that object scale is the primary factor in the recall bottleneck, with small objects being particularly challenging.

While we do not achieve strong absolute performance in these challenging cases, CQ-DINO still consistently outperforms both baseline DINO and DINO using carefully tuned focal loss across all categories and scales. This demonstrates that these are common difficulties in vast vocabulary detection rather than unique limitations of our approach.

Table R-4: Analysis on low recall categories.

Table R-5: Analysis on object scales.

Q5: Does CQ-DINO's 4.9% AP improvement stem from gradient balancing or increased model complexity?

A4: The performance gain is fundamentally owing to the gradient balancing, not model complexity. We provide comprehensive evidence through detailed ablation studies:

(1) Our baseline without image-guided query selection (47.3% AP in Table 4, row 1) achieves nearly identical performance to DINO trained with optimally-tuned Focal Loss (47.4% AP in Table 9). This demonstrates that the model architecture itself does not provide inherent advantages.

(2) Table 4 shows that adding image-guided query selection alone improves AP from 47.3% to 51.1% (+3.8%), which directly addresses gradient dilution by reducing the negative search space. Table R-1 shows that training-stage gradient dilution mitigation alone (row 4) provides +1.9% AP. This further validates that gradient rebalancing is the core mechanism.

(3) Encoding category correlations contributes +1.2% AP in Table 4 (52.3% vs 51.1%, row 6 vs row 5).

Table R-1. Additional ablations on isolated component contributions. 0 indicates the component is disabled and 1 indicates it is enabled.

Thank you for your insightful comments and kind words. We greatly appreciate your recognition of our significant performance improvements. Below we will address your concerns and provide further clarification.

Q1: Lack of discussion on prior works.

A1: Thank you for your valuable feedback regarding the relevant literature [2, 8, 10], particularly the similarity between RankSeg's top-k category selection and our image-guided query selection. We will add a dedicated "Category Query-based Methods" section to our related work.

In the following, we respectfully clarify several key distinctions that differentiate our approach:

1. Novel theoretical motivation: To the best of our knowledge, our work provides the first systematic analysis of gradient dilution challenges in vast vocabulary detection. This theoretical foundation directly motivates our image-guided query selection design. In contrast, RankSeg employs top-k selection primarily for reducing the negative sample space during inference, which decreases computational cost and improves accuracy.
2. Scale-specific challenges: While top-k selection during inference works for semantic segmentation with limited vocabularies, vast vocabularies present challenges. Our analysis in Table R-1 reveals: inference-stage category selection harms performance in V3Det (vast vocabulary: 49.4% → 46.6%, rows 2 → 3) while helping in COCO (limited vocabulary: 57.9% → 58.2%). This demonstrates the necessity of addressing gradient dilution during training, not just inference optimization.
3. Hierarchical relationship modeling: Unlike previous works operating on relatively flat category structures, we specifically address the complex hierarchical relationships inherent in vast vocabulary datasets through our novel hierarchical tree construction method.

Table R-1. Additional ablations on isolated component contributions.

We will incorporate the following related work section:

Category Query-based Methods

The paradigm of learnable queries emerged in computer vision with DETR [1], demonstrating a shift from fixed components to learnable representations that capture task-specific patterns. DETR's query mechanism inspired similar strategies across diverse tasks, including classification [2], segmentation [3], and multimodal learning [4].

Among these developments, category queries represent an innovation introduced by Query2Label [2]. Rather than relying on fixed classification heads, Query2Label proposed learnable category embeddings to capture category-specific features. Subsequent works such as ML-Decoder [5] have outperformed conventional classification methods. The effectiveness of category queries in classification motivated their extension to dense prediction tasks. CQL [7] employs category queries for human-object interaction classification. Similarly, ControlCap [8] leverages category queries to provide semantic guidance for region captioning tasks. RankSeg [9] integrates category queries into semantic segmentation by performing pixel-level classification while dynamically selecting only the top-k most relevant categories during inference. This selective approach reduces the negative search space, improving both computational efficiency and segmentation accuracy.

While prior works have explored category queries in various contexts, our work addresses a fundamentally different challenge specific to vast vocabulary scenarios. We provide the first, to-our-knowledge, systematic theoretical analysis of gradient dilution issues that arise when dealing with vast category vocabularies, which motivates our image-guided query selection design. Additionally, unlike previous methods working with limited category scales, we specifically tackle the complex hierarchical relationships inherent in vast vocabulary datasets through our hierarchical tree construction method, designed to encode category correlations across thousands of semantically related categories.

Reference

[1] End-to-End Object Detection with Transformers

[2] Query2Label: A Simple Transformer Way to Multi-Label Classification

[3] Per-Pixel Classification is Not All You Need….

[4] BLIP-2: Bootstrapping Language-Image….

[5] ML-Decoder: Scalable and Versatile Classification Head

[6] Category Query Learning for Human-Object Interaction Classification

[7] ControlCap: Controllable Region-level Captioning

[8] RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentation

Q2: Incomplete baseline comparison with Grounding DINO.

A3: As suggested, we provide the following clarification. Detailed in lines 115-119, Grounding DINO faces scalability limitations for vast vocabulary detection. With a token limit of 128 per inference pass and V3Det's 13204 categories, detecting all categories requires over 331 sequential inference passes per image. For the full V3Det val set (30k images), this translates to approximately 768 GPU hours on H800 GPUs for inference alone, making comprehensive evaluation computationally prohibitive.

To provide meaningful comparison while addressing computational constraints, we evaluate on V3Det's rare classes in Table R-2. This subset represents the most challenging categories where gradient dilution effects are most pronounced. Grounding DINO is a pretrained model from mmdetection, trained on O365, GoldG, and V3Det datasets. DINO and CQ-DINO are trained on the V3Det dataset.

Table R-2: Performance on V3Det rare classes (1072 categories).

Grounding DINO underperforms DINO (-13.0%), revealing limitations in its training paradigm in vast vocabulary settings. Grounding DINO uses positive labels with only randomly sampled small sets of negative labels during training. With few similar categories sampled during training, Grounding DINO fails to learn fine-grained distinctions between semantically similar categories—the core challenge in vast vocabulary detection.

To isolate Grounding DINO’s component impacts, we conduct ablation studies in Table 4 (row 1) and Table R-1 (row 1). Substantial improvements show our method focuses on addressing the fundamental challenges of vast vocabulary detection rather than simply benefiting from architectural choices.

Q3: Adding ablations isolating contributions from: (a) Grounding DINO, (b) inference-stage category selection, and (c) training-stage gradient dilution mitigation.

A2: Thank you for this insightful suggestion. We now provide additional ablation studies in Table R-1 to isolate the contributions of each component and validate our main claim about gradient dilution mitigation.

Component Definitions: (a) Grounding DINO components: The baseline architecture incorporating feature enhancer and decoder components from GroundingDINO (row 1 in Table R-1).

(b) Inference-stage category selection: Selecting only top-K relevant categories based on similarity of category queries and image features during inference.

(c) Training-stage gradient dilution mitigation: Using image-guided query selection during training to mitigate gradient dilution.

Key observations from Table R-1:

(1) Training-stage gradient dilution mitigation is the primary contributor. On V3Det, this component alone provides a 1.9% AP improvement (comparing rows 2 and 4). This validates our hypothesis that gradient dilution is a challenge in vast vocabulary detection. On COCO, the improvement is modest (0.3% AP) since gradient dilution is less severe with only 80 categories.

(2) Inference-stage category selection has different effects with respect to vocabulary size.

• V3Det: Inference selection without gradient dilution mitigation actually hurts performance (rows 2 vs 3), demonstrating that simply reducing categories at inference cannot solve the underlying gradient dilution problem.

• COCO: Inference selection provides modest improvements as expected when the negative space is reduced.

(3) The combination achieves optimal performance and efficiency. When both training-stage gradient dilution mitigation and inference-stage selection are combined (row 6), we achieve the best results on both datasets. Inference-stage selection improves inference speed (0.6 → 10.6 FPS on V3Det).

These additional ablation studies further demonstrate that (c) training-stage gradient dilution mitigation is the core contribution, while (a) Grounding DINO components provide the architectural foundation and (b) inference-stage category selection offers computational efficiency. Table R-1 will be incorporated into Table 4 in the revised paper.

Q4: Are Focal loss and CQ-DINO equally effective in solving gradient dilution problems in vast vocabulary detection?

No, CQ-DINO and Focal Loss address gradient dilution problems from different aspects. Our theoretical analysis in Sec. 3.1 reveals that the positive-to-negative gradient ratio scales as ρ ∝ 1/(C·ε), where C is the vocabulary size and ε represents the average activation probability of negative classes.

While Focal Loss can mitigate gradient dilution by adjusting its parameters to reduce ε, it cannot address the core 1/C scaling problem inherent to vast vocabulary detection. As vocabulary size C grows beyond 10⁴ categories, this scaling factor becomes the dominant bottleneck regardless of ε optimization. In contrast, CQ-DINO fundamentally solves the scaling issue by reducing the effective vocabulary size from C to C' (where C' ≪ C) through image-guided query selection.