D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement

Thank you for your detailed review and for highlighting the strengths of our work. We are pleased that you find our method innovative, systematic, and highly reproducible.

Regarding the smaller mAP improvement in Table 3 compared to Table 2:

Thank you for your observation. It's worth noting that as the baseline mAP increases, achieving further improvements becomes more challenging—a phenomenon often referred to as diminishing returns. Additionally, the models in Table 2 (now updated as Table 3) have not undergone the lightweight optimizations applied to the models in Table 3 (now updated as Table 4). When these lightweight modifications are excluded, as shown in Table 5, our FDR and GO-LSD modules improve the baseline model (RT-DETR-HGNetv2-L) from 53.0% to 54.5% mAP, demonstrating their effectiveness in enhancing performance.

Regarding the need for language refinement due to potential ambiguity:

Thank you for pointing out the ambiguities in our phrasing, particularly in lines 206–208 ("Initially, the first ...") and line 209 ("... one for each edge"). We appreciate your feedback and have revised the text for clarity to ensure that our explanations are precise and easy to understand. We will continue to review the manuscript to address any similar issues.

Regarding consistency in variable names in the Method section:

Thank you for your valuable suggestion. Initially, we intended to differentiate the notations for width and height between the Preliminaries and Method sections. However, we agree that consistency is important to avoid confusion. We have now unified all such notations to lowercase letters (i.e., {w, h}) throughout the manuscript for consistency and clarity.

Thank you again for your constructive feedback. We believe that these revisions enhance the clarity and quality of our paper, and we appreciate your assistance in improving it.

Thank you for your valuable suggestion. We agree that evaluating D-FINE across a broader range of datasets would strengthen our claims about generalizability and robustness. In response, we have updated the manuscript to include comparisons of D-FINE with other real-time object detectors on both the CrowdHuman and Objects365 datasets (see Table 2 and Table 8):

Table 2: Performance Comparison of Real-Time Object Detectors on CrowdHuman

Table 8: Performance Comparison of Real-Time Object Detectors on Objects365

These results demonstrate the significant superiority of D-FINE on dense scenes and large-scale complex datasets. We have also incorporated comparisons with RT-DETRv3 in Table 1 and Table 7 of the revised manuscript. We plan to further expand our evaluation with more datasets and will provide additional configurations and pretrained weights in future updates.

Thank you again for your constructive feedback. We believe these enhancements improve the presentation and strengthen the generalizability claims of our work.

Dear Reviewer 5Uux,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission1611

Thank you for your detailed review. We appreciate your thoughtful comments.

Regarding the claim that using fixed coordinates results in poor performance due to inadequate modeling of localization uncertainty:

We absolutely acknowledge the great success of fixed-coordinate methods like Faster R-CNN and RetinaNet, which have significantly advanced both speed and detection performance. We have revised the manuscript to clarify that our approach aims to optimize by better modeling localization uncertainty, rather than suggesting that fixed-coordinate regression methods are fundamentally limited. Our method builds upon these advancements by introducing a different perspective on handling localization, which we believe complements existing approaches.

Regarding the phrase "refining smaller residuals":

As shown in Figure 2, D-FINE performs layer-wise optimization of the bounding boxes, progressively bringing them closer to the objects. By "refining smaller residuals," we mean that as the earlier layers produce more accurate bounding box predictions, the difference (or residual) between the predicted boxes and the ground truth becomes smaller. This allows the subsequent layers to focus on correcting these smaller errors rather than making large corrections, effectively fine-tuning the predictions.

Regarding the use of Softmax over other normalization methods like Entmax in Eq. (3):

1. Consistency with KL Divergence: We chose to use Softmax to maintain consistency with the KL divergence calculation in Eq. (6), where Softmax and Log-Softmax are typically used to normalize logits, preserving more "knowledge" in the process.

2. Experiments with Alternatives: During the development of D-FINE, we experimented with several alternatives, including Sigmoid and Entmax. While Entmax also ensures the output sums to 1 (with the default parameter $1 < \alpha < 2$, where $\alpha = 1$ is equivalent to Softmax), it introduces a sparsity mechanism. The larger $\alpha$ is, the more likely small probabilities are set to zero, and larger probabilities become more pronounced. While this can simplify training and improve generalization in some cases, we preferred Softmax for D-FINE because it allows us to preserve even small probabilities. Since D-FINE involves a weighted sum of probabilities, preserving small probabilities ensures they contribute to the final bounding box prediction rather than being ignored.

3. Computational Efficiency: Additionally, Softmax has been widely used in numerous models, and its underlying implementation has been highly optimized. In our tests on a NVIDIA RTX 4090 GPU, for a sequence of length 10,000, the time for 1,000 iterations of Entmax was 3.0389 seconds, while for Softmax it was only 0.00472 seconds. Given that D-FINE is a real-time detector and Softmax performs well in our context, we chose Softmax for its efficiency and effectiveness.

Regarding the explanation of Figure 4 and its subplots:

In the Visualization Analysis section, we mentioned that the curves represent "probability distributions for the four edges (left, top, right, bottom)." These subplots illustrate how D-FINE adjusts each side of the bounding box based on the model's confidence. The unweighted distributions reflect the model's confidence in applying various levels of adjustment to each edge, while the weighted distributions show the actual adjustments made.

Here's what the different elements represent:

• Subplots (Left, Top, Right, Bottom): Each subplot corresponds to one edge of the bounding box and shows how adjustments are made for that specific edge.
• Peaks and Shifts: When the initial bounding box is sufficiently accurate, the unweighted distribution tends to be symmetric and centered near 16, leading to a weighted adjustment close to 0 (i.e., no adjustment needed). Peaks shifting to the left or right indicate that the edge should contract inward or expand outward, respectively. Peaks near the extremes of the distribution represent more significant adjustments.
• Colored Lines: The red and green lines represent these boxes and distributions belonging to the first and last layers.

We will revise the manuscript to include a more detailed explanation of these plots, along with an expanded caption for Figure 4, to clarify their purpose and interpretation. Thank you for bringing this to our attention.

Thank you again for your valuable feedback. We believe that these clarifications and revisions strengthen our paper, and we appreciate your help in improving its quality.

Thank you for your feedback and for pointing out your concerns, we address the main concerns as follows.

Regarding Figure 4 and Independent Modeling of Bounding Box Edges:

We agree that bounding box edges are influenced by each other during the refinement process. However, one of the key motivations of our work, as stated in Introduction (L062-L065) and Method (L188-L190), is to independently model each bounding box edge. This design allows us to explicitly capture the uncertainties and refinements needed for individual edges.

1. Dependent or Independent on Bounding Box Edges: You may have misunderstood the process of edge refinement. Each edge does not require determining two endpoints. Instead, D-FINE independently adjusts the distance of each edge relative to the center of the bounding box. This concept is defined in PRELIMINARIES (L138-L155) and Method (L206-L227), where the bounding box distances are explained.

2. Rationale Behind current Edge_Specific Visualizations:
   Figure 4 was designed to highlight the edge-specific adjustments made by D-FINE. In many cases, initial bounding box predictions are already accurate for some edges, requiring no significant adjustments, while other challenging edges with larger errors undergo targeted refinements. By visualizing the probability distributions of individual edges, we demonstrate how FDR effectively prioritizes the necessary corrections without introducing unnecessary changes.

3. Concerns on Whole-Box Visualization:
   While whole-box confidence distributions may provide a valuable global perspective, they do not align with the core functionality of FDR, which is designed to independently refine each edge. Such visualizations may overlook the fine-grained adjustments and edge-specific optimizations that are central to our approach.

Regarding Figure 3:

Thank you for pointing out your concerns. We agree that additional clarification can improve comprehension. We will revise the caption to include more details about the elements and interactions depicted in the figure.

The updated caption is as follows:

Figure 3: Overview of the GO-LSD process. Localization knowledge from the final layer (teacher) is distilled into shallower layers (students) through DDF loss with decoupled weighting strategies. Orange and green curves represent the probability distributions of unmatched and matched predictions, respectively, with decoupled weights $\alpha_k$ and $\beta_k$ controlling their contributions. Arrows indicate the flow of self-distillation, showing how refined distributions at the final layer guide shallower layers’ optimization.

These updates will be reflected in the revised manuscript after reformatting. Thank you for your valuable suggestion, which helps to improve the clarity and accessibility of our work.

Thanks to the authors for the rebuttal and responding to the comments. I am still not comfortable with Figure 4. Indeed, bbox edges are not independent and they are influenced by each other during the refinement process. So, separately visualising them misrepresents the interdependencies and shows an incomplete picture of the bbox. Moreover, subplots for each edge lack coherence in showing how the whole bbox improves during refinement. Maybe using confidence distributions for the whole bbox is a better choice. Further, Figure 3 is difficult to understand. Additional details in the caption can improve comprehension.

Dear Reviewer oAYL,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission1611

Dear Reviewer oAYL,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback. Below is a summary of how we have addressed your concerns:

1. Fixed Coordinates and Localization Uncertainty

   • We clarified that our approach complements, rather than undermines, fixed-coordinate methods. We acknowledge the success of methods like Faster R-CNN and RetinaNet and emphasize that our work seeks to enhance localization uncertainty modeling, building on existing advancements.

2. Refining Smaller Residuals

   • We provided a clearer explanation of "refining smaller residuals," highlighting that as early layers make more accurate bounding box predictions, subsequent layers focus on fine-tuning smaller residuals rather than correcting large errors.

3. Softmax vs. Other Normalization Methods (e.g., Entmax)

   • We explained our choice of Softmax over alternatives like Entmax, citing its consistency with KL divergence and the need to preserve small probabilities. We also discussed the computational efficiency of Softmax, particularly in real-time detection tasks, where it outperforms Entmax.

4. Figure 4 and Edge-Specific Refinement

   • We clarified that Figure 4 visualizes the independent adjustment of each bounding box edge. This approach helps capture edge-specific uncertainties and ensures targeted refinements. We also explained that visualizing whole-box confidence distributions may not align with the core functionality of our method, which focuses on edge-specific adjustments.

5. Figure 3 and Caption Update

   • We agreed that Figure 3 needed further clarification and revised its caption to include more details about the GO-LSD process. The updated caption now provides a clearer explanation of the interactions between matched and unmatched predictions, as well as the flow of self-distillation.

We hope that these revisions adequately address your concerns. Please let us know if there are any remaining issues or further questions. We look forward to your feedback.

Best regards,
D-FINE Authors

Thank you for your insightful comments and for pointing out areas that needed clarification. We appreciate your thorough review and are glad that you find the proposed method technically sound and well-presented.

Regarding the potential confusion in Figure 2 and the meaning of "layer" in FDR versus GO-LSD:

We apologize for the confusion caused by Figure 2. To address this, we have redrawn Figure 2 and explicitly numbered the different decoder layers for better clarity.

1. Clarification on Figure 2 and Decoder Layers:

   • In our model, every decoder layer outputs probability distributions through one D-FINE head. The first decoder layer includes an additional traditional bounding box regression head, which generates preliminary bounding boxes.
   • This means that within each decoder layer, there is one D-FINE head, and in the first decoder layer, there is an extra traditional head. The D-FINE heads iteratively refine the probability distributions across the decoder layers.

2. Consistency of the Term "Layer":

   • The term "layer" is used consistently across both Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD).
   • Specifically, FDR operates across all decoder layers, refining the probability distributions iteratively within each layer.
   • The outputs from these layers are then used in GO-LSD to distill localization knowledge from deeper layers to shallower ones. Thus, both FDR and GO-LSD involve operations across the same decoder layers, and the term "layer" refers to the layers of the decoder in both contexts.

Regarding Figure 3 and the representation of predictions:

Thank you for bringing this to our attention. We have revised Figure 3 to remove the potentially misleading gray squares and have unified the representation using unmatched predictions. Additionally, we have included annotations indicating the respective numbers of matched and unmatched predictions for clarity. This should make the figure more understandable and accurately convey the intended information.

Questions:

1. How many D-FINE heads are used in the first decoder layer, i.e., the value of $L$? Is FDR applied within each decoder layer or across decoder layers?

   • In our implementation, each decoder layer has one D-FINE head, so the value of $L$ corresponds to the number of decoder layers, which is three in our case.
   • FDR is applied across decoder layers, where each D-FINE head refines the probability distributions iteratively based on the previous layer's output.

2. If FDR is applied differently across different decoders, is the number of D-FINE heads consistent across all decoder layers or does it vary? How and why does it differ?

   • The number of D-FINE heads is consistent across all decoder layers; each layer has one D-FINE head.
   • The exception is the first decoder layer, which also includes an additional traditional regression head for generating preliminary bounding boxes.
   • We have added this information to the manuscript to provide a clearer picture of the model's architecture and operation. This enhancement should improve the reproducibility of our proposed method.

Thank you again for your valuable feedback. We believe that these clarifications and revisions will improve the clarity of our paper, and we appreciate your assistance in enhancing its quality.