Union-over-Intersections: Object Detection beyond Winner-Takes-All

We thank the reviewer for their appreciation and the actionable feedback to improve our experimental rigor.

IoU-based loss functions

We clarify our method is complementary to existing IoU-based loss functions, not a competitor. To address the reviewer's comment, we performed additional experiments combining our method with the publicly available implementations of GIoU [a], DIoU [b], and Alpha-IoU [c] . Results are presented in the table below, using a ResNet-50 backbone on the COCO-val dataset. Integrating these losses into our approach is straightforward and just requires replacing the L1 loss with the respective IoU-based loss during the regression stage.The results confirm consistent improvements, demonstrating that our approach not only works with these advances but also enhances their performance. We will include this experiment and discussion in the supplemental material.

[a] Rezatofighi H, et al. Generalized intersection over union: A metric and a loss for bounding box regression. CVPR 2019.

[b] Zheng Z, et al. Distance-IoU loss: Faster and better learning for bounding box regression. AAAI 2020.

[c] He J, et al. $\alpha$-IoU: A family of power intersection over union losses for bounding box regression. NeurIPS 2021.

---

Effect of different nms variants

Our method is indeed compatible with various NMS variants, as they primarily influence the grouping stage. Following the reviewer suggestion, we conducted experiments with Cluster-NMS [d], comparing it to NMS and Soft-NMS [e]. The results confirm consistent improvements across all variants, demonstrating the robustness of our approach. We will include this ablation in the supplemental material.

We use a Resnet50 backbone for the Faster-RCNN and the results are on the COCO val set. All variants are trained with L1 regression loss.

[d] Zheng Z et al. Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation. IEEE transactions on cybernetics, 2021

[e] Bodla N, et al. Soft-NMS--improving object detection with one line of code. ICCV 2017

---

Number of proposals

Overall, we consider our approach robust, as we consistently use a maximum of 5 top proposals per region across all datasets and architectures. This hyperparameter ensures a balance between having sufficient information for accurate box merging and avoiding overly large boxes when too many proposals are combined. If there are fewer than 5 proposals for a ground truth target, all available proposals are used; if there are more, only the top 5 are considered. We will clarify this robustness and consistency in the paper.

---

More consistency between datasets used for ablations

We thank the reviewer for raising this concern. To clarify, the majority of our ablation studies are conducted on the COCO dataset for consistency except for Table 4 in the main paper, where we present an ablation on Soft-NMS using the VOC dataset. To address this concern, we have now included the same ablation on the COCO dataset in this rebuttal. Moving forward, we will ensure that results for all ablation studies are presented consistently across datasets, with detailed COCO results included in the supplemental material.

---

Font size in Figure 3

We will increase the font size in Fig. 3 to improve readability in the revised version.

---

Soft NMS in Table 4

Indeed, the difference between Tables 1 and 4 for the Faster R-CNN backbone is due to the use of Soft NMS. We will make sure to include NMS and Cluster-NMS results in Table 4 as well to have an easy overview of this comparison.

---

We hope to have addressed the concerns and demonstrated the robustness, flexibility, and broad applicability of our approach, including its compatibility with IoU-based losses, multiple NMS variants, and diverse datasets. We gladly clarify any remaining questions.

Dear Reviewer JFjT,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission10907

There are still important ablations in Appendix that do not appear in the main paper. The Fig. 4 is too large. I suggest removing half of Fig. 4 to save more room for presenting those experiments. Fig.5 is showing something similar to Fig.4 and can be removed too. I argue that those ablation studies are inspiring that provide more insights of the method.

The followings are minor concerns:

1. There are no related works of bbox regression in Sec.2. I suggest giving a comprehensive introduction to that.

2. All the algorithms engaging in comparison must be appears in Sec.2 Related Works. Please check them carefully.

I increase my rating score to 7.

Applying union-over-intersection to full ground truth predictions

Following the reviewer suggestion, we apply the Union of Proposals while maintaining the original regression objective (Full-object regression), where the model predicts full ground truth bounding boxes instead of part-based regions. As shown in the table below, this approach produces looser bounding boxes after combining proposals, even with a second-stage regression, resulting in lower AP scores. This confirms the necessity of our part-based regression strategy (UoI), which achieves finer localization and better overall accuracy by focusing on parts of the objects. We will include this analysis in the supplemental material.

---

We hope to have clarified the novelty of our method, its distinction from IoU-based approaches and traditional voting mechanisms, and its adaptability to various architectures, including proposal-free methods. We gladly clarify any remaining questions.

We thank the reviewer for thoughtful feedback and the opportunity for reconsideration.

Difference between an Iou-loss regression objective and our approach

We clarify that the key novelty of our method lies in its redefinition of how object detection is approached. Unlike IoU-based methods that focus on optimizing bounding box accuracy through new loss functions, our approach shifts the focus to learning part-wise representations of an object. By fostering collaboration among these parts, we unify and reconstruct the final object, offering a fundamentally different perspective on object detection.

In fact, our framework’s flexibility allows it to incorporate any regression loss function or grouping strategy and integrate seamlessly with various detection architectures. To showcase this, we conducted an additional experiment where we replaced the L1 loss in our method with IoU-based losses like GIoU [a], DIoU [b], and Alpha-IoU [c]. The results, shown in the table below, confirm consistent improvements, demonstrating that our approach not only works with these advances but also enhances their performance. We will include this experiment and discussion in the supplemental. Thank you.

[a] Rezatofighi H, et al. Generalized intersection over union: A metric and a loss for bounding box regression. CVPR 2019.

[b] Zheng Z, et al. Distance-IoU loss: Faster and better learning for bounding box regression. AAAI 2020.

[c] He J, et al. $\alpha$-IoU: A family of power intersection over union losses for bounding box regression. NeurIPS 2021.

---

Box voting

We acknowledge that "box voting for accuracy improvement" is an established and extensively validated method, that is why we use it, but to group and merge proposals, instead of selecting only the top one. This is where we are different. We agree that learning the complete ground truth will facilitate in selecting the best candidate. We note that we do not select the best candidate. We merge information from the top-k best candidates, thus making our combined candidate better than the top-1 voted best candidate as done in the existing literature. We will better stress this difference in the introduction.

---

Use beyond proposal-based methods

We clarify our method is not limited to traditional proposals and can be applied to methods with regions, grids, or queries, as demonstrated in the main paper (Tables 1 and 2). We have already shown its integration with grid-based YOLO and query-based Deformable DETR, highlighting its flexibility. The concept of "proposals" is used loosely to includede regions and anchors, which are common across most detection frameworks, making our approach broadly applicable with minimal modifications.

Details of modifications: For YOLO, we modify its object-to-grid assignment strategy by using an IoU-based criterion instead of the traditional center-based approach, assigning objects to multiple grid cells if they overlap sufficiently. Each anchor is tasked with regressing the part of the object it overlaps with best, and during inference, our Intersection-based Grouping merges these predictions. For Deformable DETR, we treat queries as proposals and redefine the matching process to focus on object parts, assigning queries to specific regions of the ground truth and combining them during inference.

While most methods, including YOLO and Deformable DETR, can incorporate our Union-over-Intersection with minor modifications, unique architectures like FCOS, which rely on keypoints rather than rectangular anchors, may require additional adaptations. We will clarify the adaptability of our approach in the main paper.

---

<CONTINUED>

Dear Reviewer fZZc,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission10907

Confidence scores in methods description

The reviewer is correct that assigning new confidence scores might improve $mAP$. To evaluate this, we added a classification head for the combined boxes for the FRCNN baseline, resulting in marginal improvements in $mAP$ and $AP_{75}$, as shown in the table below. Since these gains are minimal and our primary improvements come from the localization, (Table 5 of the main paper), we chose to omit the additional classification stage in the main method. We will add the ablation to the supplemental.

---

Same number of predictions as baselines

We confirm the total number of final predictions fed to the $mAP$ evaluator for the baseline and our approach are the same, as decided by the number of regions formed during the grouping stage.

---

Extra regression head

Yes, the regression function $r$ is a learnable regression head. Our method includes an additional regression stage after grouping to refine the combined boxes into tighter-fitting bounding boxes. For baseline methods, adding a second regression head does not yield significant benefits, as both heads perform the same task—predicting the full object bounding box. In contrast, our approach assigns distinct roles: the first regression head focuses on parts of the object, while the second refines the union of these parts into an accurate bounding box. This makes the additional regression effective, as shown in the table below. We will add this ablation to the supplemental material.

---

Adaptability to other methods

We clarify our method is not limited to traditional proposals and can be applied to methods with regions, grids, or queries, as demonstrated in the main paper (Tables 1 and 2). We have already shown its integration with grid-based YOLO and query-based Deformable DETR, highlighting its flexibility. The concept of "proposals" is used loosely to include regions and anchors, which are common across most detection frameworks, making our approach broadly applicable with minimal modifications.

Details of modifications: For YOLO, we modify its object-to-grid assignment strategy by using an IoU-based criterion instead of the traditional center-based approach, assigning objects to multiple grid cells if they overlap sufficiently. Each anchor is tasked with regressing the part of the object it overlaps with best, and during inference, our Intersection-based Grouping merges these predictions. For Deformable DETR, we treat queries as proposals and redefine the matching process to focus on object parts, assigning queries to specific regions of the ground truth and combining them during inference.

While most methods, including YOLO and Deformable DETR, can incorporate our Union-over-Intersection with minor modifications, unique architectures like FCOS, which rely on keypoints rather than rectangular anchors, may require additional adaptations. We will clarify the adaptability of our approach in the main paper.

---

We hope our responses have clarified concerns regarding object sizes, dynamic regression targets, and integration with various detection architectures. Additionally, we have addressed aspects like confidence scoring and compatibility with non-box-proposal methods, demonstrating the flexibility and robustness of our plug-and-play design. We are happy to provide further clarifications if needed.

We thank the reviewer for their appreciation and the opportunity to clarify our intuitions, methodological choices and expand our experimental justification.

True alternative to original paradigm

To further validate the flexibility of our framework, we performed additional experiments showing that modifying individual components, such as the regression loss and the grouping strategy, still preserves the efficacy of our approach. For example, replacing the L1 loss with IoU-based losses GIoU [a], DIoU [b] and Alpha-IoU [c] enhances performance, as shown in the table below:

Similarly, our grouping mechanism remains effective across NMS variants like Cluster-NMS [d] and Soft-NMS [e], as demonstrated in the following table:

We hope these results and analyses convincingly justify the viability and robustness of our approach as a true alternative to the conventional box refinement paradigm.

[a] Rezatofighi H, et al. Generalized intersection over union: A metric and a loss for bounding box regression. CVPR 2019.

[b] Zheng Z, et al. Distance-IoU loss: Faster and better learning for bounding box regression. AAAI 2020.

[c] He J, et al. $\alpha$-IoU: A family of power intersection over union losses for bounding box regression. NeurIPS 2021.

[d] Zheng Z et al. Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation. IEEE transactions on cybernetics, 2021

[e] Bodla N, et al. Soft-NMS--improving object detection with one line of code. ICCV 2017

---

Performance analysis on different object sizes

We thank the reviewer for sharing the insight. Our approach indeed profits from having multiple object proposals available. For the COCO dataset, the average number of proposals is 12, 8 and 3 for large, medium and small objects, respectively. Small objects typically have fewer proposals, and in cases with only one proposal, our method defaults to the baseline (see similar comment by Reviewer zZMj). To analyze this further, we grouped small objects based on the number of proposals they received (e.g., 1, 3, or 5 proposals) and calculated the corresponding AP scores for each group. The table below shows that AP_small improves as the number of proposals increases, highlighting that our method effectively utilizes multiple proposals to enhance performance for small objects. We will include a discussion on very small objects and the anticipated effects in the main paper.

---

Stability in light of dynamic target allocation

The reviewer correctly observes that our regression targets are dynamically assigned to parts of objects based on proposals, unlike the fixed object-level targets in the baseline. To analyze the stability of this assignment, we examined both training and testing stages. During training, our regression loss—associated with these dynamic targets—is approximately half of the baseline regression loss, indicating improved stability (as shown in Figure 3(b) of the main paper). During testing, Table 5 demonstrates that our method achieves a $10.7\%$ higher localization accuracy than the baseline ($64.4\%$ vs. $53.7\%$), ensuring proper localization capability to dynamic targets. We will update the text in these ablations in the main paper to explicitly reflect the stability of our dynamic target assignment, ensuring this concern is fully addressed.

---

<CONTINUED>

Dear Reviewer GWg2,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission10907

Thanks for the feedback from the authors. I'm still concern on the adaptability.

For Deformable DETR, we treat queries as proposals and redefine the matching process to focus on object parts, assigning queries to specific regions of the ground truth and combining them during inference.

So the query need to be box proposal? So this is DAB-Deformable-DETR, right?

What about FCOS? I have not found the description in the revised manuscript.

What I still care about is: if a method like Deformable DETR (require modifying query designing like DAB-DETR) and FCOS (point based) need so many designing modification, the description

Our plug-and-play method integrates seamlessly into any detection architecture with minimal modifications, significantly improving object localization and instance segmentation.

is not sound.

Effects on tiny objects

Thank you for sharing the insight on very tiny objects. Indeed, our approach profits from having multiple object proposals available. For the COCO dataset, the average number of proposals is 12, 8 and 3 for large, medium and small objects respectively.

Small objects typically have fewer proposals, and in cases with only one proposal, our method defaults to the baseline (see similar comment by Reviewer GWg2). On COCO the small objects still have an average of three proposals, enabling our Union-over-Intersection to unify their information and contribute to the reported improvements. For even smaller objects (<10×10 pixels), the method's performance will depend on the base network and the number of intersecting proposals. Our approach is never worse than the baseline and will demonstrate an improvement when there is more than 1 proposal available. We will include a discussion on very small objects and the anticipated effect on our approach in the main paper.

---

Effect of varying IoU threshold

We thank the reviewer for this observation. The number of proposals is influenced by the IoU threshold (k) used for grouping in the UoI strategy. We set this threshold to 0.5, consistent with standard NMS. Lowering it (e.g., 0.3) increases group sizes, while raising it (e.g., 0.7) reduces them. As shown in the table below for FRCNN with a Resnet-101 backbone, the optimal performance (mAP) shifts to 4 proposals for a 0.7 threshold but remains at 5 for a 0.3 threshold. However, optimal mAP values for both 0.3 and 0.7 are lower than 0.5, reaffirming the effectiveness of the default setting. We will add this ablation to the supplemental material.

---

Crowded Scenes impact

We thank the reviewer for this insightful observation. Our limitation in crowded scenes specifically concerns multiple instances of the same class. Instances of different classes are naturally separated during grouping. However, for same-class objects positioned closely, candidate boxes may merge into a single large box, as shown in Figure 4's "3 birds" example.

Regarding object size, grouping is scale-invariant as it relies solely on IoU overlap. The challenge arises in the refinement stage after grouping, where regressing to the nearest object performs better for distinct, larger objects. Hence, this limitation is most evident with small, closely spaced objects of the same class, while larger or less crowded objects are less affected. We will add this discussion to the main paper.

---

We hope our responses adequately address the concerns raised, particularly around the generalization of our method across datasets, architectures, and object sizes, as well as its robustness in crowded scenes and various IoU threshold settings. We are happy to provide further clarifications if needed.

We thank the reviewer for their encouragement and the suggestions for further improvement.

Additional comparisons per table

Following the reviewer suggestion, we present an updated version of Table 1, which now includes results for Mask R-CNN, Cascade R-CNN, and YOLOv3 on the PASCAL VOC dataset, in addition to Faster R-CNN and Deformable DETR. In all cases, adding our Union-over-Intersections improves detection results, with stronger improvements for higher overlap thresholds. We will put this table in the supplemental material.

---

Additional architectures for ablations

As requested, we provide additional results for the localization effectiveness of UoI on Faster R-CNN and Cascade R-CNN. The first table evaluates classification accuracy and localization mIoU (Table 5 of main paper), while the second examines LRP metrics (Table 6 of main paper), demonstrating that main improvement for results with the proposed Union-over-Intersections (UoI) indeed comes from localization, irrespective of the base architecture.

We are running additional experiments to validate the UoI strategy on YOLO v3 and Deformable DETR for all datasets for results on Tables 4-7 and will include these results in the supplemental material.

---

Aesthetic consistency of the paper

Thank you for highlighting the formatting inconsistencies. We have aligned paragraph titles with their content, standardized title casing, and ensured consistent formatting across figures and captions in the updated version of the paper.

---

Content related queries

1. By the regression function $r: B \to R$, we mean that the combined boxes $B$ are refined into final regressed boxes $R$, which are compared to the ground truth boxes, denoted as $G$ or $T$, using the loss function: $L_{\text{refinement}} = \sum_{j=1}^{m} \sum_{i \in \{x, y, w, h\}} |r(B_{ji}) - T_{ji}|$. In this context, $R$ is not intended to represent the ground truth but rather the set of final predictions obtained after refinement. We will revise the wording in the main manuscript for improved clarity.

2. We confirm that it is not necessary to include "P ← P \ p_i" in the conditional statement within the red dashed box. The red dashed box represents standard NMS, which groups based on the predicted bounding boxes $B$, not the proposals $P$. Our approach instead uses the proposals $P$—the boxes before regression—for grouping. Since the red dashed box exclusively operates on $B$, including $P$ there is unnecessary.

---

Conditional statement in pseudocode

The use of "iou(H, p_i) >= k" in the green solid box reflects a key distinction in our method: while standard NMS does grouping based on regressed boxes $B$, our approach uses original proposals $P$ for grouping. This is because our regression stage outputs intersections, not full bounding boxes. Grouping by IoU between intersections $I$ (as in "iou(M, b_i) >= k") is ineffective, as intersections often lack sufficient overlap. We tested this alternative and confirmed it performed poorly: grouping by regressed intersections $I$ resulted in 33.6 $mAP$, while grouping by proposals $P$ achieved 38.1 $mAP$. Grouping by proposals $P$, which better represent object regions, is more effective for our part-based approach. We will clarify this in the main paper.

---

<CONTINUED>