Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation

We sincerely appreciate reviewer ARZw's thorough review and highly positive assessment of our work. Below, we address each point specifically.

Q1: More discussion on limitations and future works.

A1: We appreciate this valuable suggestion and will expand our discussion on limitations and future works in Appendix Sec D in the revised version. The new additions will be marked in italics:

Limitation Analysis:

…, where encountering previously unseen objects or relations is common. In addition, since the current One-to-Many Relation Assignment rules are manually designed, they may lack flexibility and potentially introduce some noise to the training process.

Future Work:

… This advancement would allow Hydra-SGG to recognize and predict objects and relationships beyond the predefined closed set of categories used in training.

As mentioned in the limitations, since the current One-to-Many Relation Assignment rules are manually designed, they may introduce noise during training that affects learning. Therefore, another direction for future work could focus on designing better one-to-many strategies to further improve performance. For instance, we could develop a feature composition module to generate new relation triplets, or we can incorporate a relationship refine module to transfer coarse-grained relations into fine-grained relations.

Q2:Strategies to manage noises introduced by One-to-Many Relation Assignment?

A2: Thank you for this important question. Our One-to-Many Relation Assignment currently relies on heuristic rules (i.e., IoU-based), which may introduce noisy triplets. We propose two potential directions for future improvement:

1. Generate new triplets by feature composition :

   Feature composition methods [ref1] can create new relation triplets at a feature level, which can be integrated into our One-to-Many Relation Assignment.

2. Refine the triplets:

   We can develop a relation transfer mechanism [ref2] that focuses on refining general relations into more specific ones (e.g., "people-on-bicycle" → "people-riding-bicycle") and incorporate fine-grained triplets in One-to-Many Relation Assignment.

Q3: How do different loss weights between One-to-One and One-to-Many affect performance?

A3: We sincerely appreciate your important suggestion. We conduct an ablation study on different loss weights between One-to-One and One-to-Many assignments. As shown in the table, setting the loss ratio to 1:1 achieves the best performance (mR@50: 16.0, mR@100: 19.7). When One-to-Many loss weight is too small (e.g., 1:0.5), the model underutilizes additional positive samples. Conversely, when the weight is too large (e.g., 1:3), it may overemphasize One-to-Many triplets, which have a relatively lower quality compared to the One-to-One triplets. In One-to-One assignment, each relation label is only matched with its optimal query, while in One-to-Many assignment, a relation label is matched with multiple queries that satisfy certain criteria. As a result, the overall quality of One-to-Many triplets tends to be lower than One-to-One triplets, since the latter only contains the most optimal matches. We add analysis in Table 4 in the revised manuscript.

Ref:

[ref1] Compositional Feature Augmentation for Unbiased Scene Graph Generation. ICCV 2023.

[ref2] Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022.

Thank you for your sincere response. All my questions have been resolved, and I also reviewed feedback from other reviewers that I had overlooked. However, I think you have addressed it well, so I will keep my score.

Dear reviewer ARZw,

We appreciate again your thoughtful review and help us further improve the quality of our work! Please let us know if any other concerns.

Best regards.

Authors.

Q3: Does faster convergence come from more diverse samples or more training data within an epoch?

A3: From our perspective, the faster convergence comes from a “better utilization” of training images. Firstly, our training process is exactly the same as that of previous works (same batch size, total training images), and does NOT generate more training images within an epoch. Secondly, compared to other Hungarian matching-based SGG methods [ref6, ref7], our framework generates more positive samples by mining false negative samples. The total number of training samples remains unchanged. The performance gains are from better, high-quality training samples in each optimization step. Therefore, the comparison is fair and our improvements are credible.

Q4: Adding comparisons with unbiased SGG methods or methods that adopt unbiasing strategies.

A4: We had already included comparisons between Hydra-SGG and methods that adopt debiasing strategies, such as CFA [ref11], in Table 3 of our submission. Debiasing techniques represent an orthogonal research direction to our work. They specifically address data imbalance issues and typically improve mRecall scores.

As requested, we add additional comparisons with debiasing methods including TDE [ref4], NICE [ref7], IETrans [ref8], CFA [ref9], and VETO [ref10], NICEST [ref11] in the table below.

As seen, Hydra-SGG achieves very competitive performance compared to these specialized debiasing methods. Note that, our method does not incorporate any debiasing strategies, therefore this comparison is unfair. We add analysis in Table 4 in the revised manuscript.

Ref:

[ref1] Faster R-CNN: Towards Real-Time Object. NeurIPS 2015.

[ref2] Training Region-based Object Detectors with Online Hard Example Mining. CVPR2016.

[ref3] Focal Loss for Dense Object Detection. ICCV 2017.

[ref4] Unbiased scene graph generation from biased training.

[ref5] SGTR: End-to-End Scene Graph Generation With Transformer. CVPR 2022.

[ref6] RelTR: Relation Transformer for Scene Graph Generation. TPAMI 2023.

[ref7] The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation. CVPR 2022

[ref8] Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022.

[ref9] Compositional Feature Augmentation for Unbiased Scene Graph Generation. ICCV 2023.

[ref10] Vision Relation Transformer for Unbiased Scene Graph Generation. ICCV 2023.

[ref11] Noisy label correction and training for robust scene graph generation. TPAMI 2024.

We thank reviewer KZeA for the valuable time and thoughtful feedback. We provide a point-to-point response below.

Q1: Lacking theory/analysis, technical solutions appear to be intuitive and empirical.

A1: Respectfully disagree. Our work begins by identifying and analyzing a critical issue in DETR-based SGG models: the slow convergence caused by the One-to-One constraint, which results in sparse relation supervision, providing only 5.5 positive samples (L 077) per image on average for model training. This bottleneck significantly limits the model’s ability to learn effectively, highlighting a key challenge in current frameworks.

Based on this analysis, we developed our solution by drawing inspiration from well-established principles in the community, where increasing positive training samples and reducing false negatives are widely used and shown to be effective [ref1, ref2, ref3]. We are the first to introduce such concepts to one-stage DETR-based SGG and explore several variations (L 066) from scratch. While our work is not purely theory-oriented, it follows a systematic approach: from problem identification, to solution design guided by established principles and empirical validation. The effectiveness of our approach is demonstrated by achieving SOTA results across multiple SGG benchmarks with only 12 epochs of training. Given such fresh insights and the substantial improvements demonstrated, this study deserves to be shared with our community.

Q2: Limited exploration of how HydraBranch enhances the proposed One-to-Many assignment.

A2: We first clarify the analyses and explorations that have been done on how HydraBranch enhances the proposed One-to-Many assignment.

Quantitative analysis on the optimized training signals (L 270), We analyze the average number of positive samples in VG150 train and val for both One-to-One and Hybrid Relation Assignment, as illustrated in Fig. 4 (a). In addition, Fig. 4 (b) shows the percentage increase in positive samples achieved by the Hybrid Relation Assignment compared to the One-to-One baseline (Sec. 3.3).

Quantitative analysis on removing self-attention layers (L 270, L 285). We introduce a Diversity Score (DS) (Sec. 3.3) to quantify the presence and absence of the self-attention layer on the diversity of relation predictions. As shown in Fig. 4 (c), our experiments demonstrated that the presence of the self-attention layer significantly increases the diversity of predicted relations, as evidenced by higher ADS (Average Diversity Score) values on VG150.

Qualitative analysis on removing self-attention layers (L 270). We included visual examples (Fig.4 (d) (e)) illustrating how the self-attention layer affects relation predictions. These examples clearly demonstrate that without self-attention, queries tend to converge on the same relation.

The above analyses showcase our efforts in designing HydraBranch and validating Hybrid Relation Assignment through both quantitative and qualitative perspectives. Our One-to-Many assignment strategy is designed to assign a single GT relation to multiple queries. To enhance this strategy, we intentionally designed a module to further encourage queries to predict duplicated relations. Through empirical analysis (L 285-L 349), we found that removing self-attention in the decoder effectively serves this purpose by making queries more similar to each other, thus creating a natural synergy with our One-to-Many assignment approach.

However, we acknowledge that a deeper investigation would be valuable. To better explore how HydraBranch enhances the proposed One-to-Many assignment, we conducted additional statistical analysis on query embeddings. We analyzed the Euclidean distances between query pairs in each image. With 300 queries per image, we computed 300*299/2 pairwise distances. We conducted this analysis on 5,000 images from the VG150 val, comparing scenarios with and without self-attention layers. By employing a paired t-test and calculating Cohen's d, we ensure that our findings are statistically robust and meaningful.

T-statistic: 54.502

P-value: < 1e-3

Overall mean distance:

• With SA: 7.55
• Without SA: 7.23 (-5%)

Cohen's d: 0.771

The large T-statistic (54.502) and extremely small p-value (1e-3) indicate that the difference between the two conditions is highly statistically significant. In addition, Cohen's d = 0.771, approaching the threshold for a large effect (0.8). These statistical results demonstrate that HydraBranch, by removing self-attention, effectively reduces query distances and encourages duplicated predictions, thereby enhancing our One-to-Many assignment strategy as intended in our design. We add analysis in Sec. 3.4 in the revised manuscript.

We thank reviewer o4Ve for the valuable time and thoughtful feedback. We provide a point-to-point response below.

Q1: This work lacks novelty, especially compared to SpeaQ [ref11].

A1: Sorry for this confusion. First of all, we want to clarify that one-to-many assignment is a fundamental concept that has been extensively applied in the community [ref1-ref10] for decades. SpeaQ [ref11] was NOT the first to propose and apply one-to-many assignment. Compared to SpeaQ [ref11], our motivation (L 90) (addressing slow convergence caused by sparse supervision vs addressing the problem of unspecialized relation queries), and core techniques (L 285) (mining false negative samples to increase positive samples for each training step vs dividing queries and relations into disjoint groups to train specialized queries for specific relations), implementation details (L 315) (architectural design and training framework vs query grouping and assignment strategies) are DIFFERENT.

Furthermore, Hydra-SGG differs from other existing one-stage DETR-based SGG models. Existing DETR-based SGG methods focus on designing queries [ref12, ref15], architectures [ref13], lightweight frameworks [ref14], etc. We emphasize that Hydra-SGG's motivation is to address the slow convergence (L 090) problem in one-stage DETR-based SGG models. Methodologically, Hydra-SGG approaches this through a hybrid query-label assignment and architectural design (L 237 and L285). Hydra-SGG significantly improves the training efficacy of one-stage SGG models, achieving SOTA performance on three common SGG datasets with only 12 epochs (VG150 16.0 mR@50, Open Images V6 50.1 weighted score, and GQA 12.7 mR@50). Given such fresh insights and the substantial improvements demonstrated, this study deserves to be shared with our community.

Q2: Removing self-attention among relation queries does not constitute a significant advancement in one-stage SGG research.

A2: Sorry for the misunderstanding. Our main contribution is proposing a comprehensive framework that addresses the slow convergence problem in one-stage DETR-based SGG models (L 100). Concretely, we propose a Hybrid Relation Assignment to increase relation supervision signals and HydraBranch, a decoder specifically designed to synergize with our proposed assignment strategy. Removing self-attention in HydraBranch is intentionally designed to encourage different queries to predict duplicated relations (L 288), enhancing our relation assignment strategy. This design serves specifically our Hybrid Relation Assignment strategy rather than being a general advancement for one-stage SGG research.

Q3: Unexpected Improvement in mRecall.

A3: We speculate the improvement in mRecall can be attributed to two main aspects:

1. Shorter training epochs reduce overfitting: Our model trains for only 12 epochs, compared to SpeaQ's 52 epochs. This shorter training period mitigates the risk of overfitting to high-frequency (common) relations. Furthermore, the relatively higher mRecall and lower overall Recall demonstrate that our model does not overfit to frequent ones.

2. Our One-to-Many assignment enhances training signals for rare relations:

While One-to-Many Assignment increases the number of queries for both rare and common relations, it avoids overemphasizing common relationships. As mentioned in line 259, we assign a fixed 6 queries per relation category. For example, in an image with 10 "on" relations and 2 "sit" relations:

• One-to-One Assignment would allocate 10 queries to "on" and only 2 queries to "sit".
• Our One-to-Many Assignment assigns 6 queries for both "on" (a 60% increase) and 6 for "sit" (a 300% increase).

Therefore, One-to-Many assignment enhances training signals for rare relations by providing them with a larger relative gain, while avoiding overemphasis on common relations.

Q4: Inadequate model performance and missing F-Recall results.

A4: As requested, we have added F-Recall results and comparisons with SSR-CNN [ref16]. Hydra-SGG achieves SOTA F-Recall with substantially fewer parameters (67.6M). This suggests that Hydra-SGG achieved the highest overall performance. We add comparisons in Table 1 in the revised manuscript.

Ref:

[ref1] Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR 2014.

[ref2] Fast R-CNN. ICCV2015.

[ref3] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NeurIPS2015.

[ref4] You Only Look Once: Unified, Real-Time Object Detection. CVPR2017.

[ref5] Mask R-CNN. ICCV2017.

[ref6] Focal Loss for Dense Object Detection. ICCV2017.

[ref7] OTA: Optimal Transport Assignment for Object Detection. CVPR2021.

[ref8] YOLOX: Exceeding YOLO Series in 2021. arxiv 2021.

[ref9] Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment. ICCV2023.

[ref10] YOLOv10: Real-Time End-to-End Object Detection. arxiv 2024.

[ref11] Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection. CVPR 2024.

[ref12] SGTR: End-to-End Scene Graph Generation With Transformer. CVPR 2022.

[ref13] RelTR: Relation Transformer for Scene Graph Generation. TPAMI 2023.

[ref14] EGTR: Extracting Graph from Transformer for Scene Graph Generation. CVPR 2024.

[ref15] DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation. CVPR 2024.

[ref16] Structured Sparse R-CNN for Direct Scene Graph Generation. CVPR 2022.

[ref17] Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022.

Dear reviewer o4Ve,

First, we want to thank you again for reviewing our work. We have provided point-to-point responses to your comments. However, we have not received any feedback yet since the open discussion phase. We appreciate your new input based on our responses.

Thank you!

Authors.

Dear Reviewer o4Ve,

This is a gentle reminder that today (December 2nd, Anywhere on Earth) marks the deadline for review feedback. We are available to address any questions or concerns until tomorrow. We have addressed your comments in detail, added new evaluations (F-Recall), and more comparisons, and made significant updates. Please don't hesitate to reach out if you need any clarification.

Best regards,

Authors