We thank Reviewer S4eL for the valuable time and constructive feedback. We address the raised concerns as follows:

1. Clarification of Experimental Settings (Tab. 1 & Tab. 2): All experiments follow the standard SGDET protocol [34,30], which jointly evaluates object detection and relationship prediction without pre‑defined boxes. Traditional task‑specific SGG models output triplets with confidence scores, so R@K and mR@K (K=20/50/100) are computed by ranking predictions by score and taking the top‑k triplets. However, MLLM generates triplets without scores, making top‑k ranking meaningless. Following prior works using MLLMs for SGG (e.g., LLaVA‑SpaceSGG [35]), we directly report R@20 and mR@20 as the main recall metrics. We will clarify this in the next version.

2. Contributions Beyond “Training Qwen2‑VL with RL”: Our contributions go beyond applying GRPO [28] to Qwen2‑VL [32]:

   • Graph‑centric reward design. We introduce three rule‑based reward variants (Hard Recall, Hard Recall+Relax, Soft Recall) that explicitly align with SGG evaluation metrics. These rewards are novel for structured vision tasks and enable fine‑grained node/edge‑level feedback.
   • Comprehensive analysis of reward sparsity and cross‑dataset generalization: We systematically study the impact of reward alignment (Tab. 4) and show that RL enhances domain transferability and tail‑class recall (Fig. 3 & 4).
   • A unified end‑to‑end MLLM framework (R1‑SGG): To our knowledge, this is the first work to integrate visual instruction tuning and reinforcement learning for end‑to‑end SGG with MLLMs, going beyond caption‑based or text‑only LLM methods.

3. Comparison with 2024 SOTA: We already include results of OvSGTR [ECCV 2024], ASMv2 [ECCV 2024], and LLaVA‑SpaceSGG [WACV 2025] in Tab. 1 and Tab. 2. These methods represent recent SOTA approaches for open‑vocabulary or instruction‑tuned SGG.

4. Inconsistent results (why validation set?): The VG150 test set has ~26k images. VS3 [CVPR 2023] and OvSGTR [ECCV 2024] retain ~15k images that were not used in GLIP pre‑training, but running inference on the remaining 15k images is still extremely expensive for large MLLMs. To reduce computational burden while ensuring fair comparison, we report results on the 5k‑image validation set. Importantly, all models in Tab. 1 are evaluated on the same validation split, which is i.i.d. to the test set, so the comparison remains fair. To demonstrate this, we report the performance on the VG150 test set as follows. | Split | Model | Failure Rate| AP@50 | Recall | mRecall | |:------|:-------------|------:|------:|-------:|--------:| | Test | IMP | - | – | 17.7 | 2.7 | | Test | MOTIFS | -| – | 25.5 | 5.0 | | Test | OvSGTR |-| 36.8 | 27.8 | 5.2 | | Test | R1‑SGG (7B) | 0.08%| 20.8 | 27.5 | 11.2 | | Val. | IMP | -|20.9 | 17.9 | 2.7 | | Val. | MOTIFS | -|29.6 | 27.2 | 7.8 | | Val. | OvSGTR | -| 33.4 | 26.7 | 5.8 | | Val. | R1‑SGG (7B) | 0.08%| 19.5 | 23.8 | 11.4 |

For results like PSGFormer in Table 2, we follow previous work LLaVA‑SpaceSGG [WACV 2025] which reports the R@20 as the recall and mR@20 as the mRecall. The reason is explained in the Clarification of Experimental Settings (Tab. 1 & Tab. 2) section.

5. Parameter Gap: It is obvious that MLLMs have more parameters than non-LLM models. However, the goal of R1‑SGG is to study whether RL can enhance structured reasoning in MLLMs, not to directly compete with lightweight SGG detectors.

6. Missing Related Works: We thank the reviewer for pointing this out. We will cite and discuss VLPrompt (arXiv:2311.16492) and Julian Lorenz et al. (ECCV 2024)

7. Discussion of Limitations: As discussed in To Think or Not Think? (Sec. 4.5), RL alone is not sufficient to trigger abstract reasoning patterns such as CoT. Additional SFT on CoT‑specific data may further enhance reasoning.

We thank Reviewer nLRL for the valuable time and constructive feedback. We address the raised concerns as follows:

1. Necessity of RL: First, it is reasonable that task‑specific models with dense object detectors achieve higher AP and recall. However, our goal is to develop an effective MLLM for SGG, not to outperform specialized SGG models. Second, SFT cannot directly optimize recall‑based SGG metrics, as all tokens are equally weighted in cross‑entropy loss. Our RL stage introduces graph‑centric rewards aligned with SGDET metrics, yielding large gains (e.g., Recall 5.47% → 21.09%, failure rate 72.42% → 0.10%, Table 1). RL is therefore essential for enabling MLLMs to generate accurate, structurally valid scene graphs in an end‑to‑end way.

2. Format Failure Rate: We report format failure rate not as the main contribution but as a supporting result. MLLMs often fail to follow structured output formats, leading to unusable scene graphs. RL greatly reduces such failures (e.g., 72.42% → 0.10%, Table 1), which is critical for practical deployment of end‑to‑end SGG.

3. Novelty: Beyond using GRPO, we design and study a series of scene‑graph‑specific rewards (Hard Recall, Hard Recall+Relax, Soft Recall, and format consistency). We analyze their sparsity and alignment with SGDET metrics (Table 4). Our framework is the first to achieve end‑to‑end SGG with an MLLM using RL, demonstrating substantial gains over SFT and zero‑shot MLLMs.

4. Domain Generalization: Table 3 shows that R1‑SGG‑Zero (RL‑only) generalizes better across datasets (e.g., Recall 13.83% vs. 3.03% for SFT when trained on VG150 and tested on PSG). Furthermore, Figures 3 and 4 show per‑predicate recall, where R1‑SGG achieves higher recall, especially on tail classes. This explains why R1‑SGG obtains higher mean recall than all baselines—it is more effective at recognizing rare predicates.

5. Baselines like GPT‑4o: Commercial MLLMs are included to illustrate that even strong models fail in zero‑shot SGG, motivating the need for SFT + RL. Their low AP and recall stem from limited grounding ability—we have no control over how they process images or map predicted bounding boxes to correct locations. This highlights the necessity of training an MLLM specifically for SGG.

6. Discussion of Limitations: As discussed in To Think or Not Think? (Sec. 4.5), RL alone is not sufficient to trigger abstract reasoning patterns such as CoT. Additional SFT on CoT‑specific data may further enhance reasoning.

Thank the authors for the rebuttal. However, there are no additional experiments to support the authors' claims about domain generalization, and the evaluation in the current manuscripts may not be comprehensive enough, which is also mentioned by other reviewers. I would recommend that the authors polish the experiment design to make the claim of this paper more solid.

We thank Reviewer wFe2 for the valuable time and constructive feedback. We address the raised concerns as follows:

1. Insufficient evaluation metric: We follow the standard SGDET protocol, which jointly evaluates object detection and relationship prediction. SGDET is the most challenging SGG setting, as it requires the model to predict object categories, bounding boxes, and relationships simultaneously, unlike PredCls (predicting only relationships given ground-truth boxes and labels) or SGCls (predicting relationships and object labels given ground-truth boxes). Since our goal is to achieve end-to-end SGG, SGDET is the most suitable evaluation protocol. The evaluation cases the reviewer mentions (e.g., SGCls) are less practical for MLLMs, as they require explicit prompts with ground‑truth bounding boxes.

2. Soft vs. Hard Recall (Tab. 4): As shown in Table 4, the failure rates are nearly identical for Hard Recall and Soft Recall (0.08 % vs. 0.06 %). However, the ablation study demonstrates that aligning the reward function with the evaluation metric (Hard Recall) leads to higher mRecall (11.43 % vs. 5.61 %). This confirms that although Soft Recall provides denser reward signals, metric‑aligned rewards are more effective for improving final evaluation performance.

3. Format following and <think> tokens: Our method includes a format reward (Sec. 3.3.1) encouraging adherence to the <think>...</think><answer>...</answer> structure. However, Qwen2‑VL‑2B/7B still struggles to produce CoT consistently (Sec. 4.5). We agree that larger models (e.g., Qwen 32B) could further improve format following; we will mention this as a limitation and a promising direction.

4. Object‑detection capability and performance gap to traditional SGG: We acknowledge that the current AP@50 is lower than that of detector‑based methods. However, R1‑SGG achieves substantial improvements in AP@50, Recall, and mRecall over both SFT and zero‑shot MLLMs (Tables 1–2) and effectively reduces bias on tail predicates (Figs. 3–4). These gains result from enhancements in both object detection and relation reasoning. For example, with qwen2vl‑2b + SFT, AP@50 increases from 36.05 to 39.28, and Recall rises from 22.06 to 38.49 after RL fine‑tuning.

5. Report performance: Following prior work such as LLaVA‑SpaceSGG [35], we report R/mR@20 as Recall/mRecall for non‑LLM baselines. For MLLMs, however, the models generate unordered triplet sets without confidence scores, making top‑k ranking (R/mR@K) ill‑defined. Therefore, we adopt Recall and mean Recall without top‑k filtering to provide a fair and consistent evaluation across all MLLM settings. We will clarify this in next version.

6. Chain‑of‑Thought (CoT) with RL: As discussed in Sec. 4.5, current models seldom produce <think> content even with the format reward. We plan to explore CoT‑specific SFT before RL as a potential way to enhance structured reasoning, and we will clarify this as a future direction. However, as highlighted in [1], CoT brings the largest benefits in tasks requiring multi‑step reasoning such as math or logic, while the gains are much smaller for other tasks. Since SGG is primarily a perception task (predicting objects, boxes, and relations), the benefits of CoT are expected to be limited compared to multi-step reasoning tasks.

7. why the scene graph task is still worth studying: Scene graphs remain valuable as structured, compositional representations for tasks such as embodied AI, semantic communication, and medical report generation, etc. Saying that the advent of LLMs makes the task meaningless is like claiming that “because LLMs exist, NLP is dead.” It is unreasonable to assume that a task becomes obsolete simply because we now have powerful LLMs.

8. Discussion of Limitations: As discussed in To Think or Not Think? (Sec. 4.5), RL alone is not sufficient to trigger abstract reasoning patterns such as CoT. Additional SFT on CoT‑specific data may further enhance reasoning.

[1] Sprague, Zayne, et al. "To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning." ICLR 2025.

We sincerely thank Reviewer gRpx for the constructive feedback and positive evaluation of our work. We address the raised concerns as follows:

Q1: In SGG, recall is the primary evaluation metric. For triplets that do not exactly match the ground truth, they are naturally excluded from the recall computation and do not contribute to the final score. However, during inference our models can still generate semantically correct but unmatched triplets, such as “<woman.13, wearing, pant.8>” shown in Fig. 6. These triplets are plausible yet absent from the ground truth annotations, which highlights that our approach can capture additional valid relationships beyond the labeled dataset.

Q2: Our method already investigates open-domain capability by evaluating models under different settings in Table 3, including cases without predefined object and relation categories (“w/o cats.”) and cross-dataset generalization (training on VG150 and evaluating on PSG, and vice versa). The results demonstrate that R1-SGG-Zero shows stronger cross-domain transfer, while R1-SGG achieves higher in-domain performance when initialized with SFT.

Q3: We follow the standard SGDET protocol, which jointly evaluates object detection and relationship prediction. SGDET is the most challenging SGG setting, as it requires the model to predict object categories, bounding boxes, and relationships simultaneously, unlike PredCls (predicting only relationships given ground-truth boxes and labels) or SGCls (predicting relationships and object labels given ground-truth boxes). Since our goal is to achieve end-to-end SGG, SGDET is the most suitable evaluation protocol.

Dear Reviewer gRpx,

This is the last call for author-reviewer discussion, which will ends on Aug 8. Could you please read authors' rebuttal and confirm whether your concerns have been addressed asap? Thank you.

Best,

AC