Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation

We deeply appreciate reviewer E7mD for the valuable time and constructive feedback. We provide point-to-point responses below.

Q1: Generalization study of bidirectional prompt training.

Thank you for your valuable suggestion. To validate the effect of our bidirectional prompt on the model's generalization ability, we conduct a direct ablation study at the object detection level. The results in Table_R 12 clearly show that using our bidirectional prompt leads to a significant improvement across all key AP metrics. We will incorporate these results into the revised manuscript.

Table_R 12: Object detection results on the VG dataset.

Q2: Student model with the fully annotated SGG dataset.

Thank you for the helpful comments. We would like to clarify the following points:

• Open-Vocabulary Ability. This capability originates from the model's fundamental vision-language architecture. Unlike a traditional classifier with a fixed output category, our model works by computing a similarity score between visual features and the text embedding of any potential relation. This enables it to perform zero-shot predictions for “novel” classes that were never seen during training.
• Performance of Base Classes. Yes, if fine-tuned on the SGG dataset with only standard SGG losses (w/o KD), the model would indeed learn the “base” relations in the training set very well compared to unseen classes (cf. row 2 in Table_R 13). But this comes at a huge cost: the model would catastrophically forget its general knowledge, leading to a severe collapse in its ability to recognize “novel” classes. This overfitting to base classes is the central challenge in OvSGG, leading directly to why the teacher-student distillation paradigm is not just helpful, but necessary for OvSGG (cf. row 3 in Tabel_R 13).

Table_R 13: Ablation study on fully SGG annotation and ICKD under OvR-SGG setting.

Q3: The necessity of ICKD.

Thank you for this question. We would like to clarify that VRD also follows a typical teacher-student paradigm, which does not operate on a single model in isolation:

• The Teacher Model is from the large-scale pre-training (knowledge infusion). It possesses rich, generalizable knowledge that can be applied to novel categories.
• The Student Model is initialized with the teacher's parameters, and it is then fine-tuned on the fully-annotated SGG dataset (knowledge transfer).

In the open-vocabulary setting, if we were to fine-tune the pre-trained model directly (i.e., w/o a teacher's guidance), it would easily overfit to the “base classes” in the SGG dataset. This would severely degrade or even erase its ability to recognize “novel categories”—this is catastrophic forgetting. The teacher-student distillation framework acts as a regularizer. The teacher guides the student, forcing its feature space to remain consistent with the teacher's general feature space even as it learns the new task. This preserves the invaluable generalization ability. Within this teacher-student framework, we devise ICKD to optimize the knowledge transfer process:

• VRD, ensures the student retains the teacher's understanding of “background” relations (i.e., negative samples), which is fundamental for retaining general knowledge.
• RRD, not only transfers knowledge but also teaches the student the “relative structure” among these background relations. This helps the student build a more structured concept of the background, enabling it to more sharply distinguish true novel interactions (the foreground) from the background.

We have performed an ablation study to provide the definitive proof. As shown in Table_R 13 and Sec. E.4 in the Appendix, when our ICKD is removed, performance on novel relations collapses. This empirically demonstrates that distillation is the critical component that prevents the model from overfitting to base classes, thereby unlocking its true and effective open-vocabulary generalization power.

Dear Reviewer E7mD,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC

We deeply appreciate reviewer 9mXA for the valuable time and constructive feedback. We provide point-to-point responses below.

Q1: Missing mR@K metric.

Our apologies. We have reported both recall and mean Recall in Table_R 7. It can be seen that our ACC outperforms the previous SOTA method (i.e., OvSGTR) in both metrics.

Table_R 7: Experimental results of OvD+R-SGG setting on the VG dataset.

Q2: Standard deviations.

Thanks for your keen observation. We have repeated the main experiments five times, and the results (Table_R 8) demonstrated high robustness across runs — the standard deviations range from 0.01 to 0.03. Due to the consistently low variance, we initially omitted the standard deviations from the main tables for clarity. However, we acknowledge that this information should have been included in the appendix. In the final version, we will add a dedicated table in the appendix, reporting both the mean and standard deviation for all key results.

Table_R 8: The 𝐾-times experiment of ACC in OVD+R SGG setting (Joint Base+Novel).

Q3: Computational overhead.

Good suggestion! We conduct a time analysis on VG, with training on the entire dataset and testing on 20 images. We report the mean value in Table_R 9 of our ACC (w/o and with Step II in IGQS). We would like to claim that: 1) Our Step I in IGQS just introduces minor computational complexity in elementary matrix operations (cf. Eq. 1). 1) Due to the requirement of forward prediction for self-enhancement, Step II will induce extra computational overhead, but the performance gain brought by Step II is optional.

Table_R 9: Inference costs on the VG dataset under the OVD+R SGG setting (Joint Base & Novel).

Q4: More comparisons with recent baselines.

To address the reviewer's concern, we have included a comparison with a newly released method, OwSGG [3] (June 9, 2025), which adopts a multimodal prompting and embedding alignment strategy to enable pre-trained models such as LLaVA-Next and Qwen to generate scene graphs under an open-world setting.

We report its performance on the VG dataset under two evaluation protocols: 1) the Open-vocabulary Relation (OvR) setting, and 2) the Open-vocabulary Detection + Relation (OvD+R) setting. The comparison results are shown below:

Table_R 10a: More experimental comparison results of OvR-SGG setting on VG test set.

Table_R 10b: More experimental comparison results of OvD+R-SGG setting on VG test set.

[3] Dutta, Amartya, et al. Open World Scene Graph Generation using Vision Language Models. CVPR Workshop 2025.

Q5: ACC vs. OvSGTR on out-of-distribution data.

Thank you for your valuable suggestion. We have tested both the OvSGTR and ACC models, trained on the VG dataset, on out-of-distribution datasets including PSG (Table_R 11a) and HICO-DET (Table_R 11b). The results are presented in the tables below:

Table_R 11a: Test on out-of-distribution PSG dataset.

Table_R 11b: Test on out-of-distribution HICO-DET dataset.

As seen, our ACC demonstrates a striking performance advantage, significantly and consistently outperforming OvSGTR. This result provides strong evidence for the superior generalization capability of our approach, proving its effectiveness extends well beyond the original training domain. We will incorporate these compelling new results into the revised manuscript.

Q6: Key notations in Figure 2.

Thanks for your helpful suggestion. Due to the limitation of rebuttal guidelines, we cannot include an external link / figure here. We will add refined Figure 2 with key notations in the revision.

Q7: Typical failure scenarios.

Good suggestion! We analyze the examples where ACC misidentifies non-interacting object pairs, and find that:

1. For Interaction-Centric Knowledge Infusion, it is difficult to correctly match small objects and their related objects through bidirectional interaction prompts.
2. For Interaction-Centric Knowledge Transfer, when multiple subject-object pairs with the same relational triplet categories appear in the same image, the model might mistakenly match the subject in one triplet to the object in another triplet.

Since including an external link in the rebuttal is forbidden, we will add the visualization and discussion of failure cases in the revision.

Q8: Minor points.

Thanks for your insightful comments. We will carefully refine the revision by:

1. Completing the sentence: "ACC achieves 29.28% R@100, which is higher than OvSGTR across both base and novel relations."
2. Adding quantitative results of Figure 5. We count the number of queries that match the corresponding GT's bbox (IoU > 0.5). The query quantities of matched GT's bbox in Figure 5 are 5 (original) vs. 10 (holding-guided) and 4 (original) vs. 11 (laying on-guided), respectively. It proves the effectiveness of our IGQS strategy in query initialization.
3. Correcting all reference formatting issues.

Sorry for the misunderstanding. Since the setting and splitting of OvD-R (with both novel objects and novel relations) were proposed by the recent OvSGTR (ECCV 2024), there are relatively few comparable baselines. The low performance result of OwSGG was directly copied from their released paper [3]. In addition, the baselines they compared (VS, OvSGTR, RAHP, cf. Table 3 in [3]) are ALL included in our paper (cf. Table 1 and Table 2).

Furthermore, we reviewed the recently released papers and updated the more recent baselines (i.e., OpenSGen[4] and RTHP[5]) for comparison in Table_R 10. We hope the new comparison will satisfy you. Thank you!

[4] Kong, Z. and Zhang, H., OpenSGen: Fine-Grained Relation-Aware Prompt for Open-Vocabulary Scene Graph Generation, ICMR 2025.

[5] Feng, C., Xu, T., Wu, S., Xu, D. and Chen, E., Adaptive Hierarchical Prompt for Open-Vocabulary Scene Graph Generation, ACM Transactions on Asian and Low-Resource Language Information Processing 2025.

Table_R 10 More experimental comparison results of OvR-SGG setting on VG test set.

We deeply appreciate reviewer Y5Z7 for the valuable time and constructive feedback. We provide point-to-point responses below.

Q1: Incremental extensions of existing techniques.

Sorry for the misunderstanding. Our core contributions lie in proposing a unified interaction-centric framework, designed specifically to solve the overlooked yet critical problem of “relation pair mismatches” in OVSGG. We would like to clarify two crucial techniques:

1. Bidirectional Interaction Prompt in Interaction-Centric Knowledge Infusion. Its novelty lies NOT in the linguistic/semantic transformation, but in how it strategically leverages the attention mechanism of the grounding model's text encoder (e.g., Grounding DINO). As mentioned in Line 187~190, in the passive form “surfboard held by man”, “surfboard” becomes the syntactic subject, thereby being more contextualized by “held”. We present the attention variance of the passive form and related discussion in Q3.

2. Query Selection Mechanism in Interaction-Centric Knowledge Transfer. Since both OvSGTR and ACC are based on Grounding DINO, language-guided query selection (object-centric) is used as the default. This work further proposed IGQS, an interaction-centric selection scheme for OVSGG:

   • Objective: Traditional query selection aims to improve general object detection or segmentation. Our IGQS is explicitly designed to resolve the interaction ambiguity among objects inherent in SGG.
   • Interaction Modeling: We devise a unique two-step mechanism for modeling interaction. It first leverages both object and relation semantics for an initial ranking. More importantly, the second step is a self-reflective process that uses relations predicted from an initial pass to dynamically construct “interaction-pair” prompts. This interaction-focused query refinement is a novel design for the essential “mismatched” problem in OVSGG.

   Furthermore, the ablation studies in Table 4 and the qualitative results in Figure 5 demonstrate the effectiveness of IGQS compared with the baseline (language-guided query selection in Grounding DINO). We provide more explanations of the IGQS design in Q3.

   Nevertheless, we agree that IGQS’s query initialization draws high-level inspiration from the DETR-series in detection and segmentation. We will add relevant discussion in the revision. Thanks!

Q2: Missing citation of RKD.

Thanks for bringing this excellent work (i.e., RKD [2]) to our attention. We would like to clarify the main difference between RKD and our RRD:

• Motivation. While RKD and our RRD use relationship distillation, they are motivated by different core problems. The original RKD was proposed as a general method for transferring knowledge by preserving the inter-sample relations of a teacher model's output. In contrast, our RRD is specifically motivated by a key challenge in OvSGTR, distinguishing meaningful relational foregrounds (true novel interactions) from a vast number of background object pairs. Our goal is to use relationship distillation as a targeted mechanism to help the model learn the structural differences between true interactions and random co-occurrences.
• Technique. Our RRD is fundamentally different from RKD in that: The distillation loss is computed on the set of negative samples ($\mathcal{N}$). By aligning the structural similarity of these background triplets between the teacher and student, we explicitly encourage the student model to push the embeddings for true, positive relations away from the dense space of irrelevant background pairs.

In addition, we conduct an experiment to empirically validate the advantage of our RRD. Concretely, we implemented RKD loss within our framework. As seen from Table_R 3, our RRD can get higher results compared with RKD. We will add these discussions and cite this work in the revision.

Table_R 3: Comparison of RKD [2] and RRD (ours) on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).

[2] W. Park, D. Kim, Y. Lu, M. Cho. Relational Knowledge Distillation. CVPR 2019.

Q3: Deeper reasoning behind the design choices.

1. Passive triplets. The benefit stems not from new semantic information but from the asymmetry of the attention mechanism within the text encoder (e.g., BERT). In the passive form “surfboard held by man”, “surfboard” becomes the syntactic subject, which absorbs richer contextual semantics from the predicate (e.g., “held”). This more discriminative textual feature is used to perform cross-attention with the visual patches from the image, which directly improves grounding accuracy. To empirically validate this, we display the attention scores of the active phrase (Table_R 4a) and the passive phrase (Table_R 4b). As seen, the subject in the passive phrase absorbs more from the predicate (”held”).

   Table_R 4a: Attention score across different tokens in “man hold surfboard”.

   token	man	hold	surf	##board
   man	0.0891	0.2062	0.0307	0.0448
   surf	0.0125	0.0160	0.0589	0.2721
   ##board	0.0227	0.0324	0.1365	0.0724

   Table_R 4b: Attention score across different tokens in “surfboard held by man”.

   token	surf	##board	held	by	man
   man	0.0395	0.0352	0.0514	0.0495	0.0622
   surf	0.0706	0.2585	0.0252	0.0396	0.0186
   ##board	0.1642	0.0768	0.2096	0.0363	0.0212

2. Decomposed strategy in query selection. The core motivation for decomposing triplets is to prevent the subject and object queries from interfering with each other during the initialization phase, ensuring both are properly localized. When using a full <subject, predicate, object> prompt (e.g., “man hold racket”) to guide query selection, the initial queries concentrate on the union box of interacting pairs, leading to the omission of relatively small objects within the union box. To provide clear evidence, we visualize the query initialization results of the first image in Figure 5 under three different prompts (<subject, predicate, object>, <subject, predicate>, <predicate, object>), and count the number of queries that successfully matched GT boxes (IoU > 0.5).

   • The composed prompt “man hold racket” led to queries focusing on the union box of the man and the racket (8 matched “man”, 3 matched “racket”).
   • The decomposed subject-predicate prompt “man hold” focused queries on the subject region (10 matched “man”).
   • The predicate-object prompt “hold racket” successfully focused queries on the object region (9 matched “racket”).

   This analysis shows that our decomposed strategy is crucial for more accurate query initialization for both subjects and objects. Since including an external link in the rebuttal is forbidden, we will add the visualization of query comparison in the revision.

   In addition, we conduct an experiment to study the effect of the decomposed strategy. As shown in Table_R 5, our decomposed strategy can lead to higher performance.

   Table_R 5: Ablation study on the effect of the decomposed strategy on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).

   Decomposed Strategy	R@20	R@50	R@100
   ✕	10.89	15.17	18.72
   ✓	11.37	15.71	19.37

Q4: Technical novelty.

Our main novelty and contributions lie in proposing a unified interaction-centric framework designed to solve the overlooked problem of “relation pair mismatches” in OVSGG. The innovation is not just an isolated component, but a cohesive, complete solution that synergistically addresses this mismatch problem:

• During the knowledge infusion stage, we enhance the quality of weak supervision signals through our bidirectional interaction prompts.
• During the knowledge transfer stage, we devise interaction-guided query selection and interaction-consistent knowledge distillation to ensure the model focuses on true interactions while retaining its generalization capabilities during fine-tuning.

This end-to-end, interaction-centric paradigm, from data generation to model optimization, represents our unique contribution to OVSGG.

Q5: Ablation study of object similarity and interaction similarity.

Thanks for the excellent suggestion. To study the contributions of object and relation similarities in our initial query selection, we conduct a detailed ablation study on the weight hyperparameter $γ$ (cf. Eq. 1). The results in Table_R 6 clearly show that combining both similarities (0 < $γ$ < 1) outperforms using either component alone. Performance peaks at $γ$=0.7 (our default implementation), empirically validating our core hypothesis, i.e., incorporating relational context is critical for identifying the most relevant and interacting object candidates, leading to superior performance.

Table_R 6: Ablation study of similarity weight $γ$ on the VG dataset under the OVD+R-SGG setting (Joint Base & Novel).

Q6: Limitations.

Thanks for your helpful suggestion. We discussed the limitations in Appendix G:

“it also inherits inductive biases from the teacher model. Like two sides of a coin, any biases in the vision-language model toward specific feature traits or classes may propagate to our model. Besides, our method can alleviate mismatched relational pairs, but cannot avoid all mismatches.”

We will add failure cases and discussion in the revision. For analysis of failure cases, please refer to Q7 in Reviewer E7mD.

Dear Reviewer Y5Z7,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC

Thank you for your valuable time and feedback! We would address your remaining concerns as follows. We hope our follow-up responses have fully addressed your concerns.

For Q1, in the first response, we have committed to explicitly citing this paper. Once again, we thank the reviewer for bringing this outstanding work to our attention.

For Q3-1, thank you for your further illustration of your concern. We would like to clarify that the overall intention of Bidirectional Interaction Prompt is to reduce mismatch, but passive-voice is designed to enable more interacting objects to be detected (relevant discussions have been given Sec. F.1 in the Appendix).

• Problem with the Baseline ("Object Prompt"). In this paper, we argue that the original object prompt (i.e., directly concatenating the subject & object entities, e.g., “man. surfboard.”) generates redundant object boxes, easily causing mismatches in associating object pairs and noisy pseudo supervision (cf. L49-54 and Figure 1).
• Design 1. Adding Interaction Context ("Interaction Prompt"). Interaction Prompt can alleviate the mismatches by modeling interaction information to distinguish interacting objects (e.g., man involved in holding action) from non-interacting ones through the attention mechanism. (cf. L68-68, L184-187, and L965-970)
• Design 2. Augment the object’s role with Passive-voice ("Bidirectional Interaction Prompt"). While interaction prompt significantly reduces the number of redundant object boxes, it often over-focuses on the subject (e.g., detecting only the "man" subject bounding box), leading to the omission of critical object boxes (cf. Figure S3 in the Appendix). That is the reason for introducing the passive-voice verbs to augment the object role (cf. L187-190 and L973-977 and rebuttal).
• Rule-based combination. After the boxes for the interacting subject/object were detected, we adopted a rule-based combination to enable matches with IoU overlap (cf. L191-192).

To empirically validate this, we have conducted experiments to compare three prompts in Table_R 14. As the results clearly show, each component of our design provides a significant boost.

Admittedly, the model cannot accurately attend to ALL objects in prompts. Nevertheless, we do our best to ensure that more interacting object pairs can be detected and matched.

Table_R 14. Comparison of different pre-training prompt strategies, evaluated directly on the VG150 test set.

For Q3-2, as said, since including an external link (Figure) in the rebuttal is forbidden, we will add the visualization in the revision.

For Q5, we will add the reported experiments in the main paper of the revision.

We deeply appreciate reviewer Ae9W for the valuable time and constructive feedback. We provide point-to-point responses below.

Q1: Evaluation on the base relation.

As suggested, we have added the evaluation on base object and relation categories in Table_R 1. The results clearly show that the proposed ACC significantly outperforms previous SOTA (i.e., OvSGTR) on both novel and base classes. This demonstrates that our approach provides a more comprehensive and powerful generalization capability, enhancing performance across the board, not just for unseen classes. We will incorporate these metrics into the revised manuscript.

Table_R 1: Experimental results of OvD+R-SGG setting on VG test set.

Q2: Comparison with the HOI task.

Thanks for your valuable feedback. We agree that a comparison with the HOI detection task provides valuable context for our work. We will incorporate a detailed discussion clarifying the distinctions between these two tasks in the revised manuscript. In brief, the key difference lies in their scope and objective. HOI detection, particularly on benchmarks like HICO-DET [1], is primarily a detection task over a predefined, closed set of specific human-centric interactions (e.g., 600 <action, object> pairs). In contrast, SGG addresses a more general and compositional challenge: generating <subject, action, object> triplets between any pair of objects, making the potential output space combinatorially large (e.g., 150x50x150 potential triplets for VG).

Furthermore, to empirically validate the effectiveness of the proposed ACC, we evaluate ACC and OvSGTR on the HICO-DET benchmark. As shown in Table_R 2, ACC consistently outperforms OvSGTR, achieving a 2.54% absolute improvement in R@100 of novel classes. This result is significant: it demonstrates that our model's core principles are so robust. They excel not only on the general OVSGG task they were designed for, but also on the specialized HOI task. We will incorporate these new compelling results into the revised manuscript.

Table_R 2: Performance comparison on the HICO-DET [1] dataset under the OvR-SGG setting.

---

[1] Chao YW, Liu Y, Liu X, Zeng H, Deng J. Learning to detect human-object interactions. WACV 2018.

Q3: Claim challenges in OVSGG in Abstract.

Good suggestion! We will revise the abstract to clearly highlight the challenges in OVSGG. Below is the updated abstract, with bold text indicating the revised parts:

Open-vocabulary scene graph generation (OVSGG) extends traditional SGG by recognizing novel objects and relationships beyond predefined categories, leveraging the knowledge from pre-trained large-scale models. Existing OVSGG methods always adopt a two-stage pipeline: 1) Infusing knowledge into large-scale models via pre-training on large datasets; 2) Transferring knowledge from pre-trained models with fully annotated scene graphs during supervised fine-tuning. However, due to a lack of explicit interaction modeling, these methods struggle to distinguish between interacting and non-interacting instances of the same object category. This limitation induces critical issues in both stages of OVSGG: it generates noisy pseudo-supervision from mismatched objects during knowledge infusion, and causes ambiguous query matching during knowledge transfer.

Dear Reviewer Ae9W,

The author-rebuttal phase is now underway, and the authors have provided additional clarifications and performance results in their rebuttal. Could you please take a moment to review their response and engage in the discussion? In particular, we’d appreciate your thoughts on whether their revisions adequately address your initial concerns. Thank you for your time and valuable contributions.

Best, Your AC