Scene Graph Generation with Role-Playing Large Language Models

We thank reviewer xMLQ for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Clarifying the rule of prompt construction.

A1: The prompts are like example #1 and are generated offline. The "{scene content to be discussed}" is constructed based on a given subject-predicate or predicate-object pair. For instance, in L167 (example #1), the "{scene content to be discussed}" is constructed based on a subject-predicate pair where the subject is “animal” and the predicate is “eating”. Details about the scalability and cost of generating descriptions are provided in Q2.

Regarding Fig. 1c, it is indeed misleading. In Fig. 1c, we specify a triplet instead of a pair for easier understanding of the later discussion process. We appreciate the keen observations and recognize that this creates an inconsistency. We will revise Fig. 1c for clarity. The revised version is shown in the PDF (Fig._R1). Thank you.

Q2: Scalability and cost of generating descriptions.

A2: As you noticed, iterating all possible triples of <subject, predicate, object> is computationally unaffordable (150 * 50 * 150 for 150 object categories and 50 predicate categories). Therefore, we use subject-predicate and predicate-object pairs to generate descriptions. Following [a], we use 3 object categories (i.e., human, animal, and product) instead of 150 object categories to save computational costs. The description generation process (as mentioned in L653-667) involves the following steps:

1. Initial Description Generation. This step calls OpenAI's API 2 * 3 * 50 times (2 stands for subject-predicate and predicate-object pairs).
2. Summarizing Descriptions. This step calls OpenAI's API 1 time.
3. Description-Relation Association. This step calls OpenAI's API 50 times.
4. Opposite Description Generation. This step calls OpenAI's API 1 time.

As illustrated, the primary cost is in the initial description generation (i.e., discussion by multi-persona collaboration). The generation is one-time and is completed offline, and would not delay the inference stage. Since NOT all triples are iterated, the cost remains acceptable even when the number of object/predicate categories increases. We respectfully refer the reviewer to {reviewer RK4c Q2} for the cost of online deployment.

Q3: Usage of possible coexistence descriptions.

A3: The goal of making the prediction of possible coexistence descriptions close to those of CLIP is to regularize the training process and avoid overfitting (L255-257). Such an approach brings more supervision signals during training and is inspired by the knowledge distillation-based open-vocabulary frameworks [b]. It is important to note that possible coexistence descriptions are used only during training. During inference, they DO NOT contribute to the similarity measurement, as the correlation $C_{r}^{n}$ equals 0 (see Eq. 1 & 2).

Q4: More ablative experiments on possible coexistence descriptions.

A4: In Table 8, $\alpha_{indef}$ is a scaling factor (Eq. 8, L258) for training targets rather than a weight of a loss. Per your request, we remove $\mathcal{L}_{indef}$ in Eq. 9 to study its influence. The results are as follows:

As seen, the $\mathcal{L}_{indef}$ leads to considerable improvements in the novel set (e.g., 1.6% mR@50 and 1.4% mR@100) and does not affect the performance of the base set. This further varifies our hypothesis (i.e., regularize the training process and avoid overfitting as stated in L255-257). We will incorporate the above results into the appendix of our revision. Thanks.

Q5: Discussion on descriptions and CLIP's recognition capability.

A5: The generated description can be viewed as a semantically informative representation compared to the raw representation with only category names. They can convey extra concepts without modifying model architecture or introducing additional learning processes. Concretely, they 1) provide comprehensive concepts that help CLIP differentiate between relations (see Fig. 1 in [c]), and 2) offer simpler and more common concepts for relation categories that may not have been encountered during CLIP's pre-training (see Fig. 5 in [d]).

Note that these detailed, fine-grained descriptions may not align perfectly with CLIP's pre-training pipeline. By collecting 10 horse images from the Web and comparing the vision-language similarities, we empirically find that CLIP struggles to properly distinguish between concepts with slight differences (e.g., "with four legs" vs. "with two legs"). This indicates an inherent drawback of previous description-based methods that uniformly process all descriptions as affirmative classifiers (L50), and motivates us to introduce the renormalization mechanism (L69-76) so as to adaptively weight the generated descriptions. We will incorporate the discussions into the appendix of our revision. Thank you.

References:

[a] Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network, AAAI 2023.

[b] Towards Open Vocabulary Learning: A Survey, TPAMI 2024.

[c] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.

[d] Visual Classification via Description from Large Language Models, ICLR 2022.

We thank reviewer RK4c for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Computational complexity of description generation (offline).

A1: Suppose there are 3 common object categories (i.e., human, animal, and product [a,b]) and 50 predicate categories. The description generation process (L653-667) involves the following steps:

1. Initial Description Generation. This step calls OpenAI's API 2 * 3 * 50 times.
2. Summarizing Descriptions. This step calls OpenAI's API 1 time.
3. Description-Relation Association. This step calls OpenAI's API 50 times.
4. Opposite Description Generation. This step calls OpenAI's API 1 time.

The execution time of the above steps depends on the network and the speed of OpenAI's response. Since these descriptions are generated offline, they DO NOT incur any computational cost during deployment. We respectfully refer the reviewer to {reviewer xMLQ Q2} for the scalability and cost of generating descriptions.

Q2: Computational complexity of model (online).

A2: Since the renormalization and similarity measurement (Eq. 2) involve only a few matrix operations that can be omitted from the complexity analysis, we will focus on reporting the inference time of the following three main modules:

As seen, the inference time of CLIP’s visual and text encoder is significantly higher than that of our mutual visual adapter. Compared to [b] which also uses CLIP, the delay of the newly-introduced MVA (0.2ms) is neglectable. In addition, it is important to note that during deployment, only the visual part (i.e., CLIP’s visual encoder and our mutual visual adapter) requires computational resources. This is because the descriptions are generated offline for all categories and remain unchanged during deployment so that their text embedding can be pre-computed and stored. Therefore, the overall framework can meet the requirement of real-time applications or resource-limited scenarios. We will incorporate the results about computational complexity into the appendix of the revision.

Reference:

[a] Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network, AAAI 2023.

[b] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.

We thank reviewer qa1m for the valuable time and constructive feedback. We provide point-to-point response below.

Q1: Presentation.

A1: Our apologies. We will revise Section 3.1 to improve clarity and coherence. Our revisions will focus on:

1. Streamlining the naming conventions: We will eliminate the term "text classifiers" and consistently use "descriptions" throughout.
2. Delineating modules: We will add a summary of the newly-involved modules and their utility for description generation.
3. Ensuring consistency: We will standardize the terminology across the manuscript, aligning it with the definitions provided in L64~77.

Q2: Comparison with conventional SGG methods.

A2: Thanks for your suggestion. Per your request, we trained our model with frequency bias on the full set of relations. The results are as follows:

As seen, our model also demonstrates good performance under standard settings. We will incorporate the above results into the appendix of our revision. Thanks.

Q3: Experiments on different base/novel splits.

A3: Good suggestion! As you noticed, the mean and variance are reported in Tables 1-3 to ensure the performance advantage is reliable. Per your request, we trained our model on different base/novel splits to investigate the robustness further. Specifically, we 1) change the proportion of the base and novel split and 2) change the categories within the base and novel split (i.e., different No. for the same ratio). The results are as follows:

As seen, our model is robust to different base/novel splits. We will incorporate the above results into the appendix of our revision. Thank you.

Q4: Experiments on panoptic scene graph generation.

A4: Thanks for your suggestion. Our current framework is not directly applicable to PSG. Due to time constraints, we are unable to provide empirical results on the PSG dataset, as significant modifications and engineering efforts are needed. Nonetheless, we recognize the importance of exploring this direction and will definitely consider this as our future work. Thanks.

Q5: Standard prompts in L329.

A5: Given a subject-predicate or predicate-object pair, we ask LLM to generate corresponding descriptions. The prompt is defined as:

Imagine [subject-predicate / predicate-object pair]. Think about what the scene should look like. Summarize each descriptive statement of the scene in about 15 words each.

We respectfully refer the reviewer to {reviewer xMLQ Q1} for the rule of prompt construction.

Q6: Clarifying multi-persona collaboration.

A6: As mentioned in L336, the involvement of multiple personas in the discussion process enhances the diversity of generated descriptions. The key point of our multi-persona collaboration is about the “collaboration” rather than a specific persona. Actually, using only one persona can even decrease the diversity of generated descriptions and hurt the performance, as it can only generate descriptions from its own viewpoint without discussion with others. From the perspective of the number of generated descriptions, each persona's contribution is almost the same.

In addition, we use the standard prompts and change the system prompt of LLM from the default like “you are a helpful AI assistant” into a persona-specific one like “you are a biologist”. We then evaluate the performance of our model with these generated descriptions. We only report the effect of the biologist and engineer persona because of the urgent due. The results are as follows:

As seen, using only one persona results in a significant performance decrease, which is consistent with the findings in Table 4. We will give the results of the physicist persona asap. Thank you.

References:

[a] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.