LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision

Q1:

"The performance of LASER heavily relies on the quality of video captions. If captions are sparse or inaccurate, the resulting generated STSGs may also suffer in quality."

Answer:

Thank you for pointing out the potential issue that when captions lack sufficient detail, our method may struggle to extract high-quality temporal STSL descriptions. To address this, we can leverage a visual-language model pretrained on video-caption datasets to generate detailed captions for a given video. To alleviate the reviewer’s concern, we have already conducted experiments using LLaVA-Video [1], which introduces a prompting technique to synthesize such detailed captions effectively with GPT-4o. The results are very promising and can be found in the overall rebuttal section under “Training with Noisy Object Trajectories and Long Captions on LLaVA-Video.”

[1] Zhang, Yuanhan, et al. "Video Instruction Tuning With Synthetic Data." arXiv preprint arXiv:2410.02713 (2024).

Q2:

"The model seems to have a problem building scene graphs with longer videos and more instances. ""

"While the framework is promising, the scalability of the approach to very large datasets or real-time applications may pose challenges to the model."

Answer:

To address scalability concerns, we include an experiment with the LLaVA-Video dataset, which contains significantly more instances and richer temporal information. This experiment is detailed in the overall rebuttal section under “Extra Experimental Results: Training on the 10K LLaVA-Video Dataset Showing Generalizability, Robustness, and Scalability.” Furthermore, we have optimized the training process by incorporating the DistributedDataParallel module, which enhances training efficiency on large-scale datasets.

It is also important to note that our LASER pipeline impacts only the training phase. During evaluation, the trained model is directly deployed to recognize STSGs, and the test-time efficiency is entirely dependent on the backbone STSG recognition model. If the backbone model supports real-time inference, the LASER fine-tuned model will retain this real-time capability.

Q3:

"In Table 3, Setup 1 & 2 have the same F1 score, while the full setting of losses is higher than Setup 3. Can you explain this phenomenon?"

Answer:

In general, all three loss terms contribute significantly, although their individual contributions may differ across datasets. The primary reason for the observed differences lies in the inherent distinctions of the datasets, including variations in video content, caption content, and temporal characteristics.

In the 20BN dataset, temporal loss is the most critical factor. The videos in this dataset capture specific actions, with specifications describing the pre- and post-condition of these actions. Naturally:

• The precondition should occur in the early stages of the video.
• The postcondition should occur in the late stages.

Without the guidance of temporal loss, the alignment checker compares the specification against the entire probabilistic STSG, retaining only the top five likely traces. When training from scratch with only the contrastive and semantic losses, the model struggles to converge on appropriate matches for early video frames corresponding to the precondition and late video frames corresponding to the postcondition. Therefore, incorporating temporal loss is crucial for LASER to achieve strong performance on the 20BN dataset, with further improvements achieved by adding contrastive and semantic losses.

As the temporal loss is necessary for learning on the 20BN dataset, and both Setup 1 and Setup 2 do not include temporal loss, it explains the relatively low performance observed in these setups. We have also included a qualitative study on the contribution of each loss term for 20BN in Figure 12 in Appendix B.4, which provides an intuitive understanding of each loss's impact.

Q1:

"The significance of the target task spatiotemporal scene graphs needs to be discussed further. ... It would be better to compare the proposed approach more explicitly to directly using LLMs to generate scene graphs. If the experiments are too expensive, the authors can discuss the potential advantages of the proposed method over state-of-the-art MLLMs. The authors can also explain why training a separate STSG model is beneficial."

"The paper does not present state-of-the-art MLLM results. To demonstrate the advantage of the proposed method, authors should consider some multimodal large language models, such as BLIP-2, InstructBLIP, MiniGPT-4, and LLaVA-1.5. "

Answer:

In predicting video scene graphs, our observations are that MLLMs face significant challenges in localizing textual descriptions to specific parts of images and videos due to the lack of precise labels. In practice, when attempting to leverage MLLMs to predict video scene graphs, we found that prompting them often yields suboptimal results. For example:

• VideoLLaMA [3] frequently produces mismatched outputs, such as incorrect object colors or inaccurate spatial locations.
• While GPT-4o has not yet released its video-language API, our experiments with image prompts revealed difficulties in extracting precise relationships between objects, such as determining which object a person is looking at.

We will include a qualitative analysis of these failure cases in the revised paper to further illustrate existing MLLM limitations.

Additionally, a major advantage of training a separate STSG model instead of invoking the MLLM is speed. During inference, we call the STSG backbone model directly. On average:

• Calling the full MLLM pipeline takes about 34.5 seconds per video for extracting STSL from the video.
• Directly calling the CLIP model takes only about 2.6 seconds per video on the OpenPVSG dataset.

[3] Zhang, Hang, Xin Li, and Lidong Bing. "Video-llama: An instruction-tuned audio-visual language model for video understanding." arXiv preprint arXiv:2306.02858 (2023).

Q2:

"The title uses the keyword "neuro-symbolic," but it seems that the proposed method doesn't emphasize this point. It deserves further discussion."

"Some technical details are not clear enough. The symbols in Equation (1) are not explained well. I am unclear about what the $ψ$ stands for and why $M_θ(X)$ needs to be conditioned on 1. What does the "1" stand for?
There are many key concepts and methodologies that are not explained in the method section."

Answer:

To understand Equation (1), it is important to clarify the symbol $⊨$ (models), which denotes semantic implication. The notation $A ⊨ B$ means that $B$ is true in every model where $A$ is true.

Here:

• $M_θ(X)$ represents the predicted probabilistic STSG.
• $ψ$ denotes the STSL specification.

Thus, $Mθ(X) ⊨ ψ$ implies that the relationships, actions, and temporal constraints specified in $ψ$ are consistent with the probabilistic trajectories and interactions predicted by $M_θ(X)$. Consequently, $Pr(Mθ(X) ⊨ ψ)$ represents the alignment score between the predicted STSG and the specification $ψ$.

A detailed example of alignment score calculation and explanation for neural-symbolic components is provided in the overall rebuttal under “Clarifying Symbolic Components and Their Role in Neural-Symbolic Programming”.

Q3:

"The spatiotemporal scene graph part in the right-up corner of Figure 1 is unclear. The same problem appears in Figure 6. In Figure 1, in the third row, there are "¬touching (B, H)" both on the left and the right side. What this means seems to be ambiguous."

Answer:

In Figure 1, the x-axis represents time, while the color indicates the likelihood of a given description being true. For example:

• The third row shows the probability progression for touching(B, H). Initially, touching(B, H) has a low probability, suggesting a high likelihood for its negation ¬touching(B, H).
• Over time, the probability of touching(B, H) increases before gradually decreasing again, returning to a low likelihood.

We will enhance the visual representation of this figure in the revised version to make these concepts clearer.

Q1:

"The reliance on large language models to extract logical specifications may introduce noise and inaccuracies, especially if the captions lack detailed descriptions. Enhancing this extraction process or adding error-handling mechanisms could improve overall model performance."

"Could you clarify the specific criteria or methods used by the language model to extract logical specifications? Additional details on handling ambiguous or underspecified captions would be helpful.""

Answer:

Thank you for pointing out the potential issue that when captions lack sufficient detail, our method may struggle to extract high-quality temporal STSL descriptions. To address this, we can leverage a visual-language model pretrained on video-caption datasets to generate detailed captions for a given video. To alleviate the reviewer’s concern, we have already conducted experiments using LLaVA-Video [2], which introduces a prompting technique to synthesize such detailed captions effectively with GPT-4o. The results are very promising and can be found in the overall rebuttal section under “Training with Noisy Object Trajectories and Long Captions on LLaVA-Video.”

[2] Zhang, Yuanhan, et al. "Video Instruction Tuning With Synthetic Data." arXiv preprint arXiv:2410.02713 (2024).

Q2:

"The evaluation covers three datasets but could be strengthened by testing on additional complex datasets for scene graph generation to validate generalizability, such as Action Genome, VidVRD, and VidOR."

"How well does the approach generalize to more complex, real-world videos beyond the tested datasets? Testing on diverse, large-scale datasets could provide valuable insights."

"Have you considered using additional forms of weak supervision, such as visual cues or scene descriptions, to compensate for the limitations of captions alone?"

Answer:

In the overall rebuttal section under “Training with Noisy Object Trajectories and Long Captions on LLaVA-Video”, we demonstrate that LASER can fine-tune its backbone model using the long and detailed captions, as well as noisy object trajectories, provided by the LLaVA-Video dataset. These detailed captions can be viewed as comprehensive scene descriptions, further enriching the training process. Moreover, LASER showcases strong generalizability across various downstream evaluation datasets, including Action Genome and VidVRD. It is worth noting that VidOR is already included as a subset of the OpenPVSG dataset.

Q3:

"The paper does not include strong works on scene-graph generation, such as STTran, TRACE. It is hard to evaluate the robustness of the proposed framework without the comparison with such methods."

Answer:

While STTran and TRACE demonstrate strong performance in video scene graph generation, they are closed-domain models with fixed classification layers, limiting their ability to transfer knowledge across datasets. Additionally, their architectures are not designed to handle noisy labels directly extracted from captions, making them less robust in such scenarios.

It is important to note that the primary contributions of STTran and TRACE lie in their carefully designed model architectures for capturing visual features. In contrast, our work focuses on an orthogonal problem: developing a model-agnostic weak supervision training pipeline. Our approach is not in competition with STTran or TRACE but rather complements them. In fact, STTran and TRACE can be adopted as backbone models within our LASER framework, demonstrating that our method enhances their capabilities rather than being mutually exclusive.

Nevertheless, we report unary and binary predicate recall for STTran using the author-provided checkpoint. Note that STTran is trained in-domain on the Action Genome dataset in a fully supervised manner, while our models are trained on out-of-domain datasets using a weakly supervised approach.

Q1:

"The design of this approach seems puzzling. If the STSG can be generated based on a series of advanced models such as CLIP and LLM, why is there a need to train an additional model to make predictions based on the generated data? What are the problems with directly using the generated data as the results, and what is its performance?"

Answer:

Imagine the caption: “A boy is climbing a tree, and another boy is watching him.” In STSL, this is expressed as:
name(v1, “boy”), name(v2, “boy”), name(v3, “tree”), climbing(v1, v3), watching(v2, v1).

Note that STSL differentiates between the two boys using variable names. Without fine-tuning CLIP, the original pretrained CLIP (see Fig. 2) cannot make this distinction. For instance, CLIP will likely assign the same probability to “first boy watching second boy” and “second boy watching first boy.” This is why a simple combination of CLIP and LLM does not suffice. Using STSL to supervise the fine-tuning of CLIP imparts the capability to distinguish such relationships, enabling our CLIP-based SGG model to make accurate predictions.

Q2:

"The description of the experimental section is also quite unclear. For instance, it is not specified which state-of-the-art (SOTA) methods are used for comparison or whether they represent the most advanced approaches. Additionally, there are no specific references provided, making it difficult to assess the relative effectiveness of the proposed method."

Answer:

To the best of our knowledge, our method is the first framework to utilize weak supervision from captions to learn video scene graphs. As such, there are no direct state-of-the-art (SOTA) baselines available for an apple-to-apple comparison. In the OpenPVSG experiment, we benchmarked our performance against 8 fully supervised baselines provided with the dataset [1]. Unlike these baselines, which rely on full supervision, our method exclusively uses captions as the learning signal.

[1] Yang, Jingkang, et al. "Panoptic video scene graph generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q3:

"The motivation behind STSL is not very clear. Its relationship with STSG is also unclear, as well as why this step is necessary."

Answer:

The Q1 answer provides an explanation of why STSL is essential.

Further, another key feature of LASER is the “tracking” of objects along the temporal dimension. For example, consider the video caption:

“A man swims freestyle with fluid precision. The man pushes off the wall, gliding underwater before breaking the surface to take his first breath.”

In STSL, this is represented as:
Until( Finally(name(v1, “man”), name(v2, “wall”), push_off(v1, v2)), Until( Finally(name(v3, “water”), beneath(v1, v3)), Finally(break(v1, v3))) )

Note that STSL captures the temporal sequence.

The CLIP-based SGG model we trained operates similarly to image caption-based methods, predicting object(s) and relationships for each frame. However, it lacks the ability to track objects across time. The STSL and symbolic checker (Section 3.4) address this by projecting the temporal sequence captured in STSL onto the CLIP frame-by-frame predictions. Both the STSL and the symbolic checker are non-trivial and significant contributions of our paper.

Q1:

"Although the authors claimed that they propose the first framework to learn weakly-supervised scene graph generation from video caption data, the utilization of image caption data in weakly-supervised scene graph generation has been widely explored [1, 2]. I think this work falls into a straightforward adaptation to the video domain, so this contribution is not that appealing."

Answer:

The main difference between LASER and an image caption-based method is the “tracking” of objects along the temporal dimension.

Imagine a video caption:

A man swims freestyle with fluid precision. The man pushes off the wall, gliding underwater before breaking the surface to take his first breath.

In STSL, this will become:

Until(Finally(name(v1, “man”), name(v2, “wall”), push_off(v1, v2)) Until(Finally(name(v3, “water”), beneath(v1, v3)), Finally(break(v1, v3))))

Note that STSL captures the temporal sequence.

Now, the CLIP-based SGG that we trained is similar to the image caption-based method you mentioned, predicting, for each frame, the object(s) and relationships. However, it does not understand the “tracking”. It is not hard to imagine that the checker (Section 3.4) could take the STSL, which captures the temporal sequence, and project it onto the CLIP frame-by-frame predictions. The STSL and the symbolic checker are non-trivial and significant contributions of our paper.

Nevertheless, at the reviewer’s suggestion, we conducted experiments to compare our approach with [2]. Using the caption processing strategy outlined in [2] on the LLaVA-Video dataset, we obtained 25,685 unique unary keywords and 2,263 unique binary keywords, but without any temporal information linking relations to specific video frames. Notably, LASER demonstrates superior performance in extracting high-quality binary relation keywords.

Q2:

"The idea to incorporate LLMs into scene graph generation has also been investigated by several previous works [3, 4]. This makes the novelty of this framework less significant."

Answer:

We note that faithfully converting the intermediate representation from a caption into an executable specification that respects the temporal sequence is nontrivial. Our compilation design, described in the overall rebuttal section “Compiling GPT-Generated Intermediate Representations into Executable STSL Specifications”, takes in the GPT representation and converts it into an STSL program. This process includes lexical analysis, syntactic and semantic analysis, and an error handling and recovery system for generating valid and rich STSL programs.

As shown in the overall rebuttal section “Extracted STSL Program Quality”, our compilation process achieves a failure rate of 0.78% on OpenPVSG and 0% on LLaVA-Video. Importantly, our STSL program supports open-domain keywords, enabling learning from noisy weak supervision labels, which sets it apart from works like [3, 4] that focus on predicting keywords within a target dataset.

Nevertheless, we followed the reviewer’s suggestion and ran experiments using the method from [4]. We preprocessed long captions using the LLM4SGG [4] prompting technique but found its post-processing script for cleansing GPT-generated results to be highly error-prone and prone to hallucinations, especially in scenarios involving long captions. When replicating their method on the LLaVA-Video dataset, only 3.11% of the GPT-generated responses could be successfully parsed into a list of relational triplets. In contrast, our parser achieves 100% conversion of GPT responses into executable STSL programs. Further details are available in the overall rebuttal section “Clarification on Compiling GPT-Generated Intermediate Representations into Executable STSL Specifications”.

3. Extra Experimental Results Trained on 10K LLaVA-Video Dataset Showing Generalizability, Robustness, and Scalability (gWyP, sYch, ym8p)

We present additional results on the 10K LLaVA-Video [1] dataset, showcasing the generalizability, robustness, and scalability of our approach. We evaluate the quality of extracted STSL programs, coverage of ground-truth keywords across datasets, and the pipeline’s ability to handle domain shifts and complex captions. These findings highlight the flexibility and effectiveness of our method in diverse and noisy real-world scenarios.

(a) OpenPVSG Caption to STSL Program

• Captions average 12.69 words, containing an average of 1.69 events in the GPT-extracted representations.
• The failure rate for converting captions into valid executable STSL formulas is 0.78%.
• A total of 453 unique unary keywords and 679 unique binary keywords were identified, covering:
  • 92.86% ground-truth unary and 94.74% ground-truth binary keywords in the OpenPVSG dataset.
  • 55.56% ground-truth unary and 61.54% ground-truth binary keywords in the Action Genome dataset.
  • 20% ground-truth unary and 12.88% ground-truth binary keywords in the VidVRD dataset.
• Notably, these closed-world datasets include keywords like "unsure," "other relation," and "not contacting," which rarely occur in natural scenes.

(b) LLaVA-Video 10K Dataset Caption to STSL Program

To demonstrate the capability of our LLM + compiler pipeline in handling long and complex caption descriptions, we evaluated it on LLaVA-Video, a dataset with detailed captions released on October 4, 2024. From a randomly sampled subset of 10,000 video clips (each under 30 seconds):

• Captions average 233.05 words, and the pipeline extracted 4.03 events per video on average.
• The failure rate for converting captions into valid executable STSL formulas was 0%.
• A total of 18,458 unique unary keywords and 4,492 unique binary keywords were identified, covering:
  • 91.27% ground-truth unary and 89.47% ground-truth binary keywords in the OpenPVSG dataset.
  • 80.56% ground-truth unary and 69.23% ground-truth binary keywords in the Action Genome dataset.
  • 82.86% ground-truth unary and 31.82% ground-truth binary keywords in the VidVRD dataset.
• Notably, 91.99% of LLaVA-Video samples originate from YouTube, with the remaining samples sourced from ActivityNet, Charades, Ego4D, NextQA, and YouCook2. This highlights a significant domain shift between the LLaVA-Video dataset and all evaluation datasets.

(c) Keyword Analysis

We present the top 10 most frequent unary and binary keywords extracted from captions in both datasets to further illustrate the diversity and robustness of our method.

(d) Cost of Generating the STSL from Intermediate Representations

• Generating all intermediate representations with GPT for the 10K dataset costs approximately $50 and takes around 15 minutes using parallelization techniques.
• The compilation process from the 10K GPT-generated structured representations to executable STSL programs requires about 1 minute.

[1] Zhang, Yuanhan, et al. "Video Instruction Tuning With Synthetic Data." arXiv preprint arXiv:2410.02713 (2024).

2. Clarification on Compiling GPT-Generated Intermediate Representations into Executable STSL Specifications (gWyP, ym8p, sYch, nqrt)

We will include a comprehensive section on STSL parsing and compilation in our revised paper. Converting GPT-generated intermediate representations into executable STSL specifications is a nontrivial task. Below, we outline our compilation pipeline, which translates the structured representation from GPT into an STSL program for video analysis:

(a) Parsing

The parser ingests the GPT-generated intermediate structured representation, provided in JSON format, and constructs an Abstract Syntax Tree (AST). This step includes:

• Normalizing JSON dictionary keys for consistency.
• Removing invalid or unrecognized JSON dictionary keys.
• Constructing and parsing the program using PyParse.

(b) Analyzing

The analyzer processes the AST to perform validity checks and necessary transformations, producing a validated AST. Key validation tasks include:

• Checking the logical correctness of operations.
• Verifying the sizes of unary and binary tuples.
• Ensuring the consistency of time descriptions.

(c) Synthesizing

The synthesizer uses the validated AST to generate an executable STSL program. This step ensures the output program adheres to the formal syntax and semantics of STSL.

Throughout these steps, robust error-handling and recovery mechanisms are integrated to mitigate the potential effects of hallucinations from the large language model. The pipeline guarantees that the resulting STSL program is executable while preserving the open-world diversity of keywords inherent in the GPT-generated representation.