Adaptive Visual Scene Understanding: Incremental Scene Graph Generation

[XGhv.Weakness.1 - CSEGG is limited] We would like to address the concerns regarding the difference of CSEGG compared to open-vocabulary SGG [a] and zero-shot SGG [b].

First, we respectfully disagree with the reviewer that the significance of CSEGG is limited. The problem setting in open-vocabulary SGG [a] and zero-shot SGG [b] differs fundamentally from the problem setting addressed by CSEGG. In CSEGG, we aim to simulate scenarios where the “new” predicates or “new” objects encountered by the model are completely novel and have not been seen by any models before including the existing language models or models with multiple modalities.

For example, if a new drug called XYZ is discovered, this drug has not been introduced to any AI models, whether they are language, vision, or multi-modal models. As a result, all the zero-shot and open-vocabulary models referenced in the two citations would fail to identify this new drug, as there is NO prior knowledge about this drug to transfer knowledge at the first place.

Second, open-vocabulary SGG [a] uses a frozen text encoder with pre-learned weights, assuming prior knowledge of objects or predicates. Similarly, zero-shot SGG [b] relies on large language models for description-based prompts, assuming these models have encountered the new objects or predicates. In contrast, CSEGG simulates scenarios where completely novel objects or predicates are introduced. Relying on pre-learned information from large language models or frozen text encoders would violate the continual learning setting.

Both [a] and [b] are fantastic works. We will cite them and discuss the differences of these works from our work in the related work section. However, our work focuses on continual learning in SGG, which is NOT addressed by these methods as it is a different problem setting altogether.

[XGhv.Weakness.2 - Fine-grained controls for RAS] We appreciate the reviewer’s suggestion. However, we would like to point out that even without fine-grained controls as suggested in [c], our approach has already demonstrated decent performance outperforming all the existing continual learning baselines.

As we are the first to address continual learning in SGG, we introduced a proof-of-concept model called RAS. Our main goal was to show that generative modeling can prevent catastrophic forgetting in CSEGG.

We agree that using generative models with fine-grained control signals like boxes and captions could improve our method. We will cite the suggested work in the related works section and incorporate it in our model in the final version.

[XGhv.Weakness.3 - Limited Choice of SGG Models] We understand the reviewer's concern that the two-stage SGG baseline (IMP, CVPR'17) is old. However, with 1359 citations, it remains a pioneering model in SGG and a strong baseline for comparison in subsequent works. We included it to show that our method works on classical SGG models and that the CSEGG benchmark can be extended to other SGG models. Additionally, we included a one-stage baseline (SGTR, CVPR'22) to validate our benchmark with more recent models.

We agree that PE-Net [d] is a very interesting work, and we plan to try and include this in the CSEGG benchmark for the final version.

We included only one model from each category (one-stage and two-stage) due to hardware and time constraints. For Learning Scenario 1 (S1), we conducted 120 experiments per model, including 5 tasks for each of the 8 baselines (excluding joint models), with 3 runs per baseline for statistical significance. For Learning Scenario 2 (S2), we performed 48 experiments per model, involving 2 tasks for each of the 8 baselines, also with 3 runs per baseline. This totals 168 experiments per SGG model, and with 2 models, we conducted 336 experiments. Each experiment takes about 2 days on a single machine with 4 A5000 GPUs. Using 3 machines, training for S1 and S2 alone took approximately 7 months. Including Learning Scenario 3 (S3) and additional ablation studies, the entire set of experiments extended to around 9-10 months. Thus, we included only one model from each category (one-stage and two-stage) in our study. We invite the community to contribute by incorporating more SGG backbones.

[XGhv.Weakness.4 - No generality] We agree with the reviewer that studying model generalization in scene graph generation is interesting. This includes training a SGG model in VG dataset and testing on other SGG datasets, such as OpenImages. However, we argue that model generalization and continual learning are completely different problems. In our work, we focus on continual learning.

Ideally, we would like to include other SGG datasets such as OpenImages, for our Continual SGG problem. However, our current analysis is limited to the Visual Genome (VG) dataset due to hardware restrictions and time constraints as mentioned in [XGhv.Weakness.3].

We appreciate the list of references related to open-vocabulary, open-set, and zero-shot problems in SGG. However, we would like to emphasize again that none of these works look into the problem of continual learning, which is the main focus of our work. We will cite these papers and discuss the differences from our work in the related work section. As mentioned in [XGhv.Weakness.1], our work studies continual learning problems in SGG, which is valuable, unique, and challenging. It is different from open-vocabulary, open-set, and zero-shot SGG.

[XGhv.Question.1 - "long-tailed distribution" in L43] We refer “long-tailed distribution” to distributions in both objects and relationships; and hence, predicates altogether. We will clarify this statement in the final version. We present histograms of object and relationship distributions in each task of each learning scenario in Fig. 5 in the Appendix. The histograms show that the long-tailed distribution of objects and relationships (predicates) persists across all tasks and scenarios.

[myHM.Weakness.1 - Focus on Image SGG explanation] We agree with the reviewer that advancing continual learning in Scene Graph Generation (SGG) to include dynamic or video SGG is a natural and beneficial progression, as it more closely aligns with real-world settings.

However, this research topic on continual learning in SGG is relatively unexplored, as no one has studied it before. Therefore, we need to begin with basic and straightforward settings, i.e. continual learning in scene graph generations on static images. Here, we aimed to establish a comprehensive benchmark of baselines and datasets to provide a foundation for the research community to start developing more advanced continual learning methods for SGG on static images. With all these foundations developed in this work, we can then tackle more complex aspects, such as dynamic video SGG in the next phase.

[myHM.Questions.1 - Opinion on Dynamic SGG Benchmark] We agree that creating a similar benchmark for dynamic scene-graphs is an excellent idea, and we definitely plan to extend our benchmark dataset and to incorporate dynamic scene-graph models. As mentioned in our previous response, we are the first team to address SGG in a continual learning setting, and our initial goal was to begin with basic and straightforward settings on static images. Only after this, we can move on to incorporate dynamic SGG models into our benchmark. Thank you for bringing Action Genome to our attention; it has inspired us for future developments in our benchmark. We will certainly cite this work in our final version.

[myHM.Limitations.1 - Continual learning for dynamic SGG] We agree with the reviewer that advancing continual learning in Scene Graph Generation (SGG) to include dynamic or video SGG is a natural and beneficial progression, as it more closely aligns with real-world settings. As mentioned earlier, we are the first team to address SGG in a challenging continual learning setting, and our initial goal was to establish a comprehensive benchmark for image-level SGG models. Only from here, we can then move on to develop continual learning SGG in dynamic and video settings.

We thank the reviewer for the clear directions in continual learning for dynamic scene graphs. In response, we will include a paragraph in the related works section to survey all the dynamic scene graph generation works, such as the three works here: Action Genome, HOMAGE, and EASG. We will cite these papers in the related works. However, do note that all these three works do not study continual learning problems. Neither do these works look into the importance of studying continual learning in SGG in a dynamic setting. Expanding our benchmark to incorporate continual learning in dynamic SGG models will be a key focus of our future work. Thank you for the valuable suggestions and for highlighting important works in this area.

[Y6eG.Weakness.1-Poor paper presentation] We appreciate the reviewer’s feedback regarding the presentation of our paper. We would like to note that other reviewers did not mention any issues with the clarity or presentation of our work. However, we value your input and it would be great if you could specify which parts of the paper were difficult to follow. This would help us make the necessary revisions and improvements for the final version. Thank you for your constructive feedback.

[Y6eG.Weakness.2-Lack of Mathematical Formulation] For each scenario in CSEGG, we have task-specific dataset D_t \, where $t = 1, 2, 3, 4, 5$ for Learning Scenario 1 (S1), $t = 1, 2$ for Learning Scenario 2 (S2), and $t = 1, 2, 3, 4$ for Learning Scenario 3 (S3).

At task $t$, we take the previously trained model $M_{t-1}$ (with weights $W_{t-1}$) and train it on the dataset for task $t$, $D_t$, to obtain weights $W_t$ and model $M_t$. If using an exemplar-based approach (Replays and RAS), the model is also trained on $E_t$, where $E_t$ is the exemplar for task $t$. Note that $E_1$ doesn’t exist as no exemplar is needed for task 1 in all scenarios.

For replay-based methods, we create exemplar $E_t$ after training of task $t-1$ by storing data points from $D_{t-1}$. Therefore, after training of task $t$, we create $E_{t+1}$ from $D_t$. At task $t+1$, we obtain $M_{t+1}$ by training $M_t$ (with weights $W_t$) on $E_{t+1}$ and $D_{t+1}$.

For RAS, we create $E_{t+1}$ as shown in Fig 3 in the manuscript. Following that, at task $t+1$, we obtain $M_{t+1}$ by training $M_t$ (with weights $W_t$) on $E_{t+1}$ and $D_{t+1}$.

If we are using EWC, we use $W_t$ and $M_t$ to calculate $F_{t+1}$, where $F$ is the Fisher information matrix using the equation, $F_{t+1} = -E[\frac{\partial^2 }{\partial W_{t}^2}log(M_t(x)|W_t)]$, where $M_t$ is the model of task t and $W_t$ is the weights of task $t$. Now during training of task $t+1$, we add $L_{EWC}$ to the training loss. We calculate $L_{EWC}$ using the equation, $L_{EWC} = \sum_{}^{}F_{t+1}(W_{t+1} - W_{t})^2$. While training for task $t+1$ we modify the loss $L_{t+1} = L_{train} + L_{EWC}$.

If we are using PackNet, after training of task $t$ we take $W_t$ and apply a pruning algorithm to obtain $(W_t)’$, where $(W_t)’$ are the pruned weights. At task $t+1$, we obtain $M_{t+1}$ by training $M_t$ with $(W_t)’$ and training on $D_{t+1}$.

We are also attaching a Figure R1, in the rebuttal pdf to show the entire schematic of CSEGG.

[Y6eG.Weakness.3-Gen Rbbox@K explanation] Gen Rbbox@K is specifically defined for Learning Scenario 3 (S3) to evaluate the CSEGG model’s performance in locating bounding box locations. As described in Section 3.1, S3 consists of 4 tasks, with each task incrementally introducing 30 new objects. The relationship classes remain consistent across all tasks. In S3, there is a standalone test set where novel objects, absent from any training set of S3, appear with the same set of common relationship classes used in training.

During evaluation, to calculate Gen Rbbox@K, for each predicted bounding box, we calculate the Intersection over Union (IoU) between the predicted bounding box and the ground truth boxes. If the IoU exceeds the predetermined threshold, it is considered a true positive (TP). We used IoU thresholds of 0.3, 0.5, and 0.7 for our experiments. If the IoU exceeds the threshold multiple times for the same predicted bounding box, we consider it a single positive prediction. This is because the metric aims to define the model’s performance in locating bounding boxes of objects that have not been learned during training. Counting multiple times for the same box would be misleading, as it would inflate the number of TPs and recall, while the actual number of unknown bounding boxes the model can generate might be low. The total possible positives (TP + FN) are determined by the total number of ground truth boxes in the image.

This metric helps evaluate how well the CSEGG model locates unknown objects within an image. True positives (TP) represent successful identification of the location of an unknown object, while (TP + FN) represents all possible unknown objects the CSEGG model could locate. Thus, object classification labels are not required to calculate Gen Rbbox@K. Instead, we need the total number of ground truth bounding boxes in an image and the number of predicted boxes that meet the IoU threshold as explained in Section 3.3.

[Y6eG.Question.1- Gen Rbbox@K explanation] Same as [Y6eG.Weakness.3]