Adaptive Visual Scene Understanding: Incremental Scene Graph Generation

SDSd.1 Weaknesses: The writing is convoluted in some places. Specially the scene graph generation backbone 3.2 wasn't clear after the first read. The references to CNN-SGG and SGTR aren't separated, and caused a bit of confusion.

We acknowledge the reviewer's feedback regarding the convoluted explanation of the scene graph generation backbones. To provide a clearer understanding, we offer a more concise and separate explanation for each backbone.

For SGTR, the approach uniquely formulates the task as a bipartite graph construction problem. Starting with a scene image (Ii), SGTR utilizes a 2D-CNN and transformer-based encoder to extract image features. These features are then incorporated into a transformer-based decoder, predicting object and subject nodes (Oi). Predicate nodes (Ri) are formed based on both image features and object node features, and a bipartite graph (Gi) is constructed to represent the scene collectively. The correspondence between object nodes (oi) and predicate nodes (rk) is established using the Hungarian matching algorithm (Kuhn, 1955). Experimental results are based on the average over three runs, and the implementation leverages public source codes from (Li et al., 2022b) and (Wang et al., 2021b) with default hyperparameters. Refer to Section A.2.1 for detailed training and implementation specifics.

As for CNN-SGG, it employs Faster-RCNN (Girshick, 2015) to generate object proposals from a scene image (Ii). The model extracts visual features for nodes and edges from these proposals, and through message passing, both edge and node GRUs output a structured scene graph. The training occurs in two stages within a supervised framework. In the initial stage, only object detection losses from Faster-RCNN (Ren et al., 2015) are applied to Oi, incorporating cross-entropy loss for object class and L1 loss for bounding box offsets. In the subsequent stage, the visual feature extractor (VGG-16, pre-trained on ImageNet) and GRU layers are trained to predict final object classes, bounding boxes, and relationship predicates using cross-entropy loss and L1 loss.

We have also clearly separated the explanation of each backbone in the revised main manuscript in Sec. 3.2. We have clearly referenced the training and implementation details for both backbones in the revised main manuscript as well in Sec. 3.2.

FdjE.1 Weaknesses: Clarity on Relationship with SGG Tasks: The paper lacks clarity in establishing the direct correlation between its observations and the Scene Graph Generation (SGG) tasks. While the identified issues like catastrophic forgetting, the efficacy of replay methods, and addressing long-tail problems are extensively explored in existing research, the unique challenges in integrating continual learning, long-tail problems, and SGG remain unclear. The paper falls short in delineating the specific challenges arising from the amalgamation of these factors.

FdjE.1 Questions: Could the authors delve further into elucidating the unique challenges that arise in SGG tasks due to continual learning scenarios? Clarifying these challenges could significantly strengthen the paper's contribution.

We appreciate the reviewer's feedback and would like to provide a detailed response to address the concerns. Challenges faced by us are as follows :-

• Creating diverse tasks for CSEGG poses a non-trivial challenge, primarily due to the intricate relationships between objects and the potential entanglement of tasks. An essential aspect of task creation involves meticulous data analysis to address this complexity. For instance, dividing objects like "Man" and "Horse" into separate tasks can diminish the frequency of the relevant relationship "riding" since a new task may lack either "Man" or "Horse." Blindly partitioning the dataset based on random assignment of objects or relations to tasks risks eliminating numerous pertinent triplets. Careful analysis was indispensable in crafting each learning scenario to ensure the maximum inclusion of relevant training examples in each task. Consider, for instance, a task with objects like "Horse," "Car," and relationships like "on." Such a scenario fails to yield contextually relevant triplets. Therefore, meticulous data analysis was crucial during the creation of each learning scenario. In designing Learning Scenario 1, preserving the long-tailed nature of the dataset across all tasks was a priority. This was imperative to prevent an imbalance where one task would be overloaded with long-tailed classes, leaving the remaining tasks with a shortage of training examples. In Learning Scenario 2, the division of objects and relations was carefully orchestrated to maximize the number of images for both tasks while ensuring the inclusion of contextually relevant triplets. In the case of CSEGG, Learning Scenario 3 involved a detailed analysis to determine the optimal number of objects in each task. This aimed at maximizing the number of training examples and ensuring an equitable distribution of common relations across all tasks. The intricate analysis undertaken to design each task within the learning scenarios is a pivotal challenge specific to CSEGG. Along with this, we overcame implementation challenges as we are the first to implement various SGG backbones in the continual learning setting.
• A significant hurdle we encountered in our study was the absence of continual learning methods specifically tailored to minimize forgetting and optimize the performance of CSEGG models. In response to this challenge, we introduced a novel generative replay method termed "Replays via Analysis by Synthesis" (RAS) detailed in Sec. 4. This method harnesses the capabilities of generative models to artificially produce exemplars representing the classes present in each task. Subsequently, these exemplars are employed to train the model in subsequent tasks, effectively mitigating catastrophic forgetting. The application of RAS is particularly pertinent to SGG, as evident from our observations with CSEGG baselines (which is consistent with continual learning literature covering tasks like detection and classification), where replay methods exhibited superior performance in addressing catastrophic forgetting. However, practical considerations, such as limited storage and privacy concerns in real-world scenarios, posed challenges for traditional replay methods. To address these limitations, we adopted a generative replay approach that generates images solely from the textual descriptions of triplets in each task. This not only enhances user privacy but also allows for the generation of a flexible number of images, eliminating storage constraints as these images are used once and not stored. A unique challenge specific to CSEGG arose during this process. In contrast to image classification scenarios where generated images typically require no annotations, the case of SGG demands annotations for object locations, object labels, and relation labels. To surmount this challenge, we leveraged each task's trained model to generate annotations for the synthetically created images. This combined notation, along with the generated image, formed the basis for training the SGG model, effectively addressing the issue of catastrophic forgetting in the CSEGG context.

pXA5.1 Weaknesses: It seems the organization and writing are quite disordered and difficult to follow. For example, the authors claim scenario 1 has 5 tasks. However, detailed definitions of these tasks are missing. Similar problems exist for the other two scenarios.

We apologize to the reviewer for any confusion resulting from the organization and writing of the manuscript. The meaning of each learning scenario is now clarified in Sec. 3.1 of the main manuscript. Detailed definitions of all tasks for each learning scenario, along with data statistics, are provided in Sec. A.1 and Sec. A.3 and visualized in Figures S4, S5, S6. We acknowledge the oversight of not referencing these details in Sec. 3.1 and have rectified it in the revised manuscript by adding references to the detailed definition section for each learning scenario in Sec. 3.1.

pXA5.2 Weaknesses: The first contribution seems to over-claim. It seems the images, object classes and relationships inherit from the visual genome dataset. I do not know what is new.

We acknowledge the reviewer's concern regarding the first contribution and appreciate the opportunity to clarify. Our primary contribution, as outlined in "Main Contribution 1" in the main manuscript, is not centered around providing a new dataset with images and notations for Scene Graph Generation (SGG). Our focus uniquely centers on the intricacies and dynamics of continual learning within the domain of Scene Graph Generation (SGG). By introducing innovative splits for Visual Genome (VG), we have tailored the dataset to better support the demands of continual learning. Consequently, our efforts extend the capabilities of VG, resulting in the creation of the novel dataset, CSEGG. This dataset stands as the pioneering resource designed specifically for examining the nuances of continual learning within the realm of SGG.

The challenge arises from the inherent long-tail nature of the VG dataset, making it non-trivial to divide. Our main contribution lies in carefully designing and creating the splits for each continual learning scenario, addressing the long-tailed distribution within VG. This involves selecting combinations of triplets that enable meaningful training samples for each task, ensuring the model can effectively learn from the dataset. For example, in creating Learning Scenario S3, detailed data analysis was conducted to identify splits that maximize common relations across tasks while also providing testing samples with entirely new objects and the same relations.

Along with specific data division, the design of the individual scenarios is quite unique and specific to study a fixed purpose. Learning Scenario 1 (S1) simulates a real-world scenario where an agent learns new relations over time, focusing on the continual learning impact on SGG when introducing new relations. This is essential for a model to identify new relations between known objects without relearning them, as detailed in Sec. A.1.1. Learning Scenario 2 (S2) replicates scenarios in which new objects and relationships are gradually introduced over time. This study is crucial for real-world applications, ensuring a model can perform a specific task with new objects and relationships, as discussed in Sec. A.1.2. Learning Scenario 3 (S3) evaluates the model's generalization capability, testing its ability to apply existing knowledge to unseen data. This is vital for practical applications, such as a robot using existing spatial relation knowledge to navigate unfamiliar objects, detailed in Sec. A.1.3.

In summary, our contribution is not the images or notations themselves but rather the thoughtful design of the continual learning scenarios and the specific data division within VG to analyze CSEGG models under the challenging conditions posed by the dataset's long-tailed nature.

LwTF.1 Weaknesses: The authors missed connections to the meta-learning and curriculum design literature. From that lens, claims such as "CSEGG methods improve generalization abilities" seem a bit unsurprising.

We highly appreciate your detailed feedback, and we would like to provide additional clarification on how our work distinguishes itself from traditional meta-learning and incorporates curriculum design elements.

Our primary focus is on evaluating the generalization testing performance of CSEGG models, specifically their ability to perform on unseen data. As outlined in Sec. 3.1, In S3 our models are trained across four tasks, each introducing 30 new object classes and a consistent set of 35 relationships. Notably, the object classes in each task are unique and not repeated in any other task. The test set is a standalone set comprising 30 objects not present in the training data, with only the relationships being common across all tasks and the standalone test. This design allows us to assess the model's capability to identify known relationships between objects it has never encountered during training. Importantly, our approach does not involve task adaptation during training and our model undergoes no training on data from the same domain as the test data, highlighting its domain independence and setting it apart from traditional meta-learning methods

Furthermore, we have incorporated the impact of curriculum learning in Learning Scenario 1 (S1). In Sec. A.5.4 of the manuscript, we discuss the effect of curriculum learning in S1, identifying that curriculum learning affects the performances of CSEGG models as seen in Fig. S16. It also aligns with the conclusions in [1], where curriculum learning was found to have a substantial effect on class-incremental learning. We hope these clarifications address your concerns and provide a more comprehensive understanding of our approach.

[1] Parantak Singh, You Li, Ankur Sikarwar, Weixian Lei, Daniel Gao, Morgan Bruce Talbot, Ying Sun, Mike Zheng Shou, Gabriel Kreiman, and Mengmi Zhang. Learning to learn: How to continuously teach humans and machines. arXiv preprint arXiv:2211.15470, 2022

LwTF.2 Weaknesses: The dataset is somewhat small in scale. 150 objects and 50 relationships might not be enough to pose a sufficient continual learning channel. The benchmark also reuses images between tasks in a learning scenario, which is not ideal.

We respectfully disagree with the reviewer's comments on the scale of the dataset. The term "small" is relative and context-dependent. For instance, MS COCO [1] is considered a substantial dataset in computer vision, playing a crucial role in numerous advancements in the field. While it may be deemed "small" when compared to ImageNet [2], its size is still significant and allows for quality research and development.

Additionally, our choice of the Visual Genome (VG) dataset [3] is intentional and aligned with the specific requirements of our research. VG is a dedicated scene graph generation dataset designed for tasks similar to SGG. Our aim is to explore the effects of continual learning on SGG, a novel approach that hasn't been extensively studied before. The size of the dataset is not necessarily a limitation; rather, it aligns with the scope of our research objectives. VG provides a sufficient basis for studying continual learning in SGG, allowing us to investigate the model's performance across different tasks.

The reuse of images between tasks is a deliberate choice, as explained in Sec. A.1.5 of our manuscript. We draw parallels to human learning scenarios, where a parent guides a baby to recognize various objects in a bedroom. Despite encountering the same scenes, the focus is on continual learning, instructing the baby to detect and recognize individual objects sequentially. This intentional reuse of images serves as a valuable aspect of our experimental design, reflecting real-world learning scenarios.

[1] Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014. [2] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009. [3] Krishna, Ranjay, et al. "Visual genome: Connecting language and vision using crowdsourced dense image annotations." International journal of computer vision 123 (2017): 32-73.

TSA3.1 Weaknesses: The overall presenation of the paper is difficult to follow and not organized well. (Also too many bold letters for section and figure names)

We apologize to the reviewer for any confusion resulting from the organization and writing of the manuscript. We have addressed the issues in the revised manuscript.

TSA3.2 Weaknesses: The authors has proposed three learning scenarios and 8 evaluation metrics. All the learning scenarios has multiple tasks (data separations). However, given that there is a lot to report and include, the results are not summarized well in a tabular form. And it is very difficult to follow how each component is contributing in different metrics and leanring scenrios over the tasks.

We apologize to the reviewer for any confusion resulting from the organization and writing of the manuscript. We have addressed the issues in the revised manuscript. We have included Tab. 1 giving an overview of all learning scenarios.

TSA3.3 Weaknesses: Most of the results are written in textual description. Summarizing them in a tabular form and discussing the interesting finding might help the readers to understand the numbers better

We apologize to the reviewer for any confusion resulting from the organization and writing of the manuscript. We have addressed the issues in the revised manuscript. We have also included results of S1 and S2 in tabular form in Tab. 2 with all the metrics.