Multiview Scene Graph

Q4: Experiment. The baselines in the paper were from early ages

A4: We thank the reviewer for this question. All the baselines are from recent years. The proposed MSG task involves two parts of baselines: visual place recognition (VPR) and object association. For the VPR part, our NetVlad baseline is adapted from the widely used baseline deep VG benchmark [1] (reference [8] in the manuscript), which was published in 2022. Anyloc was published in 2023. For the object association part, UniTrack is a widely used multi-object tracking method released in 2021. DEVA is from ICCV 2023.

We appreciate the reviewer’s suggestion and have also evaluated optimal transport aggregation (Salad) [2] as an additional and more recent baseline for the VPR part. The results are reported below. We find its performance comparable to Anyloc on the recall while slightly better on PP IoU. We note that both of the baselines are evaluated off-the-shelf.

We would like to emphasize that the baseline methods we proposed (AoMSGs) are designed to be straightforward and easily extensible. Future work can incorporate more advanced techniques and insights from the VPR and object association fields, including but not limited to the Sinkhorn algorithm[3] used in the Salad paper. It would be exciting to explore more ways of combining the wisdom in both place and object association fields for MSG in future work.

[1] Berton, Gabriele, et al. "Deep visual geo-localization benchmark." CVPR. 2022.

[2] Izquierdo S, Civera J. Optimal transport aggregation for visual place recognition. CVPR 2024.

[3] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.

Q5: Do the authors want to modify the title a bit?

A5: Thank you for the question. We chose “Multiview Scene Graph” as the title because we believe it best describes the contribution of this work while maintaining conciseness. A multiview scene graph refers to our proposed task, where a place-object graph is inferred by associating unposed images across views, predicting their topological connectivity, and associating objects across multiple views simultaneously. So multiview association is the key characteristic and essence of the proposed scene graph and the task and we believe that naming the work “Multiview Scene Graph” highlights its core focus and makes it both memorable and easy to understand.

However, we are more than happy to consider any advice on better titles from the reviewer.

Q6: What Object Detector is used in figure 2?

A6: Any detector can be used as long as it proposes object bounding boxes. The detector is frozen and bounding boxes are the only thing our method needs from the detector. For training, we use groundtruth detection for efficient supervision. We also used Grounding DINO as our off-the-shelf object detector for open-set detection and it shows consistent performance order for all the methods.

Q7: The README is a void file in the supplementary materials.

A7: Thanks for pointing it out. We deleted README while trying to keep anonymity. We will upload the complete code base and model checkpoint when our paper is camera-ready.

Q8: typos.

A8: We thank the reviewer for pointing this out. We will fix all typos for the camera-ready version.

Q1: the current scene graph extracted from multiple views is a bit simple

A1: We acknowledge the fact that the proposed MSG has a rather simple format as it is free of spatial or high-level semantic relationships between objects. However, this does not mean this task is simple, insignificant, or can be readily solved. As acknowledged by reviewer 2 (AiA2) and demonstrated by the baseline performances in the paper, building topological connectivity between the unposed images and associating objects across frames and long stretches of time without observation is a challenging and important task. Constructing such multiview scene graphs from unposed images is fundamental to many tasks, such as 3D reconstruction[1], visual navigation[2], visual slam[3], etc. Having such a graph can be beneficial to all these applications.

Furthermore, the proposed MSG is complementary to the existing scene graphs. Edges for object-object relationships can be a seamless add-on to extend the MSG with more semantic information. Therefore, we believe our work adds a meaningful contribution to the scene graph community.

We also demonstrate in the next response that current Vision-Language Models (VLMs) are not yet capable of solving the MSG task.

[1]Xiao, Jianxiong, Andrew Owens, and Antonio Torralba. "Sun3d: A database of big spaces reconstructed using sfm and object labels." Proceedings of the IEEE international conference on computer vision. 2013.

[2] Chaplot, Devendra Singh, et al. "Neural topological slam for visual navigation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[3] Salas-Moreno, Renato F., et al. "Slam++: Simultaneous localisation and mapping at the level of objects." Proceedings of the IEEE conference on computer vision and pattern recognition. 2013.

Q2: Do authors think one can leverage a VLM to do a similar job?

A2: We appreciate the suggestion. VLMs exhibit strong emergent abilities for many tasks, and we agree it would be interesting to try them out for MSG. However, querying VLMs with every image pair is a huge amount of work and cost, and sending all images together poses a challenge to the context length limit while also hurting performance. Therefore, we conducted a case study with one scene as a pilot study.

Specifically, we sampled a scene with a relatively small number of images and further subsampled all the images containing annotated objects, resulting in 22 images in total. We then queried the GPT-4o[1] 231 times with each image pair annotated with object bounding boxes as visual prompts, the corresponding box coordinates, and the task prompt. By parsing the GPT-4o outputs, we obtained the following results:

The model total represents the performance of our model on the entire scene, and model adjusted represents the performance evaluated only on those 22 subsampled images for a fair comparison. Besides the issues with computation cost and context limits, we note that a common failure pattern of VLM is the failure to maintain consistent object associations. The limitations of VLM in VPR are also discussed in the literature [2].

Nevertheless, we acknowledge that this is only a small-scale pilot study. It is well possible to have better VLMs in the future and come up with better prompts, and we are excited about the future possibilities of VLM + MSG.

[1] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

[2] Lyu, Zonglin, et al. "Tell Me Where You Are: Multimodal LLMs Meet Place Recognition." arXiv preprint arXiv:2406.17520 (2024).

Q3: do the authors think the proposed graph can somehow have a better ability to evaluate these tasks (VPR and object association)? … Motivation and usage. Can existing scene graph formats do similar tasks in this paper?

A3: Thank you for the question. The proposed MSG encompasses both visual place recognition (VPR) and object association. Our proposed evaluation metrics, PP IoU and PO IoU, reflect the quality of place recognition and object association. Our proposed baseline combines the two tasks in a joint architecture. Therefore, we believe training and evaluating on the task of MSG will help both VPR and object association.

We are motivated by the fact that building spatial correspondence by associating images and objects is fundamental for scene understanding. The MSG takes unposed RGB image sets as input, allowing it to be applied to any arbitrary image sets and video streams. As discussed in Q1, obtaining MSG from the unposed RGB images will provide a foundation for many downstream tasks in computer vision and robotics, such as loop closure, SfM, visual SLAM, navigation, and mobile manipulation.

Existing scene graphs focus on describing the spatial and semantic relationships between different objects, which is different from the MSG task. Meanwhile, existing 3D scene graphs require having 3D scene representations before obtaining the scene graphs. This, where object association has been given, is already a good starting point for many vision tasks. However, such 3D scenes are not always available or can be reliably obtained from unposed RGB observations. Therefore, we believe our MSG is a complementary contribution to the existing scene graph literature.

Continue in the official comment

Dear reviewer o1XP

Thanks for your reviews, which are valuable for improving the overall quality of this manuscript.

To address your concerns, we have added discussions and explanations of the definition and format of our proposed Multivew Scene Graph (MSG) and how it is different from other existing formats of scene graphs. We have also discussed the potential applications and benefits of the MSG.

For the experiments, we appreciate your constructive suggestions and have conducted an additional baseline based on the recent VPR literature you recommended. We will also add these references to the revised version. We would also like to thank you for suggesting the idea of using VLM. We like this idea and conducted a pilot study. We have presented the results and discussion in the response above as well as in the attached PDF file.

We have also provided answers and clarifications to the other questions of your concerns. We thank you for pointing out the typos and we promise to fix them in the revised version. We have also explained our motivation for choosing the title and we would love to hear from you for any advice on this.

Could we kindly ask if our responses have addressed your concerns and if you have any new questions? Thanks for your time and effort in reviewing our submission.

Authors of Submission ID 18923

Q5: Seeing the gap between w/ GT detection w/ GDino, it seems a better detector is more important than the proposed detector itself, have you tried more powerful detectors?

A5: Our MSG task and model tackle object association rather than object detection. Association happens after object detection and refers to identifying whether the object detected in one image is the same object detected in another. While it is true that a better detector will enhance the overall performance of MSG, it is inaccurate to say that the detector is more important than MSG and object association itself, as they focus on different tasks.

The gap between ground truth detection and Grounding Dino (GDino) is expected since GDino is used off-the-shelf and in a zero-shot manner. Our aim is to demonstrate that although MSG uses ground truth detections during training, any available detector can be incorporated into the framework, and the performance order remains consistent. Thus, we anticipate that a more powerful detector than GDino or one fine-tuned on the same data should yield better results.

The task focuses on associating places and objects, and we believe having ground truth detections and consistent results with the commonly used GDino is sufficient to convey our contributions. We appreciate the reviewer’s suggestion and will consider training or benchmarking detectors in future explorations.

Q6: if overlapped unposed images are needed:

A6: No, we do not make such overlapping assumptions for the unposed images. The input of the MSG task is just a set of unposed RGB images. The model learns to figure out whether the images are taken from the closeby positions and associate the object appearances, i.e. whether they are the same object. The input of the entire process is simply unposed RGB images, with no overlapping required.

Q7: Have you tried SAM? do you think DinoV2 performs best because the detector's feature is close to the image encoder when using DINOv2 as the image encoder?

A7: Yes, our baseline DEVA uses SAM to obtain object segmentations and perform video segmentation by tracking these object segments across the frames. Detector’s features are not used in our model. The detector only provides bounding boxes. Grounding Dino is different from DINOv2 although bearing a similar name. The former is a Transformer-based detection model and the latter is a Transformer-based self-supervised vision pretraining model. DINOv2’s features have shown great performance in downstream tasks, such as in Anyloc. In this work, we have a similar observation that DINOv2 as the encoder backbone produces the best performance. We think the reason for its strong performance is possibly due to its vast and diverse pretraining data.

Q1:The task is not novel, see EgoSG.

A1: Thank you for the question. We will cite EgoSG as a related work. However, while both carry “scene graph” in the names, the MSG task is completely different from the EgoSG and other existing 3D scene graphs.

Firstly, the definition of the graph is different. The existing scene graph connects objects and describes their relationships with the graph edges. We propose MSG as place-object graphs, where edges between images reflect the topological connectivity, and edges between objects and images reflect the object locations after resolving the object association. As reviewer 2 (AiA2) acknowledges, associating objects across frames and long periods without observation is a challenging and important task that does not truly require 3D information. We believe introducing and studying the multiview unposed scene graph will provide a meaningful topological scene representation beneficial to other downstream tasks in computer vision and robotics, such as loop closure, SfM[1], visual SLAM[2], visual navigation[3], and mobile manipulation.

Secondly, the input data format is different. EgoSG and other 3D scene graphs require having 3D scene representations before obtaining the scene graphs, which is already a very good starting point for many vision tasks and is not always available. Our proposed MSG focuses on a different challenge. The input is purely unposed RGB images without any additional spatial knowledge. MSG exactly aims to extract topological spatial knowledge from these inputs. This allows it to work on arbitrary videos when depth information is not given or needs to be estimated and lays a good foundation for reconstructing 3D information from 2D inputs. While estimating depth and poses to convert the problem into RGB-D could improve performance, it does not solely solve our task as the estimation itself can introduce errors. As suggested by the reviewer(discussed in the following response Q3A3), we have tried a recent strong pose estimation method [4] and found the performance less satisfactory due to the accumulated drifting pose errors. The loop closure provided by the MSG will in fact help better estimate poses.

Additionally, our MSG will be complementary to the existing scene graphs. Edges for object-object relationships can be a seamless add-on to extend the MSG with more semantic information. Therefore, we believe our work adds a meaningful contribution to the scene graph community.

In conclusion, we believe this is a novel task different from the previous scene graphs and also important for many downstream applications in computer vision and robotics. This paper contributes by introducing the MSG task, the set of new baselines, and benchmarks to stimulate future research.

[1]Xiao, Jianxiong, Andrew Owens, and Antonio Torralba. "Sun3d: A database of big spaces reconstructed using sfm and object labels." ICCV. 2013.

[2] Salas-Moreno, Renato F., et al. "Slam++: Simultaneous localisation and mapping at the level of objects." CVPR. 2013.

[3] Chaplot, Devendra Singh, et al. "Neural topological slam for visual navigation." CVPR. 2020.

[4] Barroso-Laguna, Axel, et al. "Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences." CVPR. 2024.

Q2: how was ViT pre-trained:

A2: Thank you for the question. We will update the manuscript with this information. For the ViT model, we are using the base model with the default weights pretrained on ImageNet-21K. It is not a masked auto-encoder (MAE). MAE is also an important image pretraining model. We would like to note that any visual encoders that can produce feature maps or tokens are adaptable to our method. However, we did not sweep through more available choices as we found that DINOv2 gives significantly better performance.

Q3: other baselines with existing methods.

A3: Thank you for the suggestion. We agree adding more baselines linked to existing tasks and methods will benefit our work. Therefore, We have added a new baseline based on pose estimation. We are happy to explore other possible directions in the future. Specifically, We use a pretrained Mickey model [4] and provide the image data sequentially in temporal order and also the corresponding intrinsics ( not provided to MSG). We compute relative poses and convert them to absolute poses w.r.t the first frame. Then use the same threshold as the data set hyperparameter to obtain the p-p adjacency matrix and P-P IoU. The results are listed in the table below. We see that the performance is close to that of Anyloc. Recall@1 is easily 1.0 since the data is given in temporal order, and consecutive frames are trivially recalled. But its P-P IoU is inferior to Anyloc, suggesting the drifting issue of estimated poses and the need for loop closure, which is precisely MSG’s strength.

[4] Barroso-Laguna, Axel, et al. "Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences." CVPR. 2024.

Q4: why explicit leveraging detection predictions by cropping features? What if you just feed in the image features? How much does "explicit leveraging detection results" help?

A4: The task of MSG involves associating objects from images across views. Therefore, we leverage detection bounding boxes to crop features to obtain features of object appearances. If directly feeding the image features as a whole, it would be difficult and unnatural for object association since the model would have no prior knowledge of what are the interested objects. So explicitly leveraging detection results will help the model locate the object appearances in the input images and associate them to build the multiview scene graph.

Continue in the official comment.

Dear reviewer 9wC7

Thanks for your comments, which are valuable for improving the overall quality of this manuscript.

To address your concerns, we have provided a thorough discussion and explanation of similarities and differences between our proposed Multivew Scene Graph (MSG) and other existing formats of scene graphs such as EgoSG. We really appreciate your thoughts and will include this in our references as well as the discussion of it in the related works section in the revised version. We have also conducted an additional baseline based on pose estimation for the place recognition part of MSG, thanks to your suggestion.

We have also provided a thorough explanation to the other questions of your concerns, such as the use of the detectors in our proposed method, the design choices, encoder choices, and image choices.

Could we kindly ask if our responses have addressed your concerns and if you have any new questions? Thanks for your time and effort.

Authors of Submission ID 18923

Q1: Example of qualitative experiments on some arbitrary video sequences.

A1: Thank you for the great suggestion! We have self-recorded an unposed video with an iPhone in a household scenario and run our trained AoMSG model with a pretrained Grounding DINO detector on it. In the attached PDF file we show some resulting images with object instance id labeled. Results suggest that our model is able to obtain sensible outputs on arbitrary videos outside of the dataset. We have also made a simple frontend tool for interactive visualization with MSG, which we show a screenshot in the PDF. We will release the tool along with the full source code when camera-ready.

Q2: How to train the model without poses or other 3D annotations.

A2: Thank you for the question. Like many other new tasks, certain requirements of annotation exist in the current MSG task. We rely on explicit camera poses and 3D object annotations in the ARKitScene to generate our dataset since marking the same places and objects requires knowing camera poses and object instances. We note that any 3D dataset or environment providing such annotations can be leveraged for the MSG task and there are many large-scale 3D datasets available for use such as ScanNet [1], ScanNet++[2], and HM3D[3]. Nevertheless, we acknowledge that this is a limitation and in the future, we will try to explore training on a combined recipe of different datasets for strongly generalizable performance or explore training with less annotation.

[1] Dai, Angela, et al. "Scannet: Richly-annotated 3d reconstructions of indoor scenes." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[2] Yeshwanth, Chandan, et al. "Scannet++: A high-fidelity dataset of 3d indoor scenes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Ramakrishnan, Santhosh K., et al. "Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai." arXiv preprint arXiv:2109.08238 (2021).

Q3: What is the benchmark VG Pipeline

A3: We thank the reviewer and apologize for the confusion. VG stands for Visual Geo-localization[1] which refers to the same task of Visual Place Recognition (VPR). This pipeline is referenced as [8] in the manuscript. We will revise this line in the manuscript for better clarity.

[1] Berton, Gabriele, et al. "Deep visual geo-localization benchmark." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Q4: Encoder of SepMSG

A4: Thank you for the question. To clarify, SepMSG baselines use the same image encoder as the AoMSG methods. We use pretrained encoders and keep them frozen while training all the methods. Specifically, SepMSG-direct refers to directly evaluating the output feature vector of the encoders, SepMSG-linear/MLP trains linear probing or MLP on top of the encoders and AoMSG trains the Transformer-based decoder. We empirically find that the pretrained DINO-base model produces the best performance in direct evaluation, so all the experiments in Table 1 use it as the baseline. Below is a table for the detailed performance of different backbone choices when used in SepMSG-direct. We will revise the manuscript with better clarification and the results.

Q5: if features just learn those spatial thresholds? Calculate the relative pose distribution

A5: Thank you for the great suggestion. Annotating based on spatial thresholds is a conventional setup in visual place recognition (VPR) tasks and we chose to follow this convention. While training the features, we adopt a metric learning approach instead of simple binary prediction. During training, embeddings positive pairs are drawn closer in terms of cosine similarity, while negative pairs are pushed further. We also monitor the training dynamic using the Total Coding Rate as described in [1] to alert possible feature collapse. Thus, the learned features are not confined to just those spatial thresholds.

We have also added 4 plots for relative pose distribution (both orientation and translation) in the attached PDF file. They are obtained by taking the model’s predictions on the test set and calculating the histograms of the relative orientation and translation differences of the predicted connected and not-connected places. From the plots, we can see that while the model makes some mistakes (currently the PP IoU is around 0.40), the distribution difference between the connected and the not-connected places is clear. There is a clear yet smooth separation at the spatial thresholds on all the plotted distributions.

[1] Yu, Yaodong, et al. "Learning diverse and discriminative representations via the principle of maximal coding rate reduction." Advances in neural information processing systems 33 (2020): 9422-9434.

Q1: The current task and model do not seem to allow for an adjustable "place" definition beyond the train set constraints

A1: Thank you for the question. The thresholds are dataset hyperparameters which is a conventional setup in visual place recognition (VPR) tasks and datasets [1, 2]. Since our MSG task involves place recognition, we choose to adopt this convention. VPR tasks require a model to classify whether two images are taken from the same place or not. The concept of “place” is a discretization of the space which is continuous by nature, which necessitates the use of thresholds in the VPR setup.

The proposed baseline model conducts supervised training, so it is possible to set different hyperparameters for different datasets and train the model on multiple datasets. To give a closer look at the effect the threshold has on the model, in the attached PDF file, we also included figures of relative pose distributions (orientation and translation) for the connected and non-connect nodes based on our model’s prediction. The figures show that instead of collapsing to only represent the fixed thresholds, the pose distributions have a clear yet smooth separation across the spatial thresholds.

[1] Lowry, Stephanie, et al. "Visual place recognition: A survey." ieee transactions on robotics 32.1 (2015): 1-19.

[2] Zaffar, Mubariz, et al. "Vpr-bench: An open-source visual place recognition evaluation framework with quantifiable viewpoint and appearance change." International Journal of Computer Vision 129.7 (2021): 2136-2174.

Q2: Give further insights about the implication of the proposed metric … provide failure cases that result in low IoU scores.

A2: Thank you for the suggestion. We have added visualizations of some failure cases in the attached PDF file. We observe that most failure cases can be attributed to having very similar visual features with relatively large pose differences (false positives), such as observing a room from two opposite sides, or having fewer similar visual features with relatively smaller pose differences (false negatives). We note that the recall metric, conventional in VPR, is straightforward and effective for image retrieval against a database. However, it falls short in reflecting challenging false positives and negatives, especially when constructing a topological graph like MSG where the number of positives varies. This highlights the usefulness of our proposed IoU metric, which consistently evaluates the quality of the graph.

Q3: Missing details in Figure 3.

A3: Thank you for pointing it out. We made a mistake with the numbers and y-axis. We have uploaded a new figure with more information in the attached PDF file to replace the original one. The numbers marked as direct are obtained by directly evaluating the pretrained encoders corresponding to the SepMSG-direct baseline. Those marked as AoMSG are obtained by training the AoMSG-4 model on top of the frozen encoders, following the same setup as the main experiments in Table 1 in the manuscript. We chose the DINOv2 base as the default encoder for our main experiments as it gives the best performance when evaluated directly. We will revise the manuscript with the updated Figure 3.