CityAnchor: City-scale 3D Visual Grounding with Multi-modality LLMs

We sincerely appreciate the reviewer's insightful comments and time dedicated to evaluating our work. We address all concerns below.

Q1: Limitations of 2D Mapping: Projecting 3D space onto a 2D map often results in a loss of spatial information, particularly in cases of occlusions or overlapping instances. This pipeline, especially the CLM component, operates under the significant assumption that all objects are nearly flat, which inherently limits its applicability to more complex environments, such as indoor spaces or densely populated urban areas like NYC.

A1: Thanks for pointing out the limitations of 2D mapping. We agree that the proposed method using 2D projection works in most cities similar to our two datasets but not in extremely complex 3D cities like NYC. Current existing 3D visual grounding only focuses on small-scale scenes like a room while our method CityAnchor extends our baseline methods to real city-scale visual grounding, which is a substantial step toward intelligent geospatial analysis. We agree that generalizing such visual grounding to complex 3D cities like NYC could be an important and interesting future topic. Though the current CLM may not be applicable to such complex 3D cities, incorporating an LLM like our FMM or a more generalized CLM is still a feasible and promising starting point for these complex 3D cities. We have added these discussions in the revision of Sec.5 (Limitations).

Q2: Dependence on Point Segmentation Model: The pipeline requires users to first segment the scene point cloud into objects using a pretrained 3D segmentation model. As a result, it heavily depends on the granularity and accuracy of this pretrained model, setting an upper performance limit for the proposed approach.

A2: Thanks for pointing out the CityAnchor's dependence on a pretrained 3D segmentation model. Below, we provide a detailed response to address this concern.

a) CityAnchor has the flexibility in selecting 3D segmentation models. CityAnchor follows the CityRefer to segment the scene point cloud into objects using a pretrained 3D segmentation model but can be incorporated with different instance segmentation methods. We do not rely on any specific type of instance segmentation algorithms.

b) CityAnchor has the potential to generalize to unknown objects. In the revision, we additionally conduct a generalization ability experiment to assess CityAnchor's ability to recognize unknown objects in Figure 8 of the Appendix. On the CityRefer dataset, CityAnchor is trained with only four object categories: Building, Car, Parking, and Ground, and we introduce three objects of unknown object categories (Road Intersection, Woodland, and River) that CityAnchor has not seen before. We show that CityAnchor is still able to determine a reasonable matching score between the text descriptions and the given 3D objects due to the strong generalization ability of LLMs, even though these objects may not be well segmented by the 3D segmentation model.

Q3: Justification for the Proposed Dataset: While introducing a new dataset is valuable, the necessity of the CityAnchor dataset for this task or the proposed method requires further clarification. It is essential to address whether the method’s performance would significantly degrade without this dataset, whether it enhances diversity or robustness, and if it offers any unique attributes that the CityRefer dataset lacks. Providing these details would strengthen the contribution of the dataset to the paper.

A3: Thanks for your comments. We utilize the CityAnchor dataset to illustrate the effective design of our method because 1) the CityAnchor dataset has a wider range of object categories, covering nine categories including Building, Vegetation, Aircraft, Truck, Vehicle, LightPole, Fence, StreetSign, and Bike, which demonstrates the effectiveness of the grounding algorithm on more categories. 2) the CityAnchor dataset is based on synthetic data and thus has accurate instance labels, which helps us evaluate the visual grounding more accurately without other distracting factors. Thus, we have provided this dataset for the evaluation of our visual grounding performance.

Q4: Inference time and hyperparameter tuning: These two limitations were briefly discussed in the paper.

A4: Thanks for your comments. We have quantitatively discussed inference time and hyperparameter tuning for RoI threshold as much detail as possible in the main paper. Below, we provide a detailed response to clarify these points.

a) Inference time. We have discussed the inference time in Table 2 and Table 3 of the main paper. CityAnchor takes only $32.45s$ and $51.72s$ to localize an object in a scene of the CityRefer and CityAnchor dataset, which is $1.3\times$ and $1.8\times$ faster than the CityRefer method, respectively. This is mainly attributed to our coarse-to-fine strategy for filtering out most irrelevant objects. Additionally, utilizing the CLM in CityAnchor accelerates inference time by $2.5 \times$ compared to the configuration without CLM.

b) RoI threshold in hyperparameter tuning. As shown in Table 4 of the main paper, we have discussed the RoI threshold in hyperparameter tuning. A low threshold results in numerous candidate objects being involved in fine-grained comparisons, thereby leading to long grounding times. Conversely, setting a stricter threshold results in the exclusion of correct target objects, leading to a decrease in grounding accuracy. To strike a balance between grounding accuracy and time consumption, we select $0.3$ as the RoI threshold for the CLM in the first stage.

From the above analysis, our CityAnchor is faster and better than all the baseline methods, because the CLM effectively filters out erroneous objects.

Q5: Effectiveness of CLM: In Figure 4, it appears that CLM is not highly effective, as it predominantly highlights houses and roads. Could the authors provide additional results or offer an explanation for this behavior?

A5: Thanks for your comments. The rasterized map shown in Figure 4 is mainly dominated by small objects, which shows the limited effectiveness of CLM in this case. As illustrated in Figure 2 of the Appendix, we provide additional qualitative results of our method on the CityRefer dataset in "Novel Objects" setting, which shows that CLM can effectively filter our erroneous objects.

As illustrated in Figure 6 of the Appendix, we provide heat map visualizations for both simple and complex query texts, which shows that CLM performs reasonably even on complex texts. We have shown in Table 3 of the main paper that CLM significantly reduces computation time while improving grounding accuracy, with 46.86% of Acc@0.50 and only $32.45s$ in localizing an object.

The coarse localization module (CLM) is designed to filter out irrelevant objects, thereby enhancing the model efficiency in city-scale visual grounding tasks. However, we must acknowledge that the filter effectiveness can vary depending on the size and attribute of target objects. Large objects (e.g., factories, athletic fields, parking lots) can be easily identified and excluded through the heat map output by CLM. In contrast, small objects (e.g., cars, residential buildings) are more challenging to distinguish, requiring further detailed comparison in the fine-grained matching module (FMM).

We have revised the paper to include a discussion of this phenomenon in Sec.5 (Limitations) and plan to address these issues in future work.

Q6: Prompt Engineering Requirements: It seems that users must invest effort in engineering their prompts to achieve accurate results, which may pose usability challenges. The authors could consider conducting a user study to gather prompts from a diverse set of users to assess the method’s robustness when different descriptions target the same object.

A6: Thanks for your comments. We have performed experiments to analyze how specific parts of text prompts influence grounding performances, which serve as the reference in prompt engineering for different categories of objects. Additionally, we assess the method’s robustness when diverse text prompts from different users target the same object.

a) Analysis for specific parts of text prompts

As shown in Figure 6 of the main paper, we visualize the grounding results of systematically removing text descriptions related to color, shape, and neighborhood contextual information. Color description plays an important role in visual grounding and the lack of color information often leads to incorrect results. In contrast, while shape descriptions are less critical for cars, they become more significant for buildings due to the larger variability of shapes. In a scene with many similar objects (e.g., red cars), color and shape descriptions are inadequate to distinguish these objects, and integrating neighborhood contextual information is vital for achieving accurate grounding.

b) Analysis for diverse text prompts from different users

As illustrated in Figure 7 of the Appendix, we provide qualitative results for diverse text prompts toward same object. In addition to thorough object feature extraction, the text description plays a crucial role in visual grounding. For the successful grounding cases, text descriptions related color and category are often indispensable.

Q7: Progressive Prompting Approach: It would be intuitive to structure prompts in a coarse-to-fine manner—beginning with a broad query that identifies multiple potential candidates and gradually introducing constraints to precisely locate the target.

A7: We greatly appreciate the reviewer’s insightful suggestion in the progressive prompting approach, and it holds great potential for enhancing the usability and flexibility in city-scale grounding.

While current CityAnchor employs a two-stage pipeline for coarse localization and fine-grained matching, integrating progressive prompting approach offers a promising area for future work. This process could begin with a broad query to identify multiple potential candidates, followed by progressively refining constraints (such as category, spatial relation, attribute cues) to precisely locate the target object.

For example, consider a city scene where the user queries, “The tall and white-roofed building near the One Stop Building (Landmark)”. In a progressive prompting query, we might have the following steps:

(1) Initial Prompt (Broad Query): “The building in this city scene”. The system identifies all the building candidate objects in the city-scale point cloud.

(2) Refined Prompt (Spatial Constraint): “The building near the One Stop Building”. The system filters most of the building candidate objects based on their proximity to the “One Stop Building”, progressively localizing the target object.

(3) Final Prompt (Attribute Matching): “The tall and white-roofed building”. Additional attributes (e.g., roof color or specific textural features) could further confirm the target object among the filtered building candidate objects.

Progressive prompting aligns naturally with human reasoning and interaction patterns, making it an inspiring way to enhance CityAnchor’s adaptability in real-world scenarios.

Thank the authors for the detailed and comprehensive response! I have some follow-up questions. Re: A2(b) The setting of this experiment is a bit confusing to me. I cannot tell if you are only testing the FMM, or the entire pipeline. My question is, it seems CLM relies on off-the-shelf point cloud seg model a lot, so if CLM cannot identify the novel categories, there's not much FMM can do. A most straightforward way to proof the generalizability of your approach might be something like in Fig.4, but with prompts of novel objects/neighborhood/landmarks.

Also a suggestion for figures like Fig.4, Fig.A1, Fig.A2, it would be better if authors can also draw a target box on CLM results. Right now I'm having a hard time finding the query object on the map (i.e. cannot verify if CLM is able to correctly include the target in the candidates)

As the author-reviewer discussion period approaches its end, we want to kindly remind you of our responses to your follow-up questions and thoughtful suggestion. We sincerely appreciate your considerate feedback and are eager to address any additional questions or concerns you may have.

If there is any additional information or clarification we could provide to support the discussion, we would be delighted if you could let us know.

Thank you once again for your considerate feedback and the time you have devoted to reviewing our work.

Thank you for the discussion. We sincerely appreciate your time and effort in reviewing our work and helping us improve our paper!

We sincerely appreciate the reviewer's insightful comments and time dedicated to evaluating our work. Our responses are listed below.

Q1: The method seems heavily reliant on large language models, and the need for manual adjustment of the Region of Interest (RoI) threshold in the coarse localization stage suggests a lack of generalizability.

A1: Thanks for your comments and our responses are in the following.

a) Reasons for using LLMs.

LLMs have proven effectiveness across a variety of domains due to their powerful text understanding and reasoning capabilities, and a promising direction to improve visual grounding is to design a multi-modality large language model to process both the text prompts and city-scale 3D point clouds. While the previous city-scale grounding method is trained from scratch using text prompts and 3D point clouds, one of our contributions is to introduce the powerful LLMs in city-scale visual grounding, which outperforms them by 36% ~ 51% on Acc@0.25 and 31% ~ 48% on Acc@0.50 on two city-scale datasets as shown in Table 1 of the main paper. We believe that our method is aligned with the ongoing trend in the research community to incorporate LLMs into multi-modal applications, where the complementary strengths of language modeling and 3D understanding are combined to tackle complex problems in city-scale scenarios.

b) We do not manually tune the threshold in CLM.

In all the experiments conducted on the CityRefer and CityAnchor datasets (e.g., Tables 1, 2, and 3 of the main paper), we employed a fixed RoI threshold (0.3) without adjusting the threshold. As shown in Table 1 of the main paper, when the RoI threshold is fixed to 0.3, CityAnchor already performs well and achieve at least 46% and 35% on Acc@0.50 on both CityRefer and CityAnchor datasets, respectively. Table 4 uses the results of different thresholds to analyze the efficiency and effectiveness of our method but is not our main results. We have clarified this in Sec.4.1.4 (Implementation details) of the revised paper.

Q2: While the coarse-to-fine localization design is effective, it builds on existing multi-stage approaches without introducing fundamentally new concepts in 3D visual grounding, which suggests that the method's strengths lack a degree of innovation.

A2: Thanks for the feedback. We agree that the coarse-to-fine strategy is widely adopted in various parts of Computer Vision but how to utilize such a strategy in city-scale geospatial analysis is unexplored. Previous method CityRefer randomly selects objects for the visual grounding task, which is slow and inaccurate. Thus, we introduce the coarse-to-fine strategy to improve both accuracy and efficiency.

Designing such a coarse searching module for the city-scale point cloud is also non-trivial. We effectively project the point clouds onto the ground and formulate it as a more manageable 2D coarse-searching problem. All of these techniques are unexplored and a part of our innovation.

Q3: The layout could be improved; for instance, all teaser images should be embedded as PDFs rather than PNGs to ensure the text within images is selectable.

A3: Thanks for your suggestion. we have adjusted the paper to use PDFs instead of PNGs as the teaser and experiment figures.

Q4: It’s unclear how the method addresses dynamic objects in urban environments, such as moving vehicles or pedestrians, which could affect the reliability of grounding results.

A4: Thanks for your comments. In our current implementation, we remove dynamic objects, such as moving vehicles, pedestrians, and ongoing constructions, following the baseline method CityRafer. By removing these dynamic objects, we can reduce the risk of grounding errors caused by the unpredictable motion or temporary presence of such objects.

We agree with the reviewer that handling dynamic objects, such as moving vehicles, pedestrians, and active construction sites, is important in the data preprocessing of our method. For future works, it could be an interesting topic to conduct city-scale visual grounding for dynamic 3D point clouds. We have added these discussions in the revision of Sec.5 (Limitations).

Dear Reviewer YNjM,

As the author-reviewer discussion period is coming to an end, we wanted to kindly remind you of our responses to your valuable comments. We sincerely appreciate your considerate feedback and are eager to address any additional questions or concerns you may have.

If there is any additional information or clarification we could provide to support the discussion, we would be grateful if you could let us know.

Thank you once again for your considerate feedback and the time you have devoted to reviewing our work.

Thank you for your further explanation. I have raised my scores.

Thank you for the discussion. We sincerely appreciate the time and effort you dedicated to reviewing our work and helping us improve the paper!

We sincerely appreciate the reviewer's encouraging comments and time dedicated to evaluating our work. We make the responses to each question below.

Q1: One concern is the validity of the coarse localization module. As shown Fig.4, I am curious whether CLM operates as intended. It appears to activate too many candidates, as if it were merely distinguishing between urban and non-urban areas. This raises doubts about whether it significantly aids in the efficient operating in the overall framework.

A1: Thanks for your comments. The rasterized map shown in Figure 4 is mainly dominated by small objects, which shows the limited effectiveness of CLM in this case. As illustrated in Figure 2 of the Appendix, we provide additional qualitative results of our method on the CityRefer dataset in "Novel Objects" setting, which shows that CLM can effectively filter our erroneous objects. As illustrated in Figure 6 of the Appendix, we provide heat map visualizations for both simple and complex query texts, which shows that CLM performs reasonably even on complex texts. We have shown in Table 3 of the main paper that CLM significantly reduces computation time while improving grounding accuracy, with 46.86% of Acc@0.50 and only 32.45s in localizing an object.

The coarse localization module (CLM) is designed to filter out irrelevant objects, thereby enhancing the model efficiency in city-scale visual grounding tasks. However, we must acknowledge that the filter effectiveness can vary depending on the size and attribute of target objects. Large objects (e.g., factories, athletic fields, parking lots) can be easily identified and excluded through the heat map output by CLM. In contrast, small objects (e.g., cars, residential buildings) are more challenging to distinguish, requiring further detailed comparison in the fine-grained matching module (FMM).

We have revised the paper to include a discussion of this phenomenon in Sec.5 (Limitations) and plan to address these issues in future work.

Q2: Experiment analysis and description are not specific and descriptive. For example, a detailed explanation based on specific examples is needed to clarify the novel objects and novel descriptions, whether the reason they are not in the training data is due to an out-of-distribution (OOD) scenario.

A2: We appreciate the reviewer’s insightful feedback and acknowledge the need for more detailed experimental analysis and clearer descriptions regarding novel objects, novel descriptions, and out-of-distribution (OOD) scenarios. In the revised version paper, we have added more specific examples and explanations to address these concerns.

a) Novel Descriptions (ND).

"Novel Descriptions" setting indicates that the evaluation objects are seen in the training set, but the query text descriptions are different in test set. This variance in text description may lead to different grounding results, even for the same object.

To clarify "ND" setting, we provide a specific example in Sec.4.1.3 (Metrics and experimental setting) of the revised version. For example, a building object may be described in the training set as “A building with white roof near the road”, but in the test set, the same object may be described as “Along with the street, there is a white-roofed building”.

b) Novel Objects (NO).

"Novel Objects" setting means the objects in the test set are unseen in the training set. This variance in both object and text description is likely to lead to different grounding results, even for these objects belong the same semantic category (e.g., Building or Car).

To clarify "NO" setting, we provide a specific example in Sec.4.1.3 (Metrics and experimental setting) of the revised version. For example, a blue car object may appear in the test set, while the same blue car object and its corresponding text description is not present during training process.

c) Out-of-distribution (OOD) scenarios.

To evaluate CityAnchor's ability to recognize "OOD" objects of novel categories, we conducted a generality experiment as illustrated in Figure 8 of the Appendix. In the CityRefer dataset, CityAnchor was trained with only four object categories: Building, Car, Parking, and Ground, and we introduced unknown object categories (such as road Intersection, Woodland, and River) that CityAnchor has not seen before. We evaluated the CityAnchor’s ability to predict the similarity between these unknown objects and both the correct text descriptions (Positive Sample) and incorrect text descriptions (Negative Sample). The grounding results for unknown objects demonstrate that CityAnchor exhibits strong generalization capabilities, enabling it to perform 3D visual grounding across a broader range of objects.

Q3: Also, there are little vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.

A3: Thanks for your comments. We have carefully checked and corrected the vague sentences and grammatical errors in the paper, and the PDF has been updated.

Dear Reviewer 8f4q,

As the author-reviewer discussion period is coming to an end, we wanted to kindly remind you of our responses to your valuable comments. We sincerely appreciate your thoughtful feedback and are eager to address any additional questions or concerns you might have.

Please let us know if there's any further information or clarification we can provide to facilitate the discussion process.

Thank you once again for your encouraging comments and time dedicated to evaluating our work.

We sincerely appreciate the reviewer for their invaluable comments and time dedicated to evaluating our work. We provide our responses to each question below.

Q1: As also mentioned in Sec.5, the efficiency of the proposed method is a major concern. It involves two LLMs and multiple forward passes in the fine-grained matching stage. Are there any possible solutions to improve the efficiency?

A1: Thanks for your insightful comments. Our method achieves better efficiency than baselines due to the coarse-to-fine strategy. We agree that there is still room for improvement. We provide two possible solutions in the following. We have added this discussion in the revision of Sec.5 (Limitations).

a) RoI Enhancement. In the first stage of CityAnchor, the Coarse Localization Module (CLM) is designed to regress the regions of interest (RoI) of target object for filtering out irrelevant objects. We can train the CLM using more diverse datasets to enhance the accuracy of the RoIs, enabling them to comprehend complex query texts, while effectively excluding irrelevant objects and purposefully retaining only the relevant objects.

b) Landmark Information Guidance. We observe that landmarks are frequently mentioned in query text, such as “The big oval green ground that is nearby the Perry BarrGreyhound Stadium”. Considering that Retrieval-Augmented Generation (RAG) emerges as a promising solution to mitigate issues of limited expertise in highly specialized queries, we can incorporate landmark information for each object into the CityAnchor as external knowledge using RAG, utilizing landmark information for efficient grounding. Specifically, we plan to select only the objects near the landmark to compare with the query text and avoid too many candidate objects being involved in fine-grained comparisons.

Q2: How to choose the positive and the negative samples during the training of FMM?

A2: Thanks for your comments. We select positive and negative samples in a ratio of 1:3 for FMM training, and the negative samples are randomly chosen among the candidate objects after excluding the target object. We have included this clarification in Sec.4.1.4 (Implementation details).

Q3: The objects are generated by a pretrained 3D segmentation model which can only recognize a closed set of targets. So I expect to see the generality of CityAnchor to unknown objects.

A3: Thanks for your comments. We follow the experimental setting of CityAnchor to rely on a pre-trained 3D segmentation model to extract objects.

We show that CityAnchor has the potential to generalize to unknown objects. In the revision, we additionally conduct a generality experiment to assess CityAnchor's ability to recognize unknown objects in Figure 8 of the Appendix. On the CityRefer dataset, CityAnchor is trained with only four object categories: Building, Car, Parking, and Ground, and we introduce three objects of unknown object categories (Road Intersection, Woodland, and River) that CityAnchor has not seen before. We show that CityAnchor is still able to determine a reasonable matching score between the text descriptions and the given 3D objects due to the strong generalization ability of LLMs.

Q4: Some important details are missing:

Line 157, what is $T_c$?

Line 138 & 209, how to determine whether an object has a landmark name?

Line 206, what is $c$ in $E^s_x$?

Line 232, which encoder is the concatenated feature vector fed into?

A4: Thanks for your comments. We have added these missing details in the revised version.

a) Line 157, what is $T_c$?

$T_c$ denotes the output texts (including <RoI> token) in Coarse Localization Module (CLM). We have added a representation of $T_c$ in Figure 2 in revised version paper.

b) Line 138 & 209, how to determine whether an object has a landmark name?

The CityRefer dataset provides landmark names for some objects, with approximately 10% of the objects already assigned a landmark name, while the remaining objects lack such identifiers. We follow the CityRefer dataset and assign a landmark name to an object based on the geospatial information retrieved from OpenStreetMap.

c) Line 206, what is $c$ in $E^s_x$?

$c$ in $E^{s}_x\in \mathbb{R}^{c \times d}$ represents the feature-length of the 2D CLIP feature, which is 256.

d) Line 232, which encoder is the concatenated feature vector fed into?

The concatenated feature vector is input into the transformer encoder of the Fine-grained Matching Module (FMM), which is fine-tuned from LLaVA. The FMM predicts the similarity between the query text and the objects based on the concatenated feature vector and then selects the object with the largest similarity as the final grounding result for the query text.

Dear Reviewer 6qAx,

As the author-reviewer discussion period is coming to a close, we would like to provide a gentle reminder that we have posted a response to your valuable comments. We sincerely appreciate your encouraging feedback and are eager to address any additional questions or concerns you may have.

If there is any additional information or clarification we might provide to support the discussion, we would be grateful if you could let us know.

Thank you once again for your encouraging comments and time dedicated to evaluating our work.

Thank you for the discussion. We really appreciate your efforts in reviewing our work and the insightful comments that help us improve the paper!