UrbanMLLM: Joint Learning of Cross-view Imagery for Urban Understanding

Q4: Can you clarify how the cross-view perceiver module compares to more advanced fusion mechanisms used in similar multimodal tasks? While the cross-attention mechanism is effective, it seems like a relatively simple approach. Have you considered more sophisticated alternatives for deeper integration between satellite and street-view imagery? If so, what were the reasons for opting for the current design?

Response: It's true that there's some more sophisticated fusion methods of satellite and street-view images for CV tasks. For example, SG-BEV [1] introduces a complex satellite-guided re-projection operation that transforms street-view features into satellite-view features and then fuses them with convolutional layers. However, such approach relies on depth information and high-resolution satellite images for image alignment, inefficient to be applied to large-scale data collection. Moreover, existing methods for urban understanding take very simple fusion methods such as feature concatenation for fusion in UrbanVLP. By comparison, we adopts the more advanced cross-attention fusion and found that it performs effectively in our framework, thus we decide to choose it as the final implementation.

[1] Ye J, Luo Q, Yu J, et al. SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 27748-27757.
[2] Hao, Xixuan, et al. UrbanVLP: A Multi-Granularity Vision-Language Pre-Trained Foundation Model for Urban Indicator Prediction" arXiv preprint arXiv:2403.16831 (2024).

Q5: What impact do the model's size and dataset scale have on the performance? It's not entirely clear whether the performance improvements are primarily driven by the large dataset and increased model parameters. Have you conducted any comparisons or ablation studies that analyze the performance of smaller models or less data? How would a smaller model perform under the same conditions?

Response: Please refer to the response to Q2.

Q6: Why wasn't there a comparison with domain-specific models like UrbanCLIP or UrbanVLP in the indicator prediction task?Given the task-specific nature of indicator prediction, models like UrbanCLIP and UrbanVLP could provide a more relevant baseline. Could you provide insights into why these comparisons were not included, and how you believe UrbanMLLM would fare against such models?

Response: This is mainly because that these two models do not provide open-source codes or weights for reproducing. Please see the response to Q2 for more details.

Q7: Can you elaborate on how the model handles simpler tasks, such as geographic location prediction? The paper mentions that the model falls slightly behind some closed-source models in certain tasks like geographic location prediction. Could you explain why this might be the case, and whether there are specific limitations in UrbanMLLM that contribute to this performance gap?

Response: Since geo-location information is very common online, most MLLMs might have include such knowledge in their training corpus, thus can achieve superior performance on the geographic location prediction task. However, in more complex tasks like object reasoning and indicator prediction, UrbanMLLM outperforms existing models, demonstrating its more specialized urban understanding ability. A more carefully-tuned data mixture might help UrbanMLLM achieve a better balance among all urban understanding tasks.

Q2: Limited Model Comparisons: The experimental section does not sufficiently compare the model with others in terms of model size and training data scale. The paper showcases UrbanMLLM's superior performance, but it remains unclear whether this improvement stems primarily from the large dataset or the model's architecture. A more in-depth discussion on these factors is necessary. Furthermore, the paper lacks comparisons with specific domain models, particularly in critical tasks such as indicator prediction. A comparison with models like UrbanCLIP and UrbanVLP would provide a more comprehensive evaluation of UrbanMLLM's performance in urban understanding tasks.

Response: In Table 2-4, we have provided the results of UrbanMLLM with different model sizes, including 3B, 8B, and 13B. The results show that the overall performance of UrbanMLLM improves with the increase of model size. And we also provide the results of UrbanMLLM trained on different dataset scales, including 0.35M, 0.92M, and 1.86M images. The results show that the performance of UrbanMLLM improves a little with the increase of data scale. The reason might be that our pre-training data is still less than data size that scaling law requires. As for the comparison with other models, we have compared UrbanMLLM with other CLIP-based models, including RemoteCLIP and OpenAICLIP. The results are shown in Table. UrbanMLLM outperforms these CLIP-based models on all tasks, demonstrating its effectiveness in urban understanding tasks. UrbanCLIP and UrbanVLP do not provide trained model for fine-tuning, so we can't compare them on our dataset.

Results of satellite imagery-based urban understanding tasks:

Results of street view imagery-based urban understanding tasks:

Results of cross-view imagery-based urban understanding tasks:

Performance comparison with CLIP-based models on satellite imagery-based indicator prediction tasks:

Performance comparison with CLIP-based models on street-view imagery-based indicator prediction tasks:

Q3: Lack of Depth in Experimental Analysis: While the experiments cover multiple task types, there is a lack of in-depth analysis on how the model performs across different subtasks. For example, the model excels in complex reasoning tasks like population density prediction, but it does not outperform certain closed-source models in simpler tasks like geographic location prediction. A more thorough discussion of the model's strengths and weaknesses across various tasks would add valuable insights into its overall performance.

Response: In the satellite imagery understanding tasks, UrbanMLLM excels in complex reasoning tasks in all tasks. The main reason is that the most of models are not pretrained on the large-scale dataset of satellite images. However, in the street-view imagery understanding tasks, UrbanMLLM does not outperform other models in simpler tasks like geographic location prediction and scene classification. This is because geo-tagged street-view images are more likely to be involved as training corpus of existing MLLMs, enabling most general MLLMs to achieve superior performance on such tasks. However, in more complex tasks like object reasoning and indicator prediction, UrbanMLLM outperforms existing models, demonstrating its superior specialized ability. In the cross-view tasks, UrbanMLLM excels in most of the tasks, including depression rate prediction, median income prediction, and population density prediction and spatial relationship reasoning. However, it falls slightly behind some closed-source models in certain tasks like poverty rate prediction.

Q1: Lack of Novelty: While the paper provides a solid technical solution, it lacks innovation on the methodological front. The idea of combining satellite and street view images for urban understanding has been explored extensively in previous works. Although the cross-view perceiver module is introduced, its design is relatively simple, relying mainly on cross-attention mechanisms without deeper integration. Additionally, the interleaved data pre-training strategy is common and does not bring any groundbreaking technical advancement.

Response: Difference between our module and previous works: There are some works that have expored cross attention for image and text fusion, such as Flamingo [1] , BLIP2 [2], Qwen-VL [3], CogVLM [4]. However, these models are designed for bootstrapping language-image pre-training, and cannot be generalized to urban understanding tasks. For example, Flamingo uses cross-attention to extract fixed-length feature vectors from single images. BLIP2 and QwenVL uses the learnable query to get fusion features from image and text. CogVLM use visual-expert to align the image and text deeply. These methods are all designed for extracting features from a single image and can not be applied for the joint learning process involving two types of input images.

In contrast, our module is designed to conduct the mutual fusion of cross-view image features and break the isolated visual knowledge in different views, which previous designs can not achieve. We also introduce a gating mechanism to adaptively fuse the features from different views, producing semantically rich cross-view image features.

Novelty of our interleaved pre-training: Although the interleaved pre-training strategy has been adopted by some MLLMs, but the interleaved data is limited to general domains, with limited help for solving urban enderstanding tasks. Differently, we are the first to use geo-tagged satellite and street-view images to construct a large-scale interleaved dataset in the urban domain and develop a powerful MLLM to universally solve urban understanding tasks. We believe this training corpus is valuable for the community, which has been validated to bring clear performance gain on wide urban understanding tasks.

[1] Alayrac J B, Donahue J, Luc P, et al. Flamingo: a visual language model for few-shot learning[J]. Advances in neural information processing systems, 2022, 35: 23716-23736.
[2] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.
[3] Bai J, Bai S, Yang S, et al. Qwen-vl: A frontier large vision-language model with versatile abilities[J]. arXiv preprint arXiv:2308.12966, 2023.
[4] Wang W, Lv Q, Yu W, et al. Cogvlm: Visual expert for pretrained language models[J]. arXiv preprint arXiv:2311.03079, 2023.

Q5: Lack of bad case analysis.

Response: Thanks for your suggestion. We have added a bad case analysis in the revised paper. We show some examples of bad cases in scene classification and indicator prediction tasks. The results are shown in Figure 6-8. Firstly, in the scene classification task, the model misclassifies the image with a truck parking as a car parking. Although there are a few differences between the two classes, the more granular understanding of the urban environment is required to distinguish them. Secondly, in the indicator prediction task, as shown in Figure 7-8, the model predicts the population density of an urban area as 6.8 and the actual value is 9.9 using a satellite image. The model fails to capture detailed information with a single-view image, which makes it challenging for the model to learn from the limited dataset. For poverty rate prediction, the model gets a high score of 5.4, but the poverty rate is 2.6. It's may be due to the number of street view images in the dataset is not enough to learn the detailed information of the urban environment.

Q6: I think you mentioned the UrbanVLP using satellite and street-view images in Table 1, so I wonder if it is possible that the CLIP-based pertaining model can outperform the MLLM-based pertaining model.

Response: Thanks for your suggestion. UrbanVLP is not open-source thus hard to directly compare with it. We have compared UrbanMLLM with other CLIP-based models including RemoteCLIP and CLIP (OpenAI version). The results are shown in our response to Q2. UrbanMLLM outperforms these CLIP-based models on all tasks, demonstrating its effectiveness in urban understanding tasks.

Q7: Is the vision encoder the same for both satellite and street-view images?

Response: Yes, the vision encoder is the same for both satellite and street-view images. The previous remote sensing MLLMs like GeoChat and LHRS-Bot have shown that the general vision encoder can also be used to deal with satellite images, thus both types of image can share the same encoder. The difference lies in the subsequent processing of the features extracted by the vision encoder, where the satellite and street-view features have two different branches for later fusion.

Q8: It is recommended to validate/evaluate the quality of text parts from the UrbanView dataset. Or how do you refine the text?

Response: Please see the response to Q3.

Q9: You should include a bad case analysis of UrbanMLLM, so the community can know the gap of the current best model towards comprehensive urban understanding.

Response: Please see the response to Q5.

Q10: Does UrbanMLLM support multi-image as input? If so, possible to include an ablation study on the number of images?

Response: Yes, our model can support multi-image as input. We conduct an ablation study on the number of images (N = 2, 4, 6) for indicator prediction tasks. For example, 6 images mean one satellite image and five street-view images as input. The results are as follows. It can be seen that more images bring certain performance gain for the indicator prediction task.

Q11: It seems that the scaling law does not totally apply to UrbanMLLM across all urban tasks. We expect you to dive into the explanation of the difference among urban tasks in terms of UrbanMLLM performance.

Response: As for the scaling law, we have conducted experiments on different model sizes and data scales. The results show that the performance of UrbanMLLM improves with the increase in model size, but the performance of UrbanMLLM has limited improvement with the increase of data scale. The reason might be that our pre-training data amount has not reached the required magnitude (e.g., tens-of-millions level). As for the difference performance among tasks, it may be due to the complexity of the tasks and corresponding data scale. For example, in the satellite imagery-based scene classification task, the distribution of images across categories is uneven, making the long-tail class prediction very challenging. In this case, the performance improvement requires much more data to train the model rather than purely increasing model size, so the scaling law may be not apparent.

Q1: Architecture-wise, I think the cross-attention mechanism in the cross-view perceiver is not a novel part. Hence, mode-side novelty is limited, although the interleaved image-text dataset is beneficial for the MLLM community.

Response: The perceiver resampler in Flamingo [1] from DeepMind is proposed to extract fixed-length feature vectors from a single image. Its structure is similar to Q-former [2], both adopting the learnable query to sample feature vectors from the input image. It's not designed and not able to handle the challenges of modality fusion (cross-view imagery fusion in our work).

In contrast, our method is designed for cross-view images to jointly learn the region embeddings. Our module is designed to enable the complementary fusion of cross-view knowledge and get semantically rich urban imagery features. We also introduce a gating mechanism to adaptively fuse the features from different views, which can effectively solve the problem of isolated visual knowledge in different views.

[1] Alayrac J B, Donahue J, Luc P, et al. Flamingo: a visual language model for few-shot learning[J]. Advances in neural information processing systems, 2022, 35: 23716-23736.
[2] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.

Q2: In the evaluation, the baselines you chose are MLLMs, but it could be more comprehensive to compare yours with CLIP-based models.

Response:

Here we add more CLIP-based models for comparison, including RemoteCLIP and OpenAICLIP. The results have also been updated in Table 3 of the paper. From the result, UrbanMLLM outperforms these CLIP-based models on all tasks, demonstrating its effectiveness in urban understanding tasks. As for UrbanCLIP, they did not provide trained model for fine-tuning on our dataset, so we can't provide objective results of it.

Results on satellite imagery understanding tasks:

Results on street view imagery understanding tasks:

Q3: You used InternVL2-40B to generate the captions for the UrbanView dataset. The dataset quality, especially the text quality, has not been verified/evaluated yet.

Response: During our dataset establishment, we have conducted manual examination of the text quality and found that it's OK to accurately describe the content. Therefore, we did not take complex operations to further improve it at that time. The superior performance of our model pre-trained on this dataset also confirms the quality of these data to some extent. To further enhance the text quality, we are now taking a Human-AI collaboration pipeline for further refine the annotated texts. Specifically, we first use two other powerful open-source MLLMs, VILA-1.5-40B and LLaVA-Next-34B to judge if the annotated text matches with the image. If either of them thinks it is not a match, we will proceed to send it to GPT-4o for further examination. If GPT-4o still judges the image-caption pair mismatched, we will conduct manual refinement about the text. We preliminarily randomly sample 10000 images for test and observe that the original text quality is already good, where only around 0.13% images are judged as mismatched. We are working to polish all texts in the dataset via the above pipeline and will update the refined text data in our dataset.

Q4: The statement from your abstract "MLLMs in urban studies are only developed focusing on remote sensing imagery" may be incorrect, unless you illustrate the comparison of the pertaining corpus proportion of different MLLMs.

Response: Sorry for the confusion. Here, we want to emphasize that the MLLMs in urban studies are mainly developed focusing on addressing remote sensing imagery-based tasks. These models are developed by fine-tuning general MLLMs on remote sensing data and actually perform poorly on street-view understanding tasks (see the performance of GeoChat in Table 3). By comparison, our UrbanMLLM is pretrained on a large-scale interleaved image-text corpus involving both satellite and street-view images, which achieves superior performance on urban understanding tasks across satellite, street-view and cross-view domains.

Thank you for the reply.

I am concerned about [1] because it seems that the novelty lies in the functionality, rather than the architecture.

[3]: "we first use two other powerful open-source MLLMs, VILA-1.5-40B and LLaVA-Next-34B to judge if the annotated text matches with the image. " What does this procedure mean? Do you use two judges for each pair (i.e. two matching scores)? Please elaborate on this. Besides, why not use gpt4o as a judge at first, if you believe it is the 'fairest' one? Overall, I think an end-to-end evaluation pipeline and quantitative metrics for text quality should be needed (an evaluation of the image captioning field may be a good reference).

Dear Reviewer Sahu,

Thank you again for your thoughtful review of our manuscript. As the discussion deadline approaches, we would appreciate your feedback on whether our revisions have addressed the concerns you raised. If any further clarification is needed, we are more than happy to provide additional information.

If you find the revisions satisfactory, we would be grateful if you could reconsider your score.

Thank you for your time and consideration.

Best regards,

Submission4194 authors

Dear Reviewer Sahu,

Thank you once again for your valuable feedback during the review process. We have carefully addressed the concerns you mentioned and provided a detailed, point-by-point response in the rebuttal. The revised paper is available at https://anonymous.4open.science/r/UrbanMLLM-2F1E, and the following revisions have been made:

1. Cross-view perceiver: We have added a more detailed discussion on this topic in Lines 186-191 and Lines 219-223 of Section 3.2 "Cross-view Fusion-enhanced UrbanMLLM".

2. CLIP-based baselines: RemoteCLIP and OpenAICLIP have been included in Tables 2-3 of Section 4.2 "Results".

3. Dataset quality: A comprehensive description of the dataset has been added in Section A.6.2 "Dataset Structure and Construction", covering Lines 1022-1054.

4. Related work: Revisions have been made to the Abstract (Lines 13-14) and the Introduction (Lines 59-60) to clarify the statement of related work.

5. Bad case analysis: Updates have been provided in Section A.4.5 "Case Study", specifically in Lines 795-803.

6. Vision encoder: The introduction of the vision encoder has been included in Lines 199-200 of Section 3.2 "Cross-view Fusion-enhanced UrbanMLLM".

7. Quality validation/evaluation: A detailed description of the dataset’s structure and evaluation has been added in Section A.6.2, covering Lines 1022-1054.

8. Multi-image study: The relevant update has been incorporated into Section A.4.2 "Ablation Study", specifically in Lines 758-770.

9. Urban tasks and UrbanMLLM performance: Revisions clarifying the differences in UrbanMLLM performance across urban tasks have been made in Section A.4.1 "Experimental Analysis", Lines 746-755, and Section A.4.3 "Data Scale Study", Lines 773-783.

We sincerely hope that the revisions and clarifications meet your expectations. As the discussion phase nears its end, we kindly ask if you could take a moment to review our responses. We understand the time constraints but hope that the revisions address your concerns and will be reflected in the final scoring.

Thank you once again for your time and support in refining our work.

Best regards,

The Authors

Q4: For a work targeting urban understanding, providing information on data collection, refinement, and model training time would be highly beneficial.

Response: Thanks for the suggestion. We have updated a detailed description of the dataset in A.6.1 Data Collection and A.6.2 Dataset Structure and Construction section.

About data collection: The images are primarily collected from two sources: Google Maps API for street view images, and ESRI for satellite imagery. For street view imagery collection, we randomly generate 2,000 random points in each census tract polygon and use their coordinates to query Google Maps API, returning street view image patches and real coordinates. We scrape over 2 million street view images and all satellite imagery of zoom level 15 across the United States. The ground truth of the dataset comes from a variety of sources, including US Census Bureau, Safegraph, VIIRS, and some open-source datasets.

About data refinement: During our dataset establishment, we have conducted manual examination of the text quality and found that it's OK to accurately describe the content. Therefore, we did not take complex operations to further improve it at that time. The superior performance of our model pre-trained on this dataset also confirms the quality of these data to some extent. To further enhance the text quality, we are now taking a Human-AI collaboration pipeline for further refine the annotated texts. Specifically, we first use two other powerful open-source MLLMs, VILA-1.5-40B and LLaVA-Next-34B to judge if the annotated text matches with the image. If either of them thinks it is not a match, we will proceed to send it to GPT-4o for further examination. If GPT-4o still judges the image-caption pair mismatched, we will conduct manual refinement about the text. We preliminarily randomly sample 10000 images for test and observe that the original text quality is already good, where only around 0.13% images are judged as mismatched. We are working to polish all texts in the dataset via the above pipeline and will update the refined text data in our dataset.

We mostly use MLLMs to generate question variants in the instruct-tuning dataset, and most ground truths come from open-source data such as U.S. Census Bureau, VIIRS and WorldPop, thus the quality is reliable. For a few street-view imagery-based tasks without ground truths such as scene classification, we try various prompting methods and different MLLM architectures to generate the label with detailed human examinations to guarantee the quality.

Training Details: As for training details, we train the 8B model on 8 NVIDIA A100 GPUs. In the pre-training stage, the model is trained for 3,200 steps with a batch size of 8. In the fine-tuning stage, the model is fine-tuned for 7,200 steps with a batch size of 16. The training time for the 8B model is approximately 8 hours for pre-training and 8 hours for fine-tuning. We have added it on the section of 4.1 EXPERIMENTAL SETUP.

Q5: The paper introduces a new benchmark to comprehensively assess urban understanding capabilities. However, it lacks results on previous benchmarks, which, if included, would further validate the advantages of the proposed approach.

Response: We now provide the results of our model on another relevant benchmark CityBench [1]. We test the performance of our model on three tasks including CityInfer, LocInfer, and Population and use Accuracy, Accuracy@25km, and R-square as evaluation metrics. The results are as follows and it can be seen that our model outperforms SOTA models reported in their paper, demonstrating the high generalization ability of our model on urban understanding tasks.

[1] Feng J, Zhang J, Yan J, et al. CityBench: Evaluating the Capabilities of Large Language Model as World Model[J]. arXiv preprint arXiv:2406.13945, 2024.

Q1: The paper emphasizes that prior works have largely overlooked street-view images; however, this argument alone does not fully justify the motivation, especially as prior works are limited in number, and UrbanVLP has already explored this area to some extent. Furthermore, the three tasks—perception, reasoning, and prediction—do not seem to introduce substantial innovation or expansion.

Response: Yes, it's true that there are some works that have explored using both street-view images and satellite images in urban understanding tasks, such as UrbanVLP. However, these works can only deal with prediction tasks such as indicator prediction via end-to-end fine-tuning, but fail to conduct perception and reasoning tasks. By comparison, our UrbanMLLM provides a more general solution toward wide range of urban perception, reasoning and prediction tasks.

We build UrbanView Benchmark with 13 different tasks including urban perception (scene classification, geo-localization), reasoning (object reasoning, spatial relationship reasoning, landmark reasoning) and prediction (indicator prediction) based on single-view or cross-view urban imagery. A relevant work CityBench [1] only involves very limited reasoning and prediction tasks including city inference, location inference, and population prediction. Another work Urbench [2] focuses on detailed cross-view tasks on different urban scenarios without prediction tasks. By comparison, Our benchmark is more comprehensive and has a wider coverage.

[1] Feng J, Zhang J, Yan J, et al. CityBench: Evaluating the Capabilities of Large Language Model as World Model[J]. arXiv preprint arXiv:2406.13945, 2024. [2] Zhou B, Yang H, Chen D, et al. UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios[J]. arXiv preprint arXiv:2408.17267, 2024.

Q2: While the appendix includes high-quality visualizations of the dataset, more details are needed for a new benchmark, particularly on data refinement, which is crucial.

Response: We have updated a detailed description of the dataset and benchmark in A.6.1 Data Collection and introduce the data refinement in A.6.2 Dataset Structure and Construction section of the paper.

Q3: The paper does not discuss limitations or future work. Additionally, no accompanying code is provided to support its methodology and results.

Response:

Discussion of limitations and future work: Although our model achieves state-of-the-art performance on a wide range of urban understanding tasks, there are still some limitations. Our dataset is limited to the United States, and the generalization of the model to other countries may require additional data collection and pre-training. Therefore, we plan to extend the dataset to cover more regions and improve the general applicability of the model, and explore the use of additional modalities for urban understanding tasks. Moreover, we will investigate the use of more advanced MLLM architectures for urban understanding tasks and provide more results to demonstrate the generalization of our method.

The discussion has been added to the revised paper. We have also provided an anonymous link including our codes: https://anonymous.4open.science/r/UrbanMLLM-2F1E.

Thank you for your careful review and suggestions. We hope that the revised version of the paper and the response meet your expectations. If you have any further questions, please let us know and we will provide detailed response to address your concerns. If you are satisfied with our response, please consider revising your score. Thanks again for your insightful comments and suggestions.

Dear Reviewer t4j3,

Thank you once again for your valuable review of our manuscript. As the discussion deadline approaches, we would like to confirm whether our responses have fully addressed your concerns. If any clarification is needed, we would be happy to provide further details. Additionally, if the revisions are satisfactory, we respectfully request that you reconsider your score.

We appreciate your time and consideration.

Best regards,

Submission4194 authors

Dear Reviewer t4j3,

Thank you once again for your valuable feedback during the review process. We have carefully addressed the concerns you mentioned, and a detailed point-by-point response is provided in the rebuttal. Additionally, we have made the following revisions to the paper (https://anonymous.4open.science/r/UrbanMLLM-2F1E/ICLR_2025_UrbanMLLM-V1201.pdf):
(1) About data refinement, we have added a detailed description of the dataset in Section A.6.1 (Data Collection) and introduced data refinement in Section A.6.2 (Dataset Structure and Construction).
(2) About limitations and future work, we have added a discussion on Section A.2 (Limitations and Future Work).
(3) About training details, we have added training information in Line 321-373 of Section 4.1 (Experimental Setup).
(4) About benchmark comparison, we have included results of our model on the CityBench benchmark in Section A.4.4 (Evaluation on CityBench) and discussed our advantage over other benchmarks in Line 1119-1126 of Section A.6.3 (UrbanView Benchmark and Evaluation).
We sincerely hope that the revisions and clarifications we've made meet your expectations. As the discussion phase is nearly ending, we kindly ask you to take a moment to review our responses. While we understand the time constraints, we are hopeful that these revisions address your concerns and will be reflected in the final evaluation. We deeply appreciate your time and support in helping to refine our work.

Best regards,
Authors

Q3: During the experiments, only UrbanMLLM was trained on the dataset collected by the authors, which is extremely unfair, and it is difficult to judge whether the difference in performance between different MLLMs is due to missing data or structural problems, or whether it is a problem with the pre-training process.

Response: In the ablation study, we have provided the result comparison of UrbanMLLM and the only fine-tuned VILA on our dataset. We now provide a more comprehensive comparison including (1) VILA without any further training (2) VILA fine-tuned with our instruction tuning data and (3) our UrbanMLLM. The results demonstrate that UrbanMLLM outperforms VILA with fine-tuning on the same dataset, demonstrating the effectiveness of our interleaved pre-training process. Besides, the fine-tuned VILA performs better than zero-shot version, indicating the necessity of fine-tuning. We also agree that the structure of MLLMs has influence on the performance and will apply our pre-training and cross-view perceiver designs to more MLLM structures to demonstrate the generalization of our method.

Results of satellite imagery-based urban understanding tasks:

Results of street view imagery-based urban understanding tasks:

Results of cross-view imagery-based urban understanding tasks:

Q4: The results of LHRS-Bot and SkysenseGPT are not shown in Table II and in Table III. What is the reason for this?

Response: We have added LHRS-Bot as a baseline. The results have been updated in Table 2-3 of the paper, demonstrating that UrbanMLLM outperforms LHRS-Bot on all tasks. There is no open-source codes and trained models for SkysenseGPT thus it's hard for comparison.

Q5: Although the authors have validated the effectiveness of the proposed dataset to some extent, the validation set in the experiments has labels, which are captions generated by MLLM. How can the accuracy of the benchmark be guaranteed?

Response: We mostly use MLLMs to generate question variants in this part, and most ground truths come from open-source data such as U.S. Census Bureau, VIIRS and WorldPop, thus the quality is reliable. For a few street-view imagery-based tasks without ground truths such as scene classification, we try various prompting methods and different MLLM architectures to generate the label with detailed human examinations to guarantee the quality.

Q1: The data obtained by MLLM is inherently noisy and limited, and introducing it for training without special processing can result in models with a significant upper bound.

Response:

About the quality of text annotations of images: During our dataset establishment, we have conducted manual examination of the text quality and found that it's OK to accurately describe the content. Therefore, we did not take complex operations to further improve it at that time. The superior performance of our model pre-trained on this dataset also confirms the quality of these data to some extent. To further enhance the text quality, we are now taking a Human-AI collaboration pipeline for further refine the annotated texts. Specifically, we first use two other powerful open-source MLLMs, VILA-1.5-40B and LLaVA-Next-34B to judge if the annotated text matches with the image. If either of them thinks it is not a match, we will proceed to send it to GPT-4o for further examination. If GPT-4o still judges the image-caption pair mismatched, we will conduct manual refinement about the text. We preliminarily randomly sample 10000 images for test and observe that the original text quality is already good, where only around 0.13% images are judged as mismatched. We are working to polish all texts in the dataset via the above pipeline and will update the refined text data in our dataset.

About the quality of benchmark: We mostly use MLLMs to generate question variants in this part, and most ground truths come from open-source data such as U.S. Census Bureau, VIIRS and WorldPop, thus the quality is reliable. For a few street-view imagery-based tasks without ground truths such as scene classification, we try various prompting methods and different MLLM architectures to generate the label with detailed human examinations to guarantee the quality.

Q2: There does not seem to be a straightforward relationship between the direct information interaction of the perspective images and the performance of the larger model (e.g., Figure 13), and many MLLMs can produce correct answers without this mechanism. This may be because UrbanMLLM only focuses on the relationship between different perspective images of the same area and ignores the connection between the corresponding captions.

Response: In urban understanding tasks, the model needs to understand the urban environment from both satellite and street-view images. During the interleaved pre-training process, our model can learn both textual and visual knowledge from the knowledge of another view. Taking the street view imagery-based geo-localization task in Figure 13 as an example, our model can capture both object details in street-view images and region function knowledge from satellite images, enabling complementary learning and making it easier to accurately determine the location. Without our design, the model would be limited to extracting knowledge solely from the street view itself.

Q6: The performance of the proposed dataset under other model architectures needs to be verified, and also how the dataset performs under other benchmarks after pre-training needs to be given.

Response: Thank you for your suggestion. We have added result comparison of fine-tuned VILA-8B and Qwen-2 VL (the most advanced open-sourced 7B MLLM) as follows. Our model outperforms VILA and is competitive with Qwen-2 VL (while we use much smaller pre-training data to achieve this).

Results of satellite imagery-based urban understanding tasks:

Results of street view imagery-based urban understanding tasks:

Results of cross-view imagery-based urban understanding tasks:

We also provide the results of our model on another relevant benchmark CityBench [1]. We test the performance of our model on three tasks including CityInfer, LocInfer, and Population and use Accuracy, Accuracy@25km, and R-square as evaluation metrics. The results are as follows and it can be seen that our model outperforms SOTA models reported in their paper, demonstrating the high generalization ability of our model on urban understanding tasks.

[1] Feng J, Zhang J, Yan J, et al. CityBench: Evaluating the Capabilities of Large Language Model as World Model[J]. arXiv preprint arXiv:2406.13945, 2024.

Q7: What is the reason for the fact that in Table 7, none of the components of UrbanMLLM-8B have a significant impact on the performance, and there is even a large gap for certain tasks?

Response: In the ablation study, we conduct experiments to evaluate the effectiveness of the components of UrbanMLLM. The results show that the perceiver module and the interleaved pre-training strategy are effective for urban understanding tasks. By comparing w/o Perceiver and w/o SI+SVI+Perceiver in Table 5-7, it can be observed that the performance of the model drops significantly when the perceiver module or the interleaved pre-training strategy is removed. And we found that the perceiver module and the interleaved pre-training strategy need to be combined to achieve the best performance.

Thank you for your thoughtful suggestions. We have revised the paper according to your comments. We hope that our revised version and response meet your expectations. If you have any further questions, please let us know and we will provide detailed response to address your concerns. If you are satisfied with our response, please consider revising your evaluation. Thanks again for your insightful comments and suggestions.

Dear Reviewer CuDA,

Thank you once again for your thoughtful review of our manuscript. With the discussion deadline approaching, we would appreciate it if you could let us know whether our responses have addressed your concerns. If further clarification is needed, we are happy to provide additional information. If the revisions meet your expectations, we kindly ask that you reconsider the paper’s score.

Thank you for your time and feedback.

Best regards,

Submission4194 authors

Dear Reviewer CuDA,

Thanks again for your valuable and continued feedback during the review process, which is very constructive to improve this work. We have carefully addressed the concerns you mentioned, and provided a detailed point-by-point response during the discussion phase. In addition, we have made the corresponding revisions in the paper (please see: https://anonymous.4open.science/r/UrbanMLLM-2F1E/ICLR_2025_UrbanMLLM-V1201.pdf), which are summarized as follows:

(1) About more details especially quality control of our dataset and benchmark, we have added a detailed description of the dataset in Section A.6.1 (Data Collection) and introduced data refinement in Section A.6.2 (Dataset Structure and Construction).

(2) About the experimental comparison with other trained models, we have updated results in Table 2-4 and added more discussion in Section 4.2.

(3) About results on other benchmarks, we have included results of our model on the CityBench benchmark in Section A.4.4 (Evaluation on CityBench) and discussed our advantage over other benchmarks in Line 1119-1126 of Section A.6.3 (UrbanView Benchmark and Evaluation).

(4) About the limitation of our method for large models or models pre-trained with larger data, we have added more discussion and results in Section A.4.1 and Section A.4.3 of the Appendix.

We sincerely hope that these revisions align with your expectations. As the discussion phase is nearly ended, we kindly ask you to take a moment to further consider our responses. While we understand the time constraints, we hope that these revisions address your concerns and will be reflected in the final scoring. Thank you again for your valuable time and effort in refining our work.

Best regards,

Authors

Q1: Other than stating that 2 million satellite and cross-view images were used, not much information is presented about the dataset. For example, where did the ground truth information to calculate accuracy come from? Also, the authors state that the dataset covers the whole of the US. I am sure that, just for street-view images, there are more than 2 million in the US. How did the authors select the used images and how uniform or concentrated did they select images for certain cities?

Response: We have updated a detailed description of the dataset in A.6.2 Dataset Structure and Constrcution section.

About the source of ground truths: The ground truth of the dataset comes from a variety of sources, including US Census Bureau, Safegraph, VIIRS, and a lot of open-source datasets. We do normalization for numerical indicator ground truth and reorganize the format of these ground truth data to build the instruct-tuning dataset.

About the coverage: The street view imagery cover 71,433 out of all 73,868 census tracts in the United States (96.7% coverage), and each satellite imagery is matched with street view imagery using a 1km $\times$ 1km grid.

About the imagery selection: The street view imagery amount in U.S. is indeed much larger than 2 million, however, due to the coordinate query method of Google API, we can only randomly generate thousands of points in each census tract polygon and use these points to query street view images, so it is not quite possible to collect all the street view images in a census tract considering the massive amount of time it requires. In fact, in some less populated areas, it is quite hard to get street view images because the randomly generated query coordinate points in these areas are always off-road, which is also the main reason for the missing 3.3% coverage of census tracts. In the end, we chose to randomly select about 200 images in each census tract, which we reckon is dense enough to make a good representation of the region according to previous works such as Urban2Vec [1] and Devil in the Landscapes [2], which both use around 40 images to represent communities with a similar size.

About the uniformity of images for certain cities: The data distribution is largely conform to the distribution of population density in the United States. Census tracts in the United States are primarily divided based on population size, and in each census tract polygon we randomly sample similar size of images. In terms of cities, larger cities usually have more census tracts, therefore more images will be used in these cities. However, as mentioned above, we still have a very good coverage of the whole country because we sample all the census tracts.

[1] Wang, Zhecheng, Haoyuan Li, and Ram Rajagopal. "Urban2vec: Incorporating street view imagery and pois for multi-modal urban neighborhood embedding." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020. [2] Han, Zhenyu, et al. "Devil in the Landscapes: Inferring Epidemic Exposure Risks from Street View Imagery." Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems. 2023.

Q2: The utilized perceiver module in the UrbanMLLM is a previously proposed component by DeepMind, hence the novelty of the architecture seems a bit limited.

Response:

The perceiver resampler in Flamingo [1] from DeepMind is proposed to extract fixed-length feature vectors from a single image. Its structure is similar to Q-former [2], both adopting the learnable query to sample feature vectors from the input image. It's not designed and not able to handle the challenges of modality fusion (cross-view imagery fusion in our work).

In contrast, our method is designed for cross-view images to jointly learn the region embeddings. Our module is designed to enable the complementary fusion of cross-view knowledge and get semantically rich urban imagery features. We also introduce a gating mechanism to adaptively fuse the features from different views, which can effectively solve the problem of isolated visual knowledge in different views.

[1] Alayrac J B, Donahue J, Luc P, et al. Flamingo: a visual language model for few-shot learning[J]. Advances in neural information processing systems, 2022, 35: 23716-23736. [2] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.

Q3: It seems the performance of UrbanMLLM was compared against general MLLMs which were not trained on the same dataset. Since only UrbanMLLM was trained on both the satellite and cross-view images from the authors dataset it is not very surprising that UrbanLLM's performance is overall better. Thus, the comparison in this paper does not seem to be quite apples-to-apples.

Response:

In the ablation study of the paper, we have provided the results of the fine-tuned version of VILA with our dataset, which has lower performance than our method. We now provide comparisons with (1) VILA without any further training (2) VILA fine-tuned with our instruction tuning data and (3) our UrbanMLLM as follows. The results demonstrate that UrbanMLLM outperforms VILA with fine-tuning on the same dataset, demonstrating the effectiveness of our method.

Results of satellite imagery-based urban understanding tasks:

Results of street view imagery-based urban understanding tasks:

Results of cross-view imagery-based urban understanding tasks:

Q4: In the abstract and introduction, I think it would be good if the authors could be a bit more specific about what kind of applications their UrbanMLLM is supposed to support. There are many urban applications and clearly UrbanMLLM is only suitable and targeted towards a subset of them.

Response: Thanks for the suggestion. We have added more details about the applications that UrbanMLLM can support in Introduction.

In summary, our model is focused on urban understanding tasks using satellite and street-view images including: scene classification, object reasoning, spatial relationship reasoning, geo-localization, indicator prediction and landmark reasoning.

Q5: The authors state that the dataset covers the whole United States. They also mention that 2 million images where used. I would assume that coverage of the whole US would require more images. So this dataset must be very sparse. How did the authors decide which images to select?

Response: As mentioned in Q1, we randomly sample thousands of points in each census tract polygon and use these points to query street view images. About 200 random but uniformly distributed images in each census tract are selected, we observe that these images have been dense enough to serve as a good representation of the region. Moreover, we notice that there are only around 36 images used in previous works to represent communities with similar size, which can already produce good results in indicator prediction tasks. Therefore, we believe the sampled images in our dataset are well representative.

Q6: For some urban computing tasks there exist specialized architectures, e.g., for cross-view localization. The authors may want to refer to some of the work, and also make it clear that they have a more general model in mind.

Response: As for other urban computing tasks using satellite imagery and street-view image like cross-view localization, there are indeed some specialized models, such as Sample4Geo [1] and MFRGN [2]. However, these models are designed for specific tasks and can not be generalized to other urban understanding tasks. By comparison, our model is more general and can be applied to a wide range of urban understanding tasks, including scene classification, object reasoning, spatial relationship reasoning, geo-localization, indicator prediction, landmark reasoning, etc.

[1] Deuser F, Habel K, Oswald N. Sample4geo: Hard negative sampling for cross-view geo-localisation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 16847-16856. [2] Wang Y, Zhang J, Wei R, et al. MFRGN: Multi-scale Feature Representation Generalization Network for Ground-to-Aerial Geo-localization[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2574-2583.

Q7: "imapact" -> impact

Response: Thanks for pointing out, we have corrected it.

Q8: "The results showcase that UrbanMLLM achieves state-of-the-art performance, which successfully ." There is some text missing here at the end of this sentence.

Response: Thanks for the pointing out and we have revised the paper to correct that. This revised sentence is "The results showcase that UrbanMLLM achieves state-of-the-art performance, which successfully demonstrates the effectiveness of the proposed model for urban understanding tasks".

Dear Reviewer iqRR,

Thank you again for reviewing our manuscript. As the discussion deadline approaches, we’d like to confirm whether our responses have addressed your concerns. If any clarification is needed, we are happy to provide further details. Additionally, if you feel the revisions are satisfactory, we would appreciate it if you could reconsider the paper’s score.

Thank you for your time and consideration.

Best regards,

Submission4194 authors