AddressVLM: Cross-view Alignment Tuning for Image Address Localization using Large Vision-Language Models

Thank you very much for the constructive feedback! Below, we address the concerns raised in the review and will adjust the paper accordingly in the revision.

Response to weakness 1: First, the reason analysis introduced by GeoReasoner in the first tuning stage is designed for coarse-grained country and city-level address localization, and no global information is introduced when expanded to fine-grained street-level address localization. In contrast, our method introduces satellite images as global information in the first tuning stage, integrating the global understanding of urban street distribution into LVLMs. This is the key factor why our method is superior to GeoReasoner. Second, due to the lack of access to GeoReasoner's data and the corresponding benchmark data for its second phase, we are unable to reproduce its results on the benchmark. We fully followed the method description in the original text and specifically constructed reason labels in the first stage for fine-tuning for GeoReasoner to reproduce the method to the greatest extent possible. Third, should GeoReasoner releases its code or data in the future, we will conduct the necessary evaluation or update our results accordingly.

Response to weakness 2: Since UrbanCLIP and UrbanVLP are designed to solve the urban indicator prediction problem, rather than our image address localization problem, these two methods are difficult to directly extend to the address localization task. Therefore, we cannot compare with UrbanCLIP and UrbanVLP: 1) In terms of input, these two methods typically require satellite images as input, while our method only requires street view images for inference. 2) In terms of output, these two methods focus on predicting numerical indicators such as Carbon, Population, and GDP in downstream tasks, while our method focuses on address related VQA.

Response to weakness 3: Thanks for your suggestion! The table below presents the results on Pitts-VQA when $\delta$ is set to 0.3, 0.5, and 0.7, as well as a comparison of the outcomes for three different grafting methods. It can be observed that the results are superior when $\delta=0.5$, it outperforms both $\delta=0.3$ and $\delta=0.7$. This is because when $\delta$ is too small, there is insufficient fine-grained information from the street view images, and when delta is too large, it obscures most of the map area, leading to a lack of information from the map. Comparing the three different grafting methods, our grafting approach yields the best results. This is attributed to the fact that LLaVA-phi3-mini is designed for single-image input scenarios, creating a significant domain gap when two images are inputted (two images stitched), deviating from its pre-training conditions. Additionally, since CLIP is adapted to square input images, images with a high aspect ratio may lose token information due to resizing and padding operations (two images stitched). In summary, we have chosen the $\delta$ of 0.5 with two images grafted for input image construction.

Response to weakness 4: In Table 1, the baseline represents the results of directly fine-tuning on VQA data (stage2) without undergoing stage 1 training. It can be observed that our cross-view alignment tuning consistently and significantly enhances performance across all metrics (e.g., +8.2% on $A_s^G$, +9.08% on $A_{sd}$). Please kindly check it.

To Reviewer K9em:

We have to remind you that you have written your comments in the reply box belonging to Reviewer pFw4. The response to your further questions has been provided in the block belonging to yourself. Please kindly check it.

To validate our assumption: We believe that by carefully adjusting the data ratio, we can strike a balance between enhancing the model's general abilities and its address localization capabilities. We conducted experiments by mixing address localization VQA data with llava_v1_5_mix665k data in a 1:1 ratio, and the results are shown in the table below. It can be observed that by increasing the proportion of address localization VQA data, the performance can be further enhanced.

Please kindly check the responses.

Looking forward to your reply！

Thank you very much for the constructive feedback! Below, we address the concerns raised in the review and will adjust the paper accordingly in the revision.

Response to weakness 1: We ensure the accuracy of alignment labels in two ways. 1) As mentioned in lines 261-265 of our paper, the precise location of the street-view image is provided to the pre-trained LVLM in advance (hint: {near Forbes Avenue, Downtown, Pittsburgh}). This ensures that the model is explaining the reasons for cross-view alignment rather than predicting the reasons, thus avoiding potential errors or biases. 2) We manually check the image descriptions and alignment reasons to ensure that they are correct. As a result, compared to the baseline of directly applying localization tuning, the proposed alignment tuning stage brings significant and consistent performance gains, e.g., +9.08% and +11.83%, on Pitts-VQA and SF-Base-VQA, respectively.

Response to weakness 2: In this paper, our goal is to train a model on question-answer pairs under an open problem setting so that it can answer address-related questions in a more interactive way during the test phase. Although our method is slightly weaker than specific discriminative models with predefined address labels, our method has the following advantages: 1) Open problem setting; 2) Easy to interact with address-related question-answering format; 3) General question-answering capabilities. The key contribution of our method is integrating city-wide address localization capabilities into LVLMs, which provides a promising way of thinking how to learn address-related information from street-view images. We believe this paradigm has the potential to outperform discriminative modeling approaches, for example through larger LVLMs or exploring better tuning approaches.

Response to weakness 3: Our approach is applicable to modern cities based on administrative address divisions, so we believe that AddressVLM will achieve consistent performances for cities outside the US, regardless of the country in which the city is located. We evaluated AddressVLM on two cities in US for two reasons. 1) Both datsets are widely used in the image geo-localization task, which is convenient for comparison. 2) Our meta address information comes from the AddressCLIP, where only Pittsburgh and San Francisco in the US are covered. We agree that evaluating on wider and diverse urban scenarios helps fully demonstrate the effectiveness of our method. We are currently constructing data and validating our method on a completely different localization dataset, the Tokoy dataset [Ref 1], which covers Tokyo, Japan. Since our method includes a series of steps such as address crawling and filtering, satellite image crawling, cross-view alignment label generation, and model fine-tuning, we have not yet obtained experimental results. Once the experiments are completed, we will add the results into the revised version.

[Ref 1] 24/7 place recognition by view synthesis. In CVPR 2015.

Response to weakness 4: Thanks for the valuable suggestion. We provide the details of the proposed Pitts-VQA and SF-Base-VQA datasets in Table.6 in Appendix. B. A preview of the tabular comparisons with existing related datasets is given below.

We use the training and testing sets in the Pitts-IAL and SF-Base-IAL datasets to construct VQA data, and use them as the training and testing sets of the Pitts-VQA and SF-Base-VQA datasets, respectively, without including the database used for the VPR task in the original datasets. We will add the table of dataset comparisons in the revised version.

We are very grateful for your feedback and suggestions on our paper and look forward to further discussions with you. Enjoy your holiday!

Thanks to the authors for responding to my questions and the listed weakness.

Re-response to W1: The author claims providing location hints ensures label accuracy. However, while this guarantees correct location information, it doesn't ensure accurate visual descriptions or alignment reasoning from the LVLM, which may still generate inaccurate or biased feature descriptions. Using performance improvements (+9.08% and +11.83%) to validate label quality is logically flawed. The gains could stem from other factors like the cross-view alignment mechanism itself, rather than label quality.

Re-response to W2: "Open problem setting" is incorrectly presented as an advantage. It actually makes the task more challenging and explains the lower performance compared to discriminative models, but shouldn't be listed as a benefit. The claim that "this paradigm has the potential to outperform discriminative modeling approaches" lacks substantive evidence. Simply mentioning larger models and better tuning is insufficient - I hope the authors explain why generative approaches could theoretically surpass discriminative methods.

Re-response to W3: The response has several questionable claims and statements: "Our approach is applicable to modern cities based on administrative address divisions" – There maybe different addressing systems for the world's diverse administrative divisions. This claim lacks evidence since the authors haven't tested on cities with different addressing systems. "AddressVLM will achieve consistent performances for cities outside the US" - Making this prediction without enough evidence is not convincing. The authors mention ongoing work with Tokyo dataset but provide no concrete timeline or preliminary results, making it unclear if this will actually materialize.

Thank you very much for the constructive feedback! Below, we address the concerns raised in the review and will adjust the paper accordingly in the revision.

Response to weakness 1: Thanks for the valuable suggestion. We appreciate the reviewer for providing the benchmark reference for a comprehensive presentation. However, the two benchmarks, OpenStreetView-5M [1] or Geoguessr [2], cover coarse-grained address localization of countries and cities around the world, while our work focuses on fine-grained address localization of urban streets and district. These two benchmarks do not match our research problem. Nevertheless, we adopt another address localization benchmark, the Tokyo dataset [Ref 1], for experiments. Since our method includes a series of steps such as address crawling and filtering, satellite image crawling, cross-view alignment label generation, and model fine-tuning, we have not yet obtained experimental results. We believe that AddressVLM will achieve consistent good performances. Once the experiments are completed, we will add the results into the final version.

[Ref 1] 24/7 place recognition by view synthesis. In CVPR 2015.

Response to weakness 2: Thanks for the valuable suggestion. The vision encoders and LLMs included in current pre-trained LVLMs are typically paired in a rigid manner. Simply replacing CLIP with SigLIP or DINOv2 means we have to retrain the entire LVLM from scratch, which is not the focus of our work and is not realistic for implementations. As an alternative, we conducted a similar experiment using MiniCPM-v2.6, which features a vision encoder of SigLIP and an LLM of Qwen2-7B. The results on Pitts-VQA are shown in the table below.

It can be observed that the performance of AddressVLM based on SigLIP and Qwen2-7B is slightly higher than that based on LLaVA-phi3, which demonstrates that adopting a more powerful LVLM can achieve better results. We will add the results in the revised version.

Response to question 1: We have present some failure cases in Fig. 11 of Appendix G in the original manuscript. Our model often faces challenges in scenarios involving low-light conditions, such as tunnels, where there is a lack of recognizable features like large areas of sky, walls, and vegetation. Additionally, when examining cases near street intersections, we've observed that the model tends to mistakenly identify them as regular street views or incorrectly recognize them as street names with road signs. The limitations in low-light conditions and at street intersections belong to visual and textual challenges, respectively, which are similar to the failure patterns of existing work like GeoReasoner. The conditions where the images were captured are beyond our control, but we are committed to encouraging the model to think more deeply about cases related to street intersections thus enhancing its ability to locate addresses at intersections.

Thank you very much for the constructive feedback! Below, we address the concerns raised in the review and will adjust the paper accordingly in the revision.

Response to weakness 1: We clarify the motivation of the cross-view alignment fine-tuning and the limitations of the proposed method.

• Motivation

To realize the street-level address localization within a city, a reasonable way involves fine-tuning a well-trained LVLM using street-view question-and-answer (VQA) data with LoRA. However, this straightforward method yields suboptimal performance because street-view images are sparsely collected in terms of both location and viewpoint, which inhibits the model’s ability to build a global understanding of street distribution across the entire city. Such global information is crucial for effective address localization since street-view images are densely sampled during testing.

To enhance the global information in fine-tuning, we introduce perspective-invariant satellite images to establish connections between sparse street-view images. Satellite images are globally consistent and exhibit overlap, allowing for a mapping of the sparse street-view images to a global framework that facilitates inter-image correlations. Previous research in cross-view geo-localization has shown the viability of correlating satellite images with street-view images. In light of this, we propose a method named cross-view alignment tuning, designed to enable LVLMs to align street-view images with street addresses on satellite images annotated with street name labels.

We are sorry that it is difficult to describe the above motivation through theoretical analysis or mathematical derivation. After all, there is no theoretical or mathematical derivation for the research work of cross-view matching. This cross-view approach has been continuously verified in practice and in the work of predecessors.

• Limitation

Our method may face challenges in scenarios involving low-light conditions and street intersections. We have present some failure cases in Fig. 11 of Appendix G. For cases under low-light conditions like tunnels, there is a lack of recognizable features like large areas of sky, walls, and vegetation. For cases near street intersections, the proposed model tends to mistakenly identify them as regular street views or incorrectly recognize them as street names with road signs.

Response to weakness 2: We have provided a demonstration of the scalability of our method on a dataset with two cities by mixing Pitts-IAL and SF-IAL-Base in Tab. 4. It can be observed that when the dataset size is doubled, our approach is able to maintain performance close to that of training on a single dataset.

Response to question 1: Thanks for your suggestions! In the introduction and Fig. 1, we reviewed the related work AddressCLIP and GeoReasoner. The former is about the discriminative address localization with CLIP. Although the authors of AddressCLIP presented an experiment using VQA data, they just used a single round of fixed question template (i.e., Where might this photo have been taken? Tell me its street level address.) and only provide qualitative results. In contrast, we focus on the open-set problem and utilize multi-round VQA data with various complex question templates. Besides, we provide detailed quantitative experimental results in terms of various metrics. The latter is designed for coarse-grained country and city levels of address localization while ours focuses on fine-grained street level address localization within a city.

Response to question 2: In traditional Visual Place Recognition, both the query images and the database images consist of ground-level street views. Consequently, the core objective is to learn a robust feature extraction model that maps images of the same location to the same position in the feature space. However, in Cross-view Geo-localization, the query images are ground-level street views, while the database images are aerial perspectives, such as bird's-eye views. The central goal is to learn a feature extractor that can map these cross-perspective images of the same location to the same position in the feature space. Therefore, a sufficiently powerful feature extraction model may be well-suited for Visual Place Recognition but may not be directly equipped to handle Cross-view Geo-localization.

Response to question 3: We clarify that the proposed grafting mechanism is only formally similar to CutMix, but not related to data augmentation or the MixUp series of methods. We mentioned CutMix in the paper only to facilitate readers' understanding of our method, and provided an example in Figure 3. We will remove the confusing expression in our revision.

We are very grateful for your feedback and suggestions on our paper and look forward to further discussions with you. Enjoy your holiday!