Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework

Q1. Table 1 involves a comparison between open/closed source single LVLMs with smileGeo-single. However, smileGeo appears to primarily focus on a multi-agent framework, without introducing any new single LVLM architectures.

Thank you for your comments. In fact, Table 1 compares the results directly generated by the LVLM used by each different LLM agent in smileGeo with the results discussed using the smileGeo framework for all LLM agents. The experimental design of Table 1 aims to demonstrate that the LVLM possesses a certain reasoning ability and can infer more accurate results by incorporating external information when discussing with other LVLMs. Additionally, we have included the comparative results of different agent frameworks in Table 2. We will refine the structure of the experiments to avoid any confusion.

Q2. The comparative results of different agent frameworks without web searching are reported in Table 2. It's important not to unfairly advantage one framework over others by using superior LVLMs.

Thank you for your comments. Sorry for the confusion about the experiment settings. To ensure the fairness of experiments, all frameworks utilize the same number and types of LLM agents. The only difference lies in the architectures used for discussion among the LVLMs. We would fill in more details of experiment settings to avoid confusion in the revised version.

2.1 How are 'acc' and 'tks' determined for each framework?

Thank you for your questions. We have clarified the meanings of 'acc' and 'tks' below Table 2: 'acc' stands for the accuracy of the framework, while 'tks' refers to the average number of tokens a framework uses per query (including image tokens). Additionally, the calculation of 'acc' is provided in Section 4.1, Evaluation Metrics. The number of tokens is determined by the length of the query sentences and the pixels of the images: the longer the sentence or the higher the image resolution, the more tokens there are.

2.2 Which types of LVLMs are employed?

Thank you for your questions. Each framework utilizes all the single LVLMs listed in Table 1 as different LLM agents, engages them in discussions, and summarizes the final results.

2.3 Did they aggregate all LVLMs and calculate averages per framework, or utilize the best-performing LVLM specific to each framework?

Thank you for your questions. All frameworks are designed to enable all LVLMs to reach a consensus and provide a unified answer. To prevent situations where a consensus cannot be reached, we implement a majority rule after a certain number of discussion rounds, ensuring a unified answer is recognized by the majority of agents.

Q3. Question about the comparison with LLM/LVLM-based agent frameworks:

3.1 For the integration frameworks you compared in Table 2, what specific LVLMs were integrated within the LLM-Blender, LLM Debate, and smileGeo frameworks?

Thank you for your questions. All frameworks use the single LVLMs listed in Table 1 as distinct agents, engage them in discussions, and summarize the final results.

3.2 Did you use the same LVLM combinations for the different frameworks in the comparison? Different LVLM combinations may have different underlying behavior on metrics.

Thank you for your questions. Yes, we use the same LVLM combinations for the different frameworks in the comparison. We apologize for any confusion caused and will refine the experimental section to ensure a clearer expression.

Q1. The authors could make the geo-localization setting more clear in the introduction, for example, the paper focuses on worldwide city-level geo-localization. There are lots of different settings for geo-localization problem and this could be confusing for some researchers.

Thank you for your suggestions. Our task aims to achieve city-level localization for any landmark worldwide, without assuming access to a limitless image database. In the revised version, we will clarify the problem of worldwide city-level geo-localization in the introduction. We will detail the differences, difficulties, and challenges of this task compared to high-precision positioning in a local area. Thank you again for the valuable feedback.

Q2. This paper provides a comparison with three traditional geo-localization methods, i.e., NetVLAD, GeM, and CosPlace. However, these three methods are either retrieval-based landmark matching methods or fine-grained classification-based place recognition methods. It would be better to provide a direct comparison with worldwide geo-localization method on city-level setting, e.g., [A]. Although I believe LVLM-based method is better at this setting, a comparison can make it more convincing.

Thank you for your advice. We have added a direct comparison with the model under the same geo-localization settings. The comparative results (accuracy in %, without web searching) on our dataset, GeoGlobe, are shown below:

Due to the limited rebuttal time, we did our best to train the ViT-based model presented in paper [A]. To ensure fairness, we trained all the models for the same length of time (3 days) using the same hardware configuration, and then used the same test dataset to test each model separately to obtain the results. The results demonstrate that our method significantly outperforms the referenced method.

Q3. There are only two qualitative results in the appendix. Given that the accuracy is over 60%, it should be easy to find successful and failed cases to demonstrate the actual output cases of the proposed methods. It can also better illustrate how multiple agents help the geo-localization process.

Thank you for your suggestions. In the revised version, we will supplement the paper with detailed case studies to describe both successful and failed cases.

An example of a successful case is illustrated in Figure 6. While a single LVLM may not directly identify local landmarks, it possesses relevant geographical knowledge about the landmarks. Our framework can stimulate reasoning capabilities among LVLMs, resulting in correct positioning.

Regarding failure cases, we will show, for example, that increasing the number of the same LLM agents in Figure 2 leads to excessive repeated and redundant information in the discussion, negatively affecting the experimental results.

Additionally, we will include cases illustrating that web search results provide extra information to our LLM agent framework. This can achieve better outcomes when our framework does not rely on Internet search tools.

Thank you again for your valuable advice. We will incorporate these changes in the revised version.

Q4. There are also some existing worldwide geo-localization datasets that could be used for more comprehensive evaluation, e.g., IM2GPS3K, YFCC4K.

Thank you for your comments. We conducted further experiments on those datasets, and the results are illustrated in the table below (accuracy in %, without web searching). We also include the comparative results of the best single LLM agent, Gemini-1.5-pro, in our framework and TransLocator as illustrated in paper [A].

Although all models, including ours, have generally lower accuracy on these datasets, our method still outperforms the others.

It is worth noting that the YFCC4K and IM2GPS3K datasets do not apply artificial filtering to the images, resulting in ambiguous images with almost no geographical clues, such as food photos and portraits. This issue is consistent with the problem mentioned in Section 4.1 of the paper [A] cited in Question 2 above. Therefore, in our research, we invested significant time and effort to construct a novel dataset, GeoGlobe, to better evaluate the worldwide city-level geo-localization task.

Q1.The paper only seems to tackle the problem of geolocalizing landmark images. While this is a challenging problem, the current literature [1-3] has already tried to address the problem of geolocalizing arbitrary ground-level images. The latter problem requires learning sophisticated geographic and visual features.

Thank you for your comments. Geo-localization of ground-level images is indeed challenging and will be a focus of our future work. Our current work primarily addresses the geo-localization of landmark images, which we mentioned in the abstract.

In our work, we tackle worldwide city-level geo-localization without relying on an extensive image database, unlike the methods in previous studies [1-3], which compare similar images from a backend database. Such a database reliance can limit model performance.

Moreover, geo-localization requires understanding complex geographic and visual features. We leverage the knowledge and reasoning abilities of LLM agents to design a new framework. We will clarify it in the introduction and apologize for any confusion.

Q2.I think even searching the internet cannot effectively solve the geolocalization problem for non-landmark images.

Thank you for your comments. I agree that it is very difficult to locate ambiguous images, such as selfies. However, as noted in the second sentence of the abstract, the focus of our work is to perform city-level geo-localization of various landmarks (not only famous landmarks) worldwide. Additionally, we manually constructed a geolocation-based dataset, GeoGlobe, to filter out most images whose locations could not be determined.

Q3.Limited applicability: The framework is built entirely upon the capabilities of different LVLMs. It seems the framework cannot generalize beyond the training data used for training LLMs.

Thank you for your concerns. A single LVLM indeed faces these challenges, as many large models attempt to consume vast amounts of data for pre-training. Our motivation is to address the biases in pre-trained LVLMs by combining their strengths. We designed smileGeo using various LLM agents, and experiments show that it outperforms any single LVLMs. Additionally, web searching results can provide extra information, making the framework more robust.

Q4.The work fails to address the practical applications and real-life use cases. Why do we require such a framework?

Thank you for your questions. Geo-localization holds significant application value. Beyond the applications such as robot navigation mentioned in the introduction, many popular consumer-facing applications also rely on geo-localization technology, as noted in the privacy policies of apps like Twitter. By geo-locating social media pictures posted by users in real time and analyzing their mobility patterns, these applications can offer tourists personalized recommendations for attractions and itinerary planning. Along this line, our framework achieves more accurate geolocation, thereby enhancing the usefulness and precision of such applications. We will refine our explanation in the paper and apologize for any confusion caused.

Q5.The limitation and failure cases are not adequately mentioned.

Thank you for your comments. We recognize the limitation mentioned in the second point of the checklist, noting that the framework currently relies solely on Internet search tools. However, we believe the framework has potential beyond this. As stated in the conclusion, "Looking ahead, we aim to expand the capabilities of smileGeo to incorporate more powerful external tools beyond just web searching."

In the revised paper, we will include failure cases in the appendix to provide readers with a better understanding of our model. For instance, we will demonstrate a scenario where increasing the number of identical LLM agents in Figure 2 leads to excessive repetition and redundancy, negatively affecting the results.

Q6.At present, each agent sees the same information. It might be interesting to incorporate different kinds of information that is revealed differently to the agents.

Thank you for your suggestions. Providing each LLM agent with a different view of the image is indeed an intriguing idea. However, it has settings that are different from our learning task. Allowing all LLM agents to access the complete image ensures that each agent can fully analyze the data from its unique perspective. Our proposed framework aims to integrate the memory and reasoning capabilities of different LLM agents, leading to improved accuracy of geo-localization tasks.

Q7.How much compute time is used for a single inference run?

Thank you for your questions. The 99% Response Time for smileGeo is less than 25 seconds. This efficiency is largely due to the agent selection model within our framework, which minimizes unnecessary question-answering and communication overhead with large models. Additionally, the slowest LLM agent in the framework has an average response time of less than 500ms, and the average latency for API calls within our servers in the data center is within 50ms. We also limit each question to a maximum of 20 rounds of discussion. Therefore, our model is computationally efficient and significantly outperforms other LLM agent-based frameworks. We will add this explanation to the final version.

Q8.Why does the performance of some LLMs decrease with web searching?

Thank you for your concerns. Some web search results introduce noise into geo-localization. Our web search primarily relies on the Google search engine, which inherently includes advertising URLs and contents similar to our queries. Models with weaker reasoning abilities are more susceptible to being influenced by this noise. It is consistent with what we highlighted in Section 4.2: "Models with larger parameters demonstrate superior reasoning abilities compared to smaller models". We will include a more detailed explanation of this in the revised version.