UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction

[W1] It seems that the generation of the corpus, which is a key ingredient of the framework, lacks a systematic and scientific methodology for its generation.

Sorry for the confusion. As mentioned by the reviewer, the corpus is very important for the UrbanKGent framework. Therefore, we construct the corpus by uniform sampling (Line 308-310) from multi-source urban data (i.e., AOI, road network, POI, review, and web page), which are widely used for urban knowledge distillation. By finetuning with the generated corpus, we demonstrate the performance of LLMs on UrbanKGC tasks could be largely improved. We will provide a more clear description of the corpus generation in our final paper version to avoid potential confusion.

[W2] The transferability is questionable – is this method useful in cities that are not finetuned?

Thanks to the reviewer's question to help us clarify potential confusion. The proposed UrbanKGent framework could enhance UrbanKGC performance in both fine-tuning and non-fine-tuning scenarios. Specifically, as demonstrated in Table 3 of our paper, the performance of various LLM backbones consistently improves only using the UrbanKGent Inference pipeline without fine-tuning, compared with zero-shot reasoning and In-context-learning paradigms. Therefore, in cities where the LLM has not been fine-tuned, our method retains practical usage value.

[W3] The description of the roles of GPT4 and the smaller LLM that is finetuned is vague.

Sorry for the confusion. As mentioned in Section 4.3 (Line 286-288) in our paper, we derive GPT-4 for trajectory generation consisting of two steps (i.e., the instruction generation and iterative trajectory refinement shown in Figure 4). The obtained trajectory will be further used to fine-tune the small LLMs. The fine-tuned smaller LLMs then will be used for UrbanKGC tasks with faster inference speed and lower cost.

[W1] The computational efficiency of the proposed algorithms and their scalability with the size of the dataset is not thoroughly discussed.

We sincerely appreciate the reviewer's valuable comment, which helps us improve the quality of our paper. As suggested, we provide detailed inference latency for the UrbanKGent family across different dataset scales in the following table. As shown in Table 1, by applying VLLM [1] techniques to accelerate our current framework, UrbanKGent-13B can automatically construct an UrbanKG with approximately one million triples (905,442 extracted from the NYC-Large dataset) in about 62 minutes. We also report the average inference time of the UrbanKGent family. For every 1,000 data records processed, the UrbanKGent takes 0.57 minutes on the 7B version, 0.76 minutes on the 8B version, and 1.53 minutes on the 13B version, respectively.

Table 1: The inference latency comparison of UrbanKGC using the UrbanKGent family before and after using VLLM acceleration. We use two middle-size datasets (i.e., NYC and CHI) and two large-scale datasets (i.e., NYC-Large and CHI-Large) for UrbanKG construction. The accelerated inference latency is bolded.

Currently, the maximum dataset scale we used is about forty thousand. We are working on processing larger-scale data and analyzing potential scalability issues.

Reference:

[1] Kwon, Woosuk, et al. "Efficient memory management for large language model serving with pagedattention." Proceedings of the 29th Symposium on Operating Systems Principles. 2023.

[Q2] As mentioned in Appendix Section D.1, human annotators are employed to fill out an evaluation form. Is it possible to provide more detailed information on human annotators, such as an exact number of human annotators?

Thanks to the reviewer's valuable suggestion. We invite 3 AI experts to fill out the evaluation form of the prediction results from various LLM variants. All of them possess work experience as algorithm engineers at Internet or AI companies. To avoid potential performance bias (as a priori, it is believed that larger-size LLMs are better), we do not reveal which results come from which LLMs. This rigorous process resulted in the annotation of 200 random samples. We will provide a more detailed description in Appendix Section D.1 in our final paper version. We sincerely appreciate the valuable suggestions provided by the reviewer.

W1&Q1

Due to the tables related to weakness 1 and question 1 are included in the Supplementary PDF of the Author Rebuttal, we have moved the corresponding responses to the Author Rebuttal for better understanding. We apologize for any inconvenience this may cause.

W2,W3&Q2

We choose geospatial relations for KGC tasks for two reasons. First, spatial relations are the majority in UrbanKG (about 60% in both datasets, as reported in Table 15 of PDF). Second, as validated by recent works [1-2], spatial relation understanding is one of the most challenging tasks for LLMs. Therefore, we use geospatial relations to demonstrate the effectiveness of the UrbanKGent in KGC. Other relations, including temporal and functional, can be predicted using this framework by extending corresponding instructions. For example, by injecting the time information (e.g., the date of building was built) of urban entities into the instruction, we can complete their missing temporal relation (e.g., built earlier than). The capability of UrbanKGent to identify other relation types is also echoed by its success in RTE tasks.

Regarding external tool use, it is possible to invoke tools to calculate certain geospatial relations, but not all, with sufficient geospatial information. For example, given the Region Connection Calculus (RCC) of two entities, we can derive up to 5 defined geospatial relations using the GIS system. However, some spatial relations may not be able to be extracted using tools, especially when entity information is incomplete, or the corresponding relation type and tool are unknown. Overall, we acknowledge the merit of using external tools to derive urban relations. In fact, we have devised tool use as an indispensable block in the tool-augmented trajectory refinement block, and are working to incorporate more spatiotemporal tools to improve the effectiveness of our framework.

Reference:

[1] GeoLM. EMNLP. 2023. [2] Are Large Language Models Geospatially Knowledgeable? 2023.

W4&Q4

We use UrbanKGent-7B as the final model, with the same setting in the overall experiment (Section 5.2) on the NYC dataset. For the ablation study (Line 720 - 724), we remove the knowledgeable instruction template ($UrbanKGent-7B^{\spadesuit }$), multi-view design ($UrbanKGent-7B^{\star }$), external geospatial tool invocation ($UrbanKGent-7B^{\ddagger}$), and iterative trajectory self-refinement ($UrbanKGent-7B^{\dagger}$) respectively from UrbanKGent-7B to validate their effectiveness. The results show that removing any of these modules will lead to performance degeneration, demonstrating their effectiveness. We will add more details in E.2 and provide more descriptions in the caption of Table 10.

W5&Q3

We prompt the backbone LLM to automatically judge if a trajectory is faithful and provide refinement feedback for the unfaithful trajectory, more details are explained in Lines 272-274. Trajectory Updater will then follow the feedback to refine the current trajectory. For the maximum iterations, as explained in Lines 280-282, we set it to 3 to avoid excessive cost. To further understand the role of iteration numbers, we set maximum iterations from 0 to 10, and report the model's performance as well as the average stopping epochs in which the predefined stopping condition is satisfied (i.e., all trajectories are faithful).

As can be seen, the model performance drops without iterations. However, larger than 3 iterations only lead to marginal improvements. This suggests that 3 iterations are cost-effective for the model to achieve the predefined stopping condition.

Moreover, we agree with the reviewer that LLM cannot work as a good judger if it lacks urban knowledge. As shown in Table 10, the $UrbanKGent-7B^{\spadesuit }$, whose knowledgeable instruction template (Line 720-721) is removed, performs poorly, although it incorporates the iterative trajectory self-refinement method. Regarding the correctness of the refined trajectory, the performance improvement reported in the above table indicates that the refined trajectory is more accurate. The reported case study in Figure 8 in our paper also echos its effectiveness, e.g., the model can identify some missing triplets after self-refinement.

W6&Q5

The constructed instruction dataset for fine-tuning can be found in Supplementary Material. As suggested, we provide statistics and the cost of prompting GPT-4 for instruction generation in the following Table:

As can be seen, 6,257 instructions are used and the cost of calling GPT-4 is 30.58 dollars.

W7&Q6

Existing UrbanKG construction methods rely heavily on manual extraction of urban entities and relations, as detailed in Lines 394-400. These manual methods, while effective, are not directly comparable to our automated approach without corresponding entities and relations annotated in the training data.

However, we agree that a comparison can still provide valuable insights. To this end, we performed a quantitative comparison between previous manual methods and our proposed automatic paradigm using the latest UrbanKG benchmark [1]. As shown in Table 4 in our paper, UrbanKGent can construct UrbanKGs with the same scale of triplets and entities in the benchmark by using only one-fifth of the data. Furthermore, our method expands the relationship types by a hundred times, demonstrating improvements in efficiency and comprehensiveness.

This comparison underscores UrbanKGent's potential for automatic UrbanKG construction, making it a competitive alternative against manual methods.

Reference:

[1] UUKG. NeurIPS 2023.

[W1] As the size of the dataset grows, the algorithmic efficiency and scalability of UrbanKGent may become an issue. Although not mentioned in the paper, in practical applications, the processing of large-scale data sets may require more computational resources and time. Although the paper notes limitations, the specifics may require more elaboration. For example, under what conditions may the model perform poorly and how these limitations affect the quality and reliability of the final knowledge graph?

Thanks for the reviewer's insightful question, which helps us improve the quality of our paper. We agree with the reviewer that efficiency is important in practical agent deployment. As suggested, we provide a more detailed efficiency analysis of UrbanKGent in the following table, and we also use LLM acceleration techniques VLLM [1] to improve the scalability of UrbanKGent. As shown in Table 1, by applying VLLM techniques to accelerate our current framework, UrbanKGent-13B can automatically construct an UrbanKG with approximately one million triples (905,442 extracted from the NYC-Large dataset) in about 62 minutes. We also report the average inference time of the UrbanKGent family. As can be seen in Table 1, For every 1,000 data records processed, UrbanKGent takes 0.57, 0.76, and 1.53 minutes on 7B, 8B, and 13B versions, respectively.

Table 1: Comparison of UrbanKGC inference latency (minutes) using the UrbanKGent family before and after using VLLM acceleration. We use two middle-size datasets (i.e., NYC and CHI) and two large-scale datasets (i.e., NYC-Large and CHI-Large) for UrbanKG construction. The accelerated inference latency is bolded.

Currently, the maximum dataset scale we used is about forty thousand. We are working on processing larger-scale data and analyzing the potential performance degradation and KG quality issues. Thanks to the reviewer for introducing these interesting research questions, and we will discuss them in the Section "Limitation and Future Work" in our paper.

[W2] UrbanKGent may face privacy and fairness challenges, especially when handling personal or sensitive information. I wonder what the author thinks about privacy and fairness. Although these issues are not discussed in detail in the paper, in actual deployment, if not properly addressed, they can lead to violations of personal privacy or unfairly affect specific groups.

Thanks for the reviewer's insightful question. Currently, we collect raw urban data from various public data providers (e.g., OSM, Wikipedia) to construct UrbanKGent. Although sensitive information is not a critical issue in these open-source data, we agree with the reviewer's concern regarding potential data privacy and fairness issues once UrbanKGent is deployed. To address this, we can further perform safety alignment [1] in UrbanKGent to control its outputs. In addition, we can subscribe to external LLM services to filter the malicious information of UrbanKGent's output. The above two strategies have been widely used in many online LLM reasoning services (e.g., ChatGPT, Ernie Bot, and Tongyi Qianwen), and we believe they would be the practical solution. We will further discuss this in the "Limitations and Future Work" section of our paper to provide more insights.

Reference:

[1] Ji, Jiaming, et al. "Beavertails: Towards improved safety alignment of llm via a human-preference dataset." Advances in Neural Information Processing Systems 36 (2024).

[W3] The UrbanKGent framework is mainly aimed at urban knowledge graph construction tasks. I wonder if the author has considered the applicability of other types of knowledge graph construction tasks. Can the UrbanKGent framework proposed in this paper be further extended to the knowledge graph construction in other fields?

Thanks for your insightful comment and question! Yes, although the proposed framework is designated to the urban domain, the construction pipeline can be easily extended to the construction of knowledge graphs for other domains. One of the most straightforward ways is to modify the instruction template to adapt this framework to other domains. For example, we can replace the urban knowledge currently encoded in the RTE instruction template with other domain knowledge, to adapt our framework for the knowledge graph construction in other fields.