UrbanPlanBench: A Comprehensive Assessment of Urban Planning Abilities in Large Language Models

1. The quality of the proposed datasets needs to be discussed and further clarified.

• For UrbanPlanBench, the paper does not mention any engagement of experts from the corresponding discipline, i.e. urban planning, making this domain-specific benchmark less authoritative and credible.
• The annotation procedure of UrbanPlanBench is not described, such as how the urban planning qualification exams are accessed, how the exam questions are selected and adapted, if there are any cross-annotator validations to ensure the quality, and what is the annotator agreement of gold answers.
• Similarly, the authors did not include any information about how they design the expert evaluation for UrbanPlanText’s quality. It is unclear how the experts were instructed to give the “correctness” and “informativeness” scores, and how they relate to the quality of UrbanPlanText.
• If the question sources are available online, the authors should also discuss potential data contamination concerns and their impact on the evaluation of LLMs. For example, would the LLMs train on Chinese corpora have already seen the questions and answers so that they perform better?
• The authors claim to use UrbanPlanBench “to assess their [LLMs’] acquisition of planning skill” (line 71), but the benchmark is only testing LLMs’ domain knowledge. Suppose an LLM can answer most of the questions in the benchmark correctly, it is still not obvious how it can help in real-world urban planning tasks, such as predicting population growth and geographic analysis.

2. Several technical details are missing from the paper.

• The implementation of Self-RAG on different LLMs is not introduced. The statistics and examples of exam_1 and exam_2 corpora used for retrieval are not provided. The configuration, training process, and inference hyperparameters of Self-RAG are missing.
• The details of few-shot and zero-shot CoT prompting are missing, e.g. the number of in-context examples in few-shot, how they are annotated/retrieved, and prompt templates.
• While it is straightforward to use output probability to select the answers for MCQ-S, it is not explained how the authors apply this strategy to MCQ-M questions, where the questions have multiple answer choices in nondeterministic numbers.

1. UrbanPlanBench seems to focus solely on processing data from the 2022 urban planning qualification exam, which limits its contributions:
   • The authors converted the original 2022 qualification exam documents into csv format, creating the UrbanPlanBench, which comprises 300 MCQs. Could you add a brief discussion highlighting your contributions to the benchmark beyond the data processing steps?
   • Additionally, since the evaluation data is sourced from publicly available texts, it is difficult to ensure that large models did not encounter this data during pre-training. This could undermine the usability of UrbanPlanBench and affect fair comparisons between models. Moreover, as the data is in Chinese, models like Qwen, trained on larger Chinese corpora, show better urban planning performance, raising concerns about whether they have already been trained on these data.
2. The experiments may be insufficient:
   • In Sec 2.3, this paper investigates prompting techniques, including RAG and CoT. In Sec 3, this paper evaluates how fine-tuning methods could enhance the capabilities of LLMs. However, while the former is based on GPT-4o-mini and the latter on other LLMs, their results are not comparable. What we may really care about is the comparison between these two types of methods.
   • The paper mentions "a challenge in sourcing relevant data for SFT". Given the difficulty in obtaining this data, why use SFT to enhance the models' capabilities if prompting techniques, such as RAG and CoT, have been proved effective?
3. The effectiveness of the SFT methods:
   • After SFT on UrbanPlanText, as illustrated in Table 4, 70%, 50% and 40% of LLMs exhibited decreased accuracy on the full questions of S1, S2 and S3, respectively. However, the models after SFT do not show performance improvements on many test sets.
4. Other weaknesses:
   • There is a typo in Line 284 and 464: "MCS-S" should be "MCQ-S".

1. The size of the benchmark is too small. It is also limited to multiple-choice questions (no short open-domain questions). The reviewer is also unsure about the diversity and generalizability of the chosen questions. It seems that all the questions are from urban planning in China. However, according to S3 Management and regulations, such questions might not be applicable to urban planning in other countries. In Table 1, we observe higher performance of models from China, which strengthens my concern.
2. The collection and quality control of the dataset is not well introduced. For example, what is the data source, human verification on the data correctness/categorization, and the inter-annotator agreement?
3. The take-aways are blurred since there are not many experimental details introduced: e.g., how the model performance varies with hyperparameters such as temperature, how the RAG and CoT methods are designed, and most importantly, how the data is collected with quality control.
4. From the reviewer’s point of view, there is a slight overclaim issue in the paper: this is not a benchmark on the general cross-culture urban planning with realistic scenarios that may assist real human experts. It is more like a QA dataset on the urban planning problem of some specific cultural background. The reviewer still acknowledges UrbanPlanText as a good contribution. It would be good if the authors could extend the benchmark to realistic settings that might help experts, beyond just question answering, e.g., retrieving useful cases.

While this benchmark introduces a new domain, urban planning, it doesn't fundamentally expand beyond the scope of existing benchmarks. Its formulation is quite similar to widely used evaluation benchmarks like BigBench and MMLU, with the primary difference being just the new domain focus. This makes the dataset's contribution feel less novel, as it isn't that different from prior works. Similarly, the training method lacks novelty, as it relies on leveraging domain-specific resources from the web, which is already a common practice.

To enhance the paper, the benchmark's evaluation could move beyond a multiple-choice QA format. A more realistic approach might involve simulating a full urban planning task: for example, an agent would plan steps for constructing a house or shopping mall in a given city or district, considering the environment state in a simulated setting. This could yield more interesting insights into agent performance, reveal complex failure patterns, and potentially offer valuable contributions to real-world urban planning.