ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning

1. The success rate of search-algorithm-powered language agents falls below expectations since DFS theoretically guarantees a solution if one exists (although the search space may be vast). The reported success rate of only 51.3% is attributed to "arbitrary descriptions for defined concepts" and the "emergence of undefined concepts." Given that LLMs cannot access the entire database, they can only plan with the information provided in the environment. This limitation is more of a design issue within the system itself than a deficiency in the LLMs, as expecting agents to plan beyond their accessible information is unreasonable, even for humans. Solutions such as fuzzy matching might mitigate this issue. I wonder if the results could approach the near 100% success rate achieved in the previous approach designed for TravelPlanner[1].
2. In the experiment, before performing DFS, all constraints are extracted, and then hard-coded rules guide the agents during the search process. This raises the question: if constraints are already extracted and hard rules still need to be manually coded, what is the added value of using language agents? Would it not be more efficient to complete the system without involving language agents at all? If the goal is to develop an autonomous agent, extracting constraints and designing code manually seems counterproductive.
3. This work seems more like an extension of the previous TravelPlanner, adding extra constraints and adapting the setting to suit the China Travel context rather than introducing a new novel and timely benchmark the community needs now.

Reference

[1] Hao Y, Chen Y, Zhang Y, et al. Large Language Models Can Plan Your Travels Rigorously with Formal Verification Tools[J]. arXiv preprint arXiv:2404.11891, 2024.

It is important to note that most of these constraints (environmental, logical) are extensions from existing works, enhancing their coverage aspect.

Major Comments

1. Comparative Baseline:The empirical study could benefit from a broader range of reasoning-action-based frameworks. In particular, it would be valuable to see a comparison with “Reflexion,” a reasoning-action framework considered in the TravelPlanner benchmark. This would offer a more comprehensive evaluation of ChinaTravelPlanner’s performance relative to established approaches.
2. Alternative Model Performance (Section 4.2, Line 432):In Section 4.2, the paper mentions that “...with many models failing entirely...”. Could you clarify whether additional models beyond those in Table 2 were evaluated? If so, details on their performance, especially regarding delivery, environmental, and logical pass rates, would add clarity and highlight the robustness of the proposed method.
3. Table 2 and Preference Requirements Constraints:The paper introduces preference requirements constraints, yet they are not referenced in Table 2. Including these would provide a fuller picture of constraint adherence and improve the interpretability of results across all introduced metrics.

Minor Comments (Spelling and Grammar Corrections)

• Line 140: Please revise “They integrates…” for subject-verb agreement to improve grammatical fluidity.
• Line 454: Correct “Nesy planning” to “NeSy Planning,” and add a missing period at the end of this sentence for completeness.

1. Compared to TravelPlanner, the differences appear to be limited, even though the authors have highlighted some distinctions in Table 1. While the most significant distinction lies in spatial coverage, other differences, such as constraints and metrics, appear less pronounced. For instance, I believe one of the most crucial metrics is the ability to guide users in articulating their requirements and to interact with them through multi-turn dialogues to refine the schedule.

2. The evaluation could be enhanced by including more results from various large language models, particularly open-source models. For example, you can consider Qwen2.5[1], LLama3.1[2] and Mistral-Small[3], in your experiments.

[1] Qwen2.5 https://qwenlm.github.io/blog/qwen2.5/

[2] Llama3.1 https://ai.meta.com/blog/meta-llama-3-1/

[3] Mistral-Small https://huggingface.co/mistralai

3. Based on the cases presented in Figure 4, the performance of large language models may be underestimated. For instance, the arbitrary descriptions in the two cases are not particularly challenging for large language models, as they simply require common sense knowledge within the travel service domain.

Although I appreciate the effort to create large-scale travel planning dataset that will be helpful for future research, I have some concerns regarding the technical and experimental novelties of the paper:

1. The constructed dataset has only 154 queries which are generated from survey with 250 users. Therefore, it is not clear to me whether the dataset covers all possible real-world travel plan constraints and user preferences.
2. Additional constraints and requirements are generated from LLMs, but it is not described how these are validated. An automated validation method would have been great to generate large-scale data with high coverage of potential constraints.
3. The proposed NeSy planning algorithm is a simple greedy extension of neuro-symbolic planning. So, it is hard to validate the quality of the solution generated from NeSy planner. Is it possible to identify some bounds on how far the solution is from optimal preference?
4. How is the preference data handled by NeSy planner? For travel planning, it is essential to understand how LLMs deal with human preferences (e.g., optimized cost, minimal travel time and so on).