Exploring and Benchmarking Planning Capabilities of Large Language Models

On the coverage of classical planning domains. This paper argues that the proposed work lays the foundation for improving the planning capabilities of LLMs by presenting three classical planning domains. From a classical planning viewpoint, those domains were more than 20 years old and they don't have many features in PDDL and only have almost the minimal subset of the language features. Namely, it is almost STRIPS with a negative precondition. If we view the direction of translating PDDL to NL, the work should consider the coverage of the translation of the language.

On the quality of translations. The paper mentioned that the translation is done by regular expressions. LLMs are supposed to be trained in human-written texts and natural languages. However, the translations are done by hand-crafted regular expressions that translate formal PDDL descriptions. It is doubtful how such a dataset would enhance the reasoning capability of LLMs. If we consider the LLM as a symbolic reasoning engine like a planner such as fast-downward, the scale of problems is so tiny that it is doubtful how the proposed methods will be useful.

The main issue with this paper is that it has very little novelty to contribute to the LLM planning community since it reproduces many results and echoes insights that have been communicated in other papers. There is a contrast in narrative to existing work that claims LLMs cannot plan using base models with no inference time techniques [1, 2]; however, plenty of work exists that individually investigate and demonstrate that prompting with examples boosts performance [3, 4], inference-time prompting techniques boosts performance [5, 6, 7, 8], finetuning boosts performance [9, 10, 11], and do similar failure analyses [2, Figure 5].

The conclusion ends with a quick note on 3 key findings regarding plan generalization: (1) the superiority of SFT (2) curriculum learning effectiveness and (3) limitations of hard example training. These points have already been brought up in [12] that uses SFT to tackle plan generalization but uses a better approach of process supervision [13].

In addition, while there are 5 datasets total experimented on in this paper, the experiments focus mainly on Blocksworld and Logistics which has already been focused on in [2], with TravelPlan and CalendarPlan only being used for many-shot and inference-time experiments and MiniGrid only being used for the many-shot experiment. It is unclear why these 5 datasets are introduced as part of a benchmark suite but only subsets are experimented on.

Overall, this paper reads primarily as a summary of well-investigated trends in the LLM planning community. To strengthen this paper, it would be useful to propose an approach atop the experiment findings to add novelty to the paper and further support the insights (for example, the balanced approach incorporating easy and hard examples in the conclusion). Since it would require non-trivial rewriting to improve this paper, I have suggested a rating of reject.

[1] On the planning abilities of large language models-a critical investigation (Valmeekam et. al 2023)

[2] Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change (Valmeekam et. al 2023)

[3] Language Models are Few-Shot Learners (Brown et. al 2020)

[4] Many-shot in-context learning (Agarwal et. al 2024)

[5] ReAct: Synergizing Reasoning and Acting in Language Models (Yao et. al 2022)

[6] Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et. al 2023)

[7] Graph of Thoughts: Solving Elaborate Problems with Large Language Models (Besta et. al 2023)

[8] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (Zhou et. al 2023)

[9] FireAct: Toward Language Agent Fine-tuning (Chen et. al 2023)

[10] Plansformer: Generating Symbolic Plans using Transformers (Pallagani et. al 2022)

[11] LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (Kambhampati et. al 2024)

[12] Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping (Lehnert et. al 2024)

[13] Let's Verify Step by Step (Lightman et. al 2023)

1. Did not find the paper to have novel results (please refer to questions)
2. No details on how the plans are being generated using FD (which heuristic is being used to generate optimal plans?)
3. The NL translated prompt does not include PDDL domain, making me question how any LLM is supposed to know all actions, preconditions, and effects for a given domain?
4. Not a complete coverage of literature on plan generation. Especially, authors in [1] introduce thought of search which steers LLMs to solve 100% planning problems. [2] already provided similar conclusions on SFT using even smaller parameter models and also talked about multiple plan generalization issues.

[1] Katz, M., Kokel, H., Srinivas, K., & Sohrabi, S. (2024). Thought of Search: Planning with Language Models Through The Lens of Efficiency. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24.

[2] Pallagani, V., Muppasani, B., Murugesan, K., Rossi, F., Srivastava, B., Horesh, L., ... & Loreggia, A. (2023). Understanding the capabilities of large language models for automated planning. arXiv preprint arXiv:2305.16151.

1. Novelty - One of the biggest contributions that the work claims is of the planning dataset. However, the paper only combines existing PDDL planning dataset (PlanBench) and travel planning domains, with the addition of some newly generated test problems. This seems to be more like packaging the two domains together than presenting a new dataset in a paper. Can the authors provide more details to counter this argument - such as - what % of problems are new in this dataset that are not present in the two existing datasets?
2. Unjustified evaluations for improved performance - It seems that the paper aims to establish that LLMs perform exceptionally well in planning domains, but at the cost of prompt engineering (line 73 - authors say that ICL methods show boost in performance with 'carefully instructing the model') and cherry-picking evaluation data (line 89 - authors say that 'data curation during training' can be helpful to avoid biases thereby helping LLMs). Why do we have to give all the benefit of the doubt to the LLMs at the expense of human effort during prompting and data processing? It is worthwhile noting the challenges that LLMs face (using failure mode analysis) without such human efforts, as they can be constructively useful for further improving their planning capabilities.
3. Fine-tuning results - The authors show near-perfect accuracy for SFT results in Tables 3 and 4, but only in the cases where the train and the eval data consisted of the similar problem cases. This is a classic over-fitting example, and thus, does not lead to any conclusions regarding the so-called 'generalization' abilities of LLMs.