System 1.x: Learning to Balance Fast and Slow Planning with Language Models

Thank you for your review!

Figure 3 seems to be cropped in a way that makes the method appear to be better than it is

Let us begin by first addressing your ethical concern. We unequivocally reject this claim as it is not a criticism of our paper but in fact calls into question the integrity of our research, which is completely unfounded and presumptuous with no evidence.

First, the figure on the right (3b) is for Blocksworld and the one on the left (3a) is for Mazes. Their experimental setups are different and completely independent. For BW, as noted in Lines 300-305, we follow a OOD setup – we train on plan lengths of 1-6 and test on longer plan lengths of 7-10. The setup for Mazes is in-distribution – we both train and test on plan lengths of 1-8.

Next, given this OOD setup for BW, our System-2 Planner (that is trained on search traces) obtains an accuracy of 28% (see the highest point on the green Sys-2 line to the far right; it overlaps with the violet line and hence might not be clearly visible; so you could refer to Table 4 in Appendix for exact numbers). This number is higher than the System-1 accuracy of 9% (the horizontal blue line) but expectedly, far worse than the high System-2 accuracy for Mazes (which followed an ID setup and hence, easier).

So, to summarize, the maximum achievable accuracy for BW is still 100% (not 30%). We happened to choose a scale of 0-30 because our System-2 Planner achieves an accuracy of 28% and by design, System-2 represents the skyline of a System 1.x Planner (when x=1 and System 1.x becomes the same as a System-2). If it helps, we are happy to change the scale of the BW plot also to 0-100 in a future version.

While this work attempts to build a neurosymbolic planning system, which is a good goal, it seems like the work may not be ready for publication at ICLR.

As pointed to in the general response, this is in fact not the goal of our work. As our title suggests and as is clearly stated in the abstract lines 19-20, our paper is about LLMs and how we can make LLMs alone more efficient in solving planning problems by internalizing (System-1) and externalizing (System-2) search for sub-goals based on a user-defined compute budget. The abstract and many other parts of the paper (Section 2 heading, etc) repeatedly uses the phrase "System-1.x Planner, a framework for controllable planning with language models" where the "with language models" part is important and should not be overlooked. If that is our goal, which we think we have clearly communicated in our paper, then the baselines of our hybrid System 1.x should be an LLM-based System-1 Planner and an LLM-based System-2 Planner. And if these are our baselines, which again we have compared to in our paper, States-explored is the right metric to compare against because all are LLM-based methods and States-explored directly correlates with token count and token count is what dictates the compute budget of an LLM. System-1.x outperforms both these baselines and hence, should be seen as our contribution towards making LLMs search more efficiently by internalizing easier sub-goals when operating under a budget.

Next, let us clarify the specific point about neuro-symbolic planning and comparison to A*. Our paper, in parts, does talk about a neuro-symbolic System 1.x variant. However, again the keyword overlooked in the review is variant. It’s a variant because it naturally emerges from the modular design of our framework wherein we disentangled the capabilities of a slow planner, a fast planner, and a controller (so the slow planner could also be A*). In our paper, we intentionally used phrases like "System-1.x can be converted into a neuro-symbolic variant" (line 115) and more generally, presented the neuro-symbolic results in a separate sub-section 4.2 to emphasize the point that we are not trying to build the best neuro-symbolic architecture for Mazes and BW. We absolutely understand that A* alone can solve a lot of these problems (with guarantees) and that our System 1.x may not be directly comparable to A* in terms of the States-explored metric because generating the tokens for a state by an LLM requires more computation than doing so with A*. However, wall-clock time would also not be appropriate because we are trying to make LLMs better and more efficient planners and an LLM’s typical unit of compute is token.

To summarize, we are happy to rephrase our neuro-symbolic part in a future version to further clarify our research goal. Regardless, System 1.x, even with its current presentation, is a novel framework for hybrid and controllable planning with LLMs that advances the state-of-the-art on full System-2 architectures (e.g., Searchformer, Stream-of-Search) that always make LLMs externalize search.

And if we are able to successfully communicate this research goal of ours, it should help address your other questions about guarantees of a System 1.x and such.

Thank you for your review!

A significant limitation of this approach is its dependency on multiple components performing flawlessly

The fact that System 1.x is built with three different modules should be seen as an advantage rather than a limitation. This means that one can directly swap in different versions of the System 1 Planner, the System 2 Planner, or the Controller and still retain the conceptualization of System 1.x, which, as we note, is our contribution. This is the reason why we start with general definitions of each of these components and experiment with different versions of the controller (by varying the hardness functions) and System-2 planners (LLM-based and A*). We want to emphasize that such variations are only possible because of the disentanglement of capabilities in our hybrid System 1.x. On the specific point about compounding errors, it’s always a possibility for any multi-model system; however, we show that our approach does better empirically than baselines. It remains to be seen how our method scales for arbitrarily hard planning tasks but that is beyond the scope of this paper.

A limitation of this hybrid approach is that selecting the x value too greedily can hurt performance and increase training computation costs.

This, again, should be seen as an advantage of our method that users can control "x" based on their compute budget and incidentally, also noted in one of your strengths – "This allows users to balance speed and accuracy". Naturally, there’s always going to be a trade-off between always doing deliberate search versus not doing so at all. However, the presence of a controller ensures that whatever compute budget is allowed, our system allocates them intelligently to solve the harder sub-goals.

For System-2, the hardness function should be explicitly defined for each specific domain in which the planner operates.

First, note that System-2 does not rely on any hardness function. We fine-tune an LLM to carry out search by training it on search trajectories, which in our case, comes from running A* but could generally be curated in any other way (e.g., expert demonstrations from humans, etc).

Next, if you meant that our controller requires defining hardness functions, we would like to clarify that is only for automatically generating training data (see that Algo 1 in the paper is only for training data construction). If a certain domain already has access to some easy/hard annotations of sub-goals (from a human or otherwise), the controller can be trained on that data alone without having to define any hardness function. However, since we may not always have access to such data, we show that it is also possible to automatically generate training data by using some domain knowledge, which in our case translates to hardness functions. Even such hardness functions can be quite flexible and need not be perfect because they are essentially heuristics for constructing training data – as we show in Table 10, System 1.x performs comparably for Mazes using either Obstacles or Manhattan Distance as the hardness function.

Overall, we again want to stress the importance of System 1.x as a novel framework for hybrid and controllable planning with LLMs. The specifics of how we obtain training data to build the controller can vary, based on the end-goal/domain. We hope that future works that build on top of our framework will explore such possibilities.

Thank you for your review!

How sensitive is this hybridization factor to changes in task complexity? Were there significant trade-offs observed with larger values of x in different domains?

With larger values of x, System 1.x will start approaching a full System-2. How fast (i.e., the slope of the curve) will depend on the complexity of the end task and the exact problem setup. For example, in our experiments with Mazes, we find that our System-1.5 achieves an accuracy of 70%, still a fair way short of the maximum 94% obtained by a full System-2 Planner (see Lines 370-381). Such high performance from a System-2 Planner is expected because we train and test on a similar distribution of mazes and plan lengths. On the other hand, for Blocksworld, we test on longer plan lengths than what we train on and thus, even a full System-2 Planner is able to solve a considerably smaller number of problems. In such a scenario, System 1.5 obtains an accuracy of 25%, quite close to the maximum 28% obtained by a System-2 Planner.

In summary, for certain domains, where it is possible to internalize a lot of the sub-goals, a System-1.x with even smaller values of x can get us quite close to the maximum System-2 performance whereas in other harder domains, the model might need to externalize its planning a lot more, thus requiring a larger value of x.

Could the authors explain how the controller identifies which sub-goals are in-distribution vs. out-of-distribution during inference?

Note that our controller is a fine-tuned LLM trained with certain heuristics/hardness functions as features. The heuristic used in Blocksworld is a distance measure based on the number of mismatches between the start and the goal state. Simply put, a trained model with such a heuristic would learn to map a greater number of mismatches to System-2 while using System-1 when the mismatches are less. Now in Blocksworld, we evaluate our model on much longer plan lengths than what we train on and plan length generally would be expected to correlate with the number of mismatches (our hardness function). Hence, a trained controller with such a hardness function would assign System-2 to goals/sub-goals with longer plan lengths, effectively translating to OOD.

What specific methods were used to prevent overfitting to the training traces, especially for tasks requiring extensive state exploration?

We diversify our training data generation process by having samples with different plan lengths, varying number of obstacles (for mazes) and varying number of blocks (for blocksworld) so that the model does not overfit to one particular configuration. Additionally, for blocksworld, we evaluate on much longer plans (up to 10) to test generalization.

Why is there no baseline comparison against other fast and slow architectures used for solving planning problems?

Thank you for the pointers, these are interesting studies that we’ll be happy to refer to in a future version. However, we don’t think these studies should be our baselines because our goals are separate. In particular, our goal is to study how we can leverage LLMs alone to solve planning problems while being able to reduce their computational cost when trained on long search traces (like Stream-of-Search/Searchformer do and make up our baselines). With that goal in mind, we point to the fact that all our components in System 1.x are fine-tuned LLMs with the same base architecture. If there are prior slow&fast architectures that also rely only on LLMs to solve planning problems, we would appreciate any citations and would be happy to compare to them.

This goal of ours is completely different from the following – what’s the best (neuro-symbolic) architecture for solving Mazes and Blockworld. We understand that A* alone can solve a lot of these Maze and Blocksworld problems but that should not stop us from studying whether LLMs alone can do so too and in what capacity/using how much compute and whether that compute can be intelligently balanced. This makes LLM-based System-1 and System-2 planners our baselines because they respectively make up the baseline and the skyline of a System 1.x Planner.

That said, our paper (in parts) also talks about a neuro-symbolic System 1.x which is when we compare it to A*. We do so for the sake of completeness and also because our modular design (with disentangled System-1/2/controller) allows us to have both LLM-based as well as symbolic System-2 solvers like A*. However, the goal of System 1.x is to build a purely LLM-based planning system that learns to balance deliberate/externalized and non-deliberate/internalized planning by intelligently allocating a user-defined compute budget. We believe that our method, baselines, and experimental design adequately address that research goal.

Thank you for your review and for acknowledging the originality and significance of our work.

However, System-1.x strongly rely on the presence of a hardness function, which is essentially a heuristic function. This assumption is too strong and cannot simply be listed as a limitation of this approach;

It is important to note that the reliance on a hardness function is only for automatically generating training data for the controller. Our controller is still a fine-tuned LLM that may internally use some of its pre-trained knowledge to make these hardness/sub-goal predictions (as an example, see the Controller output in Page 22). In domains where such training data is readily available, the LLM can be directly fine-tuned on that data without relying on any hardness function.

In this regard, your suggestion on using an LLM to explicitly generate hardness estimates is interesting. However, the focus of our paper was to propose a general 1.x framework that works flexibly with different choices of System 1/2 and the controller. It is definitely possible to improve upon these components (e.g., by making the LLM also estimate hardness) and we hope that future work can build on top of our framework.

To improve this paper, evaluating on environments where classical planners cannot be trivially applied would give more credibility to this LLM approach.

This is a well-received point. However, the primary baseline of System 1.x is a System-2 LLM planner that always does deliberate search (in token-space) and hence hard to implement on tasks outside of the scope of standard planning. Two prior works in this space are our baselines (Searchformer [1] and Stream-of-Search [2], cited in our paper) , which also experimented with tasks such as Maze Navigation and games like Countdown that one could argue can also be solved using A*. Hence it is important to clarify the goal of our study – it is to understand whether LLMs can be used for search and where such (expensive) search in the token-space can be bypassed without affecting accuracy (thus, System 1.x). This goal is different from whether these problems can be solved without using an LLM (e.g., by using A*), the answer to which is of course yes. While making LLMs search in more complex environments is definitely the long-term goal, we must note that it is challenging because of the large action space leading to really long trajectories and the consequent computational challenges (something that, in our opinion, needs an entirely separate study). Thus, techniques such as our System 1.x which help reduce computational overhead by using System-1 wherever possible, should be seen as a step forward in that direction.

[1] Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping, Lenhert et al.

[2] Stream of Search (SoS): Learning to Search in Language, Gandhi et al.

How could System-1.x be leveraged in new domains where a user is not a domain expert?

System 1.x is a framework for hybrid planning with LLMs. This means that applying it to a new domain requires training a System 1 Planner, a System 2 Planner, and a Controller. The System 1 and 2 Planners can be trained with only access to some exploration/search trajectories, which does not necessarily require a domain expert. The controller will be used for sub-goal decompositions and unless one wants to directly prompt an LLM to do so (which may work well, we don’t know), it should be trained with some sub-goal annotations. These annotations could either come from humans or be generated synthetically/automatically by using some domain-knowledge in the form of heuristics as hardness functions (like we do). It’s important to stress that these heuristics need not be perfect because as part of fine-tuning, we still want to leverage an LLM’s pre-trained knowledge while also teaching it to use some useful features for measuring hardness (e.g., learning to count obstacles or manhattan distance in Mazes – see our examples in the Appendix).