PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning

We thank the reviewer for their detailed review and positive comments on our novel data collection, interesting results, and well-written paper. We are in the process of updating our pdf, meanwhile please find our response below:

W1. Novelty and Comparison with Orca:

Thanks for pointing out Orca that we overlooked. Indeed there is a recent line of work on building general-purpose small models which warrant a discussion and reference. While our work shares similar spirit with Orca, we find the key distinction in (1) our goal is to develop a specialized small model for procedural/counterfactual planning and replanning with potential application to embodied domain, and (2) Orca is focused on learning from GPT-4 explanations (Chain of Thought) to improve models capabilities. Nonetheless, building specialized models on top of them can be explored in future works as we only worked on models that were accessible at the time of submission.

We will add this discussion in a new extended related work in the Appendix.

Thanks for bringing [B] to our attention. We believe such contemporaneous works indicate the timeliness of our contributions as they reflect the needs and interests of the research community.

W2. Re Missing Key Experiment:

Indeed we have tried to get an undistilled T5 model to work on the planning task. However, trying several prompt formats (such as “How to <goal>?”, Provide steps to achieve the <goal>”, etc.), we couldn’t get any meaningful outputs from this baseline to run human evaluation on or apply guided decoding. Most outputs were just repeating the given inputs or empty strings. However, we successfully ran a few-shot variant of undistilled T5 (with some post-processing to format the outputs). Below is the human evaluation result for this baseline on the same 250 randomly sampled test examples (Curie and PlaSma scores are compiled from Table 1):

Results of Few-shot undistilled T5 is significantly behind those of fewshot Curie (teacher) and all our model variants.

Q1. How CoPlan is different from ProScript?

We would like to clarify that ProScript is a small-scale manually-curated dataset of scripts, whereas CoPlan is (1) model generated, (2) larger scale, (3) includes counterfactual replanning, and finally (4) it broadly covers a diverse set of everyday, real-world activities both at low-level (actionable) goals and high-level long-term goals. We provided a detailed diversity analysis of goals and conditions of CoPlan in Appendix A.1 and A.2.

Q2. What is "symbolic"?

“Symbolic” (in symbolic knowledge distillation) refers to human-readable textual formats rather than the transfer of obscure/soft model weights as in standard distillation (Hinto et al. 2015). Thanks for pointing this out, we will include this explanation in the updated version.

Q3. Re "motivation for generating larger data" and impact of "model size of small LM", "amount of training data":

• Our experiment in Table 2 investigates the utility of larger CoPlan that is obtained through symbolic distillation in the presence of manually-curated ProScript. There we compare a distilled model trained only on CoPlan with a T5-11B model trained only on proscript. Results indicate that we do benefit from a larger dataset.

• We studied the impact of (student) model sizes on performance (Fig. 3 and Table 1) where we show that larger students perform relatively better but we can bridge the scale gap with multi-task distillation and our proposed verifier-guided decoding.

• We also conducted experiments on the impact of training data size on test/validation losses. However, we ended up not including the latter in the paper as we thought it might not fit well with the scope. Below are the results:

Based on decreased validation/test loss, we don't observe signs of overfitting. Nonetheless, a more comprehensive scaling law contribution in the context of distillation could be done in the future.

We will include this in the Appendix of our updated pdf.

Q4. Typo for the term "task"

Thanks, we fixed this to “three different task settings” in our updated pdf.

That's a good question and while I was aware of [A], I had not bothered to verify that. Orca [A] is quite old now given the fast-paced timeline (and is also somewhat popular), however, it seems the authors still haven't open-sourced the data (which also explains why I couldn't find a peer-reviewed version). Nevertheless, the framework is quite detailed in their paper, based on which there are (at least) a couple of open-sourced replications of Orca going back to June 2023 (see OpenOrca: https://huggingface.co/datasets/Open-Orca/OpenOrca and Dolphin) with impressive results. As you see, given the follow-up works, the idea of training smaller LMs to reason using knowledge from LLMs is not particularly new.

As for [B], as I said, I've not taken it into account given its recency, despite its overlap.

Thank you. This answer some of my questions. I’m looking forward to see the description of ProScript, including a justification of why is that a good dataset for the research question. This is a key issue: If the plans are very short, and the sample of problems is not diverse enough, the planning problem becomes a classification problem: decide which one to retrieve.

If new plans are required for novel situations, but the plans are basically composing two pieces of existing plans, the problem can be solved with two classifiers. Imagine a baseline that does directly that: choose possible few segments of a plan, and then use a ranker over potential combinations. Such baseline might have a good performance in a bad dataset for testing planning algorithms. That’s why is very risky to test planning with a passive data, without a simulator. Then only alternative is that the dataset is diverse.

An important factor are amount to overlapping of the plans in the train set, but also respect to the test set. In this case, even if the plans are not symbolic, it should be possible to compare the text of the “steps” of the plans. That’s true for both datasets.

Furthermore, given the simple structure of the sentences in both datasets, the overlapping of the sentences is not enough. It’d be more informative to see how the text compare with each other by ignoring nouns, while keeping at least the verbs and perhaps the rest of the sentence. Why? Well, the plans for these two goals is almost the same: “carry a glass from the bedroom to the kitchen”, “carry a plate from the bedroom to the kitchen”. This actually qualifies my argument about how plans in a dataset might be easier. If a LLM memorize the plan for glass to the kitchen, then plate to the kitchen is just editing an existing plan.

Such an analysis might explain why some datasets are harder than others.

I understand some papers in procedural planning are being accepted in top venues without such analysis. That’s a shame. I attribute that to a weak background in combinatorial and symbolic AI, as well as lack of awareness of the literature. We are not here to repeat those mistakes, so let’s show that the problem is interesting and cannot be solved by memorizing the plans and doing small editions.

In summary,

1. please explain ProScript in detail,
2. Analyze both datasets to disprove the hypothesis zero: planning can be done just by memorizing the plans and making few combinations or adjustments.

New:

• in the main body, Please explain briefly the three tasks used for Multi-Task distillation. Then in the appendix explain the dataset in detail, and any hyper-parameter so we can reproduce the experiment. Justify any decision.
• Please confirm that PlaSma-mul is not fine-tuned at all in the other datasets.

Others:

• the expression “original planning task” is confusing. It might be better just to say ProScript.
• In each table where PlasSa is reported, please mention the dataset used for fine-tuning.
• About the use of “symbolic”, I don’t think that 2020 reference is enough to use such a confusing expression, but I’m not requesting a change about that terminology. I do request to acknowledge the confusion about the term, and refer to work in symbolic AI. This paper accepted to AAAI 2022 might be useful: https://arxiv.org/abs/2109.09904

PS: I haven’t read the other responses, so perhaps part of my new questions are answered there. Apologies in that case.

We thank the reviewer for their detailed review and positive endorsement of our interesting research problem, our proposed methodology, the comprehensiveness of the task under study, and compelling results on VirtualHome. Please find our response below:

Re ProScript details:

Proscript is a manually-curated dataset of ~6k scripts at varying levels of granularity, for a wide range of everyday activities (e.g., bake a cake, audition for a musical, go sailing, etc.). Given the diversity and granularity of goals in ProScript, we believe it provides a good venue to study procedural knowledge in language models available through distillation.

Table 2 includes both evaluation on the ProScript as well as our collected CoPlan dataset. The goal is to investigate the utility of CoPlan that is obtained through “generation from an LLM” in the presence of manually-curated ProScript.

Re phrase "introduce the task of counterfactual planning":

While plan revision has been explored in classical AI (with restricted action vocabularies) and Robotics, here we are investigating counterfactual and constrained (re)planning capabilities of contemporary language-based agents. As LMs are gaining more attention in robotics and physically embodied application, it becomes imperative to enhance their capabilities by integrating counterfactual planning and contained planning mechanisms.

We thank the reviewer for bringing this to our attention, and we will revise this phrase accordingly in our updated version.

Re "step verifier" is not verifying:

We agree with the reviewer. While during training the verifier acts as a binary classifier to identify the validity of a next-step, we are using this as a bias/regularizer term during decoding the plan (Eq. 3). Thank you for pointing this out. We will update our paper to reflect this.

Re "Symbolic" in Symbolic Knowledge distillation:

We totally agree with the reviewer that “symbolic” became an overloaded word in AI. However, we use this term in reference to the previous work (West et al. 2020, “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”).

Q1. what does PlaSma mean when measured in different tasks?

Thank you for bringing this to our attention. Yes, except for PlaSma-Mul, the models (listed under PlaSma) in Table 1 are completely different from those in Figure 4. This distinction also applies to the models featured in the left and right plots of Figure 4. We will ensure clarity by having a different naming convention.

Q2. Describe ProScript:

Please see our response above. We will add this to our update pdf soon.

Q3. Other datasets:

For training purposes, we use our large-scale collected CoPlan dataset.

For evaluation purposes, in the general domain, we use ProScript which was the only human-written dataset of this kind we were aware of. In the embodied domain, we use RobotHow/VirtualHome (Puig et al. 2018) for conducting our experiments. There are other embodied text-based environments such as Alfworld (Shridhar et al. 2020) which are similar in nature to VirtualHome. We will discuss this in our extended related work section.

Note that we also used DeScript (Wanzare et al. 2016) which is a small-scale manually-written dataset as our seed examples during data generation.

Puig et al. 2018. “Virtualhome: Simulating household activities via programs”

Wanzare et al. ACL 2016. “A Crowdsourced Database of Event Sequence Descriptions for the Acquisition of High-quality Script Knowledge.”

Shridhar et al. 2020, ALFWorld: “Aligning Text and Embodied Environments for Interactive Learning.”

Re Can a classifier solve the problem?

• The original proScript paper proposed a prediction task to be: given a goal and an unordered list of steps, predict a set of ordered steps. (Note that in this setting the model has access to the ground-truth list of steps which is NOT the case with the end-to-end plan generation task.)
• They implemented a two-step approach baseline where they trained a binary classifier (RoBERTa-large) to predict the precedence between pairs of steps, followed by building an ordered list of steps (plan) by aggregating the predicted relations across all pairs of steps. Scores by the classifier are used as weights to create an adjacency matrix which is then automatically converted into ordered steps (plan).
• This classification baseline achieved an F1 score of 61.20% which is far behind human performance of 89.28%. This suggests that the dataset is not trivial for a classification model even with full access to ground-truth steps. In the generation task (our focus) where the model ONLY has access to the goal, the task becomes even more challenging.

We include all this discussion on the complexity of the datasets in the Appendix of our updated pdf.

Re details of crowdsourcing proScript:

First, given a scenario (e.g., bake a cake), each crowd-worker is required to describe five to seven core events (to balance the cognitive load and cost) that are essential for the scenario with the estimated time it takes to complete each event. In the second question, crowdworkers confirm the set of steps and they are asked to create a flowchart by connecting the steps possibly in partial order. When crowd-workers make a submission, a validation function is executed to check if the created flowchart is a valid dag and does not contain any shortcut edge. To have micro and microscopic scenarios, they iteratively picked events and used them as an additional source of finer-grained scenarios. For example, “turn on the oven” is a new fine-grained scenario derived from “bake a cake”. This results in 6,414 (train=3,252) valid scripts that include 311,502 pairs of events.

Re training data used for finetuning single and multitask models:

We trained all (single task) PlaSma models on their respective subsets of the CoPlan dataset. PlaSma variants (besides Plasma-Mul) in Table 1 are trained on the goal-based planning subset of CoPlan. Similarly, PlaSma models in the left plot of Figure 4 are trained on the constrained planning subset of Coplan, and so on. PlaSma-Mul, on the other hand, is jointly trained on all CoPlan subsets: goal-based planning, constrained planning, and counterfactual replanning. We will briefly explain this in the main text and move hyperparameters in the appendix.

Clarification of task settings:

We would like to clarify that in the paper we used “goal-based planning” and “original planning task” interchangeably. These two are contrasted with the new proposed task settings: “constrained planning” and “counterfactual replanning”.

We thank the reviewer for their thoughtful comments and discussion.

As the discussion period is nearing its conclusion, we would appreciate knowing if there are outstanding matters subsequent to our discussion.

We hope our response sufficiently addresses your concern. Your feedback is invaluable, and we are committed to enhancing the quality of our work. We have updated our document accordingly and summarized the changes in the general response.

Once again, we appreciate your time and consideration. Best, Authors

Thank you for your helpful review and positive feedback on the importance of our research to the community and our well-structured paper. We are in process of updating our pdf, meanwhile please see our response below:

Re W1: evaluate the models in domains which have hard executability conditions:

• We indeed conducted an extrinsic evaluation in a domain with hard executability conditions, i.e., VirtualHome (Section 3.3). This also helps to investigate the application of PlaSma and its generalization to the embodied agent planning domain. Given our strong results in the VirtualHome environment, we expect that PlaSma can serve as a strong foundation model and be easily transferred to other embodied domains with minimal domain adaptation.
• On the general domain (i.e., on CoPlan); however, we follow the common practice of Likert scale human evaluation [1-3] across multiple dimensions. However, we made sure these dimensions are aligned with plan ability in an embodied domain. For example, temporal ordering is related to executability (e.g., grasping should be done before picking up), and coverage/completeness is related to success/correctness.

We hope this clarifies the reviewer concern regarding our evaluation.

[1] Recent Advances in Neural Text Generation: A Task-Agnostic Survey. (Tang et al., 2023)

[2] Human evaluation of automatically generated text: Current trends and best practice guidelines (Van Der Lee et al., CSL 2021)

[3] The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations (Amidei et al. iNLG 2019)

W2: comparison with GPT-4:

Our data was collected using GPT-3 Curie (smaller and less powerful than GPT-4) as the teacher due to cost constraints. We do expect to get better results for our distillation if using GPT-4 as our teacher. Nonetheless, as suggested by the reviewer, we ran a human evaluation comparing GPT-4, GPT-3.5, our best PlaSma model and its teacher on 50 instances (total of 200):

As we observe, the trend remains the same, with GPT-4 surpassing its predecessor (GPT-3.5) only in the ordering dimension. We will add this comparison in the Appenidx F of our updated pdf.

W3: Re potential increased bias:

We acknowledge potential biases, and have discussed them in the limitations section, warranting the need for further investigation in future research.

Q1: If smaller language models can be effectively paired with human input or external verifiers...?

Smaller models without distillation are not competent to perform the planning task and thus do not benefit from being augmented with an external verifier or human input. In fact, in our initial experiments, we tried to get an undistilled T5 model to work on the planning task. However, trying several prompt formats (such as “How to <goal>?”, Provide steps to achieve the <goal>”, etc.), we couldn’t get any meaningful outputs from this baseline to run human evaluation on or apply guided decoding. Most outputs were just repeating the given inputs or empty strings. However, we successfully ran a few-shot variant of undistilled T5 (with some post-processing to format the outputs). Below is the human evaluation result for this baseline on the same 250 randomly sampled test examples:

Results of Few-shot undistilled T5 is significantly behind those of fewshot Curie (teacher) and all our model variants (Table 1).

Q4.1 Are human evaluation dimensions enough?

Given the open-ended nature of CoPlan goals having varying levels of actionability (not all of them are grounded), we follow the common practice of Likert human evaluation across multiple dimensions [1-3]. Notably, we make sure we evaluate plans on dimensions that are related to actual plan ability in an embodied domain. For example, temporal ordering is related to executability (e.g., grasping should be done before picking up), and coverage/completeness is related to success/correctness. Nonetheless, we believe that extrinsic evaluation is crucial; for this reason, we use VirtualHome to investigate the application of PlaSma and its generalization to the embodied agent with hard executability (Section 3.3). Given our strong VH results, we expect that PlaSma can serve as a strong foundation model and be easily transferred to other embodied domains with minimal domain adaptation.

Q4.2. Correlation between human and BLEU scores:

We indeed compute the correlation between BLEU metric and human scores. We find that BLEU has very weak correlations to human scores of coverage, ordering an overall quality, with a Pearson correlation of 7.7%, 5.9%, and 5.6%. This verifies the fact that n-gram-based metrics may not provide an informative measure of performance (as also suggested by previous works [3,4])

Q4.3: Proposal for better automatic evaluation:

We appreciate the interesting questions raised by the reviewer regarding human and automatic evaluation alignment. We also acknowledge the need for better automatic evaluation of plans, ideally those that take into account the causal, temporal, and completeness of steps towards achieving a goal. However, these are out of the scope of the current project and we hope future works can explore this direction.

[1] Huang et al. “Language models as zeroshot planners: Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agent.” ICML 2022.

[2] Sakaguchi et al. “proScript: Partially ordered scripts generation”. In Findings of the Association for Computational Linguistics: EMNLP 2021.

[3] Novikova et al. “Why We Need New Evaluation Metrics for NLG” EMNLP 2017

[4] Celikyilmaz et al. “Evaluation of Text Generation: A Survey“

Thank you for the detailed review and positive comments on our cost-efficient distillation paradigm and our verifier-guided decoding algorithm. We are working on improving the writing of the paper to better showcase the ideas/contributions as suggested by the reviewer. Meanwhile, please find our responses below:

Re importance of distillation:

Finetuning LLMs requires updating models’ parameters which is not only costly but often inaccessible for the broader community. To clarify, our goal is not to make superior large models, but to distill procedural knowledge of LLMs into smaller models. Reducing the scale and cost of strong models via teacher-distillation is the key in developing open-sourced LMs that are accessible to all, facilitating fine-tuning and seamless adaptation to various domains and custom use cases. Given the large-scale training dataset used for PlaSma, we hope it can serve as a foundation model that can be quickly adapted to specific domains with minimal additional annotation (like we demonstrate by adapting Plasma to VirtualHome). Moreover, we could not augment most of the LLMs with our decoding-time algorithm due to limited access to the model's log probabilities.

Thanks for the comment. We will update our pdf to include this.

Q1. Re details of training the verifier and perturbation strategies:

We included implementation details of the verifier in Appendix B.3 due to space limit. We will move these materials to the main text as we recognize the potential inconvenience:

• For training, we reuse 3k human-written plans from the existing ProScript dataset [2] to automatically create 47K positive and negative pairs of (plan-so-far, next-step).
• Our perturbation strategies to generate pseudo-negative examples include: (1) Reordered steps: Conflicting logical order results from inaccurate causal or temporal dependencies in a plan. Thus, we apply both near and distant reordering by randomly reordering two consecutive and two distant steps. (2) Repetitive steps: Degeneration i.e., generating repetitive text is commonly observed in language models (also observed in the planning domain [1]). Similarly, we include both near and distant repetition by repeating the immediate previous step and distant previous step as a pseudo-negative next-step. (3) Missing steps: Another common mistake made by language models is missing necessary steps, leading to (semantically) incoherent plans. To simulate this behavior, we randomly select a non-immediate step as a pseudo-negative next-step. Below we show symbolic examples of perturbed pairs:

Goal: g

Oracle Plan (steps): s1 s2 s3 s4 s5

We have the following perturbations:

Reordering: (g + s1 + s3, s2) → 0

Repetitive: (g + s1 + s2 + s3, s2) → 0

Missing: (g + s1 + s2, s4) → 0

Note that our perturbation strategies follow general rules applicable to any new domain with some task-specific adaptation.

• Using the constructed positive and negative pairs of (plan-so-far, next-step), the verifier is trained using the standard binary Cross Entropy loss to identify the validity of the candidate next-step. It achieved an F1 score of 78% on a held-out test set.

Re formal analysis of impact of the step-verifier:

In Table 1, we include several ablations to show the contribution of multitasking, step-verifier, and scale. For example, the comparison between PlaSma and PlaSma+ across different model sizes shows non-negligible performance boosts as a result of the step verifier in our proposed tree-based decoding.

Is the reviewer seeking a specific formal analysis? Thanks.

Q2. Re defining loss functions:

All models are trained with the standard Cross Entropy (CE) loss function with an early stop based on the validation loss. We will update our manuscript soon to include the loss used for distillation and verifier training.

Q3. Verifier perurtabation strategies

Please see our response to Q1

We thank the reviewer for taking the time to read and engage with us in our response.

Concerning the rigidity of the evaluation, we took substantial steps to address this issue. Firstly, we conducted an extensive human evaluation, scrutinizing various facets of a plan within a sizable subset of the test set (300 examples). Additionally, we undertook an extrinsic evaluation on VirtualHome, a platform characterized by hard executability conditions, to delve deeper into PlaSma's effectiveness within an embodied environment. We achieved 94.18% executability rate compared to previous baseline of 77.17% and 43.68% success rate compared to 18.33%. These show the adaptability of our model to new domains quickly.

To provide more clarity and enhance the robustness of our work, we would appreciate specific feedback on the aspects of the evaluation that remain worrisome.

---

Re necceessity of distillation: The essence of our approach lies in reducing the scale and cost of potent models through teacher-distillation. This, in turn, contributes to the development of open-sourced language models that are accessible to a wider audience, enabling seamless fine-tuning and adaptation to diverse domains and custom use cases (as we showed empirically). As we mentioned, fine-tuning LLMs requires updating parameters, a process both costly and often infeasible for broader community.

---

Furthermore, regarding the rigidity of the dataset, an issue raised by another reviewer. We have addressed this concern by incorporating several analyses/results that highlight the non-trivial nature of the dataset under examination. (included in Appendix G of our updated document)

---

As we continue to refine our work, we are committed to incorporating constructive feedback and further improving the merit of our contributions. We appreciate the recognition of the underlying ideas of our paper and remain dedicated to enhancing the quality of our research.

Thanks again, authors

Thank you so much for your positive comments and detailed reviews. We were waiting for some (human) experiment results. We just submitted our responses.

Please let us know if you have any further questions/concerns and we are more than happy to continue the discussion.

Thanks, authors