Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

We appreciate the insightful and valuable comments. We will carefully address your concerns one by one as follows:

W “… PPDPP is essentially a prompt selector. It would be more interesting if it can not only select but also generate prompts, and if PPDPP can be integrated into the dialogue LLM (to avoid to use another pre-trained roBERTa model).”

Response: We greatly appreciate the thoughtful suggestions.

1. As for the comment about “if it can not only select but also generate prompts”, it is totally feasible and worth studying. To this end, we just need to use a generative language model, such as T5 or Flan-T5, as the PPDPP. In order to demonstrate its adaptability into the generative policy planner, we further implement the PPDPP with a generative language model, i.e., Flan-T5. The implementation includes two different settings:

• Flan-T5$_\text{select}$: Use Flan-T5 to generate the strategy label, which performs a similar strategy prediction task as RoBERTa-based PPDPP.
• Flan-T5$_\text{generate}$: Use Flan-T5 to directly generate the natural language strategy prompt.

As presented in the Table, we observe that Flan-T5$_\text{select}$ achieves a similar performance with RoBERTa.

On the other hand, Flan-T5$_\text{generate}$ generally performs worse than the other two methods in these three datasets.

These results indicate that, under these proactive dialogue problems, a larger search space for actions may negatively affect the reinforcement learning process of the PPDPP or demand for more efficient or robust RL algorithms, which can be left for future studies.

2. As for the comment about “if PPDPP can be integrated into the dialogue LLM”, we would like to emphasize the advantages of the pluggable policy planner:

• For specific dialogue problems, only the policy planner plugin needs to be fine-tuned, which is more practical than fine-tuning the whole LLM-based dialogue system.
• Since the disentangled fine-tuning will not affect the LLM’s original capabilities of context understanding and response generation, the black-box LLM-based dialogue system can be applied to any dialogue problem by simply substituting the learned policy planner. From another perspective, if PPDPP can be integrated into the dialogue LLM, it will become similar to those prompt-based methods, such as ProCoT or Proactive, where we could only rely on the in-context learning or the tailored prompt designs to trigger the policy planning capability of the LLM dialogue model.

Revision: According to your valuable comment, we made the following revisions to address your concerns.

1. We add a supplementary experiment by replacing the RoBERTa-based PPDPP with a FlanT5-based PPDPP, which is a generative language model for generating prompts. The implementation includes two different settings: (1) Use FlanT5 to generate the strategy label. (2) Use FlanT5 to generate the natural language strategy prompt. The experimental results are presented in Appendix D.
2. We add more discussion about the advantages of using a pluggable policy planner against the prompt-based method in the Introduction and Related Work.

Q “Why do we want to sample the goal-oriented AI feedback for l times? (Equation 6). Do we observe large variance of the reward LLM?”

Response: Regarding the sampling of the goal-oriented AI feedback, we would like to clarify our two motivations:

1. The user state in the conversation is rather difficult to be determined. For example, it is hard to directly classify the emotional state of the user in emotional support dialogue. Therefore, there is indeed variance in the decision of the reward model. Inspired by the idea of Self-Consistency [1], using the sampling strategy can alleviate the variance.
2. The reward is more reasonable to be a continuous value instead of a discrete class. For example, the emotional state of the user is not supposed to be only four types. Using the average values of the sampled goal-oriented AI feedback can better represent the fine-grained state of the conversation. Besides, a continuous reward can provide more specific feedback for reinforcement learning.

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.

In order to validate the advantages of sampling goal-oriented AI feedback multiple times, we conduct an ablation study of the sampling strategy. As mentioned in Section 3, there are two functions of the reward LLM: (1) to determine the state of goal completion during the conversation; and (2) to evaluate the policy outcome with scalar rewards. Therefore, the ablation study will analyze the advantages of the sampling strategy from these two perspectives.

1. Analysis of State Prediction. Similar to the Analysis of LLMs as Reward Model in Appendix A.1, we also compute the F1 score of the prediction of the current user state versus the human-annotated labels. As shown in the Table, the sampling strategy substantially improves the F1 score of the state prediction, indicating that it effectively reduces the variance of the LLM-generated output.

2. Analysis of Reward Estimation. In this analysis, we adopt two reward LLMs to perform the two functions separately. One with the sampling strategy for state prediction to ensure the quality of state prediction, and the other one with or without the sampling strategy for reward estimation. As for the one that estimates reward without the sampling strategy, the reward will only be classified into one of the pre-defined discrete values. However, as for the one that estimates reward with the sampling strategy, the reward will be a continuous value that is averaged from the sampled results. Consequently, the Table shows that the fine-grained continuous reward contributes to better performance as the policy planning outcome will be more distinguishable during the reinforcement learning process.

Revision: In order to better clarify your concern regarding the sampling of the goal-oriented AI feedback, we further presented these two supplementary analysis in Appendix E and added more descriptions about the motivations in Section 3.

We appreciate the detailed and valuable comments. We will carefully address your concerns one by one as follows:

---

W1 “Limited action/prompt space for the dialog LLM, potentially constraining adaptability to different domains.”

Response: Thanks so much for the valuable comment. We would like to first clarify that the action/prompt space is NOT restricted, while it is determined by the concerned dialogue problems. For example, in the three dialogue problems we assessed, each one has a unique set of possible actions that have been tailored based on expert input or specialized knowledge in that area. In essence, our method is versatile and can be customized to suit a variety of proactive dialogue scenarios, each with its distinct set of possible actions. Additionally, our approach is equally applicable to a generative policy planner, which inherently allows for a vast and varied set of prompts.

In order to demonstrate its adaptability into the generative policy planner, we further implement the PPDPP with a generative language model, i.e., Flan-T5. The implementation includes two different settings:

• Flan-T5$_\text{select}$: Use Flan-T5 to generate the strategy label, which performs a similar strategy prediction task as RoBERTa-based PPDPP.
• Flan-T5$_\text{generate}$: Use Flan-T5 to directly generate the natural language strategy prompt.

As presented in the Table, we observe that Flan-T5$_\text{select}$ achieves a similar performance with RoBERTa.

On the other hand, Flan-T5$_\text{generate}$ generally performs worse than the other two methods in these three datasets.

These results indicate that, under these proactive dialogue problems, a larger search space for actions may negatively affect the reinforcement learning process of the PPDPP or demand for more efficient or robust RL algorithms, which can be left for future studies.

Revision: In the revision, we present this supplementary experiment in Appendix D, which compares the performance of classification-based policy planner and generation-based policy planner.

---

W2 “The primary distinction from other Reinforcement Learning from AI Feedback (RLAIF) works seems to be the mapping of the LLM's reward output from text space to scalar reward space, raising questions about the approach's uniqueness.”

Response: Thanks so much for the valuable comment. We would like to further elaborate the approach’s uniqueness, which is in two-fold:

1. The novel framework with a tunable language model plugins for LLM-based dialogue systems.
2. The design of goal-oriented AI feedback.

As for your concern regarding the design of goal-oriented AI feedback, we would like to highlight two distinct features from common RLAIF works:

• As you mentioned, mapping of the LLM's reward output from text space to scalar reward space for RL algorithm.
• We consider the long-term goal-oriented reward, instead of AI preference on single-turn responses. The long-term goal-oriented reward requires multi-turn interactions with a dynamic environment, i.e., the user.

Revision: In the revision, we elaborate the approach’s uniqueness more clearly in Section 2 by comparing with the related works.

---

W3 “The need for training different dialog policies for each dialog domain. This makes this system less generalizable.”

Response: We apologize for this confusion. In fact, the generality of the proposed method lies on the pluggable dialogue policy planner. With this design, we do not need to train the whole LLM-based dialogue system for each dialog domain, while we just need to train a small language model plugin. This not only makes the LLM-based dialogue system more generalizable to the new case in each dialogue domain than prompt-based approaches, but also enables the LLM-based dialogue system can be adapted to different dialogue domain by just substituting the dialogue policy planner plugin without affecting the LLM’s exceptional capabilities of response generation and context understanding.

Revision: In order to better elaborate the generality of the proposed method, we add more descriptions in the Introduction.

Q1 “How is the reward LLM utilized during inference at each turn of dialogue?”

Response: As mentioned in Section 3 “LLM as Reward Model”, the reward LLM has two functions : (1) to determine the goal completion during the conversation; (2) to evaluate the policy outcome with scalar rewards.

As shown in Figure 1(b), during the inference phase, the reward LLM will not output the reward value, which is only used for the RL training phase. However, the reward LLM will still be used to determine whether the goal has been completed during the conversation, which is used for evaluation.

Revision: In order to better present the inference process, we add more details in the caption of Figure 1 regarding the usage of the reward LLM.

---

Q2 “Could you clarify the process of mapping the reward LLM's output to scalar values and its integration into the PPDPP during each dialogue turn?”

Response: Sorry about the confusion regarding the reward mapping process. Take emotional support dialogues as an example. To assess whether the patient' emotional issue has been solved, we prompt the reward model to answer a multi-choice question "Has the patient's issue been solved?", and then generate the goal-oriented AI feedback at temperature $\tau>0$ to sample the responses for $l$ times. We define a mapping $\mathcal{M}_r(\cdot)$ to transform verbal feedback to scalar rewards, such as "the patient feels worse", "the patient feels the same", "the patient feels better", "the patient's issue has been solved" as -1.0, -0.5, 0.5, and 1.0, respectively.

When we obtain the $l=5$ sampled responses as ["the patient feels better", "the patient's issue has been solved", "the patient feels better", "the patient's issue has been solved", "the patient feels better"], we can compute a continuous scalar value $v_t=\frac{1}{5}(0.5+1.0+0.5+1.0+0.5)=0.7$. If $v_t>\epsilon$, we regard the state as GOAL-COMPLETED and set the reward $r_t=v_t$. If not, we assign a small negative reward, e.g., $r_t=-0.1$, to penalize the lengthy conversation for promoting efficient goal completion.

Revision: In order to make the description of the reward mapping process more clear, we reorganize the part of reward mapping to the same part in Appendix F.4, which presents the whole process with specific examples.

---

Q3 “Can you elaborate more on supervised fine-tuning used to PPDPP?”

Response: Sorry about the confusion regarding the supervised fine-tuning (SFT) process. The SFT process is described in the second part of Section 3. The SFT exactly follows the traditional corpus-based fine-tuning process for dialogue action prediction. In specific, we have the human-annotated dialogue corpus with dialogue action labels for each turn. Given the dialogue history $\{u_1^\text{sys}, u_1^\text{usr}, ..., u_{t-1}^\text{sys}, u_{t-1}^\text{usr}\}$, the PPDPP is fine-tuned to predict the dialogue action $a_t$ for the next turn.

Revision: In order to make the description of the SFT process more clear, we add more details in the part of "Plug-and-Play Dialogue Policy Planner" in Section 3.

We sincerely appreciate your acknowledgement of our work. We believe that this work can provide a practical and effective solution for the proactive dialogue system in the era of LLMs. And the evaluation of dialogue systems can also be benefited from the proposed interactive evaluation framework.

Q1 “I try to understand why there is negative relative success rate in Figure 2?”

Response: We apologize for the confusion. The relative success rate is calculated by subtracting the actual success rate of the Standard prompting method from that of the concerned method. Therefore, the negative relative success rate means that this method achieves a lower success rate than the Standard prompting method. This result shows that in some cases, these methods perform even worse than the vanilla ChatGPT with Standard prompts.

Revision: In the revision, we added the above descriptions on the caption of Figure 2 to better present the results.