Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

Thanks for recognizing our contribution to the direction of guided decoding. Below is our response to your concerns:

Alternatives to MCTS: First we would like to clarify that our “stepwise-value” baseline (used to compare with MCTS) uses not just values, but a combination of policy and values. It first selects top-k candidate tokens from policy model’s proposal, $p_\theta(x_t | w, x_{<t})$, and then decide the token based on the values of these k candidates. It would be interesting to explore applying other search algorithms (e.g., FUDGE, beam search) to PPO models, and we leave it as future research.

MCTS cannot be applied to DPO models: Indeed, DPO does not produce a value model required by MCTS, and MCTS can only be applied out-of-the-box to Actor-Critic methods like PPO. It is possible to post-hoc train a value model after the DPO model is trained, but that requires extra effort. Meanwhile, some recent work [1, 2] showed that online algorithms (like PPO) tend to produce better policy models than offline algorithms (like DPO). Building on such advocates for PPO, our proposed method can further boost the generation quality of PPO models.

Using reward model in MCTS: In our submission we reported this ablation of using reward model instead of the value model for MCTS. The result is in Table 6 (under the name “PPO-MCTS[R]”) and we discussed this in the first paragraph of Sec 5.1. Using reward model for MCTS resulted in lower goal satisfaction and lower fluency. We offered two potential explanations: the reward model is not trained to process partial sequences, and the reward model is not adapted to the PPO policy model.

Thanks for appreciating that our proposed method can be easily applied to post-RLHF LLMs. Below is our response to your concerns:

Discussion of efficiency: Thanks for the suggestion. We will further expand our discussion on the efficiency of PPO-MCTS in our next paper revision, and include empirical results for decoding latency.

Clarity and the issue with $\bar{V}(s)$: We will improve the clarity in our next paper revision. $\bar{V}(s)$ appeared indirectly in Eq 3, as part of the computation of the $Q(s, a)$ term in Eq 4. We will defer the mention of $\bar{V}(s)$ to reduce confusion in Eq 3.

PPO+best-of-n being worse than PPO: We were also quite surprised by this result. However, from Table 1 and 2 we see that PPO+best-of-n is consistently comparable with PPO. We think this implies that vanilla sampling is unable to produce outputs with a diverse range of rewards, and guided decoding is a better solution.

Thanks for your feedback! Below is our response to your concerns:

Computational overhead: In our submission we discussed the inference time overhead of PPO-MCTS in Sec 5.1. We see PPO-MCTS as a method to boost quality with more compute at inference time, and would like to explore techniques of reducing this overhead in the future.

Figure font too small: We will improve this as well as the overall presentation in the next paper revision.

Thanks for recognizing the importance of our studied problem and the effectiveness of our method. Below is our response to your concerns:

Comparison with using hand-crafted rules or separate discriminative models: In our submission we experimented with using the reward model to guide MCTS. (For the tasks we evaluate on, the reward model would be the separate discriminative model.) This method is reported as “PPO-MCTS[R]” in Table 6, and it resulted in lower goal satisfaction and lower fluency than our PPO-MCTS. We did not try hand-crafted rules because it is unclear how to design rule-based classifiers for these tasks.

Inference cost comparison: We mentioned the inference time of PPO-MCTS in the last paragraph of Sec 5.1. Naively, PPO-MCTS has a 2S times overhead compared to standard decoding from PPO models (S is the number of simulations), and this overhead can be mitigated by various techniques like caching and pruning. In some internal experiments, we have successfully generated sequences of 1k tokens with PPO-MCTS. We see PPO-MCTS as a method to boost quality with more compute at inference time.

Comparison with DPO and SLiC: We did not compare with these methods because they do not produce a value model, which is required to apply MCTS. In addition, some recent work [1, 2] showed that online algorithms (like PPO) tend to produce stronger policy models than offline algorithms (like DPO).