Dear reviewer XK5w, thank you for your careful review and constructive feedback. Our responses to each of your comments are presented below.

• W1: Exaggerated motivation, limited domain testing and applicability to NLP tasks. Thank you for pointing out this concern. We also acknowledge the importance of further evaluation of PFM, especially on NLP domain. To address this issue, we conducted additional experiments on the NLP task. Please refer to our common response.
  • We applied PFM to the open-source LLM (GPT-2) for text generation task.
  • We designed a pipeline to use PFM for variable-length text data.
  • We compared PFM with RLHF fine-tuned model.
• W2: Too strong assumption on conditional flow. We follow the unified framework for conditional flow matching (Tong, Alexander, et al. "Improving and generalizing flow-based generative models with minibatch optimal transport." Transactions on Machine Learning Research). We highlight that the Gaussian probability path is not a required assumption, rather it is a design choice. Following the original work of the CFM, we define the probability path between $y^+$ and $y^-$ as gaussian distribution for ease of usage and theoretical guarantee. Hence this does not hinder the applicability of PFM. All assumptions required for our framework to be theoretically guaranteed are thoroughly discussed in the Appendix A and C of our paper. We note here that these assumptions are generally valid across various ML domains.
• W3: Potential for Overfitting. If the preference data is limited or biased, preference-based methods may generally suffer from overfitting issue. This limitation is not restricted to our method; we believe that overfitting is generally inevitable in case of limited data for any existing ML frameworks.
• Q1: Overly idealistic motivation. Please refer to our common response. We applied PFM to the open-source LLM (GPT-2) for text generation task, included details of experiments, and reported the results compared with SFT and RLHF models.
• Q2: How to apply the simple assumptions from lines 116-117 to LLM? In our additional experiments included in our global response, we embedded text data to fixed-size latent vectors and learned the vector field for the probability path in the latent space.

Dear reviewer 2tji, thank you for your careful review and valuable comments. Our responses are presented below.

• W1: Main novelty of our method. The main novelty of our method comes from its capability of being added on top of any existing black-box models, without the need of fine-tuning this reference policy class. FTB for instance, requires an additional policy extraction phase, as FTB only focuses on the data augmentation for the behavior cloning policy. This means that FTB also requires a fine-tuning of the given reference policy in order to obtain the final policy.
• W1-1: Choice of conditional flow matching (CFM) Regarding the choice of conditional flow matching (CFM), our method can indeed be extended to frameworks that use rectified flow and diffusion models. As mentioned in the CFM paper (Tong, Alexander, et al. "Improving and generalizing flow-based generative models with minibatch optimal transport." Transactions on Machine Learning Research), rectified flow and conditional flow matching can be unified under a single flow-matching framework. The key difference of the rectified flow and the CFM only comes from the design choice of the flow transport model; CFM chooses a Gaussian probability path, whereas the rectified flow chooses a Direc probability path for density transportation. We would like to emphasize that the novelty of our approach comes from the probability transport module to be added on top of the existing black-box reference policy. To the best of our knowledge, we are the first to adopt such novel fine-tuning-free methods to improve alignment with the preference. This approach will be more favorable in cases where the fine-tuning of the original reference policy is unavailable, or computationally expensive.
• W2: Weak empirical results. Although PFM does not gain significant improvement over DPO in D4RL benchmarks, it is important to note that we have gained a similar performance increase from the reference policy, without any fine-tuning of the original policy. That being said, the greatest advantage of our method is its ability to be attached to any black-box model for improved preference alignment. This is verified across variable domains: vision, RL, and even language domains. Please refer to the common response above for our newly added experimental results for the NLP task. We believe these results will serve as strong enough evidence that our method can be used to improve preference alignment.
• W2-1: Regarding D4RL results As you pointed out, we observe high variance across all the results in the MuJoCo experiments. However, this is not due to limited random seeds or insufficient experiments. In offline RL tasks, compared to online RL settings, the variance may be large due to its limited access to the environment interaction. Especially in PbRL settings, the variance may be larger due to the lack of available preference-labeled data. This large variance can be observed in some other PbRL papers too; for instance, see Preference Transformer (Kim, Changyeon, et al. "Preference transformer: Modeling human preferences using transformers for rl.") for a similar scale of variance.
• W2-2: comparing our method to existing frameworks. As mentioned previously, we are the first to apply a fine-tuning-free method to improve preference alignment. Hence, our experiments mainly focused on comparing our new framework to the existing standard frameworks like RLHF and DPO. In the case of FTB, this method also requires an additional policy extraction phase that fine-tunes the reference policy. Furthermore, FTB suffers from heavy computational costs both for the policy extraction and the data augmentation using diffusion models. Therefore, we believe including FTB as a baseline is unfair, as it trains on a larger augmented dataset. Of course, our method can be applied concurrently with the FTB; FTB can be used to augment more quality datasets, where PFM can then be applied on top of the augmented preference dataset for additional performance gain.

To address your second question and provide a clearer understanding of the differences and contributions of PFM compared to FTB, we offer the following explanation:

Firstly, let us revisit FTB and its approach. FTB creates $K$ blocks of preference levels. Since the labeled or unlabeled datasets are not initially clustered into these $K$ levels, FTB requires score function estimation to assign trajectories to each block (or cluster). Once the dataset is classified into these $K$ performance-based blocks, FTB learns a diffusion model that transports a trajectory to match the score level of the neighboring better block. According to FTB, this transition between neighboring blocks ensures that the performance gap between each pair of trajectories is relatively consistent. Notably, FTB chose $K=20$ as the number of blocks and experimented with a minimum of 10 blocks even in their ablation study. The key motivation of FTB is to unconditionally generate good-quality trajectories within the fine-grained blocks. Note that this approach does not condition on the current state, but on the entire trajectory, making direct inference application challenging without additional steps. They also rely on the relatively close distance between these trajectory samples, as they do not condition on the same initial state, but simply optimizes for the complete trajectories, unconditionally. In short, FTB focuses on obtaining better trajectories, but not necessarily better trajectory for the current starting state.

In contrast, PFM aims to improve the trajectories generated by a reference policy in a manner that is immediately usable during inference. In other words, we aim to directly find the trajectory (or improve upon the current trajectory) that will lead to better results given the current state and situations. This is achieved by learning a vector field that directly enhances the trajectory according to current conditions (e.g. current initial state condition). Unlike FTB, which improves trajectories unconditionally within predefined score blocks, PFM conditions on the current state, making direct inference application more straightforward without additional steps. Notably, once we obtain a vector field that characterizes the probability transport of trajectories, we can iteratively apply this learned vector field to achieve the same improvement effect as FTB’s iterative upgrade to the neighboring higher-scored block. Our flow model improves the trajectory while preserving the conditional information of the current situation, making our method more interpretable and adaptable.

If one reduces FTB to having a block number $K=2$ and removes the policy extraction phase, FTB's approach becomes approximately similar to PFM, as you suggested. However, as mentioned above, the core motivations differ significantly. The authors of FTB did not reduce the block number to 2 (as in standard RLHF frameworks) and added a policy extraction phase due to the inability of FTB to be directly applied during inference, as its flow does not conditionally improve the trajectory.

Another primary benefit of PFM is its ability to enhance action trajectories without the need for score functions. PFM avoids overfitting in reward models by directly aligning model outputs with human preferences through flow-based models. This approach contrasts with FTB’s reliance on score functions for block assignment, which can lead to overfitting and suboptimal performance, as mentioned in our paper. PFM’s method of learning from preference data directly addresses these concerns, providing a more robust solution.

While FTB’s unconditional data augmentation approach can be beneficial in certain reinforcement learning (RL) scenarios, it is less adaptable to other tasks, such as language modeling. In language tasks, generating contextually appropriate sentences is crucial. PFM’s conditioning on the current state allows for more relevant and context-specific trajectory improvements, making it more versatile across various applications. FTB, on the other hand, focuses on generating high-return sentence sets, which is less practical for tasks requiring specific contextual appropriateness. Furthermore, directly applying FTB to language tasks requires an additional SFT phase to the obtained augmented dataset.

In summary, PFM provides a more efficient and versatile method that is particularly well-suited to black-box models and diverse applications beyond RL. By learning a vector field conditioned on the current state, PFM can directly enhance trajectories in real-time, making it a valuable contribution to general preference alignment in diverse domains. We hope this explanation clarifies the unique aspects and contributions of PFM. We appreciate your insightful questions and are happy to engage in further discussion to deepen the understanding of our work.

Dear reviewer XhGj, we thank you for your valuable comments and review. We are glad to hear that you have enjoyed reading our paper. Below, we provide our response to your comments and questions.

• W1: Shifted source distribution during inference. As you have pointed out, the shifted source distribution might affect the performance of our method. However, as studied in depth in Appendix A and B of our paper, our method is still guaranteed to work under a mild assumption, and under general non-deterministic preferences. In particular, our method is theoretically guaranteed to work if the support of the distribution $p_{0}$ (of less preferred samples) is included in the support of the distribution $p_{1}$ (of more preferred samples). This means that as long as the preference is continuous, we can guarantee that for any samples $y \sim \pi_{\mathrm{ref}}$, there is a non-zero probability of being included in the negatively labeled dataset. We have also provided empirical evidence for our choice of source distribution $\pi_{\mathrm{ref}}$ instead of $p_{0}$ in Appendix B of our paper. We will clarify this assumption in the main paper, as well as include further details in the limitation section of our paper.
• W2: Experiments for the NLP domain. We have obtained a promising result in one of the popular NLP tasks, one that is also adopted in the DPO paper. Please refer to the above common response section. We believe that the results above will serve as strong evidence and support that our method can be applied to general language-based tasks.
• Q1: Inference on D4RL tasks. For the inference on the D4RL tasks, we sample a reference trajectory $\tau$ using a copy of the current environment, with the same initial state. In scenarios where using a copy of such an environment is available (e.g., simulations), this is no problem, except for increased computation costs. However, as you pointed out, in some real-world applications, we may need an additional dynamics model in order to compute a reference trajectory. We will include this as one of our limitations. We would like to note here that this limitation is only for the case of RL tasks, and not for the image and language domains.
• Q1-1: Inference on D4RL tasks. Once the reference trajectory $\tau$ is obtained, we then apply PFM to obtain an improved trajectory $\tau^{+}$. In our implementation, we choose only the first action from the improved trajectory during inference. Of course, one may choose all actions from the improved trajectory to reduce the computational cost incurred during flow-matching, at the expense of performance loss. Hence, there is a trade-off between the computation (for flow-matching) and the performance gain, and one may find an appropriate intermediate as a design choice, suitable for their task.

Dear reviewer yRMJ, we appreciate your invaluable comments and feedback. Below, we provide our response to your comments and questions.

• Q1. Shifted source distribution during inference. As mentioned in our paper, since the source distribution $p_{0}$ is inaccessible during inference, we instead start from a shifted source distribution $\pi_{\mathrm{ref}}$. This can indeed be a factor that may deteriorate the performance of our method. However, as discussed thoroughly in Appendix A and B of our paper, it is still theoretically guaranteed to work if the support of the distribution $p_{0}$ (of less preferred samples) is included in the support of the distribution $p_{1}$ (of more preferred samples). In other words, for any sample $y \sim p_{1}$, as long as it has a non-zero probability of being labeled as less preferred (i.e. $p_{0}(y) > 0$), we can guarantee that for any samples $y \sim \pi_{\mathrm{ref}}$, there is a non-zero probability of being included in the negatively labeled dataset. Hence, we can guarantee a non-zero probability of existing learned flow from that sample $y \sim \pi_{\mathrm{ref}}$ to a more preferred sample. That being said, a failure regime of our method may include a scenario where the distribution of the ground-truth measure of the preference scores is sparse, that is, only the extremes (either very preferred or nearly non-preferred) are present in the dataset. We will include content in the limitation section.
• Q2. Future works for NLP domains. Thank you for pointing out a useful source (Diffusion Forcing) for future work. We also believe that these lines of work can be adapted to our framework and applied to NLP tasks. Aligned with this direction, we have shown a promising result of our method on an NLP task; please refer to the common response above.

Dear reviewer, we appreciate your thorough review and valuable feedback. Below are our responses to your comments and questions.

• W1: Weak empirical contribution. We have obtained promising results in a new domain, please refer to the above common response section for our results in the selected NLP task. We believe this new result is strong enough to prove the empirical usefulness of our framework.
• W2: Regarding experimental results for D4RL tasks. As you pointed out, PFM may not show significant improvement over DPO in D4RL benchmarks. However, it is important to note that we achieved similar performance increases from the reference policy without any fine-tuning of the original policy. Hence, the greatest advantage of our method is its ability to be attached to any black-box model for improved preference alignment. Combined with the new empirical results obtained in an NLP task, and experiments in the vision domain with MNIST examples, we believe it provides a comprehensive advantage for our framework in general preference alignment.
• W3: Regarding MNIST results. The RLHF and DPO results of MNIST experiments are obtained from the best working $\beta$ found from the same search space as in the D4RL tasks. We thank you for pointing out this missing detail; we will add this to our paper.
• W4: Regarding computational cost. For the computational cost incurred during the training phase, PFM is significantly more efficient than existing frameworks, as it only requires training a small add-on module. For example, please refer to the above common response section to see the difference in required parameter sizes for training in even a simple NLP task. Due to the limited time, we were not able to directly compare the training cost of RLHF fine-tuning and PFM, since we did not fine-tune the reference language models on our own, but adopted an open-source fine-tuned model that is already publically available. During inference, there is negligible computational complexity due to the small size of the attached PFM module. We believe this additional computational cost during inference will be even more negligible as the size of the base reference model increases.

For your questions regarding the NLP domains and additional experiments, we believe that our new results provided in the common response will address your concerns and demonstrate the applicability of our proposed methods across various scenarios. Thank you once again for your valuable feedback.

Thank you for your rebuttal and additional empirical results. Cognizant of the fact these take time to run, we have to stress the scale-dependent nature of those results and would reserve judgment for now as to whether the billion parameters+ LLM scale will improve as much as GPT-2. This leaves the empirical section fairly ambiguous in our view, and as such, we maintain our score.