Improving Reward Models with Proximal Policy Exploration for Preference-Based Reinforcement Learning

Dear Reviewer iwcb,

We would like to extend our sincerest gratitude for your detailed and insightful review of our manuscript. Your constructive feedback is invaluable, and we believe that addressing your comments will significantly improve the quality and clarity of our paper.

We are encouraged that you recognized the strengths of our work, particularly our findings on the importance of preference buffer coverage and the empirical performance improvements of our method. We have carefully considered all your suggestions and provide the following point-by-point responses to your questions and concerns.

---

Responses to Questions

1. Clarity of Section 3.2 (Motivation for Morse NN, RBF Kernel, and Goal of Eq. 2)

Thank you for pointing out the need for greater clarity in this section. We agree that a clearer explanation of our methodological choices is essential.

• Motivation for Morse Neural Networks: Our primary goal is to actively explore regions that are undersampled by the current policy but still close to it. To achieve this, we first need a mechanism to quantify whether a given state-action pair $(s, a)$ is "out-of-distribution" (OOD) with respect to the data currently in our preference buffer $\mathcal{D}^{p}$. As we discussed, Morse Neural Networks (Dherin et al., 2023) provide a principled way to learn an unnormalized density function $M(x)$ that is explicitly designed for this purpose. The function value approaches 1 for in-distribution data (modes of the dataset) and decreases towards 0 for OOD data. This provides a direct and continuous measure of "OOD-ness," which is precisely what we need for our exploration strategy.

• Motivation for the RBF Kernel: We use the Radial Basis Function (RBF) kernel, as shown in Eq. (1) to shape the Morse Network. This is a standard and well-established choice in kernel-based methods (Seeger, 2004). The RBF kernel is advantageous here because its value depends on the Euclidean distance, creating a smooth, localized "bump" around in-distribution data points. This property is ideal for generating the desired density field where the density value decreases as we move away from known data, which aligns perfectly with the objective of the Morse Network.

• Goal of Equation (2): Equation (2) defines the loss function used to train our Morse Neural Network. As we state on page 5 (lines 172-174) , this loss is derived from minimizing the KL divergence between the distribution of our preference buffer data and the learned density model. The loss function has two main components:

  1. The first term, $\frac{\lambda^{2}}{2}||f_{\phi}(s,a)-a||^{2}$, encourages the model to learn an identity mapping ($f_{\phi}(s,a) \approx a$) for state-action pairs $(s, a)$ that are in-distribution (i.e., from the preference buffer $\mathcal{D}^{p}$). This makes the Morse value $M_{\phi}(s,a)$ low, as desired for in-distribution samples.
  2. The second term involves sampling actions $a_{u}$ from a uniform distribution and penalizes the model if it assigns high density to these random (likely OOD) actions.

  In essence, the goal of optimizing Eq. (2) is to train a model $f_{\phi}$ that can reliably distinguish between in-distribution and out-of-distribution state-action pairs by outputting a continuous OOD score via the Morse function $M_{\phi}$.

We will revise Section 3.2 to incorporate these clarifications, ensuring the motivation behind our choices is more transparent to the reader.

2. Tightness of the Approximation in Eq. (4)

This is an excellent question regarding the theoretical guarantees of our approximation. The closed-form solution in Proposition 3.1 is derived by making two simplifying assumptions, which are detailed in Appendix D : (1) we use a first-order Taylor expansion to approximate the objective function $M_{\phi}(s,a)$, and (2) we tighten the KL divergence constraint.

Deriving a tight, analytical approximation bound is theoretically challenging due to the nature of these assumptions. However, we empirically analyze the impact of this approximation in our ablation study on the KL constraint parameter $\epsilon$, presented in Figure 2(c) . As we discuss on page 9 (lines 306-309) , the results confirm our theoretical intuition: performance degrades if $\epsilon$ is too large (as the first-order approximation becomes inaccurate) or too small (as exploration is overly restricted).

The primary motivation for this approximation was to develop a computationally efficient method suitable for real-time application in the RL loop, avoiding the high overhead of iterative solvers at each timestep. Our experiments demonstrate that with a reasonably chosen $\epsilon$ (e.g., 0.01), this approximation is highly effective in practice. We will add a brief discussion in the main text to acknowledge this trade-off between theoretical tightness and computational feasibility.

3. Collection of the Dataset $\mathcal{D}^{cp}$

We apologize for the lack of clarity on this point. The dataset $\mathcal{D}^{cp}$ represents the candidate pool of trajectories to be queried. Our method is designed to be integrated with existing query selection strategies. In our experiments, we integrated PPE with the state-of-the-art QPA algorithm.

Therefore, $\mathcal{D}^{cp}$ is first populated using the policy-aligned query technique from QPA (Hu et al., 2023), which selects trajectories relevant to the current policy. Our mixture distribution query method (Algorithm 1) then acts as a re-sampling post-processing step on this candidate pool $\mathcal{D}^{cp}$ to select the final trajectory pairs for which we request preference labels . We will revise the text to make this process explicit.

4. Preference Query Mechanism in Experiments

For our main benchmark experiments in Section 4.1, we followed the standard protocol used in prior PbRL literature (e.g., Christiano et al., 2017 Lee et al., 2021b). The preferences are provided by a simulated oracle. This oracle has access to the ground-truth returns of the trajectories. When presented with a pair of trajectory segments $(\tau^0, \tau^1)$, it provides a preference for the segment with the higher cumulative ground-truth reward. This ensures our experimental results are reproducible and allows for a fair and controlled comparison against the baseline methods. We note this practice in our motivating example on page 3 (lines 130-131).

To validate our approach in a more realistic scenario, we also conducted supplementary experiments with real human feedback, as mentioned in Appendix E.7. We will clarify the use of a simulated oracle for the main experiments in Section 4.

5. Discussion of State-of-the-Art Baselines

Thank you for this suggestion. We agree that a more explicit high-level comparison of the baselines would strengthen the paper.

6. Evaluation Metric in Table 1

We sincerely apologize for this omission. You are correct that the metric should be clearly stated. We will revise the caption of Table 1 and the main text to clarify:

• For the DMControl suite tasks, the reported values are the average episode returns.
• For the MetaWorld tasks, following the evaluation protocol of previous work, the metric is the ground truth success rate, reported as a percentage.

7. Clarification of Claims

"Limited query budget":** In our experimental setup, the agent is allowed to query for a fixed total number of preferences throughout training. This is enforced by setting a query frequency $K$ and a total number of environment steps (as seen in Algorithm 2 and the x-axis of Figure 2. This finite number of available queries constitutes the "limited budget." Our goal is to maximize performance within this constraint. We will make this definition explicit in the experimental setup section.

8. Generality of the KL Constraint $\epsilon$

This is an important question about the practical application of our method. For all experiments reported in Table 1, we used the fixed value of $\epsilon=0.01$ across all tasks and environments without any per-task tuning. Our results show that PPE consistently outperforms the baselines across a diverse set of locomotion and manipulation tasks. This demonstrates that the recommended value of $\epsilon=0.01$ is robust and generalizes well, alleviating the need for costly hyperparameter tuning for each new domain, which we consider a practical strength of our approach.

9. Bolding of Results in Table 1

Thank you for your careful eye and for pointing out this inconsistency. Your point is absolutely correct; bolding should be reserved for results that are statistically significantly superior, not just numerically higher when standard deviations overlap. As mentioned in our checklist response, we have performed Welch's T-tests to assess statistical significance (Appendix E) . In the revised manuscript, we will correct Table 1 and ensure that a result is bolded only if it is statistically significant (e.g., p-value < 0.05) compared to the next-best baseline.

10. Minor Issues

We appreciate you catching these details. We will make corrections in the revised manuscript.

Once again, we thank you for your thorough and valuable feedback. We are confident that addressing these points will make our paper stronger, clearer, and more impactful.

Sincerely,

The Authors

Dear Reviewer 1gpz,

We would like to extend our sincere gratitude for your detailed and insightful feedback on our manuscript. Your thoughtful comments and questions have been invaluable in helping us identify areas for improvement. We are encouraged that you found our paper well-written, our method novel, and our technical details sufficient.

We have carefully considered all your suggestions and provide the following point-by-point responses to your questions and concerns. We hope these clarifications will address the issues you raised and demonstrate the value of our work.

1. Terminology: "Preference Buffer"

Thank you for this question. The term "preference buffer" is a direct and intuitive adaptation of the standard "replay buffer" concept for the Preference-based RL (PbRL) setting. In PbRL, instead of storing state-action-reward-next-state tuples, the agent collects and learns from preference data. This data, consisting of pairs of trajectory segments and a human preference label, i.e., $(\tau^0, \tau^1, y_p)$, is stored in a dataset $\mathcal{D}_p$. While the exact phrase "preference buffer" may not be universally standardized, the concept of a buffer or dataset for storing preferences is fundamental to nearly all modern PbRL algorithms. Foundational works like Christiano et al. (2017) and Lee et al. (2021a, 2021b) all rely on collecting and sampling from such a dataset of preferences to train the reward model. We will clarify this in the revised text by explicitly defining it as the dataset $\mathcal{D}_p$ that stores preference tuples and citing these foundational works to ground the concept.

2. Experimental Setup in Section 3.1 (Motivating Example)

We apologize for the lack of clarity. The training and evaluation regions are distinct spatial areas within the 9x9 grid world, as depicted in Figure 1(a). The training region is centered, while the evaluation region expands outwards from the center. Therefore, even when their side lengths are equal (e.g., both are 3x3), they are not identical unless both are the full 9x9 grid. For instance, when the training region is 3x3 and the evaluation region is 3x3, the evaluation is performed on a 3x3 area that may or may not perfectly overlap with the training region, depending on the specific setup shown in Figure 1(a). We will revise the caption and text in Section 3.1 to make this distinction clearer.

This is an excellent question that highlights a key motivation for our work. The ensemble was implemented using a standard approach: we trained multiple reward models (5 in our experiment) on the exact same preference buffer, with diversity introduced only through different random network initializations and different mini-batch shuffling during training. This is a common technique for estimating uncertainty (e.g., in RUNE). Our finding, as shown in Figure 1(b), is that the variance produced by such an ensemble does not serve as a reliable indicator for distinguishing in-distribution (ID) from out-of-distribution (OOD) transitions. The variance can be high for complex ID transitions and low for simple OOD ones. This observation motivates our decision to use a dedicated OOD detection mechanism (the Morse Neural Network) rather than relying on ensemble variance for guiding exploration.

The ground-truth return for a given trajectory is calculated by summing the predefined, ground-truth rewards of each cell visited in that trajectory. In this specific experiment, trajectories are generated by a random walk (i.e., uniformly sampling actions), not by a learned policy. The goal is not to evaluate a policy, but to assess the reward model's ability to correctly rank pre-existing trajectories. Thus, the ground-truth return $G(\tau)$ for a trajectory $\tau$ is simply the sum of the true rewards encountered, and no single policy is employed to generate all trajectories for evaluation.

There may be a misunderstanding of the plot. In Figure 1(c), the blue curve (representing a tiny 1x1 training region) shows a sharp decrease in performance as the evaluation region expands. It starts with a high Spearman correlation when the evaluation region is also 1x1 (i.e., evaluating on the same single cell it was trained on) and then plummets as the evaluation region grows to include states the reward model has never seen. This result strongly supports our central claim: reward models generalize poorly to OOD trajectories, and their reliability is fundamentally tied to the coverage of the preference buffer.

When sampling 1,000 trajectory pairs from a small region, duplicates are indeed likely. We sampled with replacement. For this motivating experiment, the presence of duplicate trajectories in the preference buffer is not an issue. It simply means the reward model observes the same preference comparisons multiple times during training, which is standard. The key takeaway of the experiment is about the effect of the spatial coverage of the training data, not the diversity of trajectories within that covered region.

We agree that moving key details about the motivating example from the appendix to the main text would improve readability. In the revised manuscript, we will incorporate the most relevant information from Appendix G into Section 3.1, space permitting.

3. Methodology

In our framework, the "target policy" $\pi_T$ refers to the current actor's policy that we are actively trying to improve. In our implementation, which uses SAC (as mentioned in Algorithm 2), $\pi_T$ is the current SAC policy. Our exploration policy, $\pi_E$, is derived from $\pi_T$ using the proximal-policy extension method to gather data that is most useful for the next update of $\pi_T$. We will add this clarification to the text.

This is a crucial question regarding the practicality of our method. As shown in our ablation study in Section 4.2 (Figures 2b and 2c), we investigated the sensitivity of these parameters. We found that performance is robust within a reasonable range of values. Optimal performance was achieved around $\kappa=0.5$ and $\epsilon=0.01$. Based on this, we used these fixed values for all tasks in our experiments without any per-task tuning. This indicates that while PPE introduces new hyperparameters, they are not overly sensitive and do not require a costly tuning process for each new task, thus supporting the practicality of our approach.

• Interpretation of Table 8: We apologize for not having the appendix available to you. Table 8 in Appendix F.1 reports the performance of our Morse Network as an OOD detector on several tasks, using standard metrics like AUROC and AUPR. High values for these metrics indicate that the network is effective at distinguishing between in-distribution data (from the preference buffer) and out-of-distribution data, which is crucial for our method.
• Evaluation on More Tasks: Yes, a separate Morse Network must be trained for each task, as the distribution of state-action pairs is task-specific.
• Training Data: The Morse Network is trained using the state-action pairs $\{(s,a)\}$ available in the current preference buffer $\mathcal{D}^p$, as described by the loss function in **Eq. (2)*. The size of this buffer grows as more feedback is collected. In our experiments, the training starts with an initial set of preferences and is periodically updated using all data in $\mathcal{D}^p$. We will ensure these details are clearly stated in the appendix of the final version.

4. Ablation Study

Your understanding is perfectly correct, and this highlights the synergy between the two components. We will gladly add pseudocode for these ablations in the appendix of the revised paper.

• Ablation 1 (QPA + EXT w/o MDQ): In this setup, the agent explores using our proximal-policy extension method (EXT) to gather OOD data into the main replay buffer. However, for querying, it uses a standard method (in our case, the policy-aligned query from QPA) which does not explicitly prioritize selecting this newly found OOD data for labeling. As a result, the preference buffer's coverage does not expand effectively, and the reward model's OOD generalization does not improve, leading to minimal performance gains.
• Ablation 2 (QPA + MDQ w/o EXT): Here, the agent explores normally (via the SAC policy's stochasticity), without active OOD exploration from EXT. The MDQ mechanism is then used to select pairs for querying. However, since the replay buffer contains very little actively-sought OOD data, the OOD sampling part of MDQ ($P^{out}$) has a poor set of candidates to select from. This also leads to ineffective expansion of the preference buffer's coverage.

As you rightly pointed out, the two components are complementary. EXT provides the OOD data, and MDQ ensures this data is incorporated into the preference buffer to improve the reward model. This synergy is why the full PPE algorithm significantly outperforms either component alone, as shown empirically in Figure 2(a).

---

Once again, we thank you for your rigorous review and constructive suggestions. We believe that by incorporating these changes, we can significantly improve the clarity and impact of our paper.

Sincerely, The Authors

Dear Reviewer 1gpz,

We would like to express our sincere gratitude once more for the incredibly insightful discussion. Your detailed feedback has been truly invaluable in helping us strengthen the paper.

As the final decision phase approaches, we just wanted to gently inquire if, in light of our fruitful conversation, you might perhaps be willing to reconsider your final evaluation score.

We deeply appreciate the significant time and effort you have already dedicated to our work. Thank you for your consideration.

Sincerely, The Authors

Dear Reviewer X7X4,

We would like to express our sincere gratitude for your time and effort in providing such a detailed and constructive review of our manuscript. Your insightful comments and suggestions are invaluable to us, and they have helped us identify key areas for improvement. We are encouraged that you find our research on the exploration-exploitation trade-off to be valuable and potentially high-impact, and that you appreciate our empirical results and comprehensive appendices.

We have carefully considered all your feedback and provide a point-by-point response below. We believe that by addressing these points, we can significantly strengthen our paper.

Responses to Weaknesses

1. Regarding the sufficiency of ablation studies:

We agree with you that proper ablation studies are crucial for disentangling the contributions of the different components of our proposed algorithm. We appreciate you acknowledging the importance of the ablation study presented in Section 4.2.

As you noted, the current ablation in Figure 2(a) was conducted on the Walker-walk task. Our rationale was to use this as a representative environment to demonstrate the synergy between the proximal-policy extension method (EXT) and the mixture distribution query method (MDQ). The results clearly show that using either component in isolation does not lead to significant improvements, and their complementary nature is what drives the superior performance of the full PPE algorithm.

We take your point about the statistical significance and the desire for broader evidence. In the revised version of our manuscript, we will expand our ablation studies to include additional environments. This will provide more robust evidence for our claims and better illustrate the individual contributions of each component of PPE.

2. Regarding the level of detail in the Related Work section:

Thank you for your suggestion to expand the discussion on existing exploration approaches. In Section 5, under "Exploration in RL", we do provide a brief overview, noting that prior PbRL work has focused on reward model disagreement for exploration, while our approach focuses on maximizing preference buffer coverage. We also situate our work within the broader context of RL exploration strategies like uncertainty-driven and information-theoretic methods.

However, we agree that a more detailed comparison with the exploration strategies used in our baseline methods would strengthen the paper.

3. Regarding missing basic ablations (e.g., entropy bonus):

This is an excellent point. To ensure a fair comparison in our main experiments, we utilized the official code repositories of the baseline algorithms and their recommended hyperparameter settings wherever possible. Our method, PPE, is integrated into the QPA framework, and the underlying RL algorithm is SAC. The hyperparameters for SAC were kept consistent with the baseline to isolate the effects of our proposed exploration and query methods.

While we did not perform a specific ablation on the SAC entropy bonus coefficient, we did conduct sensitivity analyses on the key new hyperparameters introduced by PPE, namely the mixture ratio $\kappa$ and the KL constraint $\epsilon$. We agree that an ablation on the entropy coefficient would be informative. ]

4. Regarding the motivating example:

We appreciate your critical feedback on the motivating example in Section 3.1. While we acknowledge that the general principle of neural networks struggling with out-of-distribution (OOD) data is well-established, our goal with this experiment was more specific to the PbRL context. We aimed to systematically demonstrate that this "distributional mismatch" is a critical issue where reward models trained on static buffers fail on OOD trajectories from policy exploration, and that "such failures in policy-proximal regions directly misguide iterative policy updates".

The key takeaways we wanted to highlight were: • The necessity of making preference buffer coverage an explicit optimization objective, which is not standard practice. • A non-trivial finding that a common method for uncertainty estimation, ensemble variance, fails to distinguish between in-distribution and OOD transitions in this setting, as shown in Figure 1(b). • The crucial insight from Figure 1(d) that simply exploring OOD data is not enough; a balance between in-distribution and OOD feedback is essential for maintaining the reward model’s accuracy.

To address your concern about the lack of detail, we will add the full experimental parameters for this example (e.g., network architecture, training details) to the appendix in our revision to improve clarity and reproducibility.

5. Regarding the availability of source code:

We apologize for not providing the code with the initial submission. As stated in our checklist, we are fully committed to open science and will make the complete source code publicly available upon acceptance of the paper.

Responses to Questions

1. Regarding the details of the human feedback experiments:

Thank you for your interest in this aspect of our work. The primary quantitative results in our paper are based on experiments with a simulated oracle to ensure controlled and extensive comparisons, as is common practice in the field. However, we agree that validation with real human feedback is very important.

As mentioned in the checklist, we did conduct experiments with real human participants. We recruited volunteers who were compensated for their time to provide preference labels. The full instructions provided to these participants are included in the supplementary material, and Appendix E.7 details how this feedback was incorporated. The results are provided as qualitative evidence in the form of videos of learned agent behaviors. We acknowledge that quantitative details such as the number of participants and queries were not in the main text. We will add a detailed description of the human feedback study, including the number of participants, the number of queries collected per participant, and other relevant experimental details, to the appendix in the revised version.

2. Regarding the comparison to a standard SAC baseline:

The goal is not to outperform SAC trained with the ground-truth reward, but rather to learn a policy that is as close as possible to that upper bound while using a minimal amount of preference feedback. Previous work in PbRL has established the performance of SAC with ground-truth rewards on these benchmarks. However, for completeness and to provide a clear performance ceiling for our experimental setting, we agree that including this baseline is beneficial.

Responses to Limitations

We thank you for pushing us to expand our discussion on the limitations of our work. This is crucial for positioning our contributions accurately. The current discussion in Section 6 acknowledges that our query method could be improved by considering the information content of behavior pairs. We will expand this section in our revision to incorporate the excellent points you raised.

Once again, we thank you for your detailed and insightful review. We are confident that by incorporating your suggestions, we can substantially improve the quality and clarity of our paper.

Sincerely,

The Authors

Dear Reviewer Eg1k,

Thank you for your thorough review and insightful feedback on our manuscript. We are grateful for your positive comments on the motivation, ablation studies, and clarity of our work. Your constructive criticisms are invaluable, and we believe that addressing them will significantly improve the quality of our paper.

Below, we address each of the weaknesses you identified in a point-by-point manner.

---

Response to Weaknesses

1. Weakness: Scope of OOD Detection

Thank you for this insightful question. We apologize if this aspect was not made sufficiently clear in the manuscript. We would like to clarify that our out-of-distribution (OOD) detection mechanism does, in fact, account for the novelty of the state s.

The core of our OOD detection is the perturbation model $\hat{a} = f_{\phi}(s, a)$, which is a neural network that takes both the current state s and action a as input. The OOD score is then derived from the difference between the perturbed action $\hat{a}$ and the original action a, specifically $M_{\phi}(s,a)=1-K_{RBF}(f_{\phi}(s,a),a)$.

Because $f_{\phi}$ is a function of s, its output is conditioned on the state. If the agent encounters a state s that is OOD with respect to the states present in the preference buffer $\mathcal{D}^p$, the neural network $f_{\phi}$ is expected to generalize poorly. Consequently, it will likely produce a significant perturbation, meaning $f_{\phi}(s, a)$ will be far from a, even if the action a itself is a common one. This will result in a high OOD score $M_{\phi}(s,a)$, correctly identifying the transition $(s, a)$ as novel and warranting exploration.

In our revision, we will add a more detailed explanation in Section 3.2 to explicitly state that the novelty of the state s is implicitly captured by the perturbation model $f_{\phi}(s, a)$, thereby ensuring the OOD detection mechanism is sensitive to novel states as well as novel actions.

---

2. Weakness: Unclear Objective of the Loss Function

Thank you for this very sharp observation and for suggesting the connection to contrastive learning. We agree that our explanation was insufficient and could lead to confusion.

The formulation of our loss function in Equation 2 is directly adopted from the work on Morse Neural Networks by Dherin et al. (2023) . As stated in our paper, the objective is to minimize the KL divergence between unnormalized measures, i.e., $D_{KL}(\mathcal{D}^{p}(s,a)||1-M_{\phi}(s,a))$, based on principles from Information Geometry (Amari, 2016). This is distinct from the standard KL divergence between probability distributions, and we apologize for not making this critical distinction clear.

Your insight regarding its resemblance to contrastive learning is excellent and provides a very helpful intuition. Indeed, the loss function operates on similar principles:

• The first term, $\frac{\lambda^{2}}{2}||f_{\phi}(s,a)-a||^{2}$, encourages the model to produce minimal perturbation for in-distribution pairs from $\mathcal{D}^p$, effectively aligning the representation with the target (akin to positive pairs).
• The second term, involving a sum over uniformly sampled actions $a_u$, pushes the model's output for these "negative" pairs away from their corresponding actions, promoting uniformity over the action space for a given state.

This "alignment and uniformity" characteristic is the hallmark of contrastive learning.

3. Weakness: Applicability to Language Models

Thank you for raising this excellent point. We agree that extending our coverage-driven approach to the alignment of Large Language Models (LLMs) is a highly promising and important future direction. While our current experiments are in continuous control, the underlying principles of our method can be adapted to discrete domains like language.

Here is a potential path for adaptation:

• OOD Detection for Language: The state s would be the context (prompt and previously generated text), and the action a would be the next token. Our OOD detection mechanism, based on Morse Neural Networks, could operate on the continuous embedding space of these (context, token) pairs. The model $f_{\phi}$ would be trained to identify generated tokens that are semantically novel or unusual given the context, relative to the examples in the preference buffer.
• Exploration in Discrete Spaces: We acknowledge that our proximal-policy extension method (Equation 4), which relies on the gradient $\nabla_{a}M_{\phi}(s,a)$, is not directly applicable to the discrete token space. However, this can be addressed in several ways:
  1. Exploration in Logit Space: Exploration could be performed by adding a calculated perturbation to the pre-softmax logits, which exist in a continuous space. The OOD score $M_{\phi}$ could guide the direction of this perturbation.
  2. Modulating Sampling Probabilities: The OOD score $M_{\phi}(s, a)$ for each potential next token a could be used to directly modulate the final sampling probability distribution. For instance, we could up-weight the probabilities of tokens that lead to unexplored, OOD regions, thereby encouraging the LLM to generate more diverse and novel text for subsequent preference labeling.

We are very optimistic about this direction. In the "Conclusion and Discussion" section of our revised manuscript [see Section 6], we will add a detailed discussion on these potential adaptations for applying PPE to LLMs, explicitly mentioning the challenges and proposing these concrete solutions.

---

Once again, we thank you for your valuable and constructive feedback. We are confident that by incorporating these revisions, we can significantly strengthen our paper and address your concerns. We hope that our revised manuscript will meet the standards for acceptance.

Sincerely,

The Authors