CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

Dear Reviewer,

Thanks for your valuable and detailed comments. We hope the following statement clear your concern. We conducted additional experiments and the results are shown in the link.

Claim 1, Experimental Designs 1 and Q2: Preference collection and exploration-exploitation trade-off.

A for Claim 1, Experimental Designs 1 and Q2: The preference labels in our study are collected over multiple rounds, with the embedding space updated iteratively as new queries are selected. As for the exploration-exploitation trade-off, CLARIFY prioritizes exploitation by focusing on unambiguous queries, while its reject sampling strategy (Sec. 4.2) inherently supports exploration. This strategy diversifies the queries by sampling from a distribution, avoiding overemphasizing the "clearest" pairs.

As suggested, we conducted experiments to evaluate the exploration-exploitation trade-off. We compared CLARIFY against two baselines: (1) pure exploration (random query selection, referred to as "Random") and (2) pure exploitation (maximizing the density for clearly-distinguished queries $\rho_\text{clr}(d_\text{emb})$, referred to as "Exploitation"). As shown in Tables 1 and 2 in the supplement link, CLARIFY outperforms Random by over 20% in success rate, and surpasses Exploitation by over 15%. These results demonstrate an effective exploration-exploitation trade-off.

Claim 2: Label efficiency.

A for Claim 2: As suggested, we evaluate CLARIFY's label efficiency from 50 to 2000. Table 3 in the supplement link shows that CLARIFY consistently outperforms baselines under various query numbers. In addition, we observed that the performance of CLARIFY with 100 queries approaches that of MR with 1000 queries, demonstrating CLARIFY's effectiveness in achieving high performance with fewer labels.

Experimental Designs 2 and Q3: Reward fitting accuracy on the test set.

A for Experimental Designs 2 and Q3: As suggested, we evaluate CLARIFY's reward fitting accuracy on the test set. Table 4 in the supplement link reflects the effectiveness of reward fitting, showing that CLARIFY attains about 3% higher accuracy than OPRL on most tasks.

W1: How Eq. 6 resolves representation collapse.

A for W1: The quadrilateral loss $\mathcal L_\text{quad}$ prevents representation collapse by enforcing preference-aware geometry:

1. For clear preferences, it creates a hyperplane separating good and bad trajectories (Proposition 5.2).
2. For ambiguous pairs, while $\mathcal L_\text{amb}$ minimizes their embedding distance, $\mathcal L_\text{quad}$'s contrastive gradient prevents trivial clustering of these pairs.

To support this, we conduct an ablation study on $\mathcal L_\text{amb}$, as shown in Table 5 in the supplement link. Removing $\mathcal L_\text{quad}$ degrades performance by about 10% on average, validating its necessity.

A for Other Comments: Thank you for your keen attention to detail! We have corrected the positive and negative samples to $(\sigma_+, \sigma^\prime_+), (\sigma_-, \sigma^\prime_-)$ in the revised manuscript.

Q1: $\mathcal L_\text{reward}$ optimization.

A for Q1: To elaborate, we present an additional figure (Figure 1 in the supplement link) that illustrates the architecture of CLARIFY. As shown in the figure, the reward model is updated using the loss function $\mathcal L_\text{reward}$ (Eq. 2) during the reward learning phase, while the embedding space is updated according to Eq. 11 during the embedding learning phase. These two phases are strictly decoupled. Thus, the loss function $\mathcal L_\text{reward}$ is not incorporated into Eq. 11. Training the reward model solely with the Bradley-Terry loss is a standard approach in the PbRL literature [1,2,3]. This separation allows preference learning to focus on human feedback, while embeddings specialize in distinguishing trajectory pairs.

Thanks again for the valuable comments. We hope our response has cleared your concerns. We are looking forward to more discussions.

[1] Lee, Kimin, et al. "B-pref: Benchmarking preference-based reinforcement learning." arXiv preprint arXiv:2111.03026 (2021).

[2] Shin, Daniel, et al. "Benchmarks and Algorithms for Offline Preference-Based Reward Learning." Transactions on Machine Learning Research.

[3] Cheng, Jie, et al. "RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences." International Conference on Machine Learning. PMLR, 2024.

Dear Reviewer,

Thanks for finding our paper nice presentation, novel, thorough experiments and solid improvement. We hope the following statement clear your concern.

Claim and W1: Comparison to [1] and its offline applicability.

A for Claim and W1:

• While both CLARIFY and [1] attempt to tackle ambiguous queries, their approaches are fundamentally different. [1] address ambiguous queries by learning diverse skills through unsupervised exploration. The requirement for online exploration makes it incompatible with offline settings, where agents cannot interact with the environment. In contrast, CLARIFY operates entirely offline by learning preference-informed trajectory embeddings through contrastive learning. This eliminates the need for online exploration, making CLARIFY more suitable for real-world scenarios where environment interaction is costly, risky, or unavailable.

• Additionally, CLARIFY optimizes query distinguishability directly via embedding distances, whereas [1] relies on skill diversity as a proxy. This independence from skill diversity enhances CLARIFY's robustness in tasks with constrained skill exploration (e.g., limited state-space traversability), which eliminates the reliance on costly exploration for discovering diverse skills.

Methods 1: Task complexity and generalizability.

A for M1: We appreciate the reviewer’s concern regarding task complexity. Our experiments include challenging tasks, such as MetaWorld's peg-insert-side, which require multi-step reasoning and are more complex than prior RLHF [2] evaluations focused on locomotion. CLARIFY significantly outperforms baselines across these tasks, as shown in Table 1 of the paper, demonstrating its effectiveness beyond simple settings.

W2: Importance of offline setting and query sampling.

A for W2: Offline reinforcement learning assumes agents learn solely from a fixed pre-collected dataset without environment interaction. It is critical for safety-sensitive domains (e.g., healthcare, autonomous systems), where real-time exploration is either unsafe or impractical. In such cases, agents must rely on pre-collected datasets for learning, and thus, queries should be sampled from this static dataset rather than generated on-policy. This aligns with CLARIFY's approach, which leverages offline datasets and contrastive learning to ensure the method's applicability in real-world, offline settings.

We sincerely thank the reviewer again for the timely and valuable comments. We hope that our response and additional experimental results have cleared most of your concerns.

[1] Mu, Ni, et al. "S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning." arXiv preprint arXiv:2408.12130 (2024).

[2] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

Dear Reviewer,

Thanks for your valuable and detailed comments. We hope the following statement clear your concern. We conducted additional experiments and the results are shown in the link.

Claim, W1 and Q4: Embedding failure cases.

A for Claim and Q4: We would like to clarify that Figure 2(c) in the original paper visualizes embeddings trained with both losses rather than using only quadrilateral loss.

As suggested, we visualize the failure modes in Figure 1 in the supplemental link. Specifically, using only $\mathcal L_\text{amb}$ causes erroneous clustering, where low and high return trajectory embeddings are intermixed (Figure 1(a) in the supplemental link). In contrast, using only $\mathcal L_\text{quad}$ yields densely packed embeddings with insufficient separation (Figure 1(b)). Combining both losses resolves these issues and leads to smooth, coherent clusters (Figure 1(c)). This illustrates their complementary roles in structuring the embedding space.

Experimental Design 1 and Q5: Naive baseline comparison.

A for Experimental Design 1 and Q5: As suggested, we compare CLARIFY with the naive approach that rules out pairs with no_cop preference labels, as shown in Table 1 in the supplement link. CLARIFY outperforms the naive approach (the "Naive" method in the table) by over 50% on most tasks. This demonstrates the effectiveness of CLARIFY.

Experimental Design 2, W2 and Q3: Computational costs.

A for Experimental Design 2, W2 and Q3:

• As suggested, we analyze the computational cost and the effort required for hyperparameter tuning of CLARIFY, as shown in Table 2 in the supplement link. CLARIFY incurs moderate computational overhead, roughly 2-3 times that of OPRL, primarily due to embedding learning. While training time may increase, CLARIFY effectively identifies more clearly distinguished queries, accelerating the labeling process.
• On the other hand, CLARIFY demands minimal hyperparameter tuning. In the original paper, key hyperparameters ($\lambda_\text{amb},\lambda_\text{quad}$) were fixed across tasks with robust performance. To support this, we conduct additional experiments to visualize the embeddings under various hyperparameter configurations, as in Figure 2 in the supplement link, illustrating the stability of embeddings under parameter variations. We have added these results in the revised version.

Q1: Ambiguity's impact on PbRL.

A for Q1: Ambiguous queries hinder PbRL primarily in two ways:

1. Labeling efficiency: Human teachers often struggle to differentiate between similar segments, leading to skipped labels (no_cop) that waste annotation effort. As in Table 3 of the original paper, only about 50% of queries receive clear preference labels, indicating significant inefficiencies.
2. Reward learning accuracy: Labelers may produce random or incorrect preferences when segments are only marginally different. This can introduce errors to the reward model, ultimately degrading policy performance.

Q2: CLARIFY vs S-EPOA [1].

A for Q2: While both methods tackle ambiguous queries, they differ fundamentally in their approaches. S-EPOA addresses indistinguishability via unsupervised exploration and skill-based query selection in online settings. In contrast, CLARIFY utilizes contrastive learning to create trajectory embeddings informed by preferences, which enables offline query filtering. One of the key advantages of CLARIFY is that it eliminates the need for online exploration, making it particularly suitable for real-world applications, where interaction can be costly or risky. Furthermore, CLARIFY directly optimizes query distinguishability rather than relying on skill diversity as a surrogate. This independence from skill diversity enhances CLARIFY's robustness in tasks with constrained skill exploration (e.g., limited state-space traversability), eliminating reliance on costly exploration for discovering diverse skills.

Answer to Essential References and Other Comments: Thank you for your keen attention to detail! We have subscripted $\mathcal L_\text{amb}$ throughout the manuscript. Additionally, we have discussed [2] in the related work section.

Thanks again for the valuable comments. We sincerely hope our additional experimental results and explanation have cleared the concern. More comments on further improving the presentation are also very much welcomed.

[1] Mu, Ni, et al. "S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning." arXiv preprint arXiv:2408.12130 (2024).

[2] Liu, Tianqi, et al. "Statistical Rejection Sampling Improves Preference Optimization." The Twelfth International Conference on Learning Representations.

Dear Reviewer,

Thanks for your valuable and detailed comments. We conducted additional experiments and the results are shown in the link.

C1.1: Robustness to non-ideal teachers.

A for C1.1: We evaluated CLARIFY with a new flipping teacher that randomly assigns preferences for close-reward queries. Table 1 shows that CLARIFY outperforms OPRL by over 20% in success rate, confirming its robustness to varied non-ideal feedback.

C1.2: CLARIFY's behavior across $\epsilon$ values.

A for C1.2: We evaluated CLARIFY with $\epsilon$ in 0$\sim$0.7. As shown in Table 2, at $\epsilon=0.3$, CLARIFY outperforms OPRL by 8~18%. In contrast, at $\epsilon=0$, CLARIFY's query selection reverts to random sampling, leading to performance comparable to MR. This suggests that CLARIFY excels when distinguishing a query is difficult.

C2.1: Additional component analysis.

A for C2.1: We conduct component analysis on DMControl tasks with $\epsilon$=0.7. Tables 3 and 4 show that full CLARIFY ($\mathcal L_\text{amb}$ + $\mathcal L_\text{quad}$ + rejection sampling) consistently performs best, confirming component necessity.

C2.2: Comparison to PEBBLE.

A for C2.2: We compare CLARIFY to uncertainty-based sampling (PEBBLE). Table 5 shows that CLARIFY achieves better performance, demonstrating the effectiveness of our query selection strategy.

C2.3: Architecture impact.

A for C2.3: We illustrate our architecture in Figure 1, which shows that CLARIFY operates in two strictly decoupled phases. In the embedding training phase, the Bi-directional Decision Transformer encoder trains solely on trajectories using contrastive and reconstruction losses, without access to reward model or policy. In the reward learning phase, the reward model updates only on preference data without access to embedding information. This design prevents the encoder from leaking privileged information about future states or rewards.

Answer for Theoretical, Weaknesses, Other Comments:

• (Theoretical) Prop 5.2: corrected to $d(z^-,H)\ge\eta$.
• (Other Comments) no_cop denotes cases where labelers consider queries too similar to specify a preference, resulting in a skipped label.
• (W1) Query clarity ratio is defined as the proportion of clearly-distinguished queries to the total number of queries.
• (W2) Clearly-distinguished queries are those where human preferences are clear (not no_cop).
• (W3) Safety impact: A discussion of malicious use risks was added.

Q1, Q2, Q3, Q6: Clarity of the statement.

• A for Q1: Eq. 9 is integrated into the total training objective (Eq. 11) used in Algorithm 1 line 3. It is not a simple pretraining but trains the encoder continuously.
• A for Q2: Distance metric $\ell$ is the Euclidean distance. We conduct additional experiments to compare the Euclidean and Cosine distances. Figure 2 shows that points in the embedding trained with Cosine distance cluster together, while Euclidean distance is more suitable for trajectory embedding.
• A for Q3: Query number: 1000 for MetaWorld tasks, 500 for cheetah-run, 200 for walker-walk (Table 10).
• A for Q6: The thresholds of Large, Medium, and Small in MetaWorld and DMControl tasks are 300/100/10 and 30/10/1 respectively.

Q4: Performance with fewer queries.

A for Q4: We evaluate CLARIFY with 50 to 2000 queries. Table 6 shows a slight performance decline for CLARIFY with only 50 queries, though it still outperforms MR. Table 7 shows CLARIFY's performance with 100 queries on various tasks, which reaches about 70% of IQL's performance with ground truth rewards.

Q5: Statistical validation for walker-walk.

A for Q5: We conduct a statistical significance analysis for CLARIFY and OPRL.

• Table 8 shows the 95% confidence intervals (CIs) of the query clarity ratio and label accuracy. The narrow intervals validate CLARIFY’s performance. Though walker-walk shows overlapping CI due to high environment stochasticity, CLARIFY's directional improvements in both metrics demonstrate the effectiveness of its query selection.
• Additionally, we conducted independent two-sample t-tests comparing CLARIFY with OPRL. The experimental results in Table 9 show that CLARIFY achieves statistically significant improvements ($p<$0.05) in 5/6 tasks. These results confirm the advantage of CLARIFY's query selection over OPRL.

Q7: T-SNE visualization robustness.

A for Q7: Thanks for pointing out this issue, and we conduct visualizations across multiple random seeds (Figure 3), revealing consistent clustering patterns, indicating robust embeddings. Additionally, we provide PCA (Figure 4) visualizations to support our conclusion.

We hope that our response has cleared most of your concerns.