Aligning With Human Values Without Revealing Human Judgements

1. "The authors claim that the paper provides the first algorithms that can achieve provable privacy of human judgments. However, [1] studied RLHF under both local DP and central DP. And the authors didn't cite the paper and didn't compare the results with theirs.'

Please see our response to Reviewer CYVZ above. Note also that our results were online before [1].

[1] Sayak Ray Chowdhury and Xingyu Zhou and Nagarajan Natarajan. Differentially private reward estimation with preference feedback, AISTATS 2024.

2. "The concept of DP in section 2.1 is not clear for the corresponding case of RLHF. What is the output of algorithm $\mathcal{A}$? Why did you choose approximate DP, not pure DP? Why did you choose central DP, not local DP?"

In the full RLHF pipeline the algorithm $\mathcal{A}$ outputs a policy $\pi$ which assigns a probability $\pi(s,a)$ to each state-action pair. Thus $\pi \in \mathbb{R}^{S\times A}$. The minimax optimal rates achievable with approximate differential privacy are provably not achievable with pure DP, even in the simpler setting of private empirical risk minimization [1]. Hence, we chose approximate DP, as is standard in the literature on private convex optimization and private ERM. Central DP makes the most sense in the context of RLHF training where raters interact with an LLM hosted by a trusted provider (all of the strongest frontier models are hosted in this way). Thus, if these raters’ responses are subsequently used by the trusted provider to train a new LLM via RLHF, then the rater’s private data could be leaked to anyone who interacts with the new LLM. This corresponds precisely to the setting of central DP.

[1] Bassily, Raef, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. FOCS, 2014.

3. "In algorithm 1, why did you use objective perturbation in Line 4 and output perturbation for DP together? See section 5.2 in [1] for objective perturbation to guarantee DP. Why did you choose the regularizer parameter $\alpha$ to be related to private parameter $\epsilon$."

Objective perturbation alone only guarantees that the true optimum of the perturbed objective is differentially private. However, the true optimum is never exactly achieved via first-order methods, but rather an approximate optimum is computed. In this case, output perturbation is additionally necessary in order to maintain privacy.

The relationship between $\alpha$ and the privacy parameter arises due to the need to trade-off errors in the $\ell_2$-norm and the dataset-dependent norm $\lVert\cdot\rVert\_{\Sigma\_{\mathcal{D}} + \lambda I}$.

4. "The authors should cite earlier work like [2] for the pessimistic method."

Thank you for providing the reference. We can add the citation to [2].

5. ”What is the dependence on $B$ and $L$ in your final suboptimality gap?”

The dependence on $B$ and $L$ is $O(L^2 + B)$.

6. "In lines 110-111, the notation is confusing for $\sum_{j=1}^K \sum_{j=k+1}^K$. There are two "j" indexes."

Thank you for catching this typo, it should instead read $\sum_{j=1}^K \sum_{k=j+1}^K$.

Thank you for stating that our paper achieves minimax-optimal performance, ensuring the security of private data and further pointing out that our algorithm preserves privacy while matching the performance of non-private methods.

1. ”In Section 4.2, the paper discusses the case of $k$-wise comparison. Is it possible for agents to make comparisons involving different numbers of items?”

A standard approach to deal with situations where different numbers of items are compared across raters is to interpret each ranking of $K$-items as $\binom{K}{2}$ pairwise comparisons, and then to simply use the Bradley-Terry model for this new dataset of pairwise comparisons.

2. ”Is $O(\sqrt{d/n})$ optimal for both contextual bandits and MDP settings with and without differential privacy? If so, it would be helpful to mention these results in the main paper.”

Yes, indeed $O(\sqrt{d/n})$ is minimax optimal for the contextual bandits setting, and hence, because contextual bandits are a subset of MDPs, is also optimal for the MDP setting. We can add a mention of this to the main paper.

3. ”In the MDP setting in Section 5, is there a specific reason why the results focus on $K=2$ rather than a general $K$?”

This is primarily for clarity of the presentation, as larger $K$ leads to very unwieldy notation without much additional insight. A generalization to larger $K$ can be obtained via a straightforward translation of our results for $K$-wise comparisons to the case of trajectories.

4. ”The paper is limited to the Plackett-Luce model; it would be interesting to explore whether similar results could be derived for other comparison models, such as the Thurstone-Mosteller model.”

The reason to focus on the Plackett-Luce model and the pairwise special case of the Bradley-Terry model is that this is the standard model used in practice for RLHF on LLMs. Thus, in order to prove results most closely aligned with current best-practices, we chose to focus on the Plackett-Luce model.

Thank you for writing that our paper shows that it is possible to achieve differential privacy without any asymptotic loss in reward while further our paper is quite well written and easy to follow in an important area of consideration, and thank you for providing a thoughtful review.

1. ”If the authors can elaborate what makes this result novel, and not a direct consequence of a straightforward application of the gaussian mechanism, that would help me evaluate this work more fairly.”

The main difficulty encountered here not present in prior work is that there are two stages to RLHF: reward learning and policy optimization. The asymptotically best performing algorithms for the second stage require the learned reward function parameterized by $\theta$ and the data covariance $\Sigma$ in order to perform pessimistic policy optimization using the learned rewards. Our approach is to use differentially private versions of both $\theta$ and $\Sigma$ as input to the pessimistic policy optimization step. Separately one can keep $\theta$ private with small $\ell_2$-norm error with objective perturbation methods, and $\Sigma$ private with the Gaussian mechanism. However, for the pessimistic policy optimization to succeed we actually care about privacy of $\theta$ with respect to the dataset-dependent norm $\lVert \cdot \rVert\_{\Sigma\_{\mathcal{D}} + \lambda I}$. Thus, we must adapt standard objective perturbation to account for this different norm. But there is yet another obstacle: to maintain privacy our pessimistic policy optimization algorithm must not even have access to the true $\Sigma\_{\mathcal{D}}$ but only to a noisy DP version. Thus, balancing the privacy of $\theta$ and $\Sigma\_{\mathcal{D}}$ with the different norms required for optimal downstream policy performance requires a somewhat delicate analysis that is specific to RLHF.

2. ”Moreover, I find the last bit in the conclusion regarding the importance of privacy and how it can prevent data leaks by LLMs to be very much an oversell in this paper. The impact of this paper is limited to keeping the preferences of annotators who are involved in aligning these models, private. This by itself cannot and will not prevent data leaks from LLM responses due to the very nature of the way LLMs are trained. I would strongly prefer it if you could remove that statement/ reword it so as to not send a wrong message to the readers, especially ones who come from a more applied background and who may not be aware of how differential privacy works and what its limitations are.”

We will happily rephrase the last part in the conclusion to reflect the fact that our methods maintain privacy for the human raters that are used to align large language models during RLHF, and not for the data that might have been seen during pre-training.

1. "Literature review and the technical novelty"

While a preliminary version of our results were already publicly accessible online before the paper you refer to [1], please find the core foundational differences of our paper described in detail below.

The most notable differences between our results and those in [1] is first, that our results maintain privacy of the reward model for each entire preference tuple $(s,a_0,a_1,y)$ consisting of a prompt $s$ two possible responses $a_0,a_1$ the preference label $y$, and second, that we analyze the impact of privacy downstream on the RLHF training. In contrast, the paper [1] focuses on label differential privacy for the reward model alone. That is, [1] only keeps the preference labels $y$ private, and no analysis is done on how this private preference model will in fact perform downstream for RLHF.

This is an important distinction, because the best non-private RLHF algorithms in this setting utilize both the preference model and the data covariance (which depends on $s,a_0,a_1$) when optimizing the final policy, and thus the interplay between privacy for both the model and the covariance simultaneously is a core component of a full private RLHF pipeline.

Thus, our work indeed more directly addresses the potential privacy problems of the true RLHF setting, and second, the true RLHF setting privacy guarantees require a delicate and novel analysis on privacy with respect to the performance of the RLHF policy.

In more detail: first, in almost all use-cases the prompts themselves are produced by human raters or extracted from conversations with raters. Thus the prompts and the LLM responses may contain sensitive data about the rater. In fact, this sensitive data may be much more privacy-violating than the preference label alone, as it contains multiple sentences of text written by the rater.

Second, the fact that we require the entire RLHF pipeline to be private, and privacy for the entire preference tuple, indeed requires a careful and distinct analysis of the privacy. The entire point of private learning of preferences is to then actually use the learned preferences to train a model via RLHF. The best known non-private RLHF algorithm utilizes pessimistic policy optimization which uses the full data covariance matrix, and this is indeed the setting of our analysis. Thus, we must keep the data covariance private, and ensure that the pessimistic policy optimization algorithm still achieves good performance when a private covariance matrix is used. See also the discussion on line 394-400 in our paper, on balancing the privacy of the data-covariance and the private MLE estimator with the accuracy of pessimistic policy optimization.

The paper you refer to [1] as an obstacle to the novelty of our paper, which became publicly available more than a month after our paper was publicly available, seems to have added a vague sketch of how to extend the results to DP for the full RLHF pipeline in Appendix E with no formal proof, just handwaving in a single paragraph that the privacy of the covariance and the utility analysis can be done.

[1] Sayak Ray Chowdhury and Xingyu Zhou and Nagarajan Natarajan. Differentially private reward estimation with preference feedback, AISTATS 2024.

[2] Sayak Ray Chowdhury and Xingyu Zhou. Differentially Private Regret Minimization in Episodic Markov Decision Processes, AAAI 2022.

[3] Xingyu Zhou. Differentially Private Reinforcement Learning with Linear Function Approximation, ACM on Measurement and Analysis of Computing Systems, 2022.

2. "The current proof of privacy guarantee is not grounded (i.e., Theorem 4.1 and in particular Lemma A.10). The key issue is that the privacy proof in the original paper of Kifer et al 2012 has a gap."

The paper in the reference you listed [2] indeed fixes this gap for objective perturbation of GLM-structured loss functions of the form $l(\theta,x;y) = f(\theta^{\top}x;y)$. The loss function in Theorem 4.1 indeed has this form, which can be seen by letting $x = \phi(x,a_1) - \phi(x,a_0)$. Thus, Theorem 3.1 of [2] can be directly used as a drop-in replacement for the Kifer et al 2012 result to resolve this gap.

[2] Redberg, Rachel, Antti Koskela, and Yu-Xiang Wang. Improving the privacy and practicality of objective perturbation for differentially private linear learners. NeurIPS 2023.