Learning Differentially Private Rewards from Human Feedback

1. My main concern is about the technical difficulty. Given the algorithm in [1], Algorithm 1 is a standard application of objective perturbation, Algorithm 2 is a straightforward application of Gaussian mechanism, and Algorithm 3 is derived from replacing the estimations by their private counterparts. It would be better if the authors could highlight their technical contributions.

[1] Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.

2. The sub-optimality bound has dependence $\tilde{O}(1/\sqrt{n\epsilon})$, where the dependence is not optimal and the second term could dominate if $\epsilon$ goes to 0. As is shown in [2], the additional cost due to DP could be of $\tilde{O}(1/n\epsilon)$ for empirical risk minimization. In addition, [3] shows that the additional cost due to DP could be of $\tilde{O}(1/n\epsilon)$ for offline RL tasks. What is the challenge to derive such dependence?

[2] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression.

[3] Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy.

3. For the MDP setting, the reward for a whole trajectory is still a linear function of $\theta$. Given that the occupancy measure $\rho$ is an input of Algorithm 3, is this setting identical to the contextual bandit setting?

4. It would be better if the authors could discuss more about the papers about RL with JDP or LDP guarantees and the relationship to this work. For instance, would an algorithm with JDP/LDP still be JDP/LDP if the rewards are learned privately (as in this paper)? Here are some papers regarding online RL with JDP/LDP that may be relevant. [4,5] considers RL with JDP and LDP for tabular MDP, while [6,7] considers RL with JDP and LDP for linear mixture MDPs.

[4] Sayak Ray Chowdhury and Xingyu Zhou. Differentially private regret minimization in episodic markov decision processes.

[5] Dan Qiao and Yu-Xiang Wang. Near-optimal differentially private reinforcement learning.

[6] Chonghua Liao, Jiafan He, and Quanquan Gu. Locally differentially private reinforcement learning for linear mixture markov decision processes.

[7] Xingyu Zhou. Differentially private reinforcement learning with linear function approximation.

5. There are many typos in the paper. Here I list the typos I find.

For the definition of $\Sigma_{D}$, the summation should go from $k=j+1$ instead of $j=k+1$ (Page 2).

The transition $T_h$ should be a mapping from $S\times A$ (Page 3).

For the occupancy measure $\rho_\pi$, where is the dependence on $h$ (Page 3)?

For the probability of the ranking, the summation should go from $j=k$ instead of $j=m$ (Page 4).

In equation (5), the '$\leq$' is missing (Page 8).

For the definition of $\hat{J}(\pi)$, the dependence on $v$ is missing (Page 9).

1. The techniques are quite standard and the results are straightforward
2. Only upper bounds are presented, no lower bound to show the tightness of the bound, especially in terms of the dependence of the privacy parameters.
3. No simulation results, which is somehow weird in ICLR conferences

(1) My major concern is the lack of experimental results. I know that this is a theoretical paper, but I do think experimental results are necessary here, for the following reasons: The purpose of introducing differential privacy to RLHF is to protect the privacy of the human data providers in practice. This paper's theoretical results provide upper bounds on privacy loss and the learning performance loss due to privacy guarantee, under an idealized model ([Zhu et al, 2023]'s model). Humans' ranking behavior may not follow that idealized model, and the RLHF algorithms (for both the reward training and the downstream RL) used in practice are not necessarily the algorithms analyzed in this paper. Whether the theoretical results in this paper can be applied is questionable. Given that the theoretical contribution of this paper is only marginal (see below), I think an empirical contribution (trying DP on real datasets with real RLHF algorithms) is needed here.

(2) My second concern is that the technical contribution of this paper is marginal. The DP technique (adding Gaussian noises to the loss function and the solution in a convex optimization) is a standard technique from [Bassily et al, 2019a] and [Kifer et al, 2012]. The linear reward + BTL/PL model is the same as [Zhu et al, 2023]. The proofs of the theorems are basically a combination of the proofs from these two lines of previous work.

Due to the above two concerns I recommend weak reject.

1. This study lacks novelty, as the algorithm essentially integrates previous Reinforcement Learning with human feedback results [1] with a Gaussian mechanism, which is a widely employed approach for ensuring differential privacy guarantees.

[1] Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.

2. In this work, the authors assert that the sub-optimality gap is near-optimal. However, the absence of a lower bound in this study creates confusion regarding the actual near-optimality of the results.

3. In addressing the general Markov Decision Process (MDP) setting, the author exclusively focuses on estimating the reward function and undertaking pessimistic policy optimization. However, a fundamental challenge in learning an MDP lies in acquiring knowledge about the transition probability function $P$. The determination of the occupancy $\rho$ for a given policy $\pi$ remains unclear. The assumption appears to presume that the transition process is already known, effectively simplifying the MDP problem to a bandit problem and streamlining the learning process. It is imperative to explicitly state all assumptions before making any claims.

4. The proposed algorithm lacks experimental results to substantiate its efficacy.