Differentially Private Deep Model-Based Reinforcement Learning

The main concern is about technical novelty. The definition of Trajectory-level DP is directly adapted from [1]. The first part directly applies DP-FEDAVG, while the second part is about learning from the private model with pessimism. To the best of my knowledge, [1] is based on the same idea of private model + pessimism. The DP guarantee for the private model is from previous results, and the DP guarantee for learning the policy is from standard post-processing. I do not see any technical challenge in the process. It will be better if the authors could discuss about the challenges.

[1] indeed uses the idea of training a private model, but only in the context of tabular and linear MDPs. Moreover, experiments in [1] are limited to simple numerical simulations. In this work, we tackle more complex tasks in the general function approximation setting, especially neural network approximations, which introduces very different challenges that have not yet been addressed in the literature.

In the paper, we thoroughly discuss the challenges of obtaining trajectory-level DP in the context of model ensembles, where a direct use of DP-SGD would be ineffective (Section 4.2). We identified that the idea behind DP-FEDAVG, which had not been used in the context of RL, could be effectively used in our setting, and we carefully ensure that the resulting model is private by distributing the gradient clipping across the $N$ models of the ensemble, before discussing how this may impact model convergence in practice. In the following (section 4.3), we analyze theoretically the impact of private model training on policy optimization. Therefore, neither model training nor privacy guarantees is straightforward (and the DP guarantees for the policy are always obtained from standard post-processing).

In our experiments, we are also the first to address more challenging RL tasks with deep neural approximations and believe that our experimental results are overall much stronger than those from concurrent work in [1]. Indeed, our work achieves comparable privacy-utility trade-offs than [1], but on significantly more complex tasks, which we demonstrate in the following. We also updated the paper to enable a clearer comparison with [1] (see experimental section and Section F of the appendix).

[1] evaluate their algorithms on an episodic synthetic linear MDP with 2 states and 100 actions, and horizon $H=20$. In their results, they do not mention explicitly the privacy budgets $\epsilon$, but instead mention the zero-concentrated differential privacy (z-CDP) parameter $\rho$. For clarity and fair comparison, we convert the z-CDP guarantee into a DP guarantee. For this, we use Proposition 1.3 from [3]: if a mechanism is $\rho$-z-CDP, then for any $\delta > 0$ it is $(\epsilon, \delta)$-DP, with $\epsilon = \rho + 2 \sqrt{\rho \log(1 / \delta)}$. As they evaluate their algorithms for a dataset size up to 1000, we consider two values of $\delta \in \{1/100, 1/1000\}$. The table below shows the results for the various parameters $\rho$ mentioned in Figure 1 from [1].

Therefore, [1] also considers the low privacy regime with $\rho=25$ yielding $\epsilon$ close to 50, which is comparable to our low privacy variant. They indeed consider $\epsilon$ close to 1 with $\rho=0.1$, but the cost is a 2 to 3 times worse utility on this simple MDP. Other proposed configurations are close in privacy budgets to what we consider in our paper. Overall, our work achieves comparable privacy-utility trade-offs than [1], but on significantly more complex tasks.

This review is exactly the same as the one we received for NeurIPS. We have since made many modifications to the paper. In particular, motivated by your feedback, we have developed the theoretical analysis of the different components of our algorithm and provided thorough discussions about the challenges. Also, the private training algorithm has been improved, yielding stronger experimental results. Since this is a new paper, we think a new review is needed.

In the following, we nonetheless carefully address your concerns.

Proposition 4.4 only provides an error bound for estimating the value function of $\hat{\pi}$ which is not standard. Is it possible to derive any results about the sub-optimality gap $V^\star - V^{\hat{\pi}}$?

Thank you for the interesting observation. In this work, we disrupt the model convergence with private training, which directly impacts the valuation gap $\vert V^{\hat{\pi}} - \hat{V}^{\hat{\pi}} \vert$ quantifying the divergence of the value of the learned policy between the true and the estimated MDP. We therefore found particularly interesting to quantify how correctly the learned policy was evaluated under the private model, especially since we use the model as a simulator to optimize the policy without any further interactions with the environment. The underlying idea is that we want to control how privacy degrades the quality of the simulator.

However, we agree that a bound on the sub-optimality gap is worthwhile to quantify the end performance of the policy, and observe that the results from propositions 4.3 and 4.4 actually hold for any policy $\pi$. Therefore, we can use the following to derive a bound on the sub-optimality gap based on the valuation gap:

where the first inequality comes from the fact that $\hat{\pi} \in \text{arg}\max_{\pi \in \Pi} \hat{V}^\pi$, and the second inequality is due to each of the terms being the value gap between the true and the estimated dynamics for a fixed policy ($\pi^\star$ and $\hat{\pi}$, respectively).

We have clarified this point and added the above result in the updated version of the paper.

In the experiments, the privacy protection is very weak (some $\epsilon$ being close to 100). What will happen for more practical choices of $\epsilon$ ? E.g. $\epsilon \approx 1$

We acknowledge your concerns over the level of privacy protection, but these values must be appreciated within the entire context of the paper and literature. Here, we propose a framework with two distinct levels of privacy, where larger values of $\epsilon$ explicitly correspond to "low privacy regime". This allows us to clearly highlight the trade-offs between privacy and performance. As pointed out above, previous work [1] also consider a similar low privacy regime in their experiments.

With smaller $\epsilon$ values corresponding to a higher privacy regime (within the 5 to 20 range), our algorithm retains good performance. We do observe more substantial degradation as $\epsilon$ approaches 1, as shown in Figure 4, but not unlike previous algorithms from [1] which also suffers significant performance degradation for $\epsilon$ close to 1, although on much simpler tasks. Overall, our work achieves comparable privacy-utility trade-offs than [1], but on significantly more complex tasks.

Moreover, we must reconsider the context of achieving privacy in deep RL, and we have thoroughly discussed the implications of our empirical results in Sections 5.2 and 6, as well as in Appendix J. In light of recent literature on achieving practical privacy in deep learning, we argue that these $\epsilon$ values may offer an adequate level of privacy in real-world applications. [2] states $\epsilon \lessapprox 10$ as a realistic and widely used goal in DP deep learning and a "sweet spot" where it is possible to preserve acceptable utility for complex ML models. Additionally, we outline approaches to achieve better trade-offs between privacy and performance.

[1] Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy.

[2] Ponomareva et al. How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy. 2023.

[3] Bun and Steinke, Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds.

Given that we have addressed all the comments from your review in this new version of the paper, and have also included additional remarks to enable a comparison with [1] in the last revision, would you consider increasing your score?

Dear Reviewer,

We wanted to check if you have had the opportunity to consider the new version of our paper and the points discussed above. Should you have any further questions or need clarification on any of the matters you raised, we would be glad to engage in further discussion.

We hope the discussions above have adequately addressed your concerns and answered your questions. Is there any additional information that may lead you to increase your score?

While the motivation for the work is compelling, the experimental design is relatively simple and basic. I would have liked to see experiments that address the unique challenges of applying DP frameworks within the RL domain, yet this paper lacks a broader experimental analysis to underscore the real-world relevance of introducing DP into RL.

As we point out in the introduction and related work sections, RL is already used in personalized services on sensitive data, and has been shown to be vulnerable to privacy attacks, especially membership inference attacks (see [2]). We believe these are sufficient reasons in themselves to introduce DP in RL. We agree that it would be worthwhile to lead experiments based on real-world data, but this seems unrealistic at the moment. Moreover, our experimental design is in line with standard empirical practices in RL, and the empirical evaluation of our method is arguably stronger than existing works in DP RL. We would be grateful if you could elaborate on the experiments you would have liked to see in our paper, in order to better address your concerns and improve both the current study and future research.

[2] Gomrokchi et al. Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning.

In Section 4.3.2, the paper discusses handling model uncertainty under a private setting but appears to apply existing uncertainty-handling techniques from non-private settings directly to the private setting. Could you clarify any special considerations or unique aspects of handling uncertainty in the private setting? Specifically, how might model error and uncertainty differ under a private setting, given its unique constraints? We would appreciate any further insights on this point.

We indeed apply existing uncertainty-handling techniques from the non-private offline MBRL literature (reward penalization with a measure of model uncertainty). We justify this choice based on the theoretical analysis led in Section 4.3.1 and related work discussion from 4.3.2. This choice is further supported by the empirical evaluation of our method.

The theoretical analysis from Section 4.3.1 shows how and why private model training impacts the reliability of our model for evaluating policies, which can lead to misjudging the quality of a policy in the true environment. Therefore, we must carefully consider the implications in terms of how uncertainty is handled in the private setting. The simulation lemma suggests that the quality of the model as a simulator can be impacted by both the model error and the distribution shift; however, in the updated version of the paper, we show that private training only increases model error (scaling with $1/\sqrt{\epsilon}$), and not distribution shift. Interestingly, recent study [3] on uncertainty techniques in offline MBRL showed that the uncertainty measures proposed in the (non-private) literature actually correlate well with model error, more so than with distribution shift. Therefore, we believe that existing measures are appropriate for mitigating the worse reliability of the model under private training, since they will adequately capture the increased model error. We further point out that the impact of private model training also justifies increasing the reward penalty coefficient $\lambda$ to be more conservative, and a moderate increase in $\lambda$ was indeed beneficial in our preliminary experiments.

[3] Lu et al., Revisiting design choices in offline model based reinforcement learning.

Thank you for the positive comments and the constructive feedback. In the following, we carefully address your concerns.

It is unclear how the private-variant experiments control for $\epsilon$. In the theoretical guarantees section, $\epsilon$ is presented as a theoretical bound, yet here it seems to be treated as a tunable hyperparameter, with little explanation connecting these two perspectives.} AND **In Theorem 4.2, a general formula for $\epsilon^{MA**(z, q, T, \delta)$ is presented, but neither the main text nor the appendix provides a complete, detailed expression of this formula. Could you include a full derivation of this formula so we can clearly understand how $\epsilon^{MA}$ is calculated based on inputs like $z, q, T$, and $\delta$?

Thank you for your comments regarding the computation of the privacy budget $\epsilon$ in our experiments, and how it relates to the perspective of $\epsilon$ as a theoretical upper bound. In the following, we would like to clarify these points.

$\epsilon$ indeed represents a theoretical upper bound on the privacy leakage of the training mechanism and could then, at first, be interpreted as a parameter of the problem. However, in practice, $\epsilon$ depends on the scale $\sigma^2$ of the Gaussian noise added to the gradients: a larger perturbation $\sigma^2$ implies a stronger privacy and a smaller value of $\epsilon$. In our private training method, $\sigma^2$ depends directly on the tunable hyperparameters $z, q, T$ and $\delta$. Therefore, $\epsilon$ is not a hyperparameter itself, but is computed a posteriori based on the values of the above hyperparameters.

In our experiments, we fix $q, T$ and $\delta$ and tune the noise multiplier $z$ to make $\epsilon$ vary (the higher $z$, the smaller $\epsilon$), in order to showcase the performance of our algorithm under various levels of privacy, which is a standard evaluation protocol for DP algorithms. This eventually demonstrates that our method is capable of achieving good privacy-performance trade-offs. We point out that we can only compute a theoretical upper bound on the privacy leakage. There is no standardized approach to evaluate $\epsilon$ empirically, and empirical measures would only yield a lower bound which can be far from the true privacy leakage.

There are different ways of computing an $\epsilon$ bound, all using DP composition properties, and we want to compute the tightest possible bound to obtain the best possible privacy-utility trade-off for our algorithm. In our work, given hyperparameters $z, q, T, \delta$, $\epsilon^{MA}(z, q, T, \delta)$ is the $\epsilon$ value computed by the moments accountant from [1], as we state in Theorem 4.2. Unfortunately, there is no simple explicit formula for it: at each training step $t$, denoting $\mathcal{M}_t$ the Gaussian mechanism outputting the noisy gradient, the moments accountant computes a bound on the log moments of the privacy loss random variable $\log \frac{\mathcal{M}_t(d) = o}{\mathcal{M}_t(d) = o}$, using numerical integration ($d$ and $d^\prime$ are neighboring datasets, and $o$ is a given output). The procedure is detailed in Section 3.2 from [1], and we use the implementation from the DP accounting tools from Google’s Differential Privacy library which provides an improved version of the moments accountant based on the Rényi divergence. We have added an additional section (F.6) in the appendix to present the moments accountant more in details in the revised version of our paper.

[1] Deep Learning with Differential Privacy, Abadi et al., 2016.

Thank you for the positive feedback. In the following, we are pleased to provide you further insights about the points you raised.

There exist several typos. For example, in line 274, 'it does entirely remove...' I assume it should be 'it does not entirely remove...' because increasing will degrade the model performance.

Thank you for pointing this out, it is indeed 'it does not entirely remove'. We apologize for the typos and did our best to correct them in the revised version of the paper.

The major weakness is that there is a lack of a more explicit discussion on each term in the theoretical results, such as $\epsilon$ and $\epsilon^{DP}$. I would be curious about if these terms depend on privacy parameter or $N$, if so, what should be the approximate dependency.

We thank you for this feedback that made us rethink the role of the distribution shift in our results. After careful consideration, we ultimately determined that the distribution shift was not affected by the private model training. This is because the distribution shift term of the simulation lemma comes from a bound on $\vert V^\pi - V^{\pi^B}\vert$, the valuation gap between the policy $\pi$ and the data collection policy $\pi^B$, which does not depend on the model. Therefore, private training only impacts the model error term in proposition 4.3 and 4.4. We therefore have a better idea on how private training impacts the reliability of the model as a simulator in private offline MBRL. In the updated version of the paper, we have made modifications to Section 4.3 to account for these new findings.

It seems that the number of ensembles plays a vital role in the results. This work has already reduce the dependency to $\sqrt{N}$ in the sensitivity. I am just curious is it possible to further reduce it in training ensemble models.

Thank you for the interesting question. The ensemble size indeed plays an important role in the trade-off between privacy and utility, and you are right to highlight the importance of controlling this parameter. When distributing the clipping norm across the $N$ models, the $\sqrt{N}$ factor directly comes from the fact the Gaussian mechanism uses the L2-sensitivity, suggesting that we could not further reduce the dependence in $N$ while using Gaussian noise for privacy. While other mechanisms could be considered, the Gaussian mechanism is the preferred choice for high-dimensional queries, especially because it can use the L2-sensitivity (a Laplace noise, on the other hand would linear scaling in the ensemble dimensions as it uses the L1-sensitivity) and allows tight computations of the privacy bound (e.g., it enables the computations of the moments accountant). Therefore, we believe that a $\sqrt{N}$ dependence is the best we can get when using model ensembles.

We hope these discussions have appropriately answered your questions. Is there any additional information that could lead you to increase your score?

Dear Reviewer,

Have you had the opportunity to review the discussions above and the revisions to the paper? If you have any further questions or concerns regarding the points you raised, we would be happy to discuss them.

Thank you for your feedback.

First, we would like to question the relevance of the argument about the "excitement" that a paper might generate. We do not understand why such an argument is present in a review, since the quality of the paper is normally assessed based on the rigor and relevance of the model and techniques developed. Here, we address a problem with many practical implications that has not currently been addressed in the literature: the possibility of doing differentially private deep RL based on offline data. We 1) explain why this problem is important and 2) fill in the literature by providing a practical solution achieving good performance on a standard (non DP) benchmark. Once again, we are sorry if the reviewer finds the reading unexciting but we would like this work to be judged by the standard criteria: impact and scientific rigor.

It has been shown that RL is not immune to privacy attacks, especially membership inference attacks where malicious adversaries could recover entire trajectories of the training dataset (see, e.g., [1]), and which could be prevented with differential privacy. Currently, the differentially private RL literature is limited to tabular and linear MDPs, and proposed methods do not scale to standard RL benchmark tasks. Given the well-founded concerns about data leakage in the case of RL applied to real-world situations, the literature on DP RL cannot be limited to theoretical studies (although these are useful) but must progress towards the deployment of algorithms in complex environments. To achieve this, deep RL solutions are needed. In this work, we are the first to address differentially private RL in the infinite-horizon discounted setting, without making limiting assumptions on the MDP. Furthermore, this work is the first in this area to deal with general function approximation, and to introduce differentially private deep RL algorithms evaluated on standard tasks beyond numerical simulations. Therefore, this work fills an important gap in the current DP RL literature and is impactful as a first step towards deploying private RL agents in complex environments. We thus believe that this work is highly relevant to the community.

After considering these discussions and following the revisions made to our paper, is there any additional information that may make you increase your score?

Regarding the limitations of our experimental results and the lack of baseline, we would like to provide several clarifications.

Contrary to the above claim, we do perform baseline comparisons. As can be read in Section 5, we compare our algorithms to MOPO, a non-private baseline that is close to our method. Comparing a private method against a close non-private baseline is standard practice in the differentially private literature (see for instance [2], a concurrent work using a similar evaluation protocol).

Moreover, to the best of our knowledge, the only work proposing DP algorithms in the offline RL setting is [2]. This work is limited to tabular and linear episodic MDPs, is only evaluated on a small synthetic environment (two state, discrete action space and fixed horizon $H=20$), and cannot scale to the problems we consider in our work, preventing direct comparison on the same benchmark.

However, we point out that are our experimental results are overall much stronger than previous work in differentially private RL, where empirical evaluation is limited to numerical simulations in small environments (if there is any). By contrast, our work is the first to address standard control benchmarks with continuous state and action spaces using differentially private RL.

Especially, our empirical results are stronger than [2]. Our method is indeed capable of achieving similar privacy-utility trade-offs than methods from [2], but on much more complex environments. We updated the paper by providing a clearer comparison with [2] (see experimental sections and Section F of the appendix), and provide details below.

[2] do not mention explicitly the privacy budgets $\epsilon$, but instead mention the zero-concentrated differential privacy (z-CDP) parameter $\rho$. For clarity and fair comparison, we convert the z-CDP guarantee into a DP guarantee, using Proposition 1.3 from [3]: if a mechanism is $\rho$-z-CDP, then for any $\delta > 0$ it is $(\epsilon, \delta)$-DP, with $\epsilon = \rho + 2 \sqrt{\rho \log(1 / \delta)}$. As they evaluate their algorithms for a dataset size up to 1000, we consider two values of $\delta \in \{1/100, 1/1000\}$. The table below shows the results for the various parameters $\rho$ mentioned in Figure 1 from [1].

Therefore, [1] also considers the low privacy regime with $\rho=25$ yielding $\epsilon$ close to 50, which is comparable to our low privacy variant. They indeed consider $\epsilon$ close to 1 with $\rho=0.1$, but the cost is a 2 to 3 times worse utility on this simple MDP. Other configurations proposed are closed in privacy budgets to what we consider in our paper. Overall, our work achieves comparable privacy-utility trade-offs than [1], but on significantly more complex tasks.

[1] Gomrokchi et al. Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning.

[2] Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy.

[3] Bun and Steinke, Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds.

Dear Reviewer,

We wanted to follow up and see if you have had a chance to review the revisions and the discussions mentioned above. If you have any additional questions or concerns regarding the issues you raised, we would be glad to address them.

Thank you for providing a rebuttal. By saying "not excited" I mean 1) there's a lack of novelty. This work combines model learning, DP, SAC in the offline RL setting; 2) I do not see real applications that we are incentives to use offline RL facing a privacy setting. I apologize if this term caused some confusion. I hope this clarifies that we are on the same set of scientific standards.

By the same argument, I would expect the experiment to work on much more complicated tasks that showcase your method outperforms other private learning methods (within and beyond offline RL). This is why I evaluate the experiments as limited. I understand that the authors would like to compare the manuscript with some other works published at a major ML conference. Unfortunately I did not use this method to evaluate the manuscript.

By these reasons the rebuttal does not change my evaluation, at this moment.

Thank you for the further discussion. I basically agree with the arguments you have provided. I believe such arguments alone do not justify sufficient novelty and sufficient relevance, though. Therefore my evaluation remains the same. I apologize for not being able to increase the score.

On the challenges and technical contributions of this work

• While the notion of trajectory-level privacy is well-motivated, this is not straightforward to achieve in deep RL, since standard approaches like \textsc{DP-SGD} deal with privacy at the example-level. We identify that trajectory-level privacy requires partitioning the data by trajectories before computing and clipping trajectory-level updates. We build on prior work that tackles user-level privacy in the fields of federated learning and NLP. We adapt these ideas to our setting and propose a trajectory-level DP training method based on a Poisson sampling scheme to learn the dynamics model.
• The standard approach to handle model uncertainty is using bootstrap ensembles to compute uncertainty estimates. This raises additional challenges in the private setting, where the size of the ensemble impacts the privacy budget. We mitigate this issue by distributing the clipping factor across all models.
• We provide a theoretical analysis of how private training influences model reliability and its impact on the policy optimization process.
• We prove formal theoretical guarantees for the end policy by restricting further interactions with the system during policy optimization.
• We identify that current offline RL benchmarks are unsuitable for studying privacy and advocate for the use of larger datasets.
• We obtain meaningful experimental results on previously unaddressed benchmarks, paving the way for future work in private deep RL.
• We release our code to ensure reproducibility and encourage the development of private deep RL.

During this rebuttal period, we have carefully addressed the concerns and questions of all the reviewers, and these have all been taken into account in the revised version of the paper. In particular, we did our best to further highlight the motivation and contributions of our work, addressing the concerns of reviewers bCjw, h3Xi, and wk8A. We also propose promising research directions to scale towards higher-dimensional problems, following discussions with reviewers bCjw and h3Xi. We provided a detailed comparison with the work of Qiao & Wang (2023a), the closest to our work, to answer concerns from reviewers bCjw and wk8A. We are also more explicit on the role of $\epsilon$ and the computations of the moments accountant, following a question from reviewer pAHS. We thank again the reviewers for their constructive feedback, which allowed us to improve our work in these directions.