Deep Actor-Critics with Tight Risk Certificates

Thanks for your in-depth review of our submission.

Further evaluations

We provide additional results on PPO and SAC on MuJoCo in our answer to MQz1. These results support that our findings for REDQ generalize to other RL models. Our recursive PAC-Bayesian bounds (R-I T=2; R-I T=6) provide tighter risk certificates than non-recursive bounds (NR-NI; NR-I), and the tightness increases with the recursive depth. See also our answer to em5X for further results as the recursive depth $T$ increases.

To provide further evidence that they also generalize across benchmarks, we ran the same training and evaluation pipeline as reported in the paper on three environments each of the Meta-world (Yu et al., 2020) and DMC (Tassa et al., 2018) benchmarks. Our findings show the same performance pattern. The tables below show the tightness rank of each method for each environment averaged over seeds and policies. We will add the raw results, i.e., before averaging, to the camera-ready version of the paper.

MetaWorld -- REDQ (avg rank)

DMControl -- REDQ (avg rank)

Clarify data set sizes

During training the model is trained for 300k steps, i.e., it collects 300k transition tuples. The size of the evaluation data depends on the environments and the policy level. For example, in MuJoCo every episode has an upper limit of 1k transitions, but for more complicated environments, such as Hopper and Humanoid, they tend to end in failure before that, so collecting data for 100 episodes results in ~27k and ~47k respectively.

In Figure 4 (a) we evaluate the influence of the evaluation data size. Note that while the overall spread decreases as the percentage of training data increases, as expected, the median tightness of our recursive bounds after 25 episodes is already almost identical to collecting 100 episodes worth of transitions.

Literature survey lacks two references

Fard & Pineau (2010) were the first to introduce a PAC-Bayesian bound in reinforcement learning, bounding the value function. However, their guarantees were limited to small discrete domains. Their follow-up work, Fard et al. (2012), extends this to value functions of fixed policies in a continuous domain. However, their bound assumes access to the exact environment model and can only handle a single-episode setting.

We'll add this overview to the last paragraph in Appendix A, where we already cite Fard et al. (2012).

---

Fard & Pineau (2010), PAC-Bayesian Model Selection for Reinforcement Learning (NeurIPS)
Fard et al. (2012), PAC-Bayesian Policy Evaluation for Reinforcement Learning (UAI)
Tassa et al. (2018), Deepmind control suite (arXiv)
Yu et al. (2020), Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. (CoRL)

Thank you for your thorough review of our submission.

Focus on the Background discussion and additional results

We will move Section 2.2.1 to the appendix, while keeping Section 2.2.2 in the main text, as it provides the essential building block for the recursive bound, and place the focus on the current Section 2.2.3. We will use the resulting half-page of space to include figures and discussions of the additional experiments summarized in the answers to reviewers em5X, MQz1, and ZwF5. These experiments show that the results generalize to different models (PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018)) and different benchmarks (Meta-world (Yu et al., 2020) and DMC (Tassa et al., 2018)).

---

Haarnoja et al. (2018), Soft actor-critic algorithms and applications (ICML)
Schulman et al. (2017), Proximal policy optimization algorithms (arXiv)
Tassa et al. (2018), Deepmind control suite (arXiv)
Yu et al. (2020), Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. (CoRL)

We appreciate your detailed review of our submission.

PAC4SAC, PBAC, and PAC

PAC4SAC (Tasdighi et al., 2024) and PBAC (Tasdighi et al., 2025) are designed exclusively for policy optimization, using PAC-Bayesian principles as algorithmic guidelines rather than for reliable risk certificate generation, which is our objective. Both methods employ approximately calculated PAC-Bayesian bounds without quantifying the resulting approximation errors. While such approximations are acceptable in policy search contexts where learning performance is the main concern, they preclude making rigorous analytical statements about test-time performance and therefore cannot support risk certification.

Specifically, PAC4SAC's bound includes an expected variance term (see Theorem 1 in Tasdighi et al., 2024) that requires perfect knowledge of transition dynamics, making it impractical for real-world applications. PBAC approximates the KL divergence computation (see Theorem 2 in Tasdighi et al., 2025), which is intractable for most nonlinear function spaces. Since these approximation errors remain uncharacterized, the resulting bounds cannot provide the reliability guarantees necessary for risk certificates.

In contrast, our method is designed specifically for post-hoc risk certification, where bound tightness and theoretical guarantees are paramount. We will clarify this distinction in the first paragraph of Section 2.2 and Appendix A, where we cite these works.

Regarding the PAC comment: Could the reviewer specify which particular model they are requesting? If they seek a generic non-recursive bound, we already include two non-recursive PAC-Bayesian bounds (NR-NI, NR-I) as baselines in our evaluation.

Additional Environments and Baselines

Upon your suggestion, we evaluated our risk certificate generation method on additional environments (MetaWorld) and baseline models (PPO, SAC). Due to runtime constraints, we restricted ourselves to three out of the original five environments. Below we report the rank for each task and method averaged over seeds and policies. The results support our findings with REDQ. More advanced bounds provide better risk certificates, with the most recursive version ($T=6$) dominating the others.

See also results for REDQ on the Meta-world (Yu et al., 2020) and DMC (Tassa et al., 2018) benchmark environments in our answer to ZwF5. Our findings generalize across both environments and algorithms. We provide the full results and corresponding summary figures in the camera-ready version of the paper.

MuJoCo -- PPO (avg rank)

MuJoCo -- SAC (avg rank)

Empirical return distribution and CVaR-style loss

Distributional RL (Bellemare et al., 2023) aims to model the complete return distribution and approximations stemming from projection and discretization errors, the uncertainties of which it cannot quantify completely. In contrast, our method accounts for all sources of uncertainty such as function approximation and finite sample effects, deriving probabilistic bounds with specified confidence levels that enable rigorous risk certificates. This distinction is crucial for safety-critical domains that require formal risk quantification with known confidence levels over learned distributional approximations without uncertainty quantification.

Our results and discussion focus on risk certificates on the expected error, $\mathbb E[L(h)]$. However, the pipeline we introduce in Section 3 is agnostic to the chosen type of loss up to technical constraints, e.g., that it needs to be bounded. If $\text{CVaR}_\alpha(\pi)$ is the conditional value at risk for a policy $\pi$ and given confidence level $\alpha$, our approach can be directly adapted to provide a bound given an empirical estimate, $\text{CVaR}_\alpha(\hat \pi)$ (corresponding to $\hat L(h)$ in the terminology of our paper). The resulting risk certificate gives a tight bound on this risk-sensitive objective. A distributional approach would lack such formal guarantees.

Deeper recursion and runtime

Please see our runtime and recursion results in the answer to reviewer em5X. While increasing $T$ helps improve the tightness of the risk certificate, the improvements quickly become less pronounced. As the risk certificates are computed after training the model, which is the costly part, they add a negligible one-time post-training cost.
See also the last paragraph in Section 4.1 for further runtime results.

---

Bellemare et al. (2023), Distributional Reinforcement Learning (MIT Press)
Haarnoja et al. (2018), Soft actor-critic algorithms and applications (ICML)
Schulman et al. (2017), Proximal policy optimization algorithms (arXiv)
Tasdighi et al. (2024), PAC-Bayesian soft actor-critic learning (AABI)
Tasdighi et al. (2025), Deep exploration with PAC-Bayes (arXiv)
Tassa et al. (2018), Deepmind control suite (arXiv)
Yu et al. (2020), Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. (CoRL)

Thank you for taking the time to thoroughly review our submission.

Changes to the visual representation

• Figure 2: We will include a legend highlighting the color coding that is currently in the caption. We will also provide enlarged versions of these scatter plots in the appendix.
• Box plots: These reflect the relative performance differences. We will rotate Figure 5 by 90 degrees and increase its size. We will also include a larger copy of Figure 3 with the same modification in the appendix.

Runtime & Computational complexity

The training time of the algorithm and the runtime of the learned policy when it is deployed are independent of the risk certificate computation.
As such, there is no additional cost. The certificate computation itself is cheap and takes about a minute on a single GPU.

For the expert-level policy on humanoid, the runtime cost in seconds averaged over three seeds as $T$ increases is roughly linear.

See also the last paragraph of Section 4.1 (l276-282) for further runtime results.

Data efficiency

See Figure 4 (a) for an overview of the effect of validation data size on the bound tightness. Even with half the validation data the results are already close to the full validation data, with an almost identical median tightness given with as little as 25%.

Deeper recursion

The following tables provide results on increasing the recursive depth $T$ on MuJoCo Humanoid with REDQ and MetaWorld reach, each averaged over three seeds.

While increasing the recursive depth $T$ tends to provide ever tighter risk certificates, the improvements quickly become negligible.

Humanoid -- REDQ

Starter policy

Intermediate policy

Expert policy

MetaWorld Reach

Starter policy

Intermediate policy

Expert policy

Sufficiency of policy range evaluation

See the reward curves of Chen et al. (2021). Figure 1 therein shows the performance of a learned policy over a range of 300k interaction steps on four of our five reported MuJoCo environments. The policies show significant performance differences after 100k, 200k, and 300k steps, respectively, and tend to converge toward the end of training. We'll mention this in the camera-ready version in line 254.

---

Chen et al. (2021) Randomized ensembled double Q-learning: Learning fast without a model (ICLR)