Online Optimization for Offline Safe Reinforcement Learning

Thanks for your constructive feedback and review comments. We address them below.

Re: “The proposed framework and theory has been studied before in [1]”

Thanks for pointing out this paper. We will discuss [1] and its connections in the final paper.

Lagrangian formulation for offline safe RL is quite standard that both our work and [1] share. [1] and our work also share the general idea that we can use online optimization algorithms to tune the Lagrangian multiplier. The critical difference is in the design of the algorithmic frameworks: our framework requires one call to an offline RL oracle per loop and does not rely on any OPE oracle. On the other hand, [1] requires two calls to an offline RL oracle and four calls to an OPE oracle per loop. Since OPE procedures are known to be typically unstable, we believe that our O3SRL framework has unique advantages.

Re: “convergence guarantees with conditions for constraint satisfaction after deployment…main theoretical argument that is used in the paper has already been shown by [1]”

The solutions that our algorithms converge to (at the rate specified by Theorem 1 and 2) satisfy the cost constraint for deployment, by the very definition of the Langragian formulation in Eq. 3. The nature of our theoretical argument is completely different from those in [1]: our framework works for a general offline RL oracle, allowing us to integrate any state-of-the-art off the shelf offline RL algorithm and doesn’t need OPE procedures, whereas [1] only uses FQI and relies on OPE procedures, which are known to be unstable.

Re: “L.128: the objective and constraints are generally linear only when there is no (non-linear) function approximation involved”

The Lagrangian formulation in Section 4.1 has nothing to do with function approximation. When the policy class $\Pi$ is expressive enough, Slater’s condition easily holds. The linearity we exploit is the linearity of the objective with respect to λ.

Re: “L. 129: Slater's condition is typically assumed, i.e., Equation 1 not always satisfies it.”

When the policy class $\Pi$ is expressive enough, Slater’s condition easily holds.

Re: “Remark 1, L. 162-164, “notions of coverage depend very mildly on the reward functions, if at all”. I don't think that this statement is true. What about sparse reward tasks?”

We refer the reviewer to our explanation in Lines 164-167 where we say “most notions of coverage”, not “all notions of coverage”. Even in sparse reward tasks, we still believe the statement applies – though the extremity of sparse rewards would place much emphasis on the influence of a particular (sparse) reward function to the data coverage.

Re: “While the use of the bandit algorithm is novel, and seem to yield strong results, the practicality of it is somewhat limited.”

We use the multi-arm bandit approach because of the practicality it offers by avoiding the reliance on off-policy evaluation procedures. We demonstrate its practicality through experiments on a standard suite of benchmarks.

Note that the bandit algorithm is used only during offline training. The final output is a standard policy that maps states to actions and can be deployed directly. No bandits or additional machinery are involved at inference time. The bandit component is used solely during training to embed the desired behavior into the final policy.

Re: “Suggestions for presentation and experimental results”

We will adjust the focus to better highlight the empirical results and the significance of low-budget constraints. Initially, space limitations influenced our choice of presentation. We also note that, per NeurIPS recent guidelines, we are not allowed to share external visualizations or link to supplemental PDFs during the review process. However, we agree that plots can improve clarity.

Re: “Using a bandit to pick the Lagrange multiplier will require practitioners to implement an additional (non-trivial) subroutine”

The subroutine of multi-arm bandit (MAB) is standard. MAB methods are quite commonly used in practice for adaptive decision-making applications (e.g., A/B testing). The simplicity is apparent in the pseudo-code we provide, which shows this is a very small wrapper around an offline RL algorithm.

Note that the bandit component is used only during training to guide the learning process. After training, O3SRL produces a single policy that maps states to actions and can be deployed directly, with no bandits or additional components involved at inference time.

Re: “The algorithm returns a “mixing” policy that combines several policies together and draw them randomly. This is usually a limitation in practice, because such mixing policies are not easy to implement, and might execute severely unsafe actions.”

The mixture policy is only used for the theoretical analysis. In practice, we find that a single policy from the last training iteration is sufficient for strong results (see Lines 231–234 for the concrete approach used in evaluation). The final output is a standard policy that embeds the effect of the mixture during training. Although we use a mixture of shaped rewards based on sampled λ values during training, the resulting policy maps states to actions directly and requires no runtime mixing or sampling.

Re: “Using only three seeds for evaluation should be improved.”

We agree that more seeds can help provide a more reliable estimate of performance. To address this, we include results from three additional seeds (40, 50, and 60) on top of the original 10, 20, and 30. Columns 1 and 2 report the rewards and costs (mean ± standard deviation) over the original 3 seeds. Columns 3 and 4 show the same metrics computed over all 6 seeds.

We note that using three seeds with 20 evaluation episodes per seed is the standard practice in OSRL, as followed in prior works. Nonetheless, we appreciate the suggestion and have included the extended results for completeness.

Comparison over 3 Seeds vs. 6 Seeds

The results across 6 seeds confirm the stability of O3SRL, with consistent rewards and safe cost levels across tasks. Variance remains low, and performance trends observed with 3 seeds largely hold, indicating robust and reliable behavior.

Thank you for your response.

The Lagrangian formulation in Section 4.1 has nothing to do with function approximation. When the policy class $\Pi$ is expressive enough, Slater’s condition easily holds. The linearity we exploit is the linearity of the objective with respect to λ.

I believe this stands in contradiction to line 128 in the paper: “Note that the objective and the constraint in Equation (1) are linear, thus convex and differentiable.”

Equation (1)

$$\max_{D \in \Delta \Pi} \mathbb{E} [V_r^\pi] \text{ subject to } \mathbb{E} [V_c^\pi] \leq \kappa$$

Equation 1 is linear (thus convex and differentiable) w.r.t. the state-occupancy measure of the policy. Therefore:

1. I cannot find the relation to $\lambda$ you are referring to. Is it a bug in the paper and you were actually referring to the Lagrangian form in Equation 2? There indeed the relation to $\lambda$ is linear.
2. Otherwise, when the policy is learned via function approximation, this linear relation typically breaks.
3. In addition, when function approximation is involved, strong duality does not necessarily hold [1].

When the policy class $\Pi$ is expressive enough, Slater’s condition easily holds. (answering “L. 129: Slater's condition is typically assumed, i.e., Equation 1 not always satisfies it.”)

Precisely, when it is expressive enough. You implicitly assume that your policy class is expressive enough, which is not always the case. Being more explicit and clear about this point would make the paper more rigorous.

[1] Paternain, Santiago, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. "Constrained reinforcement learning has zero duality gap." Advances in Neural Information Processing Systems 32 (2019).

Thanks for your constructive feedback and review comments. We address them below.

Re: “The number and variety of the tasks evaluated on can still be improved. Missing related work (and probably a baseline).”

We have extended our evaluation by running O3SRL on additional environments from the Gymnasium Safety Suite, following the FISOR setup with a tight cost limit of 10. Results are summarized in the table below. (R) is the normalized reward and (C) is the normlaized cost (<1 means safe) (costs are not normalized in the last table).

Regarding TREBI [2], we will add it to the related work section. We attempted to include it as a baseline, but faced persistent issues with installing its dependencies and setting up the environment correctly. Unlike more recent baselines such as CAPS and CCAC [3], TREBI predates the standardized OSRL baselines framework, which makes integration and reproducibility more difficult.

Additionally, both the FISOR and CCAC papers report that TREBI underperforms compared to their approaches. We opted to include CCAC as an additional baseline in our experiments. We also already compare against CAPS, a recent method with strong performance. Both CCAC and CAPS are cost-limit agnostic, similar to TREBI, and can adapt across different budget levels.

[3] Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning. https://openreview.net/pdf?id=nrRkAAAufl

O3SRL satisfies the cost constraint on 6 out of 7 additional tasks, demonstrating strong generalization across environments. The next best method, CAPS, is safe on 5 out of 7. In terms of performance, O3SRL achieves the highest reward on 3 tasks, the most among all methods. Additionally, it remains competitive with the best-performing method on SwimmerVelocity (0.05 vs. 0.06) and Walker2dVelocity (0.77 vs. 0.78), while maintaining safety.

The table below compares O3SRL and CCAC on the Bullet Gym environments, as reported in Table 2 of the main results.

O3SRL satisfies the cost constraint on 8 out of 8 tasks, while CCAC is safe on only 4. Additionally, O3SRL achieves the highest reward on 6 out of the 8 tasks.

Re: “Line 286 concludes that higher values of K leads to diminishing returns, but from Table 1 the trend is the opposite.”

Thank you for pointing this out. The sentence was not phrased well. Our intended meaning was that increasing K beyond a certain point (e.g., from 5 to 10) provides relatively smaller gains compared to the improvement seen when increasing from K=2 to K=5.

Re: “Why does discretizing the Lagrange multiplier help stabilize learning?”

If we optimize for continuous Lagrange multiplier, we generally need Off-Policy Evaluation (OPE) procedures on top of the offline RL oracle. OPE is known to be typically unstable especially when the target policy, found as the return of the offline RL oracle, keeps changing during the iterative process. By discretizing $\lambda$, we completely avoid the need for OPE.

Re: “Could you provide a more intuitive explanation of how the EXP3 algorithm updates the distribution over arms?”

EXP3 maintains a distribution over arms, where the mass is biased toward the arms that have been observed to achieve higher rewards in the past rounds. As per your suggestion, we will provide more intuition in the main paper and detailed description in the Appendix.

Re: “Table 1: Why does increasing the granularity reduce the safety of the agent? How can interpret this empirical result?”

In our setup, safety is defined as keeping the normalized cost below 1, i.e., staying within the given cost budget. From this perspective, all policies that respect the budget are considered safe. Increasing granularity helps the agent discover better policies within the safe region. The goal is not to minimize cost to zero, but rather to maximize reward while staying under the cost limit. An ideal policy would use the available budget effectively to collect as much reward as possible. When the budget is tight (a limit of 5 means the agent is allowed only five mistakes over an entire trajectory), the agent may behave conservatively and avoid costs entirely, even if using some of the budget could improve reward.

Re: “Table 3: questions on results”

All reported costs are normalized (i.e., divided by the cost limit). For example, a normalized cost of 0.5 under a cost limit of 20 corresponds to an actual accumulated cost of 10. The goal is to stay below 1.0, and the percentage of budget used is not inherently important as long as this condition is met.

Given more budget, the agent accumulates more cost in absolute terms, which might not be fully reflected in the normalized cost values. In the table below, we report the non normalized costs. The agent typically leaves a safety margin and avoids pushing right up to the limit. This is intentional. Since offline RL uses fixed data and faces distribution shift, the agent must be pessimistic to avoid violating the constraint during evaluation. A slightly conservative policy is preferable to one that risks infeasibility.

Regarding Table 2: yes, the normalized costs are mostly close to 0, but this is because those results use a tight budget of 5, and the agent learns to avoid costs as much as possible to remain safe across episodes (200–500 steps). In that regime, even a few binary cost events can be enough to exceed the limit. We include additional experiments with a larger cost limit of 80 below, where the agent accumulates more costs while still remaining safe. This supports the observation that O3SRL adapts based on the constraint and trades off conservatism appropriately.

O3SRL Performance with Unnormalized Costs

Thanks for your constructive feedback and review comments. We address them below.

Re: “clarifications on theoretical part: offline RL oracle assumption and data coverage”

Remark 1 (Lines 158-170) clarifies that offline RL oracle absorbs the data coverage condition.

Pessimism is an algorithmic approach that an offline RL oracle might employ to have a good estimation error rate.

The assumption of offline RL oracle is to abstract away many state-of-the-art off the shelf offline RL algorithms. Any offline RL algorithm that returns a near-optimal policy, either in theory or practice, belongs to the oracle by definition. In the rich offline RL literature for both theory and practice, many choices for good offline RL algorithms exist (e.g., TD3BC and IQL). More broadly the idea of assuming oracles in theoretical work is widely used as it allows for the theoretical problems to be decomposed into pieces where independent progress can be made. In summary, we provide a reduction approach to leverage powerful offline RL algorithms to effectively solve offline safe RL problems.

Re: “unclear writing: approximate value function and parts of Algorithm 2”

The approximate value function (Line 3, Algorithm 2) is a function of $\lambda_t$ and the offline data. So the oracle for Lagrange multiplier in Definition 3 does not explicitly rely on the approximate value function.

Re: “alternative approach: offline RL with lambda as a hyper-parameter”

Note that in our framework the term "online" refers to the use of online/streaming optimization algorithms (e.g. bandits) for optimizing parameters, here Lagrange multipliers. This is different from how "online" is used in the context of online RL, where algorithms are allowed to run the policies online in the environment. Our work is within offline RL.

We conducted the experiment that the reviewer suggested, where we fix the value of $\lambda$ throughout training and evaluate the resulting policy at the end. This setup violates the spirit of offline RL, as it assumes we can execute policies in the environment to collect validation data for selection of $\lambda$. We are aware that some works tune key hyperparameters this way, but it remains inconsistent with the offline RL setting. Moreover, we cannot exhaustively search the entire $\Lambda$ space.

In our method (O3SRL), we define a discrete set of $\lambda$ values: $\lambda \in lambdas=[0, 0.4061, 0.9081, 2.0306, 5.0]$. Each value corresponds to an "arm". During training, we sample a $\lambda$ from a learned distribution over these arms and use it to shape the reward. This distribution is updated at each batch following EXP3 logic, allowing the policy to adaptively mix behaviors instead of committing to a single $\lambda$.

For the experiment, we train five separate policies, each with a fixed $\lambda=lambdas[i]$ (arms 0 through 4). No updating or mixture is used in these runs. The table below compares their final performance with O3SRL on the DroneCircle environment. Results are averaged over 3 seeds. We report normalized rewards and normalized costs (i.e., total cost divided by the budget) for three budget levels: 5, 20, and 40.

DroneCircle Results

Takeaways:

• Arm 0 (λ=0) obviously ignores cost and violates all constraints.
• Arms 1 and 2 fail to enforce the constraints reliably since the penalty is low.
• Arms 3 and 4 stay within budget but sacrifice reward, with Arm 4 being most conservative.
• O3SRL outperforms all fixed-$\lambda$ policies by dynamically adjusting $\lambda$, achieving better reward and cost trade-offs.

Re: “Is data distribution assumed to have good coverage? What assumptions will the offline RL oracle need to guarantee the requirements in Definition 2”

Yes, we assume good data coverage. We already discussed this in Remark 1 (Lines 158-170). The assumptions needed by offline RL oracle to guarantee requirements in Def 2 include the ones we have used in theory of offline RL: good data coverage and good representation assumptions. The offline RL algorithms might use the pessimism principle to implement offline RL oracle with a good estimation error rate. All these are abstracted away and inherently embedded in the offline RL oracle.

Re: “Why is the value function need to be estimated? There is no mention of this in Definition 3. Also, it is stated in line 200 that "approximation of the value function of the optimal policy" but then Equation 6 shows it is estimation of the policy $\pi$”

We employ value-based methods and thus, the value function needs to be estimated. One can also opt for model-based methods to get rid of value function estimation, at the cost of using more complex models.

Re: "What is a "stochastic approximation"?"

It was a typo. It should be “the stochastic approximation of the value function of the current policy (distribution)”.

Re: “Related work: Provable Safe Reinforcement Learning with Binary Feedback, Bennett et al. AISTATS 2023”

This paper is considering a very different framework with very different assumption. First, the paper is in the online RL setting whereas we are in the offline setting. Second, this paper assumes an offline oracle that provides binary feedback on the safety of state-action pairs. In our setting, such a safety signal is not available, instead there is a cost associated with each state-action pair. Third, the nature of our offline RL oracle is also fundamentally different from the oracle in this paper: their oracle provides safety signal, whereas our oracle abstracts away standard offline RL algorithms.

Minor

Thank you for pointing this out. We will fix those in the final paper.