Semi-gradient DICE for Offline Constrained Reinforcement Learning

We appreciate the reviewer’s comment and provide empirical evidence in Section 4 to better substantiate our work.

Q1. Empirical evidence on the claim from Section 4

We demonstrate that OptiDICE can yield a state $s$ where $d_{\pi^*}(s,a)=0\;\forall a$. The figure provided in https://imgur.com/a/7zh2zlg is based on the Four Rooms domain from Figure 1 of OptiDICE [1], using 1000 trajectories collected under a behavior policy with 0.7 optimality. In this setup, red arrows indicate the optimal policy, and the heatmap shows the state visitation frequency under the optimal policy, $\sum_{a}d_{\pi^*}(s,a)$. While all states are visited by the behavior policy, some states have no arrows, indicating they are not visited by the optimal policy ($\sum_{a}d_{\pi^*}(s,a)\approx 0$). Consequently, the optimal policy in those states cannot be recovered by computing $d_{\pi^*}(s,a)/\sum_{a}d_{\pi^*}(s,a)$. We will include this demonstration in the final version.

[1] Optidice: Offline policy optimization via stationary distribution correction estimation. ICML, 2021.

We appreciate the reviewer’s acknowledgment of our contribution: extending semi-gradient DICE to constrained offline RL, grounded in theoretical analysis of its optimal solution and stationary distribution correction approach to address the issue of Bellman flow constraint violation.

We thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.

Q1. Violation of Bellman flow (BF) constraint and the originality of Proposition 4.1

We respectfully disagree with the reviewer’s claim that replacing the term $(1-\gamma)p_0$​ with the dataset distribution $d_D(s)$ is a well-known violation of the BF constraint. To clarify, there are two distinct scenarios for this replacement, under the full-gradient DICE method:

• Direct substitution: While replacing $(1-\gamma)p_0$ with $d_D$ does violate the BF constraint, the resulting d becomes a scaled (due to removal of $1-\gamma$ factor) stationary distribution under a modified MDP with initial state distribution $d_D$ (due to $p_0\to d_D$). The constant scaling can be easily corrected during the policy extraction.
• Assumption-based substitution (as used in SemiDICE, appx. B, Eq. (31-32)): Here, we assume $d_D$ satisfies the stationary distribution condition. This method does not violate the Bellman flow constraint.

Thus, we argue that the fundamental cause of BF violation is the use of semi-gradient update—not the replacement of the initial state distribution. To our knowledge, this is the first time this specific cause has been clearly identified.

Furthermore, Proposition 4.1 does more than pinpoint the violation's cause. It shows that SemiDICE outputs the policy ratio, which is essential for applying our stationary distribution extraction technique. Without Proposition 4.1, this insight would not be possible. We believe this adds a meaningful contribution to the understanding of semi-gradient DICE methods.

Finally, α only affects the conservatism strength of DICE method—it has no impact on the BF property.

Q2. Lack of acknowledgement [1, 2] for discovering the connection

Although we cited [1, 2] and briefly discussed them in appx. C, we will revise the manuscript to better acknowledge their discovery. While earlier studies implied the connection through similar loss functions, we make it more explicit by showing that a behavior-regularized MDP with a general f-divergence can be approximated by its corresponding SemiDICE. This insight is central to our work, as it justifies the SemiDICE policy ratio and supports its constrained extension—the main contribution of our paper.

Q3. Lack of acknowledgement [3] for bias reduction technique

While our bias reduction technique was inspired by PORelDICE [3], we approximate a different expectation. PORelDICE estimates $U(s,a)\approx r(s,a) +\gamma\sum_{s'}T(s'|s,a)\nu(s')$, while we approximate $A(s)\approx\sum_a\pi(a|s)(\gamma\sum_{s'}T(s'|s,a)\mu(s')-\mu(s))$. Consequently, PORelDICE focuses on reducing transition bias to improve offline RL, whereas our approach targets bias from both the transition dynamics and the policy to improve OPE. We will acknowledge [3] appropriately and clarify this distinction in the final version.

Q4. High cost close to limit

We respectfully clarify that obtaining cost values near the constraint threshold is not a flaw but a desirable behavior in our average-constrained optimization setting (CMDP, Eq. (7)). Our objective explicitly encourages using as much of the cost budget as needed to improve the return, so long as the expected constraint is satisfied. This is a principled difference from hard or probability-constrained formulations, where any thresholds violation is unacceptable. Recent works (e.g. [4], [5]) have studied these stricter formulas; our method is complementary and could be extended to them in the future. We will update the manuscript to make this distinction clearer.

Q5. Hyperparameter tuning

While G.3 lists all potential hyperparameters, we only tuned one—α—for CORSDICE. For D-CORSDICE, we tuned two additional hyperparameters (guidance scale values, the number of inference action values), which were directly adopted from D-DICE [6]. For inference samples, we used a smaller search space than in D-DICE (Table 4 of [6]). The original paper did not specify a search space for guidance scale, so we referred to the values in their official implementation. Importantly, all methods, including baselines, were tuned with the same hyperparameter search budget and procedure to ensure fair comparison. We will clarify these points in the revised manuscript.

[1] Dual RL: Unification and New Methods for Reinforcement and Imitation Learning. ICLR 2024

[2] ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update. ICLR 2024

[3] Relaxed stationary distribution correction estimation for improved offline policy optimization. AAAI 2024

[4] Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model. ICLR 2024

[5] Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability. NeurIPS 2022

[6] Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning. NeurIPS 2025