Fat-to-Thin Policy Optimization: Offline Reinforcement Learning with Sparse Policies

We thank the reviewer for the valuable feedback and appreciation of our work.

We agree that FtTPO is indeed somewhat heuristic-driven, we would like to point out that our design choice was based on the observation that neither forward KL nor reverse KL was capable of achieving the goal. For the reason of failure of reverse KL, as we conjectured in Section 3.2, it could be due to that the support is finite, the sampled actions may concentrate around its mode and therefore no significant update can be performed, resulting in extremely slow learning. This is further compounded by the fact that the standard deviation tends to become smaller as a result of optimization, so the samples $a\sim\pi_\phi$ will cover an increasingly shrunk region.

Regarding computational considerations, we have appended the draft with the following details:

1. we have added that 10 seeds are used for Section 5.1 and 5.2
2. we have added explanation to captions of Figure 2 and 4 that the shaded area indicates 95% confidence region
3. FtTPO requires sweeping the temperature $\tau$ in addition to the learning rate. Therefore, the number of hyperparameters is the same with popular algorithms like AWAC or IQL.
4. The training for FtTPO on average took around 15 hours for 1 million steps. By contrast, among the baselines InAC took 8.5 hours,IQL 6 hours and TAWAC 6.5 hours. This confirms that by maintaining two policy networks FtTPO costs around double computation time. When interacting with the environment, we sample actions only from the actor policy, so the time taken is similar to a standard algorithm. The explanation has been added to Appendix A.2

We thank the reviewer for the helpful feedback. We would like to explain our method in more detail and point out a potential misconception about our paper.

Regarding why the results are surprising:

First we would like to point out that the paper aims to learn a sparse policy which has not been possible in the deep offline RL context before, due to the out-of-support issue incurred by the sparse policies. Therefore, the utility and performance of sparse policies have never been examined in the literature. It is commonly believed that sparse policies are not performant because they are inherently handicapped at the exploration-exploitation tradeoff due to the sparsity [Lee 2018, Nachum 2018, Zhu 2024]. Our work for the first time provides a framework to learn sparse policies in the offline context, and verifies that they can be as performant as Gaussian/heavy-tailed policies, not only on specific safety-critical tasks but also on the standard offline RL benchmarks where the Gaussian dominates. Therefore, showing sparse policies are at least as well as existing methods that use Gaussian/heavy-tailed is important, because our method opens up a possibility of utilizing sparse policies not only for safety-critical tasks where they fit more than infinite-support policies, but also for general environments where they can be as good as the Gaussian.

We thank the reviewer’s questions regarding that the reader may not fully appreciate why the result is surprising. We have correspondingly modified our text to help the reader better understand the contribution of the paper. Specifically, we added the following to the introduction:

As it is commonly perceived that the sparse policies are inherently handicapped at the exploration-exploitation tradeoff, we find it surprising that the sparse policy learned by FtTPO can outperform full-support policies.

and the following to Section 5.2:

Given that it is commonly perceived that the sparse policies are inherently handicapped at the exploration-exploitation tradeoff, and no past prior work has demonstrated competitive performance to the infinite-support policies, it may come as a surprise that FtTPO performs favorably to or sometimes better than those state-of-the-art algorithms even with a sparse actor.

Regarding the misconception and why the additional complexity on architecture is necessary:

We would like to point out a potential misconception of the reviewer: the proposal-only setting learns a heavy-tailed distribution rather than a sparse policy. This setting is actually a very strong baseline [Zhu et al. 2024] that leverages the properties of heavy-tailed distributions [Kobayashi, 2019, Zhu et al., 2024]. It cannot be used directly for learning a sparse policy due to the reason stated for Eq. (1): the log-likelihood will have undefined values for out-of-support actions. Therefore, by comparing FtTPO against the proposal-only setting, we are actually comparing a sparse policy against a known strong baseline utilizing a heavy tailed policy. Hence, we believe that being able to be on par with such baselines on standard RL problems proves the value of our method.

We acknowledge that the proposal-only setting was not very well explained in the paper, we have added the following explanatory text to Section 5.3:

Note that the proposal-only setting is a competitive baseline that learns a heavy-tailed policy rather than a sparse policy.

Regarding why not just heavy-tailed or clipping X std from the mean:

Regarding concentration of the proposal policy, by examining the rightmost of Figure 2 and Figure 15, it is visible that the heavy-tailed policy can differ considerably from the sparse policy, as they have heavy tails decaying slowly and may contribute to selecting unsafe actions.

By clipping a policy X standard deviations from the mean, we will have a biased policy with significant densities on the border of clipping, which can be considerably different from the intended policy shape. To verify this phenomenon, we have included a new Figure 7, whose LHS compares the IQL + Gaussian to the proposed method. Because on this environment the action space is bounded, density outside the range is clipped and the resulting policy is very different from FtTPO.

We have added the following explanatory text to Appendix B:

Figure 7 visualizes how the sparse policy of FtTPO provides a principled solution to ensuring safety. The left figure compares FtTPO actor against IQL + Gaussian on the HalfCheetah. It is visible that by clipping the policy, IQL Gaussian causes excessive density falling on the clipping boundary due to normalization constraint. As a result, naive clipping can cause the policy to be a Bang-bang controller (Seyde et al., 2021) that picks actions almost exclusively on the boundary, leading to unsafe actions.

Dear Reviewer,

We would like to thank you again for your thoughtful feedback. As the deadline of the discussion phase is approaching, we would like to kindly ask if our rebuttal has addressed your concerns, especially regarding the novelty of sparse policies, level of performance and additional complexity. We value the opportunity to engage with you during this rebuttal phase, as your insights are instrumental in improving the quality of our work. Please do not hesitate to let us know if you have further questions. Thank you for your support and looking forward to hearing from you.

Sincerely,
The Authors

We sincerely appreciate the reviewer’s valuable suggestion to help better position the paper. Indeed in the paper we used the words safety and performance interchangeably with an assumption that safety is coded into reward explicitly. This is the case for our simulated treatment environment as its reward recursively conditions on past actions. We agree that this may not always be true and we have modified the text accordingly to clarify this point. The beginning paragraph in Section 5.1 now writes:

In this paper we consider the case where safety is explicitly coded into reward, so higher cumulative reward suggests a safety-respecting policy. However, this may not always be true and extra care is required when reward and safety need to be considered separately. We defer such investigation to future study. We opt for the synthetic environment used in (Li et al., 2023) that simulates real circumstances for continuous treatment. Note that only the most challenging environment is used here. The environment has 8-dimensional state space and an unbounded action space. Safety is explicitly coded into the reward function, such that danger is defined as cumulative dosage exceeding a pre-specified threshold. Note that the states are also conditional on past dosage, therefore danger states can vary depending on the dosage in the beginning states, see Appendix A for detail. According to (Li et al., 2023), the optimal dosage is unique, and a high dosage leads to excessive toxicity while a lower dosage is ineffective.

Regarding the additional graphs for validating FtTPO. We have included a new Figure 7 (RHS) that helps explain why the safety is ensured by the proposed method.

The right figure shows the FtTPO actor and reward curves (gray) along with policy evolution. By the definition of reward function, it depends recursively on the past actions. Therefore, a dangerous action can lead to potential future low reward. It is crucial that the agent is capable of utilizing not overly large yet effective dosage at each step to ensure safety. Since the low point of rewards does not reach zero, FtTPO does not incur danger during the evolution.

We noted that it is not possible to directly compare all policies on one state because the state-action space is continuous, and they do not experience an exactly same state that allows the direct comparison.

We thank the reviewer for appreciating our work and providing helpful feedback. We address the reviewer's questions as follows.

Regarding insights for why the combination of forward and backward KL helps:
We attribute the success of FtTPO to the way two policies are learned: while forward KL cannot be used for learning sparse policy, we can first learn a high-quality heavy-tailed policy from data. Its location and scale provide important knowledge to the sparse policy. This greatly eases the difficulty of learning a sparse policy, whose sparse tails can significantly hinder learning efficiency. This point was briefly discussed in Section 3.2. We have modified it to the following:

We conjecture that the inability of reverse KL alone is due to the finite support of sparse policies, which cause the “raw” sampled actions to concentrate around its mode and therefore no significant update can be performed, resulting in extremely slow learning.

Regarding application to the low-data regime:
We thank the reviewer for pointing out potential experiments in the low-data regime. Our simulated treatment experiment has only 50 trajectories and each trajectory contains 24 steps. We believe FtTPO has the potential for some low-data applications. We greatly appreciate the suggestion and plan to do such experiments with more realistic data in the future.

How $q$ is used to filter out bad actions:
According to the definition in Section 4.3, the weight $\exp_q\left(\frac{Q(s,a) - V(s)}{\tau}\right) = \mathbb{1}\left[\left(1 + (1 − q) \frac{Q(s,a) - V(s)}{\tau} \right)^\frac{1}{1-q} ≥ 0 \right] · \left(1 + (1 − q) \frac{Q(s,a) - V(s)}{\tau} \right)^\frac{1}{1-q}$ i.e. bad actions are those with advantage values $Q(s,a) - V(s) < -\frac{\tau}{1-q}$ that cause the term vanish. We see the threshold is controlled by $q$ and $\tau$. When using $q=0$, the threshold is controlled by $\tau$ alone.

Regarding copying of $\mu_{\phi}$:
Copying the proposal mean to the actor only serves the purpose of facilitating actor learning, since the actor mean and standard deviation are still subject to optimization of the actor and therefore they can change. Since the optimization is performed on the actions sampled from the actor policy, it does not violate its policy parametrization.

Minor comments:
We thank the reviewer for pointing out the typo and have corrected it. We pronounce our method as F-T-T-P-O.