$q$-exponential family for policy optimization

We thank the reviewer's insightful comments regarding conclusion of the paper.

To help conclude that the Student's t as the policy to go, we have added the new Figure 11 showing the similar behavior of Student's t and Gaussian to hyperparameters. The following new paragraph in Section 5.1 states why Student's t can be used as a drop-in replacement to the Gaussian :

In Figure 11 we show the Manhattan plot of SAC with all swept hyperparameters on all environments. Though there is no a definitive winner for all cases, it is visible that the Student’s t and Gaussian have a similar behavior to hyperparameters. Therefore, if we are tackling a problem where the Gaussian works, the Student-t is very likely to work and perform better given the same hyperparameter sweeping range.

We further consolidate our conclusion by adding the following remarks to the paper.

(Abstract) In summary, we find that the Student’s t policy a strong candidate for drop-in replacement to the Gaussian.

(Conclusion) In summary, we found the Student’s t policy to be generally more performing and stable than the Gaussian and could be used as a drop-in replacement.

There could be many factors of an environment that makes one distribution better than the others. For example, our Mountain Car has cost-to-go as reward that demands exploration. The ranking of policies seems very different to the environments with normal rewards. We believe decoupling the environmental characteristics for choosing a suitable policy parametrization is an interesting and important future direction.

We thank the reviewer for the helpful feedback and pointing out an interesting and important future direction.

Indeed, there may be an exploration vs. exploitation trade-off underlying the $q$ policies. In the paper, this role was played by the Student's t whose degree of freedom was also learned, so it could switch between a Gaussian (so it could shrink into a Delta-like policy) and a heavy-tailed distribution. It is an interesting future direction to see what trade-off each policy parametrization would make. For general $q$-Gaussians, this could potentially be achieved by adaptively adjusting the $q$ parameter.

To further consolidate the paper and provide intuition to the question of "what policy should we use?", we have included additional results in Figure 11, a Manhattan plot to visualize SAC’s behaviour to hyperparameters changes. Although there is no conclusive winner, we can safely conclude from the plot of Acrobot that if we have a problem where the Gaussian works, the Student’s t is very likely to work, and from Figure 5(left), it is 75% more likely perform better given the similar hyperparameter sweeping behaviour. Combined with the existing offline results, these observations suggest that the researchers can use Student’s t as a drop-in replacement for the Gaussian and benefit from the performance improvements.

We now address the reviewer's questions.

• Q1: Do all the policy neural networks have a similar number of parameters/weights across experiments for fairness of comparison?

All policy networks share a common network architecture. q-Gaussians and Gaussian are location-scale distributions; they output two distribution parameters: mean and standard deviation. Beta is slightly different, but the actor network also outputs alpha and beta shape parameters. Student’s t, by contrast, maintains three parameters: mean, deviation and the degree of freedom. In summary, all actor neural networks are similar for fair comparison; they output two statistics except for the Student’s, which outputs three.

• Q2: Are the implementations mostly readily available in existing RL packages or other libraries that could be directly used? I see that you provide some of the sampling mechanisms but these seem to be already well known in the statistics literature given their popularity.

The standard Gaussian/Squashed Gaussian and Beta policies are already available in the existing RL packages. However, we are unaware of RL libraries that provide q-Gaussians. We have implemented q-Gaussians and their sampling methods in PyTorch and will release the GitHub code base upon acceptance of the paper.

• Q3: Is GBMM only valid for 1 < q < 3?

GBMM can be used for all $q<3$. However, it tends to produce high variance samples, especially when $q$ is close to 1. Therefore, for the sparse case ($q<1$), we resorted to the stochastic representation given by [Martins et al. 2022], which produces low variance samples only for sparse $q$-Gaussians.

• Q4: l. 41: ‘Heavy-tailed distributions could be more preferable as they are more robust’, what do you mean by robustness here? Can you elaborate more?

Heavy-tailed distributions like the Student’s t have been studied in [Kobayashi et al., 2019] to be resilient to environmental noises in that the policy is not easily affected by the noises so that it shifts to low-reward regions like the Gaussian. We have expanded our explanation in the corresponding paragraph to make it clearer.

• Q5: Can you clarify the difference between the different Data settings in Fig. 8 (Medium-Expert, Medium, Medium-Replay)?

We have added the following to Appendix D.3 for better clarification:

The D4RL offline datasets all contain 1 million samples generated by a partially trained SAC agent. The name reflects the level of the trained agent used to collect the transitions. The Medium dataset contains samples generated by a medium-level (trained halfway) SAC policy. Medium-expert mixes the trajectories from the Medium level and that produced by an expert agent. Medium-replay consists of samples in the replay buffer during training until the policy reaches the medium level of performance. In summary, the ranking of levels is Medium-expert > Medium > Medium-replay.

We sincerely appreciate the reviewer's thoughtful comments and appreciation of our work. This comment answers the reviewer's points in the weakness section. The next comment will address the reviewer's questions. All changes made to the paper were marked in blue.

Regarding behavior to hyperparameter changes:

We have included additional results in Figure 11, in the paper. It is a Manhattan plot visualizing the behaviour of SAC + all policies on all online environments. Although there is no definitive winner for all cases, we observe from Acrobot that the Student’s t and Gaussian have a similar behavior to hyperparameters changes. It may be safe to conclude that, if on a problem the Gaussian works, then the Student-t is very likely to work. And from Figure 5 (left), we know that Student-t is 75% more likely to outperform Gaussian given the same hyperparameter sweeping range. We have included this explanation to Section 5.1.

Regarding a procedure to pick policy:
Several interesting future directions are possible for determining a suitable policy parametrization, including the model selection, such as the procedures the reviewer mentioned. Heuristically, the current study suggests that Gaussian and Beta are less preferred than heavy-tailed policies, especially Student’s t, which has shown stable improvements across environments, both in online and offline experiments. These improvements might be because the Student’s t maintains three learnable statistics: mean, standard deviation and degree of freedom (DOF). Learning the DOF allows the Student’s t to interpolate between the Gaussian and heavy-tailed distributions, granting it more flexibility. However, it remains an open question and interesting future direction to have a function that inputs the setting at hand and outputs a ranking on policy parametrizations with confidences (Student’s t > Heavy-tailed/Sparse q-Gaussian > Gaussian).

Regarding relevant references:
We thank the reviewer for providing relevant references. We have incorporated them into the Introduction section.

With q > 1, we can obtain policies with heavier tails than the Gaussian, such as the Student’s t-distribution (Kobayashi, 2019) or the Levy Process distribution (Simsekli et al., 2019; Bedi et al., 2024). Heavy-tailed distributions could be more preferable as they are more robust (Lange et al., 1989), can facilitate exploration and help escape local optima in the sparse reward context (Chakraborty et al., 2023).

Regarding computational overhead:
We have added the following paragraph regarding computation time to the last paragraph of Section 5.2:

To give an intuition for sampling time, we drew $10^5$ samples from a randomly initialized actor on two environments: HalfCheetah with 17-dim state and 6-dim action. The sparse $q$-Gaussian, heavy-tailed $q$-Gaussian and Gaussian respectively cost (107.12, 72.09, 27.94) seconds. We confirmed that the sampling in Alg. 1 was on the same magnitude to the Gaussian, but the sparse $q$-Gaussian costed more than the heavy-tailed due to more computation to produce low-variance samples. This is further confirmed by Hopper with 11-dim state, 3-dim action, where they cost (98.13, 65.17, 25.17) seconds.

We expect the sampling time to be similar for common RL problems. But for the rare case where the dimension is huge, we expect that the efficiency for $q<1$ may be affected since it uses more computation to produce low-variance samples. This is because for Gaussian, we first sample from $\mathcal{N}(0, I)$ and then transform it with the location and scale transform $\mu + \Sigma^{\frac{1}{2}}z$. In the case of GBMM, we sample from a uniform distribution followed by basic computation of cosine, q-log, and finally, the location-scale transform. Therefore, the computation overhead is similar to that of the Gaussian. For q<1, we used the stochastic representation. The computational cost slightly increases as we need to sample two variables and compute the radius and scaled matrix. By Alg.1, the computation overhead centers around computing the determinant $|\Sigma|$. For every update to the policy, we compute and store this determinant. If the matrix dimension is huge, then it may become the bottleneck. Fortunately, in our case, we can simply use the GBMM instead.

Regarding unclear statement of the Gaussian:
We agree with the reviewer that our statement was not clear. We have removed this sentence and added the following explanation to the beginning of Section 3.1:

The Gaussian policy is one of the simplest distributions one can consider due to its omnipresence in statistics and parametric estimation as well as its widely available sampling procedure implementations. Since evaluating the log-partition function of BG is intractable, due to the aforementioned advantages many researchers consider the Gaussian policy instead.