Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization

We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.

Regarding reviewer's questions:

Q1: Results in the Medium maze are as follows

DrDiffus

DACER

The results demonstrate that the diversity-related performance of DrDiffus is improved by raising the number of diffusion steps. While for DACER, the diversity-related performance is not improved. This is probably due to that DACER controls diversity by scaling an additional Gaussian noise instead of optimizing diversity through gradients w.r.t. the actor's parameters.

Q2: We conduct additional experiments for this question, existing results includes:

Multi-goal Maze:

DrAmort

DrDiffus

Game Content Generation:

DrAmort

DrDiffus

MuJoCo

We will add comprehensive results and analysis to study this question in our paper.

Regarding how to tune the temperature hyperparameter, generally when we need better diversity or exploration we shall raise $\beta$, and when we need more exploitation we shall reduce $\beta$. Our definition $\beta$ is normalized so that the scale of $\beta$ is comparable to the per-dimension pairwise distance between actions (See the response to Q7). So one can roughly decide the range of $\beta$ and may be able to set $\beta$ according to the expected pairwise distance of actions. We will detail this in our paper.

Q3: In Medium Maze:

DrAmort

DrDiffus

The success rate of DrDiffus with $n=4$ is lower than that with larger values of $n$. Expect for this, the performances of DrDiffus and DrAmort are not sensitive to $n$. To provides some intuition for tuning $n$, it is most likely that raising $n$ will reduce the variance in estimating diversity, but usually it not needs to be very large.

Q4: For multi-goal PointMaze, we conducted a grid search for each algorithm before the formal evaluation. Based on the grid search, we picked the highest temperature that ensures each algorithm reaches an optimal success rate, to enable a fair and meaningful comparison. $0.5|\mathcal{A}|$ is the highest temperature value we found that still ensure them reach the optimal success rate, so we picked this value.

For game content generation, we just follows the temperature settings in multi-goal PointMaze.

For the MuJoCo experiements, we kept the default target entropy value as described in their papers. Our Table 3 lists the temperature/target entropy/entropy coefficients for all algorithms in MuJoCo.

Q5: This is a typo and we are very sorry for that. We used temperature tuning technique like SAC following the paper of DACER. DACER does not use fixed $\alpha$ so there should be N/A in the corresponding cell in Table 2, and $\alpha=0.6$ is for S2AC. We sincerely thank the reviewer for the careful review and have revised this error.

Q6: Under equation (11) of the DACER paper (the NeurIPS 2024 camera-ready version), it is stated “we employ the tricks in DSAC [11, 10] to mitigate the problem of Q-value overestimation.” And we carefully checked their official code and found that they used such a technique. That technique was introduced in [a], which is a distributional critic. We have revised our paper to make it more clear.

[a] Duan, Jingliang, et al. "Distributional soft actor-critic with three refinements." IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).

Q7: The auto-tuning technique operates on $\alpha$ to match the actual pairwise distance-based diversity meaure $D(\pi)$ with a target $\hat D$, i.e., $D(\pi)=\mathbb{E}[\log\delta(a^x, a^y)] \approx \hat D$. The diversity regularization $\mathbb{E}[\log\delta(a^x, a^y)]$ takes logarithms and it is not so straightforward to determine the proper range of it. By introducing $\beta$, we have $\mathbb{E}[\log\delta(a^x, a^y)] \approx \log(\beta|\sqrt{\mathcal{A}}|) \Rightarrow \mathbb{E}[\delta(a^x,a^y)]/\sqrt{|\mathcal{A}|} \approx \beta$, so the proper range is easier to understand and one may determine $\beta$ according to the promising value of distances between actions.

Regarding specific weaknesses:

Proposition 1 asserts that ... could the authors clarify what conceptual or algorithmic advance is provided by PGRep?

1. Amortized actors are lightweight policy models with strong multimodal expressiveness. However, algorithm for training amortized actors is underexplored in the field of multimodal RL, and to our best knowledge, no work has trained amortized actors reparameterized policy gradient. We highlight that the reparameterized policy gradient is effective for training amortized actors.
2. Though there have been some diffusion policy-based RL algorithms back-propagating the gradient of the Q-function to update the diffusion policy, the theoretical interpretation from the perspective of reparameterized policy gradient is overlooked.
3. We introduce a unified formulation for a range of intractable actors and highlight that PGRep is applicable to all these actors. This formulation serves as a useful tool for understanding and designing multimodal RL models and algorithms.

The assumption in Proposition 1 that is full rank ... almost never be satisfied in practice.

Thanks for raising this point. We acknowledge that this type of assumption, which serves as a sufficient condition to establish the result, is strong. Yet, we find that it is not necessary for the practical algorithm to work well, which is also a common theory-practice gap in deep RL literature, where theoretical guarantees often rely on stronger assumptions while practical algorithms succeed despite some violations [b]. We would like to emphasize that this provides insights for the development of our algorithm in Section 4.2 even when the assumption is not strictly satisfied, where our extensive experiments show that DrAC works effectively across diverse environments and backbone components. This consistent performance gains suggest that the core mechanism in DrAC is robust to potential violations of theoretical assumptions.

In the updated version, we will discuss the implications for Proposition 1 and explore connections to rank-deficient approximations that may explain why the method works despite assumption violations, and also further contextualize this assumption within the broader RL literature, which is a very interesting future direction for our work to explore smoother assumptions (e.g., regularity conditions for the gradients). Despite this theoretical point, our work remains significant, which aims to develop a unified framework for multimodal actors with a novel distance-based regularization approach that achieves consistent performance gains in diversity-critical domains with improved generalization ability.

[b] Tang, Y., & Agrawal, S. (2018). Boosting trust region policy optimization by normalizing flows policy. arXiv preprint arXiv:1809.10326.

Diversity is measured via the geometric mean of pairwise action distances, ... would strengthen the motivation for this choice.

In Figure 2, the ground truth of $X$ is a Gaussian distribution with $\sigma=1$, while the ground truth of $Y$ is a mixture-of-Gaussian with uniform weights and two Gaussian components with $\sigma_1=\sigma_2=0.2$. We calculated their entropy: $\mathcal{H}(X)=2.838$, and $\mathcal{H}(Y)\approx 0.3121$. This demonstrates that geometric mean aligns more closely with entropy compared with the vanilla average pairwise distance measure. We have updated Figure 3 to include such information.

We agree that separated clusters better capture multimodal behaviour, while there still exists a trade-off, since overly small clusters might harm local exploration. The regularizer presented in Equation (8) is our best practice so far.

For other concerns about missed references, understatement for prior works, minor errors: We thank for reviewer's time and rigorous review and will carefully revise them in our paper.

We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.

Regarding reviewer's questions:

Q1: Those baselines suffering from a performance drop are entropy-regularized algorithms. Based on the periodic visualization of trajectories in the mazes, we observed that the performance drop of those baselines is accompanied by exploration behaviours. At that stage, these algorithms appear to “wobble” among different goals. Possibly the gradient of entropy bonus pushes the actor to other directions and makes the policy explore in the maze and fails to reach goals. DACER handles entropy regularization by auto-adjusting the scale of an auxiliary noise instead of back-propagating the gradient of entropy regularization. Moreover, the computational cost of adjusting the scale is high, making such adjustment usually perform at a very low frequency (by default, DACER updates the scale every 10000 exploration steps). This possibly be the reason why DACER does not suffer from a performance drop. A more in-depth investigation shall be future work.

Q2: Let $n$ ($n=8$ in our experiments) denote the number of diversity estimation pairs, we summarize the number of forward inferences and back-propagation per transition in one single update as follows.

The diversity estimation introduces $4n$ additional forward inference and $2n$ additional back-propagation, without additional forward and backward of the critic. As the neural networks used for continuous control are typically small (e.g., SAC uses MLPs with two 256-dimensional hidden layers by default), we empirically found DrAmort and DrDiffusion do not induce significantly more training times and memory usage, as displayed in the following table.

This table shows average time cost per 1000 training steps in Ant-v4 environment. The test is conducted on a Linux server with 8 NVIDIA RTX 3090 GPUs and an Intel Xeon Platinum 8375C CPU. For each algorithm, we run three trials on GPU cards 0, 1 and 2, respectively, and then average the training time per 1000 steps. All algorithms are implemented with PyTorch 2.5.1, and the CUDA version is 12.5. | | SAC | SQL | DrAmort | DACER | DrDiffus | S2AC | | --- | --- | --- | --- | --- | --- | --- | | Time per 100 training steps | 22.7s | 18.9s | 21.0s | 76.9s | 84.8s | 220.5s | | Peak GPU memory usage | 352M | 472M | 392M | 364M | 608M | 3462M |

According to the table, we found that DrAmort is even faster than SAC. The reason is that SAC requires computing and back-propagating the log-probability at certain actions, which requires more sequential tensor operations. Though DrAmort requires more FLOPs, the computation is parallelized and takes fewer sequential tensor operations. As a result, DrAmort uses even fewer GPU clock ticks and is slightly faster than SAC, given that the neural network is sufficiently small compared to the GPU’s parallel computation capability. For DrDiffus, it takes about 10% more time for training compared to DACER, but its number of reachable goals in multi-goal PointMaze is much higher.

For large-scale tasks that require larger networks, such as visual control, we can inject the latent variable at the last few layers. For example, if a CNN is used, the latent variables can be injected after the convolutional layers. In this case, the computation time and memory usage will not significantly increase.

Regarding specific weakness:

The unification of amortized and diffusion actors with reparameterized policy gradient seems trivial.

We would like to justify our contribution regarding this point as follows:

1. Previous practices in training amortized actors are based on SVGD [a,b]. The effectiveness of the reparameterized policy gradient for training amortized actors is underexplored. We highlight that the reparameterized policy gradient is effective for training amortized actors.
2. Previous works did not formally discuss the theoretical connections between backpropagating $Q$ and policy gradient.
3. By formulating diffusion actors and amortized actors into a unified framework, we demonstrate the versatility of reparameterized policy gradient for training intractable multimodal actors. This formulation is general and extendable. It serve as a useful tool for understanding and devising multimodal RL models and algorithms.

We have revised section 4.1 and the contribution statement 1 in the introduction to make the contribution of this part clearer and accurate.

[a] Haarnoja, Tuomas, et al. "Reinforcement learning with deep energy-based policies." International conference on machine learning. PMLR, 2017.

[b] Messaoud, Safa, et al. "S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic." The Twelfth International Conference on Learning Representations.

The paper mentions that the sampling-based diversity measure can be parallelized over GPUs for generating multiple samples, but this inevitably introduces additional computational costs during training.

Though the estimation of our diversity measure inevitably introduces additional computations during training. The actual training time is similar to the baselines as those operations are fully parallelized. For large-scale tasks, we can inject the latent variable at the last few layers to avoid much additional computation (See the response to Q2).

On traditional MoJoCo benchmarks, the proposed methods perform worse than unimodal policy SAC.

Our core contribution is training mutlimodal actors with optimized diversity, which is verified with the other two domains. The main purpose of our experiment in MuJoCo benchmarks is to show our algorithm achieves competitive performance against SAC in conventional tasks. It is not so quite surprising that DrAC does not outperform SAC, as the only goal in MuJoCo is to move faster, making multimodality less important. In addition, while amortized actors are fast and possess strong multimodal expressiveness, there is a lack of an efficient and effective algorithm to train them prior to our work. SQL performs badly on MuJoCo and S2AC costs much more time and memory usage. Our algorithm fills this research gap.

Meanwhile, as suggested by other reviewers, we are conducting comprehensive ablation study, including in MuJoCo benchmarks. We have finished the training of DrAmort with other $\beta$ values in Ant and HalfCheetah. The results suggest that the reason that our algorithm does not outperforms SAC is probably insufficient parameter tuning

Missing references and baselines. The multimodal policy representation in RL has been studied for a long time, and I think there are quite a lot of papers missing as references and baselines from this paper. Discussion and comparisons against these baselines would be necessary to prove the advantages of the proposed methods.

We have revised our paper to discuss related works more comprehensively.

To address reviewer’s concern, we have tested PMOE [2] and CPQL [4] in our multi-goal environment. For CPQL, we run the official code with default settings as it does not provide a hyperparameter to trade off between return and diversity. For PMOE, we perform grid search as we done for other algorithms, and picked the highest target entropy that ensures it achieves the optimal success rate. The results in the Medium maze are presented as follows.

According to the results, CPQL does not exhibit diversity. PMOE demonstrates diversity but our DrAmort outperforms PMOE. Moreover, PMOE trains tractable mixture-of-Gaussian policies that only represent a fixed number of modes, while our algorithm is able to train intractable policies that can express arbitrary number of modes.

We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.

Regarding reviewer's questions::

Q1: We are running all algorithms with different temperatures in the game content generation environment. We will add these figures in our revision. We have finished partial experiments, in which we train all algorithms with three temperatures (five seeds each) and test the quality and diversity of content generated by the trained policies. To provide informative response, we compute hyper volume (HV) metric [a] to evaluate the policy population obtained by each algorithm. Specifically, we treat the 15 policies learned by each algorithm with 3 temperatures and 5 seeds each as one population.We take the global lowest quality and global lowest diversity among all policies learned by all algorithm as the reference point, then compute the HV metric for each population. The result is displayed as follows. (Training of S$^2$AC has not finished as it is too computational intensive)

MarioPuzzle

According to the table, our DrAmort demonstrates highest HV value. The HV values of DrDiffus and SAC are closed.

MultiFacet

According to the table, our DrAmort and DrDiffus achieve the top 2.

Comprehensive experiment results will be discussed in our revised version, where we will also show visualization of Pareto fronts. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.

[a] Zitzler, Eckart, and Lothar Thiele. "Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach." IEEE transactions on Evolutionary Computation 3.4 (2002): 257-271.

Q2: We conduct additional experiments to investigate this question. Based finished experiments, we present some results below to respond to this question. We will add comprehensive results and analysis in our paper.

Multi-goal Maze

We run DrAC with different values of $\beta$ in the our multipoint-goal maze environments, the final performances in the Hard maze are presented as follows.

DrAmort

DrDiffus

According to the results, success rate is not sensitive to beta within a wide range (about <0.8 for DrAmort and <0.9 for DrDiffus). diversity-related performance is sensitive to beta. Higher $\beta$ values enable DrAC to reach more modes and enhance few-shot robustness. However, overly high temperatures reduce the success rate, which also harms robustness metrics. We will discuss the sensitivity of beta comprehensively in our revised version.

Game content generation: We finished evaluation with three $\beta$ values for MarioPuzzle style. Results are presented below

DrAmort

DrDiffus

The quality of DrAmort is not sensitive to $\beta$, while diversity increases by raising $\beta$. For DrDiffus, quality increases and diversity decreases from $\beta=0.2$ to $\beta=0.5$, but $\beta=0.8$ does not show significant difference with $\beta=0.5$

There are two aspects may be related to the experiment phenomenon: (1) higher temperature also improves exploration which could be helpful to quality (2) policy diversity is correlated but not identical to the diversity of generated levels. More fine-grained ablation study will be conducted and discussed in our revision. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.

MuJoCo

We are running additional experiments to analyze the sensitivity of DrAC in MuJoCo. Such experiments will be added to our revised version. We have finished partial experiments now. The performances of DrAmort with different values of $\beta$, in the first two MuJoCo environments, are presented as follows

Performance of SAC is included as a baseline. According to the table, $\beta$ values that are higher than our original setting ($\beta=0.15$) improve the performance in Ant and HalfCheetah. Notably, the improvement in Ant from $\beta=0.15$ to $\beta=0.35$ is >36%, and with $\beta=0.35$, DrAmort outperforms SAC by >18% in HalfCheetah. More elaborate self-adaptation/annealing techniques could be researched in future work.

Q3: We train DrAmort and DrDiffus with a higher $\beta$ in Ant, and records the trajectories by evaluating the final policies. We compute average distance among those trajectories as a diversity measure, as presented below.

According to the table, higher $\beta$ make trajectory diversity increase, though the only objective in MuJoCo is moving forward faster. We will visualize the trajectories in our revision.

Regarding specific weakness:

The reparameterization trick is standard in SAC and SQL-style methods. While some diffusion-based approaches use similar ideas [1] for trajectory optimization, typical diffusion policies for imitation learning do not involve policy gradients or reparameterization.

[1] Huang, Zhiao, et al. "Reparameterized policy learning for multimodal trajectory optimization." International Conference on Machine Learning. PMLR, 2023.

We have revised our paper to discuss [1]. We would like to clarify the novelty and significance here as follows:

1. SAC applies reparameterization trick but only for tractable (squashed) Gaussian actors. SQL employs Stein variational gradient descent to back-propagate the estimated gradient of the KL-divergence objective in entropy-regularized RL. This is different from our gradient estimator formulated in Equation (7). To our best knowledge, no work has trained amortized actors with the reparameterized policy gradient as formulated in our Equation (7) before. Our work highlights the overlooked effectiveness of training amortized actors through the reparameterized policy gradient. We also demonstrates that amortized actor is highly potential for learning multimodal policy, which is underexplored before.
2. Both [1] and our work adopt the reparameterization trick, however, [1] applies the reparameterization trick to estimate the gradient of an evidence lower bound, while our work applies the reparameterization trick to estimate the standard policy gradient. Our work primarily differ from [1] in two aspects. (1) The algorithm proposed in [1] is a model-based algorithm that additionally learns a latent dynamics model, while our work focuses on the model-free setting; (2) Paper [1] reformulates the learning objective into an evidence lower bound of optimality, while our learning objective is expected diversity regularized return.
3. Recently, some online RL algorithms for diffusion policies have been proposed, in which [b,c] also backpropagates the gradient of $Q$. These methods show promising performances. However, the theoretical interpretation of this operation from the perspective of reparameterized policy gradient is overlooked. We highlights that backpropagating $Q$ is an estimator of policy gradient with reparameterization trick, which is applicable to all stochastic-mapping actors including diffusion actors and amortized actors. This formulation serves as a useful tool for understanding and designing online model-free multimodal RL algorithms.

We have revised our paper to discuss related works more comprehensively and improve the clarity of our contributions

[b] Wang, Yinuo, et al. "Diffusion actor-critic with entropy regulator." Advances in Neural Information Processing Systems 37 (2024): 54183-54204.

[c] Ding, Zihan, and Chi Jin. "Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning." The Twelfth International Conference on Learning Representations.

For typos: We thank for reviewer's time and rigorous review. We will carefully revise them and further proofread our paper rigorously.

We thank the reviewer for the insightful comments and suggestions. We run additional experiments to address reviewer’s questions and improve the thoroughness of our work. Due to the character limit, we only present most relevant and informative results using tables here. All presented results are averaged over 5 training seeds.

Regarding reviewer's questions:

Q1: We conducted a grid search for each algorithm before the formal evaluation. Based on the grid search, we picked the highest temperature that ensures each algorithm reaches an optimal success rate, to enable a fair and meaningful comparison. We have revised the paper to clarify this.

Q2: As SAC is a standard baseline in MuJoCo, we compute the average log-distance (our diversity regularization metric) of the trained SAC agents with default target entropy $-|\mathcal{A}|$ in each MuJoCo task. Then, as we set target diversity for DrAC as $\hat D = \log(\beta \sqrt{|\mathcal{A}|})$, so we have $\beta \approx \exp(D(\pi))/\sqrt{|\mathcal{A}|}$, where $D(\pi)$ is the expected pairwse log-distance. So we computed the corresponding $\beta$ value in each task as follows

Except for Humanoid, all the results are around 0.15, so we chose $\beta=0.15$ to make the target diversity comparable with the default target entropy of SAC. We will also add an evaluation of the sensitivity to temperature in MuJoCo during the revision.

Q3: We conduct additional experiments to investigate this question. Based finished experiments, we present some results below to respond to this question. We will add comprehensive results and analysis in our paper.

Multi-goal Maze:

We run DrAC with different values of $\beta$ in the our multi-goal maze environments, the final performances in the Hard maze are presented as follows.

DrAmort

DrDiffus

According to the results, success rate is not sensitive to $\beta$ within a wide range (about <0.8 for DrAmort and <0.9 for DrDiffus). diversity-related performance is sensitive to $\beta$. Higher $\beta$ values enable DrAC to reach more modes and enhance few-shot robustness. However, overly high temperatures reduce the success rate, which also harms robustness metrics. We will discuss the sensitivity of beta comprehensively in our revised version.

Game Content Generation:

We have run DrAC with three temperatures. The results for MarioPuzzle are displayed as follows

DrAmort

DrDiffus

Regarding DrAmort, the diversity of generated levels increase along with beta raises. The content quality appears not sensitive to $\beta$.

Regarding DrDiffus, when raising $\beta$ from 0.2 to 0.5, quality decreases and diversity increase. When raising $\beta$ from 0.5 to 0.8, the performance does not change much.

There are two aspects may be related to this experiment phenomenon: (1) higher temperature also improves exploration which could be helpful to quality (2) policy diversity is correlated but not identical to the diversity of generated levels. More fine-grained ablation study will be conducted and discussed in our revision. Future work in generative RL for game content generation includes multi-objecive RL, where our method could serve as a promising base algorithm.

MuJoCo

We are running additional experiments to analyze the sensitivity of DrAC in MuJoCo. Such experiments will be added in our revised version to improve thoroughness. We have finished partial experiments now. The performances of DrAmort with different values of $\beta$, in the first two MuJoCo environments, are presented as follows

We also include the performance of SAC for reference. According to the table, raising $\beta$ value (as we used $\beta=0.15$ in the paper) improves the performance in Ant and HalfCheetah. Notably, the improvement in Ant from $\beta=0.15$ to $\beta=0.35$ is >36%, and with $\beta=0.35$, DrAmort outperforms SAC by >18% in HalfCheeta. More elaborate self-adaptation/annealing techniques could be researched in future work.

Q4: The blue one ($X$) is a diagnoal Gaussian distribution $\mathcal{N}(\boldsymbol{0}, I)$, while the red one is a Gaussian mixture model with uniform weights and two components $\{\mathcal{N}([6, 6], 0.2I),\mathcal{N}([-6, -6], 0.2I)\}$. We have modified the figure to describe the two ground truth distributions in our paper.

Q5: The actor in SAC outputs the mean and per-dimension scale of a diagonal Gaussian distribution and applies $\tanh$ to bound the output action. It is closed to the concept of amortized inference. In the paper of SQL [1] and S2AC [2], the term “amortized actor” is used to refer to GAN-like actors, which take an additional random variable as input and output the action. So we follows there naming, specifically refer to SQL-style actor as “amortized actor”, which may be narrower than the concept of amortized inference. For SAC’s actor, we refer to it as “Gaussian actor”.

We define “stochastic-mapping actors” as the framework to unify amortized actors and diffusion actors. This framework is also compatible with SAC-like actor, by treating $f_\theta(s,z) = \tanh(\mu_\theta(s) + \sigma_\theta(s)z), z \sim \mathcal{N}(\boldsymbol{0}, I)$, where $\mu_\theta(s)$ and $\sigma_\theta(s)$ is the mean vector and scale vector outputs by the neural network parameterized with $\theta$. That also means it will be straight-forward to apply our algorithm to train Gaussian actors. We did not test this as our main focus is training intractable multimodal actors. We will discussed this compatibility with Gaussian actor in our revision.

[1] Haarnoja, Tuomas, et al. "Reinforcement learning with deep energy-based policies." International conference on machine learning. PMLR, 2017.

[2] Messaoud, Safa, et al. "S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic." The Twelfth International Conference on Learning Representations.

Q6: Thank for the careful review. We have updated Section 3.2 to acknowledge foundational works properly.

Regarding specific weaknesses:

In the MuJoCo experiments ... It's unclear whether this is due to insufficient hyperparameter tuning or if multimodal policies simply aren't necessary in these environment

We are addressing this by additional experiments (See response to question 2 and 3).

The process of tuning the temperature hyperparameters in the multi-goal experiment is not elaborated, potentially hindering the validity and rigorousness of the results.

We address this in the response to question 1.

One of the claimed contributions, ... to accurately establish its place within the field.

We thank the reviewer for pointing out this related work, as we were not aware of it before. (Lan et al 2022) has discussed the reparameterized policy gradient as an additional discovery, where the conclusion equation is equivalent to our Equation (7) in Proposition 1. We have revised Section 4.1 to cite Lan et al 2022 and re-clarify the contributions. We also revised our contribution statement 1 in the introduction.

After reviewing related works more comprehensively, we would like to re-clarify our contribution as follows

1. Amortized actors are lightweight actors with strong multimodal expressiveness. However, they are underexplored in the field of multimodal RL. Previous practices in training amortized actors are based on SVGD [1,2]. The applicability of the reparameterized policy gradient for training amortized actors has not been formally discussed. We highlight that the reparameterized policy gradient is effective for training amortized actors.
2. Though there have been some diffusion policy-based RL algorithms back-propagating the gradient of the Q-function [3,4] to update the diffusion policy, the theoretical perspective from reparameterized policy gradient has not been formally discussed.
3. We formulate diffusion actors and amortized actors into a unified framework to show the versatility of reparameterized policy gradient for training intractable multimodal actors. This formulation serve as a useful tool for understanding and devising multimodal RL models and algorithms.

[3] Wang, Yinuo, et al. "Diffusion actor-critic with entropy regulator." Advances in Neural Information Processing Systems 37 (2024): 54183-54204.

[4] Ding, Zihan, and Chi Jin. "Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning." The Twelfth International Conference on Learning Representations.