Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion

We sincerely appreciate Reviewer YLrf for the insightful feedback and the acknowledgment of our work's novelty and empirical effectiveness. Below, we provide a detailed response to address your concerns:

• W1 and L3: Limited evaluation: Our experiments cover 3 domains and 23 tasks, including key control tasks like tabletop manipulation and locomotion control. As acknowledged by Reviewers 8Q7f, 957J, and U6yw, our experiments are more extensive than all the baselines. We also evaluated our approach in real dynamic tasks, including sharp turns, obstacle avoidance on mats, and rapid backward movements. We present more task visualization in Figure 15 in our supplementary material. If you have any further detailed feedback about our experiments, we would be happy to discuss it further.

• W2, Q3 and L2: Reliance on input demonstrations: There may be some misunderstanding here. As stated in Line 187-188, we use trajectories from RL replay buffer to generate policies. We do not use optimal trajectories or select specific ones, ensuring availability and diversity of conditions. Experiments with noisy trajectories also show that our method can synthesize effective policies under perturbed conditions.

• W3 and L5: Overfitting to specific tasks and behaviors: In Figure 8, we compare the behaviors used as generation conditions with the behaviors of generated policies, highlighting the differences between them. Figure 10 demonstrates that our method explores the parameter space more extensively, not overfitting to specific tasks. Additionally, finetuning the model on significantly different environments (Metaworld to Robosuite) proves our generator's effectiveness across domains.

• Q1: Performance on vastly different tasks: We present additional generalization experiments on significantly different tasks like soccer, box close, and sweep into in Metaworld. Our method consistently finds optimal policies, outperforming baselines. As tasks with large differences may have significantly different parameter distributions, this issue exists in both prior policy learning methods and our method. Our work aims to minimize adaptation costs as much as possible.

• Q2 and L1: Computational requirements: In Line 186, we specify the GPU hours required for collecting policy data for training. In Lines 425, 434, and 438, we mention the GPU hours required for training each model. We compare the computational budget in Figure 14 and the below table. Our method requires relatively few GPU hours compared to baselines (much fewer than single RL training). Our training dataset will be released to the community for further valuable research, which we believe contributes to researchers who may have limited resources.

• Q4 and L4: Scalability: Our method handles policy networks with parameters ranging from 2e4 to 1.6e5, suitable for common continuous control scenarios. This is both more diverse and larger than the policy networks used in other policy generation methods. For more complex tasks, we can integrate pre-trained visual foundation models as encoders to process image features while using our method to generate actor networks.

• Q5, L7 and L9: Component contributions and ablation studies: Figure 7 shows results without behavior embedding. Since we use a latent diffusion model as the backbone, we can't remove the autoencoder to directly generate the policy parameters, as this would consume a lot of computational cost. The three parts of our method are not independent and separable but work together through the autoencoder and behavior embedding to reduce the difficulty of generating in the parameter space and to improve efficiency and effectiveness. Our ablation experiments include detailed studies on behavior embedding and parameter representations. The hyperparameters for our model are based on stable diffusion, with minimal impact on results. All models were trained using a single set of hyperparameters without tuning.

• Q6 and L6: Real-world deployment challenges: Deploying policies trained in simulators to real-world environments posed challenges due to the changing terrain and stability. We addressed this by adjusting environment randomness during policy dataset collection, making generated policies more adaptable to real-world dynamics.

• Q7 and L8: Robustness to noisy trajectories: As mentioned in Line 237, we mention that we use the common setting in adversarial RL by adding Gaussian noise with a standard deviation of 0.1 to actions in test trajectories, which can demonstrate the effect of perturbed trajectories on the generated results within a reasonable range.

In summary, we are grateful to Reviewer YLrf for the detailed feedback. Besides the questions that can be answered within our paper, we will include new discussions from the rebuttal in the appendix to enrich our paper. We hope our response sufficiently addresses your concerns. We eagerly anticipate further discussions.

Dear Reviewer YLrf,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

Dear Reviewer YLrf,

As the discussion period draws to a close in 12 hours, we are delighted to have received positive feedback from the other four reviewers and are very eager to ensure that our response has adequately addressed your concerns as well.

We believe that the clarification in our paper would solve your concerns on input demonstrations/overfitting/computational cost/ablation/robustness, along with the additional results and discussions in rebuttal could solve your questions regarding experiments/scalability/real-world deployment.

We deeply appreciate your contribution to the community through your review. Your insights are valuable to us. We eagerly await your response.

Warm Regards,

Paper 8467 Authors

We are grateful to Reviewer 8Q7F's acknowledgment of the novelty, empirical results, and presentation of our paper. Your feedback is very helpful in improving the quality of our work. Below are our detailed responses to each of your questions:

The generated policies seem unstable. The generation by diffusion models are diverse but not stable.

The instability in the generated policies primarily arises from two factors:

1. We use trajectories from the replay buffer for generation, which introduces instability in the results. Figure 13 in our supplementary material illustrates the correlation between the effectiveness of condition trajectories and the performance of generated policies. That is reasonable because failed trajectories may not provide sufficient information as prompts for generation.

2. For unseen tasks, the unfamiliarity and sensitivity in exploring the parameter space lead to greater variability in results. However, compared to other generalization methods that struggle to achieve optimal policies on unseen tasks, our approach is capable of discovering optimal policies in such scenarios.

How to evaluate which policy is best? Does it require testing all policies and finding which one is best?

To efficiently evaluate policies, we initially validate all generated policies with a single episode under 4 random seeds (only need 2 minutes). Subsequently, the qualified policies, sorted by trajectory length from shortest to longest (indicating the used steps for task completion), are tested over 10 episodes to report the final results following all the baselines. Although we need to evaluate all the policies, the GPU hours consumed are significantly less than finetuning required by baselines.

• The qualification rates are very weak (mostly less than 0.1) on unseen tasks except "drawer close". It's not practical. What if using expert demonstrations?

Figure 13 illustrates that the performance of generated policies on unseen tasks is strongly influenced by unfamiliar suboptimal condition trajectories. Using expert demonstrations can largely improve the qualification rate of generated policies on these "very weak" unseen tasks.

We further compared our method with the best baseline (Meta DT) fine-tuned using expert demonstrations. Our approach still outperforms baselines. We will include the experimental results using expert demonstrations in our paper.

• Why fix the initial locations during the training stage and finetuning/adaptation?

During training, we fix the initialization because we collect multiple policies from a single RL training, which can effectively reduce the computational cost of data collection (instead of changing the initialization each time to collect only one policy). The test trajectories used for finetuning/generating are derived from a single training buffer, thus having the same initialization. For our evaluation results, we conduct experiments in 4 randomly initialized environments over 10 episodes to obtain results. The results indicate that our generated policies do not only work on fixed initial locations.

• Similarly, why choose highly sub-optimal trajectories? If so, it may use offline RL approaches to finetune the baselines.

As mentioned in Lines 187-188, we select test data from the SAC training replay buffer within the first 0.5 million timesteps considering that expert demonstrations are not always available and using similar expert trajectories might not provide sufficient diversity. We compare our approach with multi-task offline RL methods: skills regularized task decomposition (SRTD). We train the policy using offline training data and then update the policy using test trajectories on unseen tasks. Offline multi-task RL performs worse than imitation learning methods and our approach demonstrates a significant advantage.

[1]Yoo, Minjong, Sangwoo Cho, and Honguk Woo. "Skills regularized task decomposition for multi-task offline reinforcement learning." Advances in Neural Information Processing Systems 2022.

• How to finetune the multi-task RL with fixed trajectories (10/50/100)?

Previous multi-task RL methods have poor performance on unseen tasks. We update the mixture of encoders and critic networks to improve its generalizability. We will include offline multi-task RL as baselines in our paper.

• What are the states or what would happen after the first success time?

In all simulator settings, the environment resets after reaching the maximum episode length. The agent's actions continue selected by the policy network after success. We maintain this environment setting consistent with all baselines.

We thank Reviewer 8Q7F for the detailed and valuable feedback. All the discussions and additional results will be included in the final version. We look forward to further discussions to ensure that our answers address your concerns.

Dear Reviewer 8Q7F,

Thank you for your time and effort in reviewing our work and providing insightful feedback.

Collecting data from fix locations and highly sub-optimal would hugely hurt the baselines' performance.

In our rebuttal results, we have demonstrated that our method not only excels in suboptimal data but also significantly outperforms all baselines when using expert demonstrations with random initialization. It's important to note that our method is not an imitation learning approach. Offline replay datasets are widely used in the evaluation of offline RL and meta RL, which is why we selected this evaluation metric.

We fully agree with your suggestion and will include experiments using expert demonstrations in the appendix to fairly compare our method with meta IL baselines.

It's expensive and unsafe to evaluate the policy (like real-world) while it's simpler to collect expert demonstrations with teleoperation systems.

In real-robot experiments as mentioned in our paper, we utilize the IsaacGym simulator to collect policies and trajectories and also evaluate generated policies before deploying them on the real robot, which ensures that we conduct evaluations at a relatively low cost.

There is no contradiction between using simulation for initial evaluation and using expert demonstrations for generation. If optimal real-world data is available, using it to generate policies would certainly be preferable. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still yield excellent results in real-world scenarios.

About real-world experiments, the paper seems only to provide some visualizations. What about the success rate and other baselines.

Thank you for your suggestions. Real-world visualization is the most straightforward way to observe the performance of policies, especially since designing success rate metrics for real-robot locomotion tasks is challenging as shown in prior works. Additionally, none of the baselines we listed include real-world experiments, and the network sizes used in those baselines are not applicable to real-robot scenarios.

We evaluate our generated policies against offline RL methods in low-speed testing and high-speed testing following [1]. The results demonstrate that the policies generated by our method exhibit better stability and performance on real robots.

[1] Margolis, Gabriel B., and Pulkit Agrawal. "Walk these ways: Tuning robot control for generalization with multiplicity of behavior." Conference on Robot Learning. PMLR, 2023.

We hope our responses address your concerns, and we are open to further discussion. Thank you once again for your valuable feedback.

Warm Regards,

Paper 8467 Authors

Dear Reviewer 8Q7F,

Thank you for your reply and understanding. We will update our paper accordingly based on your and other reviewers' comments.

We sincerely appreciate your positive comments and the discussions during rebuttal, which are of great help of improving our paper and also contribute to the community.

Best wishes!

Paper 8467 Authors

We sincerely appreciate the positive feedback from Reviewer Pxd2 on the originality, overall quality, and significance of our work. The valuable comments and suggestions from Reviewer Pxd2 are of great help to improve the quality of our work. Detailed responses regarding each problem are listed below.

All policies had to be evaluated in the environment to decide what these top policies are. This is an unrealistic assumption that weakens the significance of the result. The paper must analyze in more detail the distribution of performance between generated policies and compare the full statistics of all generated policies to baselines for a fair comparison.

The evaluation metrics we reported are chosen primarily for the following reasons:

1. Feasibility and efficiency of evaluation: Our method can generate one policy per trajectory, whereas previous methods can only train one policy through fine-tuning with multiple trajectories. As mentioned in our paper, the key difference is that we do not need to use trajectories for finetuning. Instead, we generate policies and evaluate them with one episode to determine their effectiveness. The final results are then reported after evaluation over 10 episodes, consistent with the baselines. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below. We also utilize IsaacGym simulator to evaluate policies before deploying them on the real robot, ensuring evaluation at a relatively low cost.

2. Effectiveness of all generated policies: We add a set of results in Figure 13 in our supplementary material to present the performance of all the generated policies compared with baselines. Since we do not actually need to use all the generated policies (just as we do not need to use all generated results), but only need to obtain one optimal policy to consistently complete the task, our method provides a new pathway for addressing unseen tasks. This is in contrast to baselines, which cannot achieve good unseen generalization even with finetuning. We also believe that using the generated multiple optimal policies to learn a mixture of experts for decision-making is a potential future direction.

A weakness of directly generating policy parameters is the challenge of scaling up to larger policy networks. For example, policies operating from egocentric visual inputs in 3D spaces typically have tens of millions of parameters. The core idea of directly generating policy parameters is likely incapable of scaling to this setting of 1000x more policy parameters.

This is indeed a very worthwhile discussion. Using visual encoders to handle image or 3D point cloud inputs is already a common approach. We can apply our method to generate actor or critic networks but with inputs processed through existing pre-trained visual encoders to manage different environmental inputs. Our paper covers a broad range of policy network sizes, from 2e4 to 16e4 parameters, which is among the widest ranges addressed by existing network generation methods. Additionally, we propose other potential solutions for addressing complex networks in our responses below.

Given that the results in Figure 11b already show performance suffering with increased policy size, how can Make-An-Actor scale to more complex tasks that require higher parameter count policies?

In Figure 11b, the decline in performance with larger policy sizes is likely due to the autoencoder's inability to provide effective parameter representations for large networks, which significantly impacts the effectiveness of the generated policies (for fairness, we did not modify any model parameters for large networks). When we scale up the hidden size of the autoencoder from 1024 to 2048, policies with 256 hidden sizes achieve very similar results to those with hidden sizes of 128. Therefore, an effective strategy for handling larger networks might be to use multiple autoencoders to encode different layers of the network separately. Alternatively, generating different layers or portions of larger policy networks tailored to specific task features could facilitate faster adaptation to various complex downstream tasks. We believe that exploring how to generate larger networks is a highly promising potential direction.

Thank you again for carefully reviewing our paper and providing very constructive suggestions. We will incorporate the above discussions into our final version. We hope that this addresses your concerns, and we are happy to engage in further discussion.

Dear Reviewer Pxd2,

Thank you for your prompt reply and constructive suggestions.

• In our GPU hours comparison, we have included the evaluation and inference costs. For very slower simulators, we strongly agree that filtering policies by selecting condition trajectories that are close to optimal may reduce evaluation time costs. Even if more interaction time is required, the evaluation cost in most environments is still significantly lower than the cost of fine-tuning.

• In our real-robot experiments, as discussed in our paper, we utilize the IsaacGym simulator to collect policies and trajectories and also evaluate the generated policies before deploying them on the real robot, which ensures that we conduct evaluations at a relatively low cost. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still achieve excellent results in real-world scenarios.

• Figure 13 presents results from the door unlock (seen) and coffee button (unseen) environments, both of which exhibit medium performance across all seen/unseen tasks.

• We chose to use trajectory length as it more accurately reflects the performance of the condition trajectories compared to the success condition. For example, two trajectories might both achieve success, but their lengths could differ, with shorter trajectories indicating closer to optimal and more efficient performance. Using trajectory length to evaluate policies also makes it easier to compare with condition trajectories.

• We fully acknowledge your suggestion, and we provide a table showing the generated policies' success rates across all condition trajectories for all seen/unseen tasks. We can see that on seen tasks, our method can generate more than 55% of policies that perform better than the best baselines. On unseen tasks, on average at least 35% of the policies perform better than the best baselines. This fully demonstrates the superiority of our method.

• Additionally, if we select trajectories with success signals for policy generation, in seen tasks, more than 82% of the policies outperform the baselines, and in unseen tasks, more than 50% do so. For environments where evaluation is challenging, using success trajectories can also more efficiently yield optimal policies. We will include these results as histograms in the final version of our paper.

In summary, we agree that the evaluation cost could become a weakness of our method when generating multiple policies. However, it is undeniable that the broader exploration of parameter spaces offers more effective solutions for policy generalization compared to prior works. We also believe that using these multiple policies for a mixture of experts could be a promising direction.

Thank you for acknowledging the novelty of our work and for providing such insightful feedback. We welcome any further discussions and greatly value the opportunity to continue improving our research.

Best Regards,

Paper 8467 Authors

Thank you for the response. While I still appreciate the novelty of the approach, like other reviewers, I am also concerned about the evaluation criteria for selecting the best policies. The GPU hours comparison in the rebuttal is not a representative comparison because for Make-An-Agent, this also includes evaluation costs, which could be prohibitively expensive depending on the environment, as in the real world or a slower simulator. Figure 13 in the rebuttal is also unclear. Which environment are these results for? Why compare episode length in the tasks? A clearer comparison would be to generate a histogram of success rates, not trajectory lengths, for the Make-An-Agent generated policies across all condition trajectories.

We sincerely appreciate Reviewer 957J for your acknowledgment of our idea's novelty and method's applicability. Thank you for your valuable comments and suggestions, which are of great help to improve the quality of our work. We carefully answer each of your concerns as below.

The reported metric (best generated policy performance, avg top 5 generated policy performance) both seem fudged. It seems like the authors are generating a large number of policies, and then picking the best generated policy out of them to get their numbers.

Regarding the reported metrics, we primarily consider the following perspectives :

1. Generate vs. Finetune: Our method can generate one policy per trajectory, while previous methods can only train one policy through fine-tuning with multiple trajectories. Even with multiple finetuning attempts to get the best result, no method can achieve a very high score. (The baseline results we report are the best models obtained during finetuning)

2. Fairness and Efficiency: The test trajectories we use are exactly the same as those used for the baselines. The difference is that we do not need to use trajectories for finetuning but generate policies and evaluate them with one episode to determine if the generated policies are effective. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below.

3. Usage: Our goal is to ultimately generate an optimal policy with few trajectories, similar to all previous policy learning methods, but our method achieves better performance. Although we obtained many policies, this paper does not further discuss what can be done with the large number of generated policies. They could potentially be used for learning the mixture of experts, which we believe is a promising future direction.

While the authors report the "qualification rate", i.e, the % of policies getting 100% success rate in test, I am not sure how this works. As they report the "best policy score", and since qualification rate is > 0 for all tasks, why is figure 5 (ours) not all 100%? What am I missing?

To efficiently evaluate policies and save on evaluation costs, we initially validate all generated policies with a single episode under 4 random seeds (consuming very little GPU time). Subsequently, the qualified policies, sorted by episode length from shortest to longest (indicating the time spent on task completion), are tested over 10 episodes to report the final results. Therefore, some policies may occasionally fail in certain episodes, resulting in a success rate that does not reach 100%. It should be noted that evaluating 10 episodes follows the standard metrics of all common baselines, ensuring fairness with other papers. We will add these detailed explanations of the evaluation process in the appendix.

The drop between top-1 and top-5 is concerning, since I cannot tell whether the authors got lucky finding a perfect policy from the long tail.

Based on the evaluation details provided above, this explains why the performance of the top 5 policies may differ from that of the best policy. The performance can fluctuate across multiple episodes due to the random initialization of the environment, which is a normal phenomenon also observed in single RL policy learning.

The authors don't mention the overall compute overhead and how it compares with the baselines, which I believe is a strong consideration for such a method. Please detail the compute budget for each of the baselines as well as your method. If possible, please release the dataset as well as the pretrained networks with your code.

In Line 186, we state the GPU hours for collecting policy data for training. In Line 425, 434, and 438, we mention the GPU hours for training each model. Our training time requires relatively few GPU hours compared to other methods (much fewer than single RL training). All of our datasets and pretrained models will be released to the community for further valuable research, which we believe contributes to researchers who may have limited resources.

Generating larger networks will be more difficult, so I wish the authors took advantage of any special properties of network weights while training.

This is a very insightful topic for discussion. In our method, we choose to encode the parameters of each layer of the network, leveraging the properties of the policy network. To handle larger networks, we believe several approaches can be considered: generating parameters for each layer individually, generating parameters for only parts of networks, or utilizing pretrained task representations to reduce the dimension of inputs to the network, allowing smaller actor networks to tackle complex tasks. These are all promising future directions worth exploring, and we will include this discussion in the appendix.

Thanks again for reviewing our paper carefully and providing very constructive suggestions. We hope the above resolves your concerns, and we are glad to have any further discussions.

Dear Reviewer 957J,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024

Dear Reviewer 957J,

Thank you for your constructive suggestions and understanding. We would like to provide further clarification on the concerns you raised:

In the real world, the majority of time and energy is spent on evaluating the robot in a real environment, which is a lot more expensive than simple GPU hours.

We completely agree with this point. To ensure a relatively low cost on evaluations, in our real-robot experiments, we utilize the IsaacGym simulator to collect policies and trajectories and evaluate the generated policies before deploying them on the real robot. Our experiments aim to demonstrate that policies generated and evaluated in simulation can still achieve excellent results in real-world scenarios.

For tasks where it is not feasible to evaluate in simulations, we try to discuss alternative methods inspired by other reviewers for utilizing multiple policies in decision-making, such as majority voting and using the mixture of experts, to improve the robustness and stability of real-robot decisions. We believe these are very promising directions and hope our work will inspire the community to explore more.

Release of full code and pretrained model library

As we have promised, we will release all datasets, pretrained models, and training code in our final version. We greatly appreciate your support.

Disambiguating between GPU hours needed for training (done offline) and rollouts/evaluation needed for model selection (done online).

Based on your suggestion, we will provide a detailed report on the computational costs associated with each part of our method. Additionally, we will include a separate discussion of the evaluation costs and alternative approaches in the limitations section.

Thank you again for your positive comments and valuable suggestions regarding real-world evaluation, which have greatly helped us further discuss the weaknesses and improve the quality of our paper. We appreciate the time and effort you’ve put into reviewing and the rebuttal process. We will supplement our paper based on all of these discussions.

Best wishes,

Paper 8467 Authors

We thank Reviewer U6yw for the positive comments to our writing and experiment thoroughness. Your questions are instrumental in helping us improve the quality of our paper. We have provided detailed responses point by point below.

Is there any way to roughly predict the performance of the generateed parameters without actually evaluating them?

Feasibility and efficiency of evaluation: We generate policies and evaluate them with only one episode to determine their effectiveness. The final results are then reported after evaluation over 10 episodes, consistent with the baselines. The GPU hours consumed by the inference process and evaluation for 100 policies are listed below. We ensure a very low cost for evaluation.

Is any of the statistics of the generated policy parameters useful? Is the variance of the generated parameters correlated with how "out-of-distribution" the behavior is? Is there any failure mode like certain type behaviors/demonstrations always lead to bad-performing generated policies?

Figure 13 in our supplementary material illustrates the relationship between condition trajectories and the performance of generated policies. It can be observed that when the condition trajectories are effective, the generated policies tend to perform better. When the behaviors used for synthesized conditions deviate significantly from the correct actions, the generated policies tend to fail. That is reasonable because failed trajectories may not provide sufficient information as prompts for generation. While condition trajectories do show some correlation with generated policies, it is difficult to use them as a metric to directly judge the performance of generated policies.

How does small perturbation in the behavior embedding affect the performance of the generated policy parameters. Is it possible to compose behaviors by messing around the conditioning of the diffusion model, which then gives a compositional policy?

We also show the impact of noisy trajectories on the generated results for unseen tasks in Figure 13, demonstrating a significant advantage over the baselines. For combined trajectories, our method can not generate compositional policies, due to the absence of multi-task compositional policies in our training dataset. However, this presents an interesting possibility that could offer new insights into exploring the parameter space for multi-task learning.

Could the authors report the performance distribution for all generations (as opposed to the best or top 5)?

As shown in Figure 13. Note that we choose average trajectory length as a metric because it more accurately reflects the effectiveness of the policies compared to the success rate. For example, policies with the same 80% success rate can exhibit significant differences in task efficiency.

For ablation of policy network, could the authors discuss how architectural difference (beyond hidden size; e.g., constructing policy as very small recurrent network or transformer) can potentially affect the results?

The choice of policy architectures determines the type of network parameter data collected for the training dataset. In our experiments, we used MLP policy structures with varying layer numbers, resulting in parameter counts ranging from 2e4 to 16e4 across three domains. The architecture variation did not affect the overall effectiveness in different domains. We believe that our method could also be applied to generate RNN or Transformer policies.

In real-world evaluation, could the authors provide more details about the behaviors and policies in training dataset and testing?

In our locomotion tasks, we collect the training dataset using the IsaacGym simulator. We train policies on flat ground to move forward stable without any obstacles and changes in terrain geometry, while randomizing dynamics parameters such as payload mass, motor strength, and gravity offset, to obtain policies under different environment settings. In the real-world deployment, we apply the generated policies to the Unitree Go2 Edu robot, issuing commands to complete behaviors such as sharp turns, obstacle avoidance on mats, and rapid backward movements. We will update the details of these real-world experiments in the appendix.

Given the proposed method being a generator of the policy (which can easily produce many models at a time), it would be also interesting to check the mixture of expert setups.

It is a valuable idea that can help us better utilize the large number of generated policies. We employed a straightforward approach by dividing the action space into over 20 discrete intervals. Then we performed majority voting to select the most frequently voted interval. The actions output by policies voting for this interval were averaged to determine the final action. We conduct experiments on three unseen tasks and evaluate using adversarial RL settings with random noise in the range of 0.1 added to actions on 4 random initializations. The results show that multiple synthesized expert policies can significantly improve robustness.

We sincerely appreciate your thorough and detailed feedback and hope our replies address all your concerns. All the discussions and experimental results will be included in the final version to further enhance our paper's quality. We look forward to further discussion.

Dear Reviewer U6yw,

Thank you for your thoughtful feedback and understanding! We further discuss the points you raised.

mixture-of-experts computational overhead

When using the mixture-of-experts approach for decision making, while it involves using multiple policies to output actions, the inference cost for these policies is very low. As a result, the computational cost is less than 1/10 of the baseline fine-tuning cost. We will include detailed statistics on this in the final version of our paper.

the statistics of the generated policy parameters in parameter space

Given the high dimensionality of the parameter space, we primarily compare the trajectories obtained from deployed policies. Directly averaging the generated parameters does not yield ideal results, mainly because the parameter distribution in the latent parameter space and the parameter space are not smooth.

the use of other architectures like RNN or transformer

If we use RNN as the policy backbone, and the parameter count of the RNN is not larger by an order of magnitude, it is feasible to use our model to generate RNNs. Our method directly flattens the network parameters into a vector before encoding and generating, without imposing special requirements on the network structure during the generation process.

We sincerely appreciate your positive comments and insightful discussions. The constructive suggestions for practical applications are crucial for improving the quality of our paper, and we will ensure that all discussions and new results are included in the final version.

Best wishes!

Paper 8467 Authors

Dear Reviewer U6yw,

As the Area Chair for NeurIPS 2024, I am writing to kindly request your attention to the authors' rebuttal for the paper you reviewed.

The authors have provided additional information and clarifications in response to the concerns raised in your initial review. Your insights and expertise are invaluable to our decision-making process, and we would greatly appreciate your assessment of whether the authors' rebuttal adequately addresses your questions or concerns.

Please review the rebuttal and provide feedback. Your continued engagement ensures a fair and thorough review process.

Thank you for your time and dedication to NeurIPS 2024.

Best regards,

Area Chair, NeurIPS 2024