HyPoGen: Optimization-Biased Hypernetworks for Generalizable Policy Generation

W3: Thank you for your feedback on the generalization claims and figures. The plotting format in Figures 5–9 follows the conventions of previous works like HyperZero. However, we understand your concern that these qualitative figures may not clearly demonstrate generalization performance.

To better address this, we report the mean rewards for MuJoCo tasks, as well as the success rate and episode length for ManiSkill tasks, under the suggested two conditions: "within one standard deviation of the training distribution" and "outside one standard deviation of the training distribution". These metrics are now presented in Table 26, 27, 28, and 29 of the revised manuscript.

The updated results show that HyPoGen outperforms all relevant methods in both in-distribution and out-of-distribution settings across most tasks. This provides stronger evidence of the proposed method's superior generalization ability compared to existing baselines. We hope these additional results clarify our claims and provide a more robust demonstration of HyPoGen’s generalization capabilities.

Q1: We conducted experiments with varying numbers of optimization steps K, with the results presented in Table 20 in the supplementary material. On the cheetah task, the optimal value for K is 8. As K increases beyond this point, the final rewards decrease, indicating potential overfitting.

Q2: The number of hyper-optimization steps (K) is chosen based on experimental results shown in Table 20. We choose K = 8, which provides the best performance, and used this value across all tasks. However, this value can be fine-tuned depending on the specific task or dataset.

To evaluate how the optimal number of steps varies in a more out-of-distribution scenario, we conducted additional experiments using only 10% of the training specifications (compared to 20% in the paper). The results, presented in the table below, show that the best performance was achieved with K = 5, which is smaller than the optimal K for tasks with more training data. This suggests that smaller datasets are more prone to overfitting, necessitating fewer optimization steps to mitigate this issue.

Q3: Yes, HyPoGen works for larger policy models and harder environments. We experiment with various sizes of policy networks, as detailed in Table 18 of the supplementary material. The best-performing configuration is a 3-layer policy network. As the number of layers and parameters increases, the learning difficulty of the hypernetwork also increases, which results in decreased performance. Nevertheless, HyPoGen consistently outperforms HyperZero in terms of rewards.

We appreciate your insightful feedback. We hope the additional variance analysis, computational cost comparisons, clearer generalization metrics, and experiments on optimization steps and larger models help address your concerns. Please feel free to let us know if you have more questions that can help improve the final rating of our work.

Dear Reviewer 2Bw1:

Thank you for the detailed feedback. We are pleased to hear that you found the method well-presented and motivated, the experimental results compelling, and the concept of neural gradients for policy generation interesting. We will address your inquiries and concerns on variance analysis, computational trade-offs, and generalization metrics point by point in the following responses.

W1: Thank you for pointing this out, the high variance in our results is due to the inclusion of failed episodes in the calculations, which is an established practice in previous work such as HyperZero. These failed episodes typically have low rewards in the MuJoCo environments or high episode lengths in the ManiSkill environments (around 200 episodes compared to the usual 10–50 for successful episodes), which significantly increase the overall variance.

To address this, we have updated our results to report the standard deviations based only on success episodes. Please refer to Table 24 and 25 in the updated manuscript.

Furthermore, we calculated the probability that HypoGen surpasses all other baselines using formulas from this resource, based on the sample sizes of 50 for MuJoCo and 100 for ManiSkill. These probabilities, presented separately for MuJoCo and ManiSkill, clearly show that HypoGen outperforms all baselines in most cases.

We hope this clarification addresses your concern regarding the comparison with previous methods.

W2: Thank you for raising the important point about computational costs and the trade-off between performance and efficiency. Below, we provide a detailed comparison of computational time between HyPoGen and HyperZero, along with additional experiments to analyze the benefits of parametrizing the full chain rule.

The exact computation costs of the proposed methods and HyperZero are listed below,

While the proposed HyPoGen is slower in the weight generation process, this process only accounts for a small fraction of the total computation time, as the majority of the time is spent on rollout trajectories, which are identical across methods. Furthermore, the policy networks generated by HyPoGen can be reused across multiple trajectory rollouts, making the additional weight generation time non-critical in practice.

To address your suggestion about simplifying the hypernetwork by replacing the full-chain rule modeling (Equation 6/7) with a black-box approach (Equation 5), such as a simple MLP, we conducted experiments on the cheetah task. The results are summarized below:

These results demonstrate that replacing the proposed hypernetwork with a simple MLP improves performance over HyperZero, indicating the utility of treating the update as a delta from the previous step. However, explicitly modeling the full chain rule with the proposed hypernetwork leads to further improvements

In terms of computational cost, the black-box hypernetwork achieves performance comparable to a three-layer HyPoGen but is slower in weight generation. This observation underscores the efficiency of parametrizing the full chain rule.

We hope this analysis addresses your concerns and provides clarity on the trade-offs involved.

Dear Reviewer RPpm:

Thank you for the thoughtful feedback and for recognizing the novelty, clear presentation, and comprehensive experiments of our work. The majority of your comments revolve around the comparison between the neural gradient and the actual gradient, as well as exploring the performance of our model when trained on ground truth gradients. Next, we address these points as follows:

W1: We calculated the cosine similarity between the neural gradients and the corresponding ground truth gradients. A value closer to 1 indicates that the two gradients are more similar in direction. The similarity of parameters after each update is shown in the following. Empirically, we found the similarities to be around 0.366. This suggests that while the neural gradients positively correlated in direction with the ground truth gradients, they differ significantly in actual values. This implies that the neural gradients capture the essence of the exact gradients but are far more effective in updating the parameters given that the neural update process has only 8 steps.

W2: We have incorporated a discussion of learned optimizers, including the work you mentioned (https://arxiv.org/pdf/2209.11208), in Line 155, Related Works section to provide additional context.

Regarding the question about directly training the hypernetwork on true gradients so that it explicitly mimics backpropagation:

We conducted experiments in the cheetah environment to explore this idea. Specifically, we used PyTorch's autograd function to compute the ground truth gradients of θ after each update and supervised the hypernetwork’s output with an L2 loss. The results are summarized in the table below. As shown, the HyPoGen model trained on ground truth gradients performed poorly in the cheetah environment. This aligns with our discussion in Sections 4.2 and 5.4, where we mentioned that the ground truth gradients are highly data-dependent and noisy, making them unsuitable for effective training.

We appreciate your detailed feedback and believe that these additions enhance the rigor and comprehensiveness of our work. We look forward to your response and further discussions.

Dear Reviewer JcCr:

Thank you for the constructive feedback.

We are grateful for your recognition of the novelty of our hypernetwork architecture and the effectiveness of our design choices. Your acknowledgment of our comprehensive experiment results and well-structured presentation is also highly appreciated. Next, we provide the required clarifications regarding the training process and data distribution and more detailed explanations in the following.

W1 & Q1: We apologize for the confusion regarding the training process. The training process is indeed end-to-end, and we have made this clearer in Line 339, Sec. 4 of the revised version.

W2 & Q2: Thank you for raising this valuable question. For your first question, in our experiments, the other hypernetwork parameters were kept fixed and were not retrained when varying the initial value of $\theta_0$. Regarding your follow-up question, we would like to clarify that the learnable initial weight $\theta_0$ is designed to better adapt in conjunction with the trained hypernetworks. Even though the hypernetworks can perform gradient descent with a random initial $\theta_0$ without retraining, the BC loss associated with our learnable $\theta_0$ is significantly lower compared to a randomized $\theta_0$.

To illustrate this, we present the average BC loss after each update for randomized $\theta_0$ and learnable $\theta_0$ for the Cheetah task below.

This result demonstrates that our model achieves optimal performance with the trained learnable $\theta_0$ while also exhibiting the ability to properly update random initial parameters.

Q3: We apologize for any confusion. The task specification does not involve random sampling but instead uses evenly spaced values across the range. For instance, in the Cheetah task, all the specifications used during training and testing are defined as np.linspace(-10, -0.5, 20) + np.linspace(0.5, 10, 20).

Regarding the source-task partition, we uniformly sample only 20% of the specifications to use as training tasks, with the remaining 80% serving as testing specifications (as detailed in the “Data Collection and Evaluation Protocols” section of the main text). Thus, for the Cheetah task, we used only 8 specifications for training, which is quite sparse compared to the 32 test specifications.

While it is possible for some test tasks to have similar source tasks, this is not the case for the majority of test tasks due to the limited portion of the training sets.

Q4: The latent dimension is 256 across all experiments.

Once again, thanks for your constructive feedback. Please let us know if there are more comments that help improve the quality of our paper.

Dear Reviewer 49Ju:

Thanks for your valuable feedback.

We sincerely appreciate your positive remarks on our paper's presentation and the recognition of significant performance improvement of our method. Since the major concerns are about more extended experiments for more diverse settings and additional variance results, we will address your inquiries and concerns point by point in the following responses.

Q1: Thank you for the insightful question. Theoretically, our hypernetwork design can accommodate both discrete and continuous action spaces. This flexibility is achieved by using one-hot encoding to convert discrete action predictions into a continuous format, making the underlying network operations similar for both action types.

Then the main distinction lies in the training objective, specifically the loss function used. For continuous action spaces, as outlined in Eq. 3, we use an L2 loss function. For discrete action spaces, however, we employ CrossEntropy loss. To provide more clarity, we compare the gradients of the two loss functions below:

• Gradient of CrossEntropy Loss (CE): $\frac{∂L}{∂z} = σ(z) - y$
• Gradient of L2 Loss: $\frac{∂L}{∂z} = 2 (z - a)$

where $\sigma$ represents the softmax function, and $y$ is the one-hot label of action $a$.

To further support our claim, we conducted experiments on the CartPole task within the MuJoCo environment. In this task, the objective is to maintain the balance of a standing pole by choosing between pushing the cart either left or right. We varied the pole length from 0.3 to 1.2 and compared the performance of our model against HyperZero. The experimental results are presented in the following.

The results above align with our theoretical analysis, demonstrating that our Hypernet outperforms traditional hypernetworks and effectively handles discrete action spaces without any issue.

Q2: Thank you for pointing this out. While the tasks may appear slightly modified at first glance, they are more difficult than they seem. As shown in Table 2 of the main paper, all baseline methods struggle with the ManiSkill tasks, even though these tasks differ by only a single parameter. Transferring between them proves to be quite challenging.

To further analyze the limitations of our methods, we conducted additional experiments in the Meta-World environment. Specifically, we trained our hypernetworks on three Meta-World tasks related to buttons: "button-press," "button-press-topdown," and "button-press-wall." We then evaluated our method on a novel task, "button-press-topdown-wall." For all these tasks, we used instruction embeddings from a CLIP-text encoder as the condition for the hypernetworks.

The table below reports the average episode return over 250 episodes.

From these results, we can observe that policy networks generated by HyPoGen is much better than HyperZero, which demonstrate the generalization capability of our approach to more distinct tasks in a complex environment like Meta-World.

Q3: Thank you for your valuable suggestion. In addition to reporting the average success rate, we now also report the variance of the success rate in the Table 23 of the revised manuscript. We can observe that the variance of our proposed are comparable with previous methods. This additional information helps to better understand the stability and robustness of our approach.

To recap, we demonstrated that our approach can effectively handle discrete action spaces, showing superior performance in the CartPole task compared to HyperZero. Additionally, we provided further experiments in the Meta-World environment, demonstrating the generalization capability of HyPoGen to more distinct tasks. Lastly, we included the variance of success rates to highlight the stability and robustness of our approach compared to other methods.

Thank you again for your constructive feedback. We hope these additional experiments and analyses address your concerns and help improve the final rating of our work. Please feel free to let us know if you have any further questions.