We thank reviewer nrWg for their suggestions on improving the paper, and will take every of their points into consideration. Please find our respective response below:

The work is often not clear enough in which parts are coming from the framework proposed by Ferreira et al. It is not elaborated on what the exact differences [between "the plain T setup" and Ferreira et al.] are.

We have made the distinction clearer throughout the paper.

In our experiments, its necessary to recreate Ferreira at al’s method within our framework, since we need to leverage our highly parallelizable implementation of RL algorithms to enable experiments on Brax. This means the “plain T setup” differs in the implementation of RL algorithms. We added clarification to the corresponding parts of the text.

Further, in the experiments there is no clear comparison to the prior work and one is left to guess on the actual performance differences.

To our best knowledge, there is no work on meta-learning synthetic environments besides Fereirra et al., who only present results for CartPole and Acrobot. We benchmark against our implementation of their method in figure 3.

Figure 3 most often not show the learning curve for the proposed training setup.

With the IC learning curve in each plot, they easily become cluttered. However, we acknowledge that this might make the plot less convincing, and created a new version including the curves with improved clarity. In Pendulum, we could not think of a sensible curriculum to apply, and therefore did not use one. We added a clarifying remark in the figure description.

[In figure 3] often the 95% confidence intervals seem wholly missing for some or even all methods.

Because of the high computational cost, we only have one run per ablation in hopper, so no confidence intervals can be drawn.

Note that while meta-evolution may only use a single seed, evaluation during meta-training uses 64 seeds.

Lastly, in this ablation it seems rather strange to not include “no curriculum” as an option when showing the impact of different curricula.

We have included your suggestion about including “no curriculum” in the last row of figure 3.

The writing could be improved as often statements are made which are not easy to follow.

We have improved the clarity of writing in a lot of places, thank you for your suggestions.

Limiting an episode length to 1 seems to fall from thin air without any comparison if maybe a slight longer episode of 2 or 3 steps might perform better or not.

We included empirical results that compare the performance when using different episode lengths in the appendix.

Questions:

Am I correct in understanding that learning an SE then is simply supervised learning of a reward function for particular policies? Would it not suffice to simply fix the initial distribution?

We kindly ask the reviewer to clarify their question. During meta-training there is no direct access to the reward function or value function of an expert agent. Instead, the environment is optimized to maximize fitness. If they mean that the learned synthetic reward is similar to the optimal state-action value function, we are happy to point them towards our comprehensive comparison between both in the appendix of the revised version of our paper.

Learning the initial distribution is critical to the success of the method. If you set the initial distribution to be that of the original environment, there are a lot of states which are never seen by the agent, and to which it cannot generalize. Parameterizing the initial distribution enables the CB to select observations which are relevant to learning the task encoded by the evaluation environment, and can therefore help with training speed.

How did you set the hyperparameters of your method?

We manually set the outer loop hyperparameters informed by exploratory experiments, since the experiments are very computationally expensive. The hyperparameters of the inner loop algorithms were set arbitrarily to reasonable values, and remains very stable to their combinations. We included this information in the Hyperparameters section.

How much tuning is necessary to achieve good results?

Our approach is relatively robust to outer loop hyperparameters. For example, we use the same hyperparameters for all Gym and small Brax environments. The exception is Pendulum, which needs a more stable fitness evaluation due to the very random nature of returns induced by favorable or unfavorable initial states. This leads to an increased number of rollouts per population member, and thereby a decreased population size due to memory constraints.

We hope that we could answer your questions and improve our paper by addressing your concerns. Please reply if anything remains unclear.

We hope we have addressed the remarks of the reviewer and if so, that they would consider updating their score. We revised our paper such that

• figure 3 includes all of the reviewers suggestions regarding its clarity
• the difference between Ferreira et al. is made explicit in the relevant parts of the text
• we include an empirical comparison of different episode lengths (figure 6)
• we visualize the initial distribution of the SCBs in the appendix (figure 10)
• statements are cleared up and include references to the relevant parts of the paper

We thank reviewer ABpb for their insightful comments and are happy to improve the paper by addressing their points.

The optimal reward of the contextual bandit seems to be the optimal value function at the context [...]. What is the benefit of meta-learning this?

When setting the synthetic reward (SR) to be the optimal value function of the target environment, agents trained in the SCB are guaranteed to transfer. However, the SR does not have to correspond to the optimal value function to achieve a high performance. RL agents that learn the value typically aim to choose actions which maximize it. This means that if $argmax_a Q(s,a)$ is equal between SCB and target environment, the behavior of trained agents should be the same. Thus the SR can be interpreted as a shaped reward. This has several advantages such as increased training speed, since the SR has an influence on the parameter updates. We acknowledge that this point has not been sufficiently addressed in the paper, and have added a dedicated part of section 3 (see "Synthetic Environment Setup"), as well as an empirical comparison to the value function in the appendix.

Important claims made at the beginning of the paper, such as improved wall-clock time, are not substantiated.

We show a significant increase in simulation speed and an example of wall clock training curves for halfcheetah in figure 1. We added more information on these experiments to the caption.

The results, do not demonstrate that [training in the SCB] provides a benefit over training in the real environment.

The significant increase in simulation speed can be leveraged in several ways, e.g. in neural architecture search. Even if training on the evaluation environment instead of the SCB achieves a higher reward, the SCB can be used for rapid pretraining. The benefits of synthetic environments in potential downstream applications are demonstrated in section 6.

The cost of [a] sweep can be much lower than the cost of the hyperparmaeter sweep and subsequent meta-learning of the synthetic environment itself.

You are right that running a hyperparameter sweep is less cost efficient than meta-learning the synthetic environment. However, it has not been our goal to train a single well-performing agent, but to train the synthetic environment itself. This holds value since it can be used in a myriad of downstream tasks, for example those shown in section 6.

Your setup involves many hyperparameter choices in both the inner and outer loop. These design decisions are not abalated.

The inner loop hyperparameters are sampled from a large range of sensible values. The performance on Brax tasks with fixed hyperparameters is shown in figure 1. Including parameters outside of this range would often lead to numerical instabilities, which is why we chose not to include it as an ablation. Since our experiments are very computationally costly, we aimed to ablate the most relevant and newly contributed outer loop parameters in figure 3. We would kindly ask the reviewer to clarify what they mean by design decisions. What other parameters are of interest to them?

Questions:

Section 2 (Curricula): you state that you learn a curriculum in this section, but in contribution 1 it seems that you in-fact design a curriculum.

In section 2 we state that "we use meta-optimization to discover state reset and state-action-dependent reward functions maximizing downstream learning progress". The reset and reward functions are not inner loop curricula. The curriculum mentioned in our contributions refers to a curriculum on the outer loop. We changed the paragraph in the background section such that this becomes clear.

Section 2 (Synthetic Data): Why not just use the real data [instead of synthetic data]?

In many cases synthetic data is used because it is cheaper to generate. After meta-training, this is also the case for our SCBs (see figure 1). We added this motivating example to the relevant paragraph in section 2.

In your experiments, is the meta-learned synthetic environment tuned to a specific real environment?

We are unsure what the reviewer means by this question. In our experiments, we train a separate synthetic environment per target environment.

Results, figure 3: why is there a discrepancy between the evaluation of the ablation and the results in Figure 2. Specifically for hopper?

Figure 2 shows the results of meta-training with sampled inner loop algorithms, while Figure 3 shows a fixed inner loop algorithm (see column title). We added this information to the caption to clear up any confusion.

I do not see how synthetic environments provide any interpretability beyond what a learned value function can provide.

When given a value function with the same parameterization as the synthetic reward, it can be analyzed in the same way. However, the two functions differ (see above).

We hope to have answered the reviewers questions and are happy to engage in further discussion.

Thank you for the reply, I see this paper more favourably given your responses but I still think it is below the acceptance threshold. Here is some of my feedback regarding individual comments:

Meta-learned synthetic reward as learned shaping

This is an interesting perspective, and one that I think has merit. But it raises the question: what is the benefit of meta-learning the shaped reward in this way vs some other meta-learning approach to reward shaping?

improved wall-clock time

It is good that training in the SCB lowers the number of env steps needed to learn a performany policy. It also makes sense that the cost of running the SCB is lower because no state transition is needed. But, I think the paper should demonstrate the meta-learning the SC + learning in the environment is comparable to baselines. Otherwise, an argument has to be made that the meta-learned SCB provides other value (beyond, say, an optimal value function).

In our experiments, we train a separate synthetic environment per target environment.

This is what I was wondering, because one of the benefits of meta-learning is generality. It needs to be demonstrated that meta-learning the SCB + learning in the SCB provides some benefit over learning in the original env alone.

We thank reviewer XskH for their comments and suggestions, and invite them to read our response to their points below.

A contextual bandit (CB) can be converted to a Markov decision process (MDP), but not vice versa, because CB is stateless. This means that the SCB model may not be able to accurately capture the dynamics of more complex environments, such as Go, where state is essential for planning and decision-making. It is also unclear how synthetic contextual bandit can work for partial observation environments, where the agent does not have access to all of the state information.

While environments with temporal dynamics are better modeled by MDPs, training a agent on a CB can imprint the policy of the optimal agent on the MDP. This can be done by setting the reward function to the value function of the optimal agent, and sampling the observation space sufficiently. By maximizing the reward in the CB, the agent will therefore maximize the value in the real environment. We recognize that this point was not addressed clearly enough in the paper, but the method remains general to MDPs, including Go. We added an explanation of this relation in our methods section (see "synthetic environment setup").

Partial observations can be handled in the same way, since the CB can present partial observations.

One important paper is not cited nor discussed, which is closely related: Ferreira et al. (2022)

We mistakenly referenced an earlier version of Ferreira et al. (2022), and have now changed it in favor of the version published in ICLR. We benchmark against their method in figure 3 and show clear improvements resulting from our contributions.

There is no comparison with the state-of-the-art results on the environments/tasks in this paper.

We are using common SotA RL algorithms in the inner loop, and compare to the current SotA work on meta-learning synthetic environments by Ferreira et al. For a comparison to training directly on the evaluation environment please refer to figure 1.

The motivation for using a synthetic environment is not entirely clear. In the first paragraph of the introduction, the authors mention the biological fact that organisms can learn from artificial stimuli. However, it is not clear how this relates to the use of synthetic environments in reinforcement learning.

In the biological analogy, artificial stimuli correspond to the synthetic environment, while the real world corresponds to the evaluation environment. Before birth, the agents are pretrained in a way that transfers well to the real world, similar to our synthetic training, which transfers well to the evaluation environment.

Besides this biological analogy, we list several motivations for training a synthetic environment, including computationally efficient simulation, faster RL training, no need for hyperparameter tuning, and the application to several downstream tasks.

Additionally, the authors do not explain why learning or meta-learning RL policies is not sufficient.

Learning RL policies on an evaluation environment directly is sufficient for achieving a well-performing agent. However, training synthetic environments has several additional benefits, such as enabling the downstream tasks we present in section 6.

The colors in Figure 5 should be revised so that synthetic curves have the same color, and the same for the real curves. This would make the figure easier to read and interpret.

We have changed the colors of figure 5 to be consistent.

We hope to have sufficiently addressed the reviewers points. If anything remains unclear, we are happy to continue the conversation.

We hope our comments have addressed the reviewer's concerns and if so, that they would consider updating the score. Also note that we have updated a our manuscript by

• including their suggestions on figures and references, and clarifying statements across the whole paper
• dedicating a part of the method section specifically to show that training on an SCB can lead to the optimal policy of an MDP
• adding a comprehensive comparison between meta-learned reward and optimal value function on the evaluation environment

We thank reviewer rbXB for their feedback and their recognition of our extensive empirical study.

Some necessary background knowledge is not introduced in the paper.

As the reviewer summarized, we discover synthetic environments (contextual bandits) using a meta-learning with an evolutionary strategy. We do so by taking advantage of recent advances in hardware accelerated RL. Section 2 already contains background on 1- reinforcement learning (RL), 2- curricula in RL, 3- RL from synthetic data, 4- meta-optimization, 5- evolutionary strategies, including evolutionary meta-learning and 6- hardware-accelerated RL. We argue this covers all required background for our paper. If the reviewer believes we omitted something, we ask that they please let us know what specifically, so that we may add it to the revised manuscript.

Although this paper gives empirical proof for the statements, the intuition behind the ideas are now explained.

We would appreciate if the reviewer could provide additional examples on which ideas they find lack intuitions. They do provide one example, which we address below, but they seem to imply there are more.

[...] Without any explanation, I am not convinced with the claimed results.

We are not sure what the reviewer means here. Is the claim that because our intuitions are not sufficiently explained, the experiments are invalidated?

The authors claimed that the state transition probability is not necessary to train well-performing agents. I read the experimental result, but I cannot understand the intuition behind it.

We have updated the manuscript to make the connection between single step MDPs and contextual bandits more explicit, and give an intuition on why the optimal policy of any MDP can be found by training on a CB. We have also clarified various sections throughout the paper. We additionally added a more comprehensive discussion of our choice to use contextual bandits to the method section, see "Synthetic Environment Setup".

If the reviewer still finds that some sections are hard to follow, we ask that they please specify which, so that we may improve the paper further.

We hope we could address the reviewer's questions by updating our manuscript, which now includes a dedicated part of the method section to justify our usage of contextual bandits, including the intuition why training on CBs leads to successful transfer to MDPs. Additionally, we added several insightful visualizations of SCB components in the appendix, such as the initial state distribution (figure 10), and the synthetic reward function as compared to the value function of an expert agent (figures 7-9).

If any questions remain, we kindly ask the reviewer to continue the discussion so that we may address them. If this is not the case, we politely request for them to update their score to reflect our improvements.