MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces

We thank reviewer pnae for their detailed reading of our manuscript, finding our method reasonable and our experiments well-designed and supporting our conclusions. We now address the reviewer’s concerns.

Novelty / contribution concerns

We acknowledge the reviewer’s concerns regarding the simplicity of our baselines and the contribution of our approach.

First, the reviewer rightfully pointed out that Online-ALP cannot estimate competence over unseen goals. However, we would also like to remind the reviewer that our literature review on LP estimators in Section 2.2 and Appendix B highlights that: 1) to this day, the baselines we implemented in our paper (including Online-ALP) are the only approaches that exist for estimating LP over discrete goal spaces and 2) the baselines based on periodic evaluation of the competence over the full goal space are intractable in large goal spaces.

Second, the reviewer mentions that our main contribution is the introduction of a parametric LP estimator able to generalize from seen goals to unseen ones. However, while few prior works studied the use of neural networks to learn latent representations of continuous goals, our work is the first to study parametric LP estimators and their generalization abilities for natural language goals (which have become ubiquitous in the era of LLM agents). Moreover, Appendix D.1 shows that learning such an estimator is not trivial. In particular, we show that learning a neural network on top of a fixed and pretrained embedding model totally fails to build semantic relationships useful for precise LP estimation. Our experiments show that MAGELLAN successfully estimates and generalizes LP by leveraging and adapting the embedding abilities of pretrained LLMs.

Explanations on Little-Zoo

We acknowledge that the section explaining Little-Zoo in the main paper (Section 3.4) is fairly short, with details left to Appendix A. As the reviewer suggested, we propose to improve this section, especially to explain how goals are constructed. In particular, the reviewer’s comment made us realize our explanation may be confusing as it led to a misunderstanding of two fundamental properties of our environment: 1) goals are not an instruction alone but rather the combination of an instruction and the description of a scene initialization (i.e. objects accessible). For instance, here is a feasible goal: “Goal: Grow deer. You see: baby deer, bookshelf, water, tomato seed.”. And here is an impossible one (water is needed to grow the seed): “Goal: Grow deer. You see: baby deer, bookshelf, baby lion, tomato seed.”. 2) Little-Zoo has multiple categories of objects (i.e. plants, herbivores and carnivores) and each category requires a specific sequence of actions to grow one of its objects.

Consequently, estimating (and generalizing) well an agent’s LP in Little-Zoo not only requires one to understand the inner families of objects but also necessitates one to discern what makes an instruction possible or not in a given scene (i.e. discovering the tech tree).

We give all these details in Appendix B, including the optimal actions sequence to solve the possible instructions, but we will update Section 3.4 so that Little-Zoo’s natural complexity is easier to grasp.

Essential References Not Discussed

We thank the reviewer for identifying references not discussed. In particular, UED approaches are Automatic Curriculum Learning (ACL) methods that do not assume a discrete set of goals to sample from and that, to the best of our knowledge, do not make use of Learning Progress. In Section 2.1, we discuss various ACL methods but do not provide an extensive review of them as our paper does not introduce yet a new ACL approach.

Indeed, we focus on 1) improving the estimation of Learning Progress on natural language goal spaces and 2) augmenting LLM agents with a metacognitive monitoring skill.

One can use this LP estimation (as we have done in Section 4.2) to scaffold an RL learner’s curriculum using an ACL method but we did not propose any contribution to the latter. As explained in Section 3.2, we reused for this prior approaches leveraging a Multi-Armed Bandit framework to sample goals according to their LP estimation.

We thus propose to add UED references to the ACL methods mentioned in Section 2.1.

We thank reviewer 3cqQ for their detailed review of our paper, finding our claims well supported by the experiments, the experimental protocol well-designed, and highlighting the interestingness of our section on automatic curriculum learning and our analysis of the LLM’s embedding space. We now address the concerns raised by the reviewer.

Single and limited environment

We understand the reviewer’s concern about the generalizability of our approach, given that only Little-Zoo is used for evaluation. As explained in the manuscript, Little-Zoo provides a controlled setting to rigorously assess MAGELLAN’s properties. Specifically, accessing the true underlying categories of objects allows us to verify whether MAGELLAN correctly identifies them.

Additionally, multiple reviewers' comments on Little-Zoo made us realize our explanations may have led to misunderstandings. Despite its name, Little-Zoo is neither trivial nor small—it consists of tens of thousands of goals. Crucially, goals are not just instructions but combinations of instructions and scene descriptions (i.e., accessible objects). Estimating an agent’s LP in Little-Zoo requires understanding object families and discerning what makes an instruction possible in a given scene (i.e., the tech tree). The combination of all possible instructions and scene initializations results in 25,000 goals, 80% of which are impossible. Such a goal space, with mostly infeasible goals, also naturally arises when using freely generated natural language goals with an LLM (e.g., in an autotelic agent), as many goals wouldn’t respect the environment’s dynamics. Finally, Little-Zoo builds on prior environments like WordCraft (Jiang et al., 2020) by introducing a key missing component: semantic relationships between goals.

Regarding the generality of our results, MAGELLAN relies on extracting structure from the semantic goal space. The LLM’s embedding space notably enables MAGELLAN to generalize (e.g., knowing which animals are herbivorous or that water is needed to grow seeds). MAGELLAN is designed for language-based goal spaces where LLM embeddings are most effective. In environments with non-language goals or sparse goal structures, the LLM’s utility is reduced, limiting generalization. In the worst case, MAGELLAN is expected to perform similarly to Online ALP.

To further support the broader applicability of MAGELLAN, especially on more complex natural language goal spaces, we conducted additional experiments. We evaluated MAGELLAN’s ability at estimating a learner’s competence on math problems from the OpenR1-Math-220k dataset. Here, MAGELLAN must deal with highly non-trivial problems and in particular identify what is the type of each problem to generalize its competence estimation. We focus on Algebra, Number theory, and Geometry, leading to more than 20,000 problems. Given the limited time allowed by the rebuttal period, we did not train an RL learner but rather simulated one that has different learning dynamics on each problem type (i.e. how fast its success probability increases). Similarly to our experiments in Section 4.1, we compare the competence estimation of Online-ALP and MAGELLAN (using Qwen2.5-0.5B) when problems are sampled using an Uniform curriculum. The code can be found on our anonymous repository: https://github.com/ghjnkmjl745678/MAGELLAN_ICML/blob/main/math_experiments/.

Figure “sr_estimation_math.png” from this repository shows that MAGELLAN accurately estimates each problem’s success probability. This indicates that it is able to find each problem’s type and generalize its competence estimation within each type. In comparison, Online-ALP leads to poor competence estimations given the large number of problems.

We would like to perform the same experiment with two other LLMs (Flan-T5 80M and Flan-T5 248M) as performed in our additional experiment for reviewers 9oss’s rebuttal. Unfortunately, our cluster is currently under maintenance but we should be able to provide these results within the coming days. If the reviewers kindly accept to respond to our rebuttal, we will provide these results.

Table 1

We agree with reviewer 3cqQ that Table 1 (and Table 4) need more explanations. We proposed changes and explanations to add to the manuscript in our rebuttal to reviewer s63G.

Generalization abilities

While Table 2 only shows the average error over several evaluations performed throughout training, Appendix D.4.1 and in particular Figure 16 shows the evolution of the error on the held-out test set. One can see that both the observed and estimated success probability of Online-ALP on “Grow Carnivore” goals is near 0. One can also see that this is the only case where this phenomenon happens (i.e. the observed success probability is greater than 0 on all the other plots). We therefore can affirm that this effect only affects the generalization results of Online-ALP on the “Grow carnivore” category.

We thank reviewer s63G for their in-depth review and comments on our work, finding the claims clearly supported by our experiments, as well as highlighting the novelty of our method. We now provide answers to the comments and questions asked by the reviewer.

Table 1

We acknowledge that Table 1 (and by extension Table 4) currently lacks an explanation on the unit and the criteria used to assign each method’s score. Note that this was also raised by reviewer 3cqQ. We propose to update the manuscript with the following changes. First, we will replace “+++” by “high” and “ ” by “low” in the Efficiency column. Then, we will add the following explanation providing the unit and criteria:

For the Efficiency property, we consider a method’s efficiency as “high” if it does not require any additional evaluation (i.e. it only uses the performance observed on goals sampled), and as “low” otherwise.

For Competence transfer tracking, the “+” are given according to this evaluation:

• absence of +: the estimated competence is updated only on sampled goals
• +: the estimated competence is updated on a predefined goal subset the sampled goal belongs to
• ++: the estimated competence is updated on a dynamically learned goal subset the sampled goal belongs to
• +++: the estimated competence is updated on all goals

Modification of the embedding space

As the reviewer mentions, it is indeed insightful to analyze how the goal embedding space evolves over time. Such an analysis can already be seen in Appendix D.4.2. The chronogram we present there shows how the different categories are identified and how they interact. We put this analysis in appendices for now as it requires a full page.

Failure of the baseline

The reviewer asks for more clarification on why impossible goals are not included in the predefined categories given to baselines that use expert knowledge. These baselines are based on groups predefined in advance by human experts with a strong assumption: the goals within a group share the same learning dynamics and therefore the agent’s competence is the same over all goals in the group. If impossible goals were included, these groups would lose their relevance. Moreover, because of the large number of impossible goals, the average competence within each group will always be very close to 0. There will be no progress niche that the method can use to generate a curriculum, and performance will probably be very close to the random baseline. We will update the sentence at line 266 to make it clearer.

Generalization to other environments

The reviewer wonders how MAGELLAN would perform in other environments, in particular with a sparser reward signal. As a similar question was raised by most reviewers, we provide a single response, included in reviewer’s 3cqQ rebuttal. This response notably shows new results on additional experiments launched according to reviewers’ comments.

Focusing on the reward signal sparsity property, we are not sure if the reviewer discusses the reward used by the RL policy or the signal used to train MAGELLAN and would like to remind that MAGELLAN assumes a goal-conditioned environment with a binary outcome on an episode (i.e. whether the goal has been reached or not). In Little-Zoo, the RL policy also uses a binary reward obtained on the final step only, but this is not mandatory (i.e. MAGELLAN does not make any assumption on the reward signal used by the RL policy).

We thank reviewer 9oss for their thorough feedback, finding our approach using an LLM to estimate LP innovative, highlighting the comprehensiveness of our experimental evaluation and acknowledging the effectiveness of MAGELLAN. In the next paragraphs, we answer reviewer 9oss’ concerns.

Single and limited environment

We appreciate the reviewer’s feedback and understand their concerns on our limited evaluation domain. As a similar question was raised by most reviewers, we provide a single response, included in reviewer’s 3cqQ rebuttal. This response notably shows new results on additional experiments launched according to reviewers’ comments.

LLM choice

We agree with the reviewer that studying MAGELLAN’s sensitivity to various LLMs would make our evidence more robust. We thus ran an additional experiment for our Section 4.1 where MAGELLAN’s error on 25k goals is measured with Flan-T5 80M and Qwen2.5-0.5B in addition to Flan-T5 248M (which is already in the paper). We show the results in the following plot: https://github.com/ghjnkmjl745678/MAGELLAN_ICML/blob/main/errors_littlezoo_goals.png.

Our results indicate that larger models lead to slightly more precise competence estimations in Little-Zoo. However, this difference is not significative as the LLM is finetuned by MAGELLAN to adapt its embedding space for competence estimation in the given environment. We will add these results to Section 4.1.

Lastly, the reviewer asks about scalability. In our experiments, we used LoRA (Hu et al., 2022), limiting the number of weights to finetune. Nevertheless, larger LLMs remain slower and more computationally consuming both at inference (i.e. to estimate the LP over the goal space and choose the next goal) and during finetuning. We will add this discussion to our conclusion.

Other methods and baselines

We thank the reviewer for their suggestion of additional references to discuss. Section 2.1 discusses goal selection approaches in general (including methods not using LP) and we would be happy to add these references (and the ones proposed by reviewer pnae on UED methods) in this section. Could the reviewer be more precise on which work we missed?

Failure modes

We thank the reviewer 9oss for their question. In our experiments, we observed no significant failure cases impacting the curriculum.

However, we identify two possible failures cases resulting of bad LP estimation:

• The LP is overestimated: In that case, the goal will be quickly sampled by the multi-armed bandit selector and its LP estimation will be reduced.
• The LP is underestimated: In that case, we must wait for the exploration mechanism in the multi-armed bandit goal sampler to eventually sample the goal, enabling MAGELLAN to adjust its LP estimation. This is the worst case among the two.

Additionally, applying MAGELLAN to a very large discrete goal space with no semantic structure would lead to poor performance (i.e. similar to Online-ALP). Moreover, we do not explicitly study the case where a goal semantic is not aligned with common knowledge internalised by the LLM (e.g. in Little Zoo, a rabbit that would act as a carnivore). However, as shown with the initial embedding space of Flan-T5 248M (Figure 5.a), MAGELLAN will progressively move it from the herbivore cluster to the carnivore one throughout training.