MCU: An Evaluation Framework for Open-Ended Game Agents

LLM diversity on task generation may be limited

Thank you for this thoughtful observation. To mitigate prompt-induced bias and encourage diversity in LLM-generated configurations, we explicitly design our prompts to promote variability in initialization elements such as biome, weather, and player state (Lines 49–54).

We validate our current prompt through:

• Case study: For the task “craft a crafting table”, we ran 10 generations. The results exhibited wide variation in commands (e.g., /time set day, /give oak_log, /setblock blue_bed), including different wood types like birch, oak, and spruce.
• Quantitative analysis: For 5 randomly selected tasks, we conducted 100 generations each. The item-level overlap with few-shot examples was only 2%, and on average, 89% of the commands in each task were unique.

We will also keep improving our prompt to achieve better configuration generation diversity.

It is not clear how creativity is measured, e.g. is it more creative if I paint 50% of the house of a different colour vs 20%? Are these steps taken? Do you measure if agents take any kind of new steps in the trajectory of building a house or if they take creative steps that, while might be problematic, could make sense of how to build a house?

Creativity is not simply a matter of percentages. Painting 50% or 20% of the house a different color doesn’t inherently determine creativity. What truly matters is the intention and impact behind the choice. If painting a portion of the house in a different color introduces a unique aesthetic, challenges norms, or enhances the overall design, then it can be considered creative, regardless of the percentage. Even if some steps are risky, they can still be creative if they contribute to the goal and add diversity.

Here are the specific criteria for "building a house" generated by the criteria prompt in Appendix G.3:

Creative Attempts: creative attempts exhibited by the agent during doing task

• e.g. using different materials to create a visually appealing design
• e.g. unique themes or styles applied to the house, use of diverse materials for decoration, innovative lighting techniques

Also note that, The VLM is guided to consider, but not be limited to, the examples provided.

Please discuss how MCU enables researchers to assess open-ended capabilities that are not present in other non-Minecraft benchmarks

In the Introduction and Section 2.1 of our paper, we explain Minecraft’s unique suitability as a benchmark for open-ended capabilities and our motivation for selecting it. In brief:

1.Vast State Space

Minecraft’s state space is extraordinarily large—reputedly surpassing the number of atoms in the universe—allowing for virtually unlimited configurations of biomes, blocks, and entities. In contrast, many alternative benchmarks (e.g., Habitat-lab) rely on pre-scanned real world data which makes memorization rather than genuine generalization feasible.

2.High Task Diversity

MCU defines 3,452 atomic tasks that can be combined into a staggering number of compositional tasks covering navigation, crafting, combat, and more. This level of intra- and inter-task diversity is uncommon in other platforms, making it difficult to replicate the breadth and depth of challenges found in Minecraft. Other open-ended benchmarks primarily focus on a final goal, such as finding the Oracle in NetHack, are limited in inter-task diversity.

3.Open-Endedness

Minecraft naturally accommodates complex, multi-step tasks (e.g., obtaining diamonds) that require agents to plan and coordinate over extended horizons, remember terrain and resource locations, and adapt to dynamic objectives. Such open-ended exploration is central to fostering agents with genuinely flexible and robust capabilities—traits that are difficult to assess using more constrained benchmarks.

About references to other open-ended benchmarks and RL literature

Thank you for your valuable suggestions. We will incorporate these references and discussions in the next version of the paper.

Due to character limitations, we regret that we can only provide a simplified version of the response below:

What was the rationale behind selecting these six specific evaluation criteria?

Some evaluation criteria may not translate well across all atomic task categories.

It is not clear how these criteria should be distinguished.

The selection of the six evaluation criteria in the MCU framework was guided by the goal of capturing a broad yet practical spectrum of agent competencies needed in open-ended, real-world environments.

Regarding your specific points:

• Creativity of simple tasks (e.g., find_pink_tulip) can involve creative strategies (e.g., climbing for better visibility).
• Error correction is most informative when agents do make mistakes, but agents that complete tasks without errors are awarded the highest possible score in this category.
• Task progress in open-ended tasks like decorate_the_cave can be quantified by tracking the extent and coherence of changes made relative to the environment’s initial state.
• Material usage in "find" tasks may involve the use of navigation items such as boats or compasses.
• On the point of overlap between criteria, such as between task efficiency and action control, we agree that correlations exist. However, they emphasize different aspects: task efficiency focuses on outcomes (e.g., time to completion), while action control emphasizes the process (e.g., avoiding redundant or counterproductive actions).

Configuration Diversity?

Please refer to the response to Reviewer TkoE in Question 1.

How extensive can the soft constraints be? How do you address scalability concerns for complex tasks?

• Soft Constraints: Defined by “bad-case analysis”, generally applicable across tasks, ensuring feasibility within Minecraft’s rules. Currently, 8 soft constraints are used.
• Scalability: The task complexity concern is related to current LLM capabilities. As LLMs improve, more sophisticated tasks will become feasible. Generating task specifications is easier than solving them, so we expect scalability to improve naturally.

Separate prompts per criterion?

• A test on 20 random samples showed F1 = 91.2% with separate prompts vs. 90.6% with a combined prompt.
• However, cost rose nearly sixfold since reprocessing was required for each criterion.
• Thus, a combined approach balances efficiency and accuracy.

Evaluation Cost & Open-Source Models

• We do not plan to evaluate 3,000 tasks at once; we are selecting a representative subset.
• For the 35 tasks in the paper, each run (10 trials per task) costs 13.2 USD per agent using GPT-4o.
• We also tested MiniCPM-V-2_6 (8B, Aug. 2024) and JarvisVLA (7B, Mar. 2025). While open-source VLMs still lag, they are catching up.

How are hard mode tasks created?

Created by prompting the LLM to add complexity and constraints (e.g., obstacles, random disturbances). e.g., In “mine_iron_ore,” visually similar ores (gold/coal) are placed nearby to increase ambiguity.

What criteria or time limits determine when an episode or task is terminated?

Following GROOT settings: 600 steps for atomic tasks, 12,000 steps for compositional tasks.

Can you provide a detailed breakdown of how many tasks originate from each source?

Why did you choose not to include tasks with additional constraints or extra criteria?

• Additional constraints would break the “atomic” nature (Section 2.3).
• Atomic tasks are core test units; overlapping tasks dilute evaluation efficiency.

Is sampling 1 frame every 30 enough?

• Too many frames (interval=20) approach GPT-4o’s limit (50 images), risking overload.
• Too few (interval=40) may miss vital details.
• Interval=30 strikes the best balance (F1=0.90).

Evaluating progress in “decorate_the_cave”?

The criteria generated by LLM: e.g. agents select a suitable cave to decorate, add decorative elements inside the cave, and ensure the cave is well-lit and visually appealing

Regarding open-sourced datasets, typos, formatting, references etc.

Thank you for your meticulous attention to detail. We will fix these issues.

The correlation metrics on absolute ratings are hard to interpret in isolation (how do I interpret a correlation of 0.71?), unless e.g. you show that the correlation between the auto-eval and human ratings is close to the correlation between humans and other humans.

We appreciate this point and would like to clarify that we already report inter-rater agreement in Lines 380–384: “we compute the *inter-rater agreement* for scoring the same trajectory, revealing a higher Pearson correlation for task progress (0.83) and a lower correlation for creativity (0.69).” This provides a meaningful reference for interpreting the correlation between auto-eval and human ratings in context.

I am not convinced that composing multiple atomic tasks increases the complexity of the challenge significantly. The ceiling of difficulty for MCU is somewhat limited given the straightforward instruction-following nature of the tasks.

Our core motivation for composing multiple atomic tasks is to introduce long-horizon dependencies that go beyond basic instruction following. These composite tasks require capabilities such as high-level task planning (e.g., determining the optimal execution order based on task dependencies), memory management (given the limited context length of policy models), and error correction (recovering from early mistakes that may impact downstream steps).

For instance, in the composed task “mine iron, craft an iron pickaxe, and mine diamond”, the agent must break this into subtasks with strict dependencies. If it fails to mine enough iron, it cannot proceed to craft the pickaxe, and thus cannot complete the final step. Success in such tasks requires more than executing steps sequentially—it demands adaptive reasoning, context-aware decision making, and robustness to cascading errors. These qualities represent a meaningful increase in complexity and present new research challenges.

How many hours would it take a human to complete the most difficult challenges in Minecraft?

As a reference, completing the “Ender Dragon” challenge—a commonly recognized long-horizon goal—typically takes a human player between 10 and 30 hours. This includes time spent exploring, gathering resources, crafting appropriate gear, locating the stronghold, and finally, fighting the dragon (which itself takes 10–30 minutes). The overall process involves multi-stage planning, resource optimization, and consistent execution across diverse subtasks.

I would recommend establishing a “canonical” setup for users wishing to benchmark agents on MCU: how many tasks, which tasks, and a single final scalar metric that aggregates performance. The current setup is flexible but leaves too many variables up to the users.

Thank you for the valuable suggestion. In response, we are working on defining a standardized benchmark configuration for MCU. Specifically, we plan to select 10 primary categories (excluding the “others” category) and curate 8 representative tasks from each, resulting in a total of 80 tasks. Each task will include both simple and hard manually verified configurations. For evaluation, we will report the average performance across all tasks and dimensions as a single scalar metric. This canonical setup will be included in the next version of the benchmark to promote consistency and comparability across future work.

My main concern regards the quality of the generated data. (How many errors are detected?) quantifying the advantages in Fig.2 (proportion of open-ended tasks, etc.).

Thank you for raising this important point.

To evaluate the quality of the generated data, we randomly sampled 100 atomic tasks and performed configuration generation. We observed a 9.8% error rate, primarily due to issues such as “unknown block_type or item”. In addition, we identified a 3.7% rate of undetected errors (e.g., omission of necessary items). After verification, over 95% of the configurations were confirmed to be accurate.

To quantify the advantages shown in Fig. 2, we processed the MineDojo creative task list using GPT-4o and filtered out tasks that are not solvable even by humans. From this, 521 tasks remained in MineDojo (note: these tasks are without runnable configurations). We further filtered programmatic tasks according to the definition of atomic tasks, resulting in 268 atomic tasks in MineDojo compared to 2,190 in MCU. Notably, every task in MCU is associated with both simple and hard variants, while MineDojo officially releases difficulty levels for only 64 tasks.

Could you please share more details on the brainstorming step (Lines 258–259)? How did this step improve task generation? Does it apply to all atomic tasks?

As mentioned in Lines 953–957, we incorporate tasks generated through both expert brainstorming and LLM collaboration. This process is useful for generating creative tasks. More specifically:

1. Expert-LLM collaboration: Experts iteratively prompt LLMs to produce creative task ideas. For example, they begin with prompts such as “Let’s brainstorm some creative tasks that intermediate-level Minecraft players could accomplish. {few examples}” and refine outputs across rounds with feedback like “Creating a Mona Lisa statue is too complex—can you offer simpler alternatives?”
2. Expert proposal: We work closely with a university Minecraft club, where experienced players propose engaging and imaginative tasks. This process contributed tasks such as “Prepare a birthday for your neighbor”.

While brainstorming significantly enriches task diversity, not all tasks are derived from this process. Appendix C.1 details the four task sources where tasks are collected.

Doesn’t the few-shot config prompt (Lines 257–261) bias generations to examples similar to those in the prompt?

Thank you for this thoughtful observation. To mitigate prompt-induced bias and encourage diversity in LLM-generated configurations, we explicitly design our prompts to promote variability in initialization elements such as biome, weather, and player state (Lines 49–54).

We validate our current prompt through:

• Case study: For the task “craft a crafting table”, we ran 10 generations. The results exhibited wide variation in commands (e.g., /time set day, /give oak_log, /setblock blue_bed), including different wood types like birch, oak, and spruce.
• Quantitative analysis: For 5 randomly selected tasks, we conducted 100 generations each. The item-level overlap with few-shot examples was only 2%, and on average, 89% of the commands in each task were unique.

We will also keep improving our prompt to achieve better configuration generation diversity.

For the comparative evaluation (Lines 348–351), why are there “tie” and “both are bad” classes? What about cases where both generations are good?

When both generated outputs are of similarly high quality and indistinguishable in effectiveness, we classify the comparison as a “tie.” Conversely, we mark them as “both are bad.” These cases are recorded solely for annotation clarity and are excluded from final comparison metrics. We only compute metrics based on pairs where a clear winner can be determined (line 351-353).

While this may be out of scope, the evaluation framework (Sec. 2.5) is only evaluated on a single game, Minecraft. Evaluating the generalizability of the method to additional multimodal tasks (e.g., web browsing, multimodal code generation, other gaming environments), can be helpful for future research.

Due to the constraints of the rebuttal period, extending AutoEval to additional multimodal domains is currently beyond our scope. However, we agree that evaluating AutoEval across different domains would strengthen the contribution. We plan to explore in future work.