On Evaluation of Generative Robotic Simulations

Task set generalization (Section 4.3): In this section, different task groups are used to measure the generalization ability (as shown in the Appendix Table 6). Given that the task groups are not identical across task sets, is it fair to conclude that GenSim exhibits better generalization while the others perform poorly? How does this variability in the definition of task groups affect the generalization conclusions?

The reason we opt for grouping the tasks is the limited capability of vanilla Diffusion Policy to handle a multi-task setting. We ackowledge that task spliting would quantitatively affect generalization results.

Definition of text embedding (Line 264): What is the formal definition of the text embedding $e_i$ used for task descriptions? More clarity here would help.

In this context, a formal definition for the text embedding $e_i$ would clarify its purpose, structure, and how it is generated. Generally, text embeddings are vector representations of text that capture semantic meaning and allow comparisons across different textual data. Here’s an example of a precise definition:

Let $e_i$ be the embedding of the task description $T_i$, where $T_i$ is a sequence of tokens representing the description. The embedding $e_i$ is a vector in $\mathbb{R}^d$ that is generated by applying a text embedding model $f$, such as a pretrained language model (e.g., BERT, GPT, etc.), to $T_i$. Formally, this can be written as: $e_i = f(T_i) \in \mathbb{R}^d$

where $f$ is a mapping function that converts the tokenized text $T_i$ into a fixed-dimensional vector representation in $\mathbb{R}^d$. The dimension $d$ depends on the embedding model used and is typically chosen to balance representational power and computational efficiency.

Image sampling (Line 246): The paper mentions extracting 8 images from the trajectory video. Are these images randomly sampled, or are they evenly spaced throughout the trajectory? Does the selection method include the final frame? Could different sampling strategies affect the task completion score?

We uniformly extract 8 images from each video, including the final frame. Different sampling methods can affect our results. For instance, in the "Open The Laptop" task, we applied the following sampling methods:

1. Uniform sampling
2. Random sampling
3. Selected sampling(Random sampling with a focus on the latter half of the video)
4. Sampling without the final frame

The results are as follows:

Thus, we believe our sampling method is superior to random and selective random sampling. However, since sampling has a relatively small impact on the results, we did not conduct large-scale experiments on sampling methods.

We sincerely appreciate your comprehensive evaluation and feedback on our paper.

Clarity of contribution: The contribution is not entirely clear. Is this the first evaluation strategy for task sets? If not, how does it compare to existing evaluation methods, such as Self-BLEU or Embedding similarity scores from RoboGen? More clarification is needed on how this approach improves upon or differs from existing evaluation techniques.

We agree that the issue you raised is indeed important. First, different generative simulation works use entirely distinct evaluation methods. For example, Robogen evaluates diversity using Self-BLEU and Embedding Similarity, while BBSEA and Gensim assess the quality of tasks generated by different large language models (LLMs). In contrast, we propose a more comprehensive and complete evaluation method suitable for all generative simulation tasks, allowing for a more accurate and appropriate assessment of these approaches' capabilities, rather than relying solely on text-level diversity comparisons among generated tasks. Our evaluation extends beyond text-level diversity, assessing factors such as task quality, the diversity of generated task trajectories, and the applicability of generated tasks in imitation learning. Therefore, we believe our work represents the first complete evaluation framework for generative simulation.

Realism of generated tasks: The claim regarding "evaluating the realism of generated tasks" seems overreaching. The approach measures scene alignment score, which reflects consistency between task descriptions and scene images. However, it's not clear why this metric necessarily reflects the "realism" of the generated tasks. A more thorough justification or evidence supporting this claim would strengthen the paper.

Regarding "evaluating the realism of generated tasks," we acknowledge that our wording may have been imprecise. Our primary focus is on the consistency between the scene and the text. While we aim for the scenes to be reasonably realistic, realism itself is not the central focus of our evaluation.

How many human evaluators were involved? Were they experts in robotic tasks or general participants? What guidelines were provided to ensure consistency in ratings, particularly for scene alignment and task completion scores?

We had 18 participants in the evaluation, primarily researchers in robotics(expert in robotic tasks). In the questionnaire, we provided guidelines on how to score correctly, so we believe we offered sufficient instructions.

Writing and presentation: The paper could benefit from improved clarity in writing, particularly in explaining key concepts and the methodology used.

We will refine the wording of our paper based on the reviewers' feedback.

Pearson correlation in Figure 3: Figure 3 shows Pearson correlation between evaluation methods and human evaluations. However, it does not quantify how much the proposed evaluation method deviates from human evaluation. Why are the scores in Figure 3 (left) generally better than those in Figure 4 (right)? Does this imply that the proposed method is more reliable for scene alignment scoring than for task completion scoring?

The Pearson correlation score is lower which means the evaluation method deviates from human evaluation. Scene alignment requires less capability from the model. In contrast, completion demands a higher level of skill, such as the ability to understand relationships between multiple images, which typically necessitates more advanced models to achieve better results in completion tasks. Therefore, we believe that as large model capabilities continue to improve, the gap in addressing these issues will be resolved.

Figure 4 method clarification: The paper mentions various LLM/VLM options in Section 4.1.1, but it’s unclear which one is used in Section 4.1.2 Figure 4. Do all VLM/LLM choices yield similar results, or do specific models shows the result trend in Figure 4? For example, does RoboGen consistently outperform in task completion scores while GenSim excels in scene alignment, regardless the choice of evaluation VLM/LLM?

We used GPT-4V, as it demonstrated the best performance in Figure 3.

Line 354 clarification: The paper attributes the shortfall in performance to the vision-language model’s “lack of access to top-view rendered data”. Is this limitation specific to the VLM used in GenSim or to the VLM used in the proposed evaluation method?

The limitation here arises because Gensim tasks are typically presented in a top-down view, while visual-language models like LLaVa and Cambrian generally lack similar top-down view data for robotics. As a result, their performance on Gensim is relatively weaker.

Dear Reviewer XFSC,

Thank you for taking the time to review our work and provide your feedback. We hope our responses have adequately addressed your concerns. If you have any further questions or wish to discuss any aspects of the paper, we would be glad to engage during the remaining discussion period.

On the other hand, if you feel that your concerns have been fully resolved, we kindly request that you consider reevaluating your rating. Your support means a great deal to us and would contribute significantly to the recognition of our work.

Thank you!

Thank you very much for thoroughly reviewing our work and providing constructive feedback.

Could the authors explain why the three selected metrics are considered essential for generative simulation? While single-task quality and task diversity are understandable, it's unclear how generalization fits within this context. How does it differ from simply having diverse tasks?

Our work primarily focuses on proposing an evaluation method for generative simulation that reflects the overall quality of the generated data. In generative simulation, enabling the data to serve general-purpose robots is a fundamental goal, making generalization a crucial evaluation criterion. Generalization is distinct from diversity: text diversity merely reflects the range of tasks generative simulation could potentially produce, while trajectory diversity reflects whether a single task can generate a sufficient variety of complex trajectories. Generalization, on the other hand, emphasizes that the generated data should effectively support imitation learning, enabling models to learn the ability to solve diverse tasks.

Could the authors clarify how scene alignment is evaluated? From my understanding of lines 348-350, it seems to rely on the robot's trajectory. If that’s the case, shouldn’t the quality of the generated trajectory also play a role? Additionally, I’m curious why scene alignment is based on video rather than image.

We did not use trajectory videos to evaluate scene alignment. Instead, we used multi-view images from different perspectives when evaluating Robogen and BBSEA, while for Gensim, we used top-down view images of the scenes. We will clarify the evaluation method in main content.

The connection between high prediction loss and task diversity is unclear. For instance, a single task could involve varied dynamics, like table rearrangement. Could the authors provide a justification for this?

The predictive capability of the same model after the same training duration is consistent. Therefore, if the model shows a higher loss on different datasets, it suggests that the dataset contains trajectories and scenes with greater diversity. Thus, we believe that high prediction loss can indicate higher trajectory and visual diversity. Additionally, if a task is highly complex, it will also yield a relatively high prediction loss, further suggesting that a dataset containing such a task has greater diversity.

Each baseline uses a different task for evaluation, but certain tasks or simulator setups (as mentioned in lines 318-319) might inherently pose challenges for LLM/VLM-based score evaluation, potentially affecting fair comparison.

The tasks generated by different generative simulation pipelines are inconsistent, so we chose multiple similar tasks within the same domain for evaluation. It is not feasible to have all generative simulation pipelines generate identical tasks in the same simulation for evaluation, as the content of the tasks they produce is entirely different.

How do humans evaluate the task? (e.g., Do they use a 1-5 scoring system, rank tasks by preference, or something else?)

Human experts used a scoring system of $1–5$ and $1–10$ for rating.

Related to the point above (in the Weaknesses section, high prediction loss), lines 274-277 weren’t clear about the exploration aspect. Could the authors clarify how diverse dynamics are connected to task diversity?

Same to the response above.

Dear Reviewer 2PAM,

Thank you for taking the time to review our work and provide your feedback. We hope our responses have adequately addressed your concerns. If you have any further questions or wish to discuss any aspects of the paper, we would be glad to engage during the remaining discussion period.

On the other hand, if you feel that your concerns have been fully resolved, we kindly request that you consider reevaluating your rating. Your support means a great deal to us and would contribute significantly to the recognition of our work.

Thank you!

Thank you authors, for providing the clarifications.

However, I am still a bit unclear about the main contribution of this work. While it seems to be motivated by the idea of a "generative simulator" that could support the application of multiple algorithms (e.g., IL, RL), I would appreciate it if the authors could elaborate on what is meant by "generative simulation that reflects the overall quality of the generated data." Specifically, is the primary focus on the "data" itself or on the simulation environment?

Additionally, I found the statement "the same model after the same training duration is consistent" a bit unclear, and I am not entirely convinced by the claim that "high prediction loss can indicate higher trajectory and visual diversity." High prediction loss could also result from the inherent complexity of the dynamics (e.g., contact-rich scenarios), and it’s not clear how this directly relates to visual diversity.

I would be happy to see further clarifications from the authors.

We genuinely thank you for your thorough assessment and encouraging remarks regarding our manuscript.

There doesn't seem to be a consistent correlation with human evaluation and mLLM proposed results in Figure 3.

Although in Figure 3, most models do not perform well, GPT-4V shows some alignment with human expert ratings in certain areas, such as scene alignment in Gensim and completion score in Robogen. We acknowledge that current large foundation models are not yet capable of adequately evaluating scene alignment and task completion scores. However, we believe that future models will better align with human expert evaluations in these areas.

The alignment score given by the mLLMs is not calibrated – some calibration would make the results more reasonable.

Thank you very much for raising this point. We did not perform calibration for each large model; however, by calculating the correlation with human behavior, we minimized issues arising from the lack of calibration. Therefore, we believe our conclusions remain reasonable. That said, adding calibration would indeed make our conclusions more convincing.

Based on our understanding of calibration, we modified the evaluation prompts by adding both positive and negative examples. For the BLIP2+GPT4 method, this was achieved by modifying the prompts alone, while for GPT4V, we incorporated additional images along with the prompts. The table below presents the experimental results after incorporating calibration. Due to time constraints, we conducted these tests only on the Robogen dataset as a reference.

Generalization evaluation would be more convincing if it was trained on some kind of task-conditioned multi-task policy like BAKU [1] rather than an unconditional policy like diffusion policy.

Thank you very much for your suggestion. We agree that BAKU is a suitable model for imitation learning; however, it was only made public three months ago, so we did not use it as a benchmark when completing our work. Although BAKU meets our needs, we believe it is reasonable to choose the more widely known diffusion policy method for our evaluation. If future work demonstrates that BAKU can effectively assess data generalization, we will add related experiments to our codebase for use by future researchers. We will discuss this work in related work.

Do the authors think the task diversity captures the overall size of the possible task sets in a generative simulation?

If "overall size of the possible task sets" refers to the number of non-semantically repetitive tasks that can be generated, then higher text-based diversity would likely result in more such tasks. However, if you are referring to tasks with different trajectories, we believe that simply running the generative simulation pipeline continuously can produce an unlimited number of tasks with varying trajectories.

Would the authors be open sourcing their evaluation suite so developers of generative simulations can develop with this benchmark in mind?

Yes. We will release the code in the camera-ready version.

Dear Reviewer MwfY,

Thank you for taking the time to review our work and provide your feedback. We hope our responses have adequately addressed your concerns. If you have any further questions or wish to discuss any aspects of the paper, we would be glad to engage during the remaining discussion period.

On the other hand, if you feel that your concerns have been fully resolved, we kindly request that you consider reevaluating your rating. Your support means a great deal to us and would contribute significantly to the recognition of our work.

Thank you!

Thank you for your thoughtful evaluation.

Have you considered metrics to assess how well the simulated tasks translate to real-world scenarios?

We agree that sim-to-real gap is an important problem should be considered. While assessing the transfer from simulation to the real world is indeed important, there is no universally accepted real-world setting. If we were to define our own sim-to-real transfer approach, it would compromise the fairness of the evaluation. Therefore, we believe that evaluating sim-to-real cannot be conducted in a fair and reasonable manner. Also we think that many of our metrics aim to minimize the sim-to-real gap.

It seems the diversity score only considers the text and trajectory. However, I think it should also consider at least visual input.

Visual diversity is important for training policies in imitation learning; however, it provides limited insight into the quality of the data itself and is not closely aligned with the aspects we aim to evaluate. Achieving visual diversity in a simulator can be relatively simple by adding data augmentation; for example, adding complex but irrelevant textures to the simulator background may quickly increase visual diversity without actually improving the quality or diversity of the generated data. Additionally, within trajectory diversity, the dynamics model also captures some basic visual diversity—tasks with relatively high visual diversity often exhibit greater trajectory diversity as well.

Visual diversity is indeed an important aspect and is actually already captured by the visual reconstruction part of the dynamics error. We further evaluate the visual diversity of Gensim through the LPIPS score and the image reconstruction part of Dreamer's loss function:

where a lower Mean/Std indicates more similar and hence less diverse tasks. The results align with our dynamics error measure that Piles is the least diverse and Placement is the most diverse.

It seems this paper did a great job of analysis of existing benchmarks, but did not propose a new benchmark to resolve these issues.

Our work aims to establish a unified evaluation standard that future work in the domain of generative simulation can build upon to achieve improvements by addressing the issues we highlight. We believe that developing a robust evaluation benchmark is more important than simply outperforming existing metrics. Many evaluation studies also did not propose better benchmarks aligned with their evaluations ~[1,2,3], which is why, in this work, we did not introduce a new generative simulation pipeline.

[1] Papineni K, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002: 311-318.

[2] He Y, Bai Y, Lin M, et al. T $^ 3$ Bench: Benchmarking Current Progress in Text-to-3D Generation[J]. arXiv preprint arXiv:2310.02977, 2023.

[3] Hu Y, Li T, Lu Q, et al. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 22170-22183.

It seems generalization and diversity has some connection. The diversity is to measure how out-of-distribution tasks are from each other while generalization is to measure how in-distribution tasks are with each other. I wonder whether we can use the same metric to measure these 2 things. Is it possible to investigate correlation between diversity and generalization scores? Or is it possible to potentially unify these two metrics?

'Generalization' is distinct from 'Diversity' under the context of generative simulation. Text diversity merely reflects the range of potential tasks that generative simulation can produce, while trajectory diversity indicates whether a single task can generate sufficiently varied and complex trajectories. In contrast, generalization emphasizes that the generated data should effectively support imitation learning, enabling models to learn the capability to handle different tasks. Though diversity and generalization can be related, they are neither identical nor unified. As shown in Tables 1, 2, and 3 of the paper, although BBSEA and RoboGen outperform GenSim in diversity, their generalization capabilities are not as strong as those of GenSim. From a qualitative perspective,both extremely low and high levels of diversity can undermine generalization capabilities. Considering a scenario where a dataset consists of a single task trajectory, representing minimal diversity, the policy will inevitably overfit, resulting in no generalization. Conversely, if data diversity is excessively high to the extent that the model or algorithm cannot effectively capture and learn from it (e.g. the case in RoboGen), generalization will also suffer due to a decline in overall training performance. This highlights the need for a balanced approach to diversity in training data to optimize generalization.

Dear Reviewer MyJX,

Thank you for taking the time to review our work and provide your feedback. We hope our responses have adequately addressed your concerns. If you have any further questions or wish to discuss any aspects of the paper, we would be glad to engage during the remaining discussion period.

On the other hand, if you feel that your concerns have been fully resolved, we kindly request that you consider reevaluating your rating. Your support means a great deal to us and would contribute significantly to the recognition of our work.

Thank you!