Virtual Community: A Generative Social World for Embodied AI

1. While the paper claims support for large-scale environments, it lacks quantitative results on how the system scales in terms of computation, memory, or performance metrics (e.g., agent count or scene complexity). There is also no mention of whether the simulator offers a ‘headless mode’ to allow multiple instances to run simultaneously on a cluster, which would be valuable for generating large-scale data or evaluating models in parallel. Additionally, it is unclear whether the simulator can be run ‘faster than real-time’ to speed up experiments.

2. The paper doesn’t explain how the movements of characters not being controlled by the baseline algorithms are managed. It is unclear whether these movements are based on a social forces model, predefined behavior heuristics, or something else.

3. The generative approach to creating agent personalities and routines may not fully capture complex or realistic social behaviors.

4. The reliance on external geospatial data and APIs like Google Maps could limit applicability in areas with incomplete or unavailable data. Exploring methods for generating environments in a more data-independent manner would improve flexibility.

5. The paper doesn’t discuss the usability of the simulator or the available APIs for developers. It’s also unclear whether there’s a built-in way to model interactions as graphs, or if users would need to design these graphs themselves. Providing tools to model and track interactions as graphs would make it more user-friendly. Clear documentation and API examples would also help developers integrate the platform into their projects more easily.

6. There is no clear mention of the intended target audience for the simulator.

7. The paper omits references and comparisons to similar platforms like SEAN 2.0 [1] and Arena 1.0-4.0 [2], specifically along the axis of including the presence of robots in their simulator.

8. The platform focuses on simulation, but there is no discussion about how these simulated behaviors or learned policies can transfer to real-world applications.

[1] Tsoi, Nathan, et al. "Sean 2.0: Formalizing and generating social situations for robot navigation." IEEE Robotics and Automation Letters 7.4 (2022): 11047-11054.

[2] Shcherbyna, Volodymyr, et al. "Arena 4.0: A Comprehensive ROS2 Development and Benchmarking Platform for Human-centric Navigation Using Generative-Model-based Environment Generation." arXiv preprint arXiv:2409.12471 (2024).

1. While the authors claim that the pipeline can generate an infinite number of scenes as shown in Table 1, it is still important to note how many assets this work has generated so far. It lacks a quantitative analysis of these created assets.
2. The paper performs almost no quantitative experiments, only the proposed route planning task. This actually shows that the benchmark cannot cover too many current research methods, which is the biggest weakness of the paper.

This type of work has seen a resurgence in interest since the development of LLM [Wan et al., 2024], although it has a long history featuring symbolic AI techniques of search-based path planning and plan-based decision making and behavior, of which not much has found its way to the paper’s reference list of analysis of previous work. Although more contemporary techniques certainly show promise including for this sort of application, this is unfortunate as i) LLM are generally weak on relevant topics such as planning (despite recents experiments such as [Nasir et al., 2024]) and ii) some VR integration problems have been previously described and it does not seem appropriate to re-invent them: in particular the relationship between AI-based action parameters and the lower-level control of action and animation in the virtual world.

Path planning is one of the two main tasks allocated to these virtual agents. However, the corresponding sections suffer from both a lack of clarity and a lack of technical details. In particular there is no clear articulation between the low-level A*-based path planning and higher level (MCTS, LLM). This is particularly confusing since there is a potential overlap between various levels, including at the level of AI techniques (see, e.g. [Xiao et al., 2023]). Moreover, since the authors only mention A*, they are not addressing the well-known issue of dynamic environments, which becomes increasingly relevant with the number of agents and autonomous vehicles.

Following Figure 2, there is a lack of details on how social relationships are established from the characters’ profiles, whose description is relatively simplistic. Reference to “open-world knowledge” does not appear sufficient in the light of the vats body of work dedicated to persona definition with LLM.

Results of Table II are difficult to interpret, in particular in terms of LLM performance. In addition, 19 personal schedules and 106 commutes can hardly be considered a large-scale simulation in view of previous work. Some explication given for the results, such as leveraging environmental resources, is simply not useful in the absence of a fair comparison of integration of high-level decision making and low level decision across the three techniques (Rules, MCTS, LLM).

Minor aside: There are some oddities in references, such as suggesting that Auerbach et al. [2014] would be a “foundational model”, in the sense of pretrained Transformers-based foundational models such as those listed in a previous section of the paper (Bubeck et al., 2023; Liu et al., 2023; Driess et al., 2023; Blattmann et al., 2023)

References

Wan, H., Zhang, J., Suria, A.A., Yao, B., Wang, D., Coady, Y. and Prpa, M., 2024, May. Building LLM-based AI Agents in Social Virtual Reality. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (pp. 1-7).

Aghzal, Mohamed, Erion Plaku, and Ziyu Yao. "Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning." arXiv preprint arXiv:2310.03249 (2023).

Song, C.H., Wu, J., Washington, C., Sadler, B.M., Chao, W.L. and Su, Y., 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2998-3009).

Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797.

Nasir, M.U., James, S. and Togelius, J., 2024. GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps. arXiv preprint arXiv:2410.07765.

Results section feels lacking. Only results for Route Planning are included; it would be nice to see results for the Election Campaign as well. But more importantly, the election campaign environment doesn't seem very useful. Route planning requires having an environment and a simulation since the agent is actively taking actions in that environment. However, the election campaign environment is more about interactions with other people - not something that immediately requires having a 3D environment.