OASIS: Open Agents Social Interaction Simulations on a Large Scale

[w1] It is very strange and may introduce bias to simulate the recommendation system in a social network, which is not so important. Users can interact with information in multiple ways, and the recommendation system is only one of them. Introducing recommendation algorithms in social simulation may introduce bias as we don't know the details of recommendation algorithms in the collected data. I have also checked the papers in Table 1, in which only RecAgent and Agent4Rec considered recommendation systems (of course, as they are about the simulation of recommendation rather than social simulation).

Thank you for your question. Our recommendation system is built based on the official open-source framework provided by Twitter. The recommendations are categorized into 'in-network' and 'out-network.' In-network recommendations are based on users' following relationships, where agents can follow users of interest to receive more tweets from them in the future. Out-network recommendations, on the other hand, are driven by interest-based suggestions, where tweets that match an agent's interests are recommended. Therefore, a recommendation system is essential for platforms like Twitter. We use Twhin-BERT for our recommendation system, which has been pre-trained on a large amount of Twitter data, enabling it to model user interests effectively and provide better interest-based recommendations. Regarding bias, we believe that our recommendation system, like those of existing social media platforms, may lead to the creation of information bubbles, where agents are predominantly exposed to information that aligns with their interests. We will further address this issue of bias in future versions.

[w2] The evaluation should be improved. Can we use the results in Figure 4 to claim that the simulation is accurate and useful? There are still errors. The authors should carefully analyze where the error comes from and help the readers know when we can use the results of OASIS. In other words, what kind of use can these imperfect simulations have? We all know that perfect simulations are not possible, but it is important to analyze what kind of impact these errors may have.

Thank you for your question. In our message propagation experiments, we compared the message propagation process in OASIS with that in the real world. We found that while the broadcast mechanism in OASIS aligns well with real-world behavior, there is a significant discrepancy in the depth of the propagation paths. We provide an explanation for these differences in Section 3.3.1 of the paper. Understanding these discrepancies can help users better tailor their use cases and further improve the capabilities of OASIS.

[w3] The authors should conduct more evaluations based on social network theory. Could the simulation system be consistent with those representative theories such as network dynamics? These important and basic questions have not been well answered.

Thank you for your question. In our message propagation experiments, we compared the propagation paths with those observed in the real world. In future versions, we plan to include additional comparisons related to social network dynamics, such as the Schelling segregation model.

[w1] I think, in a way, the paper tries to pack too much in a single paper. I think it was challenging to provide all the details of the engineering AND conduct thorough social experiments. As a result, I feel that the social phenomena simulations are somewhat shallow and not very insightful. For instance, the simulations raise the question of "null models" -- how surprising or insightful are these massive LLM-simulated results compared to a simple model? -- as well as the question of potential parameter and environment settings. Given so many parameters one should set to simulate a social media, it is really hard to know what is driving the results and how different the simulation results would be in different settings. It is simply impossible to know whether the results shown in the paper are typical across different settings or something that only arises in the specific settings used in the paper. While I think it is still meaningful to demonstrate that the OASIS framework can produce somewhat realistic social phenomena, I think it could have been more insightful if there was a separate, in-depth analysis of each social phenomena.

Thank you for your valuable suggestions. The experimental settings for our three social scenarios align with those used in social science literature. At this stage, our goal is to replicate the phenomena observed in these classic sociological experiments in order to validate the effectiveness of OASIS across different settings. In future versions, we plan to incorporate deeper analysis of each scenario. Regarding the issue of parameter control influencing results that you mentioned, I believe your thoughts are valuable, and we have also considered this. In our paper, when setting up the platform, our consideration in parameter setting was to align as closely as possible with real social media, not to manipulate the results. We also tried to avoid introducing decision bias when setting agent prompts. The reason for this setup is to faithfully explore the similarities and differences between agent societies and human societies. Furthermore, while the possibility of using a single model for group simulations is an interesting direction, we believe that the multi-agent interaction approach currently offers greater flexibility and interpretability. This approach allows us to more easily configure settings and analyze the states of individual agents.

[w2] I was not entirely sure whether the paper contributes to the advancement of representation learning. The paper uses LLMs, but the focus is more on the engineering of the simulation framework. Although it may be possible to argue that the paper enables the study of large-scale social phenomena that can be studied with LLMs and how the "representation" that LLMs use can affect the social phenomena, it is still quite indirect.

Thank you for your valuable suggestions. We believe that understanding the representations of LLMs is crucial for studying the collective behavior of LLM-based agents. Collective behavior not only helps us assess the capabilities of individual agents, but also enhances their performance through interactions within the group.

[Q1] Studies have shown that many large-scale patterns from social media can be explained by simple models and few key mechanisms. For instance, Weng et al. Sci. Rep. (2012) showed that limited attention is enough to produce the power-law distribution of propularity of hastags; the "burstiness" of human behavior (see Barabasi's work on burstiness) has been studied as a key mechanism for many patterns in social phenomena; the "homophily" mechanism has been shown to be a key mechanism for many social phenomena; There are also many cognitive biases that can explain many social phenomena

Thank you for your insightful suggestions. We will conduct further research in future versions to investigate whether the aforementioned characteristics exist within agent-based societies.

[w1] The primary contribution of this paper is significantly increasing the scale of existing social simulations to the one-million level. ...

Challenge 1: Inference Scheduling. We manage the immense volume of inference requests from one million agents using a system of asynchronous requests and concurrent processing through multithreading. This allows for efficient simulation of large agent numbers.

Challenge 2: Large-Scale Agent Interaction. To handle the complex information flow among agents, we use a recommendation system based on interest and hot-score recommendations, simplifying large-scale simulations. Future work will refine these solutions and compare them with other methods. We will further refine these solutions and provide a comparative analysis with alternative methods in the paper.

[w2] An important contribution of this paper is incorporating dynamically updated environments, diverse action spaces, and recommendation systems. ...

Thank you for your insightful question. Firstly, on dynamic environments: Agents are initialized from real-world data and interact similarly to real-world social media platforms. They receive post recommendations, decide actions based on a large language model, and dynamically update relationships and the post database through various actions.

Secondly, on the recommendation system: Traditional recommendation systems, like Light GCN, are less suitable for dynamic social media platforms like Twitter, where content is constantly evolving. To our knowledge, no research has specifically designed recommendation systems for such platforms.

[w3] This work lacks comparisons with previous methods. ...

Thank you for your question. While previous works have indeed addressed message and rumor propagation, the differences in the social simulation settings make it challenging to directly align our work with others. As a result, a fair comparison with existing studies is difficult to achieve. In future versions of this work, we plan to dedicate more effort to benchmarking and comparing our approach with other relevant research.

[w4] In section 3.4, this part explores "DOES THE NUMBER OF AGENTS AFFECT THE ACCURACY OF SIMULATING GROUP BEHAVIOR?"

Thank you for your question. In the herd behavior experiment, the accuracy of the results is assessed by the confidence intervals of the disagree score, as shown by the differently colored regions (red, green, and blue) in Figure 8. Comparing the confidence intervals, rather than just the means, helps distinguish between results caused by random fluctuations and those that exhibit statistical significance. This approach is similar to the concept of increasing sample size in sociological surveys. We observed that when the number of agents was 100, the confidence intervals for the three groups were large and overlapped, indicating that the results might be unreliable. However, when the number of agents increased to 10,000, the confidence intervals narrowed, and the down-treated group clearly separated from the other two groups, providing strong evidence for the accuracy of the experiment.

[w5] On line 180, the paper mentions a "relations network." In what format is this relations network stored? ...

Regarding the 'relations network,' we store the edges of the social network in a relational database as part of the overall social media platform database. See follow table in Apendix C.2.

The basic unit of the propagation path is the post-to-post forwarding relationship. By tracking the post-to-post forwarding graph, we can calculate the depth and breadth of propagation. The agent-to-agent forwarding relationships are recorded in the Trace table of the database.

【w6】 1. For Comment scores, why is it calculated as upvotes minus downvotes? ...

It is correct that many social media platforms display both the number of upvotes and downvotes on posts. However, Reddit follows a different approach, as it only shows the score(see https://www.reddit.com/). A study on herd behavior in humans, published in Science (https://www.science.org/doi/10.1126/science.1240466), conducted experiments on a platform similar to Reddit, where only scores are displayed. We have replicated their experimental setup in our own research.

【w7】 1. How is an “uncensored LLM” defined?

The term 'uncensored LLM' refers to a series of LLMs that have had their safety barriers removed, significantly compromising their security. For more details, refer to [https://huggingface.co/blog/mlabonne/abliteration].

【w8】 1. Why not use models like GPT-3.5-turbo, GPT-4 (possibly more costly), or GPT-4o-mini for the experiments? Given their relatively strong reasoning abilities, they might achieve higher accuracy.

Due to the large-scale user simulation we conduct, running a full experiment incurs significant costs. In future versions, we plan to incorporate powerful proprietary models to conduct large-scale message propagation experiments.

[w1] Given the complexity of LLMs, it is unclear which insights an LLM-based agent-based simulation can yield into the actual microscopic mechanism underlying collective social phenomena that emerge at the macroscopic scale.

Thanks for the reviewer’s question. The main confusion raised by the reviewer is at the micro and macro levels of analysis. OASIS, in its simulation process, records the motivation behind each agent’s actions, which is done through the COT technique. This allows us to track the fundamental behaviors of the agents. For example, our statistical analysis reveals that when agents encounter rumors, they tend to use emotional language such as "interesting" or "attractive," indicating that the rumors trigger the agents' emotions. These analyses help us to understand the underlying causes of group behaviors, which we have discussed thoroughly in the paper.

Moreover, the design of our platform aids in replicating the complexity of interaction dynamics in real social media platforms and maintains versatility for various social experiments. For instance, the number of likes on a post, the content of comments, and the recommendation system all interact with each other, mirroring the way real human social media information dissemination is influenced by these interconnected factors.

Many studies have demonstrated the ability of LLM agents to simulate human behavior[1][2], affirming the feasibility of using LLM agents for social simulation. We understand your focus on making agents more similar to humans for better simulation, but there might be a slight misunderstanding regarding our contribution. We want to emphasize that as an initial exploratory work on large-scale social simulation with LLM agents, our aim is not to make LLM agents' behavior identical to human behavior. Instead, we are exploring the similarities and differences between agent societies and human societies, investigating the potential of agent societies in predicting human society and other aspects, or applying it to other areas such as gaming.

[1] Social simulacra

[2] SOTOPIA

[w2] The description of the platform remains at a rather high and superficial level. Some details of the complex platform design are not explained in detail.

Thank you for the reviewer’s question. In Section 3.2, line 309, we describe how we collect real user data and the main data we use for the simulation. In Section 3.2, line 312, we explain that the users for the group polarization experiment are sourced from the real users involved in the message propagation experiment. For the generated users, we provide a detailed discussion in Section 2.6 and Appendix D.2. As for the prompt, it mainly consists of two parts: the user description and the action generation. Due to space limitations, we did not conduct ablation studies on the specific details of the prompt. We will conduct more ablation in the future.

[w3] I am not convinced by the way how temporal and topological characteristics of agent interactions are integrated into the simulation.

Thank you for your insightful comments. User time behavior is challenging to model due to its multifactorial nature; we use a simple activation probability model to address this. It approximates user time behavior and scales well for large agent simulations. We aim to improve temporal feature modeling in the future. As for the social network, we classify users into core and regular users, based on prior work. This method aligns with Twitter's network structure. More details are in Appendix D.2.

[w4] I do not think that the contributions summarized in this paper are of sufficient relevance for the wider ICLR research community.

Our innovation primarily lies in constructing a realistic, universal social media environment for studying large-scale agent interactions. Past work in this field has often focused on specific scenarios and oversimplified many platform features. In contrast, with the increasing capabilities of large models and agents, our work has built a highly realistic LLM agent social simulation platform based on real platform features, making a significant contribution as an agent infrastructure. We have put substantial effort into the engineering aspect, and the results not only facilitate the social simulation experiments we've mentioned but also provide a platform for other researchers to conduct a wider range of studies on subjects such as LLM, data generation, recommendation systems, and more.

Ethics Concerns Our experiments are conducted in a controlled environment, without the involvement of real human participants. We believe that our platform can serve as a valuable tool for advancing research on LLM safety, offering insights into emerging security challenges in agent-based societies. By providing a realistic simulation of agent interactions, our platform aims to guide the development of strategies to address potential safety risks in future agent-driven ecosystems.