Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancy

Dear Reviewer wsJL:

Thank you for your thorough and thoughtful review.

Scalability Analysis

We acknowledge that 100 agents is modest for large-scale social simulations. This design choice was driven by resource constraints, as LLM-powered agents are significantly more computationally intensive than rule-based agents. Nevertheless, as noted in our general response, we have run additional experiments (experiment #2) with varying numbers of agents and found that both simulation run-time and attitude dynamics scale roughly linearly. These results suggest the feasibility of scaling to larger agent populations with optimized infrastructure. To that end, we are also exploring frameworks like AgentTorch (Chopra et al., 2024), which enable scaling up to millions of agents to support future work at larger scales.

Repo and Libraries

We plan to release the codebase alongside the public version of the paper. The simulation uses a modular architecture built primarily with:

• vLLM for serving local model servers and efficient batched inference across agents
• Python multiprocessing for parallel simulation execution
• OpenAI API for accessing local model servers
• SentenceTransformers (Reimers & Gurevych, 2019) for embedding tweets/news and similarity computation.
• We will ensure that the final release includes complete documentation and pretrained configurations for replicability.

Related Work

We are also very grateful for the related work you point out. It is indeed highly relevant to our paper and we will include this in our camera-ready version.

Applications to Other Domains

Thank you for your question and for encouraging us to consider broader applications of the framework. While our current implementation is focused on vaccine attitudes, the underlying simulation infrastructure is modular, and we believe it could be adapted with additional development for use in related domains such as misinformation and disaster response.

Misinformation:

We believe that adapting our framework to simulate health-related misinformation is a natural next step. The current components, such as agent-level variation in trust, social network structure, and exposure to news, can be extended to model how false or conflicting information propagates and influences decision-making. While we have not yet implemented such a domain, we can incorporate more targeted prompts and misinformation-specific dynamics, including credibility of sources and susceptibility profiles in future iterations. By varying message content, source credibility, and network structure (such as using trust data such as from the KFF Tracking Poll: https://www.kff.org/health-information-trust/poll-finding/kff-tracking-poll-on-health-information-and-trust-the-publics-views-on-measles-outbreaks-and-misinformation/) we can assess how these factors impact vaccination decisions and test interventions (e.g., fact-checking, trusted messengers, message framing).

Disaster Response:

Another potential direction is modeling behavior under crisis conditions, such as during wildfires or other natural disasters. We sketch an example adaptation below:

A wildfire event can be introduced as a time-evolving environmental stimulus. Agents receive alerts, updates, and rumors from various sources (government, social media, neighbors), simulating the information landscape during a crisis. Factors such as household size, mobility limitations (e.g., elderly or disabled residents), and previous disaster experience can influence agent decisions to evacuate or shelter in place. Agents interact with one another, shaping risk perception and behavior through social cues, peer reassurance, or panic, capturing emergent group-level patterns like compliance or congestion. Public safety policies (e.g., targeted alerts, phased evacuation orders, or hyperlocal outreach) can be tested and compared in terms of their effect on response timing and safety outcomes.

While we have not yet tested this setting, the simulation infrastructure offers a promising foundation for exploring how individuals react to uncertainty, peer influence, and institutional broadcasts during emergencies.

We have added a discussion of these extensions in the conclusion to reflect both the flexibility and current limitations of our framework. We hope to explore these applications more fully in future work. Thank you again for highlighting this important direction.

Dear Reviewer eATj,

We are grateful for your detailed review.

Presentation Improvement

We appreciate your comments regarding the clarity of our figures. In the current version, Figure 4 aggregates multiple evaluations into a single visual to conserve space. In the camera-ready version, we will consider separating these sub-figures, improve axis labeling and legend clarity, and provide detailed captions. We will also include a compact summary figure/table to help readers understand the simulation pipeline in addition to the illustrative example in Figure 3.

Component-wise Analysis

We agree that a component-wise analysis would improve interpretability. In response, we have:

• Conducted an ablation of the fixed bias parameter used in tweet recommendations. Results in the general response show that hesitancy trends are stable across the range [0.2, 0.4], suggesting moderate reliability of the social recommendation module.
• Evaluated the quality of generated news by comparing GPT-generated articles with real-world news from the HuggingFace COVID News Dataset. We used GPT-4 to rate both sets blind to source on a scale of 1-5 (1 being poor and 5 being excellent) and control the news to have the same lengths. Results show that AI-generated news have average scores of 3.817 and 4.043, which demonstrates comparable quality.

Self-rating Bias

Thank you for highlighting the issue of LLM raters favoring their own outputs, a phenomenon that has been extensively documented in prior work (e.g., AlpacaEval). To mitigate this, in our local consistency evaluation (Section 4.3):

• We do not use GPT-family models as both evaluator and generator. No GPT-series models are among those being evaluated in the P4 of Figure 4.
• We also enforce length constraints on generated responses to control for verbosity bias in evaluation. We will clarify these methodological safeguards more explicitly in the final version.

Ablations

We appreciate the suggestion to include positive/negative controls or ablations to better isolate the contribution of each component. We have already conducted the following:

• In Figure 4 (P1), we compare simulations with and without attitude modulation, showing improved real-world alignment for several models (e.g., Llama-3.1).
• We run additional ablations of the warm-up phase. We remove the warm-up phase and perform 20-step simulations under the strong incentive policy for five different random seeds, and evaluate the MAE. It increases from 2.87% (with warm-up) to 5.00% (without warm-up). Without warm-up, the vaccine hesitancy reduction becomes more drastic with a less smooth curve, which shows that the warm-up phase is helpful for a more reliable simulation. We will include these results in the updated appendix in the camera-ready version.

Dear Reviewer 2uyB,

We appreciate your acknowledgement of our paper and your insightful feedback. We agree that a scalability study would be very interesting and clarifying. We run additional experiments with larger agent populations. Due to resource and time constraints during the discussion period, we ran on the scale of hundreds of agents. Please refer to our results and analyses in the general response (additional experiment 2), which show that the simulation behaves regularly on the scale of hundreds. We fully agree that emergent phenomena such as complex contagion, clustering, or opinion cascades require testing at larger scales. We are actively exploring frameworks like AgentTorch (Chopra et al., 2024), which offers infrastructure to scale multi-agent simulations to thousands or millions of agents. This is a key direction for future work, which we will address in our discussion of limitations.

We also thank you for suggesting we should tone down our paper. We will modify the phrasing in the contribution sections (where claims “pioneering”) and focus on highlighting the social media input and attitude modulation, following your advice.

We appreciate your concern about the potential pro-vaccine bias that may be inherited from LLM pretraining and the need for safety alignment. This is an important and underexplored issue. As a partial step, we did evaluate Llama-3-8B-abliterated, an uncensored model trained to bypass refusals to safety-related prompts (Arditi et al., 2024). As shown in Figure 4, its behavior does not substantially differ from the instructed version on vaccine hesitancy. We suspect that it is because Llama-3-8B-abliterated is trained on safety-related data, which does not include vaccine-specific data and does not debias pro-vaccine tendency.

We believe the reviewer is absolutely right in highlighting the need for a deeper investigation. This is an active area of interest for us, and we will note this limitation more explicitly in the final version. In future work, we aim to finetune agents on individual-level, vaccine-related demographic data to better capture the diversity of vaccine opinions and reduce systemic bias in simulation outcomes.

Dear Reviewer 99J4,

We genuinely thank you for your comprehensive consideration and thorough reviews.

Normalization of Saliency Scores

The saliency score (Eq. 1) is designed to rank memory lessons based on their relevance to agents, combining lesson importance and time decay (following popular work like Park et al., 2023). We normalize these scores to the [0,1] interval so that the saliency of older lessons remains comparable to newly assigned importance scores. Without normalization, older lessons with high absolute scores might dominate agent memory, even if their relative importance has decayed. This could skew the retrieval process and interfere with consistent memory-based reasoning. Normalization thus helps maintain relative comparability across time, consistent with prior work.

Order-preserving Attitude Modulation

For attitude modulation, we apply temperature scaling to the agent’s output distribution over vaccine attitudes. This method is widely used in machine learning (Hinton et al., 2015; Holtzman et al., 2019) to adjust distribution sharpness while preserving the original order of preference (when temperature > 0), as proven in Rahimi et al., 2020.

Alternative Similarity Metrics

We selected cosine similarity for news recommendation as it is simple, widely used, computationally efficient, and interpretable. Given the complexity of the overall simulation, we prioritized the simplicity of individual components. We agree that alternative similarity metrics (e.g., BERTScore, Jaccard similarity, etc.) are worth exploring, and we consider this a promising direction for future work.

Following Bias in Tweet Recommendation

For the following bias in tweet recommendations, we included a sensitivity analysis in the general response. As shown there, the system is insensitive across a range of values [0.2,0.25,0.3,0.35,0.4] with minimal difference on MAE, which supports that our system is not sensitive to perturbations in the following bias.

Average End Hesitancy

We compute end hesitancy as the average over the last 3 simulation steps to smooth high-frequency fluctuations at the end of the trajectory. Choosing a longer window (e.g., 5 steps) would represent 16–25% of our total simulation length (typically 20–30 steps), which would inaccurately represent terminal behaviors. In longer simulations (e.g., 50+ steps), a wider averaging window may be more appropriate, and we are interested in exploring variations in future work.

Warmup stage selection

The 5-step warm-up phase was selected based on preliminary runs without policy interventions. We observed that agent attitudes stabilize after approximately 5 steps. Extending this phase would increase computational cost without significantly improving convergence quality; hence, our choice represents a trade-off between stability and efficiency.

Empirical validations

Regarding empirical validations, while we acknowledge in our abstract and conclusion that VacSim is a first step and not intended for immediate policy deployment, we have incorporated multiple empirical validation strategies in the paper:

• Figure 5 shows that simulated trajectories can closely match real-world hesitancy trends (lowest MAE approximately 2.8%).
• Table 1 demonstrates that policy rankings generated by Llama-3.1 simulation correlate strongly with expert survey rankings (Kendall's τ=0.733, p = 0.056).

These results suggest the framework has meaningful application potential, which we emphasize as a foundational step for future deployment.

Readability

We appreciate the reviewer’s concern and agree that clarity is paramount. In the final draft, we will move key information, such as demographic schema, policy definitions, and prompt structures, from the appendix into the main text. In addition to Figure 3, which currently provides a detailed end-to-end example with persona, news, policy, and tweet generation, we will also add a compact summary table in the main text for reference.

Bib:

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. https://arxiv.org/abs/1904.09751

Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, Byron Boots. 2020. Intra Order-Preserving Functions for Calibration of Multi-Class Neural Networks. https://arxiv.org/pdf/2003.06820

Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. 2015. Distilling the knowledge in a neural network. https://arxiv.org/abs/1503.02531.