LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces

We greatly appreciate the reviewer’s feedback. In response, we have clarified our methodological framing of pluralistic alignment, explicitly acknowledged the limitations regarding global metrics such as CLIP and FID, and revised Figure 1 for consistency and clarity.

---

1: The claimed pluralistic alignment lacks support, relying on binary DPO without modeling intersectionality.

Response:
We propose a pluralistic alignment framework, as our dataset and workflow explicitly incorporate multiple criteria—Accessibility, Safety, Comfort, Invitingness, Inclusivity, and Diversity—each annotated independently. While we apply DPO, our primary aim is to test its capacity to capture community-driven, intersecting needs. The core contribution of this work lies not only in the dataset itself but in the participatory framework through which it was created. Our originality lies in operationalizing pluralistic alignment: from co-developing multi-criteria definitions with communities, to collecting fine-grained preference data, to empirically fine-tuning a T2I model. We further demonstrate how DPO can be extended to reflect localized values. Models fine-tuned on LIVS and other datasets developed using this framework have potential applications in community consultations and urban design processes. We hope the reviewer recognizes that this approach meaningfully advances alignment research by grounding it in real-world civic contexts. Our human preference data offer greater relevance for intersectional, community-based alignment than generic image quality metrics. The high proportion of neutral judgments, in particular, underscores the complexity of socio-spatial values—dimensions that standard T2I evaluation metrics are ill-equipped to measure.

To further clarify this distinction, we have revised the relevant text in the Introduction section as follows.

Revised text in Introduction:
“We propose a participatory data-collection framework that captures intersectional, multi-criteria feedback for T2I models in inclusive public-space contexts. While we refer to this as a pluralistic alignment approach to emphasize the local diversity of preferences, our method currently employs standard DPO with binary preference pairs, rather than a specialized multi-objective optimization algorithm. By integrating multiple locally defined criteria into preference annotations, we aim to expose both the potential and the limitations of a single-objective approach in accommodating intersectional needs.”

---

2: No CLIP or FID; relies on user qualitative analysis

Response:
We intentionally foreground community-derived judgments over global image-similarity metrics, as our primary aim is to assess how well the generated images align with local, context-specific criteria. FID, measures similarity to a global real-image distribution and is not meaningful in our context. We also avoid CLIP-based metrics, as prior work ([1], [2]) shows weak correlation with human judgments. Given our focus on subtle, value-driven distinctions across communities, relying on CLIP risks missing the very nuances we aim to evaluate.

Nonetheless, we acknowledge that such metrics can provide useful insights into broader questions—such as identifying common features shared by inclusive spaces across different typologies.

To reflect CLIP and FID limitations, we added this to the Limitations section:

“Our evaluation centers on human preference judgments rather than traditional metrics such as CLIP scores. While these metrics are useful for quantifying generative fidelity, they fall short in capturing nuanced, local, or intersectional considerations. ”

---

3: Figure 1 problem

Response:
Revised Figure 1 for clarity and consistency.

---

4: More project than a novel research contribution.

Response:
We respectfully disagree. Our framework and dataset are novel on multiple levels: (i) they capture multi-criteria feedback from diverse communities, thereby reflecting nuanced, real-world norms; (ii) they empirically demonstrate how DPO performs on intersectional needs, exposing a key gap in current alignment methods; and (iii) they introduce a participatory data collection framework that systematically grounds alignment in heterogeneous civic contexts. Unlike ImageReward and other prior efforts that rely on homogeneous annotator pools, our pluralistic dataset and analysis illuminate both the potential and the limitations of single-objective alignment when values meaningfully differ across populations. This combination of empirical insights and community-based methodology advances alignment research beyond conventional quality metrics, underscoring its broader social implications.

---

References:

[1] Ku et.al. VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. ACL 2024

[2] Ku et.al. ImagenHub: Standardizing the evaluation of conditional image generation models. ICLR 2024

We thank the reviewer for their feedback. We clarified how criteria were defined, explained neutral judgments, revised Figure 1, and added detail on axis-specific outcomes and image generation settings.

---

Question 1: Axes Relevance and Plausibility

Response:
It was both, since participants were looking at images during the focus groups. We revised our Criteria Consolidation section to clarify that the final six criteria emerged from community input on design and from probing whether each dimension could be assessed through images.

---

Question 2: Neutral Judgments and Multi-Criteria Evaluation

Response:
We clarify that neutral judgments reflect both genuinely balanced preferences and difficulties in visually encoding certain criteria, especially Inclusivity and Diversity.

New text in first paragraph in "Neutral Annotations as a Signal":
“Approximately half of the final evaluations were rated as neutral. While these outcomes may initially seem to indicate a lack of meaningful improvement, they could also stem from genuinely balanced preferences and participants’ difficulty in identifying visual cues for certain criteria (especially Inclusivity and Diversity). In several interviews, participants noted that subtle or symbolic elements did not always appear clearly. This indicates an opportunity to incorporate neutral signals explicitly into training and to explore methods that better visualize intangible attributes, such as inclusive design features, beyond purely aesthetic details.”

---

Question 3: Guidance Scale and Subtle Differences

Response:
Yes, we employed a moderate guidance scale during the image generation process. To ensure meaningful variation in outputs, we conducted pilot runs and fine-tuned hyperparameters such as seed, guidance scale, and steps. We also generated 20 images per prompt and used a greedy selection strategy, choosing the 4 most distinct images based on CLIP similarity scores, as detailed in Algorithm 1. This approach helped balance the risk of generating images with either overly subtle differences or excessively stylized divergences. We provide further clarification in Appendix (C.2. Image Generation).

---

Comment 1: Figure 1 Organization

Response:
We have revised the figure (Reviewer iv23 also finds this problematic).

---

Comment 2: More Nuanced Breakdown per Axis

Response:
We include additional discussion of axis-specific outcomes (see New Paragraph below), showing that some axes (Comfort, Invitingness) benefit more noticeably from DPO with more annotated examples.

New Paragraph in "4.1 Additional Observations":
“We further analyzed alignment improvements on each criterion by correlating annotation counts with the DPO model’s win rate. The criteria that received more annotations (Comfort and Invitingness) exhibited stronger improvements, suggesting that denser feedback can refine criteria-specific features more effectively. In contrast, Inclusivity and Safety showed a higher proportion of neutral or mixed outcomes, possibly reflecting both fewer annotations and the inherent difficulty of visually conveying representational aspects through T2I alone.”

Please see the link below, which contains a figure on the Average Tendency Toward DPO by Criterion https://anonymous.4open.science/r/livs-6E96/average-tendency.png

---

We hope these revisions address the reviewer’s concerns and enhance the clarity of our manuscript. Thank you for the valuable insights.

We thank the reviewer for their detailed and constructive feedback. Below, we respond to each comment. Where revisions are needed, we provide the updated text.

---

Comment 1: Multi-criteria preference learning but collapsing annotations, neutral annotations not leveraged; disagreements across criteria are collapsed

Response:

We acknowledge that our current method for DPO training reduces multi-criteria signals into a single reward label. While this dataset is inherently multi-criteria, this reduction reflects a pragmatic simplification for the initial phase of DPO training. However, this also underscores a broader limitation of DPO: it is not straightforward to perform multi-criteria alignment—particularly when preferences across criteria conflict or when alignment must be criteria-aware within the same model. We have clarified this in the revised text and emphasize that future work is needed to develop methods capable of capturing such partial, intersecting, or contradictory preferences in a more principled way.

Revision in 2.1. Alignment of Generative Models:
“Although the LIVS dataset contains multi-criteria feedback, we initially collapse these signals into a single preference label for each pair during DPO. This step overlooks conflicting or nuanced assessments across different criteria. Future work is needed to explore approaches that account for intersections and disagreements without forcing a single binary label, thereby preserving the richness of multi-criteria preference data.”

---

Comment 2: Figure 3 does not clearly show "comprehensiveness" of prompts

Response:
We revised the text to clarify that the word cloud is a preliminary visualization of different concepts within prompts. We rely on Jensen–Shannon Divergence (JSD) scores and scenario-based coverage to demonstrate prompt diversity. The figure is meant only as an illustrative snapshot.

---

Comment 3: Synthetic prompts not validated by humans

Response:
We acknowledge that synthetic prompts were primarily evaluated using automatic diversity checks (JSD). As noted in the limitations, this expands coverage but may lack the contextual specificity of human-authored prompts. Future work needs to incorporate human validation to improve relevance.

---

Comment 4: Use of "democratizing" (line 43) requires more context

Response:
We removed the term “democratizing” and clarified that T2I tools aim to lower barriers to community participation in design. Thank you for the reference.

Revised Sentence in Introduction:
“These developments can benefit communities by making design processes more accessible—enabling broader engagement among non-expert stakeholders in architecture, urban planning, and environmental visualization.”

---

Comment 5: Citation issue

Response:
Rectified.

---

Comment 6: Participatory action research grounding

Response:
We added the following paragraph in the Methodology: Building the LIVS Dataset section.

Revised Paragraph:
“Participatory Action Research (PAR). In line with the principles of PAR, our community-based approach centers on iterative, collaborative inquiry and reciprocal learning throughout the dataset development process (Israel et al., 1998; Cornish et al., 2023). By involving local organizations as active co-researchers, we ensured that the framing of inclusion, safety, and other design criteria emerged from lived experiences rather than external prescriptions. This iterative feedback loop aligns with PAR’s emphasis on collective problem-solving and empowerment, as participants guided each stage of data collection and model evaluation while gaining familiarity with T2I technology and its potential applications in urban contexts.”

---

Comment 7: Question—On average, how many criteria per image pair?

Response:
On average, each image pair received approximately 1.49 non-zero (decisive) annotations from community members.

---

Comment 8: Question—Refining definitions like "Inclusivity" for more distinct preferences

Response:
This is a trade-off: while detailed definitions may yield clearer preferences, our prompts were prescriptive and annotations observational to avoid biasing participants. Now, with the full dataset, we plan to analyze each criterion by identifying subdimensions and associated objects for emerging patterns.

---

Comment 9: Question—Figure 8: Do left and right images come from the same or different models?

Response:
Regarding Figure 9 (since Figure 8 pertains to prompt methods), both images were generated by the same SDXL model.

---

We appreciate the suggestions, which helped clarify multi-criteria signals, prompt diversity, and neutral annotations. We believe the revisions address the reviewer’s concerns and strengthen the paper.

References

1. Israel, B. A. et al. (1998). Annu. Rev. Public Health
2. Cornish, F. et al. (2023). Nat. Rev. Methods Primers

Thank you for the constructive feedback. We clarified the rationale behind using community-informed prompts over universal keywords, explained the three-criteria annotation design, and updated key sections for clarity.

---

Comment 1: Necessity of the LIVS Dataset vs. Enhanced Prompting

Response:
We recognize that refined prompting strategies—for instance, appending keywords such as accessibility, safety, or inclusivity—can guide generative models toward more targeted outputs. Approximately half of the prompts used to generate images included at least one criterion (see Table 1). However, LIVS emphasizes that lived experience varies considerably across social identities and contexts. In many urban settings, local communities articulate inclusivity or accessibility in ways specific to their sociocultural histories. As demonstrated in prior research (e.g., Beebeejaun, 2017; McAndrews et al., 2023), relying solely on universal keywords can impose a fixed and potentially biased interpretation of inclusive space. By contrast, our dataset incorporates diverse local knowledge and community priorities, reducing the risk of reproducing a single design standard that may neglect intersectional needs (Crenshaw, 1997; Low, 2020).

Table 1: Prompts with at least one of the six criteria vs. prompts without explicit criteria, and corresponding neutral response performance.

We adopted a localized approach without imposing a predefined notion of inclusive space on participants. Our objective was to maximize the diversity of both images and prompts, allowing for an emergent understanding of what inclusive or diverse space could mean in different contexts (Anttiroiko & De Jong, 2020; Madanipour, 2010). To our understanding, this design encourages community-driven insights rather than a one-size-fits-all approach to inclusivity.

Revised Paragraph (in Prompting and Early Feedback):
“Although refined prompting techniques can shape generative outputs, universal keywords alone may overlook local sociocultural and historical contexts (Anttiroiko & De Jong, 2020; Beebeejaun, 2017; Talen, 2012). Our goal in creating the LIVS dataset was to integrate granular, community-generated perspectives on accessibility, safety, and inclusivity. By embedding localized knowledge, we reduce the likelihood of producing a uniform design standard that might disregard certain intersectional needs (Low, 2020; Madanipour, 2010; McAndrews et al., 2023).”

---

Comment 2: Annotating Only Three of the Six Criteria per Comparison

Response:
During our tutorial workshop, participants indicated that rating fewer or more than three criteria simultaneously was cognitively demanding and reduced their focus on each image. Including more criteria also gave a sense of reduced progress, especially since the initial annotation batches comprised around 1,000 comparisons, later reduced to 750. We also observed that displaying six rating elements constrained the interface, limiting visibility of the image pairs and making it harder to assess spatial details. As a result, we randomly assigned three criteria per annotation task. This approach balanced coverage and participant effort while preserving image clarity. We confirmed that all six criteria received substantial coverage across multiple batches.

Revised Paragraph (in Annotations and Evaluation):
“To optimize data quality and minimize cognitive fatigue, we randomly presented three of the six criteria in each comparison. During pilot trials, participants found evaluating all six criteria difficult, which reduced their sense of progress and visual engagement. The multiple rating elements also constrained image size, making it harder to assess spatial details. Focusing on three criteria enabled more meaningful engagement. Over successive annotation batches, we ensured that all six criteria were robustly evaluated.”

---

We hope these clarifications and revisions address Reviewer's concerns. We appreciate their detailed comments, which have helped strengthen the methodological clarity and contextual framing of the paper.

---

References

1. Anttiroiko, A.-V., & De Jong, M. (2020). The Inclusive City. Springer.
2. Beebeejaun, Y. (2017). Gender, urban space, and everyday life. J. Urban Affairs, 39(3).
3. Crenshaw, K. (1997). Demarginalizing intersectionality. In Feminist Legal Theories. Routledge.
4. Low, S. (2020). Social justice and public space. In Companion to Public Space. Routledge.
5. Madanipour, A. (2010). Whose Public Space? Routledge.
6. McAndrews, C. et al. (2023). Toward gender-inclusive streets. J. Planning Lit., 38(1).
7. Talen (2012). Design for Diversity. Routledge.