Monoculture or Multiplicity: Which Is It?

Dear Reviewer t4zS,

Thank you for the time you spent reviewing our work and your valuable feedback and suggestions! We appreciate your positive comments on our empirical analysis and the value of our findings. We address your remaining questions or concerns in the following paragraphs:

"strict monoculture" is a bit of a straw man [...] suggest that the discussions, especially around recourse, might want to have more nuance

We thank the reviewer for raising this point. We find it helpful to contrast both extremes, strict monoculture and strict multiplicity, as a way to evaluate and reason about a given model ecosystem. To address your comments, we will more clearly term strict monoculture as the far end of a spectrum encompassing model multiplicity and monoculture and will revise our discussion of recourse in order to better reflect the nuances of our findings.

"individuals overwhelming have recourse" might be too strong a statement.

We agree that the fraction of positive instances experiencing no recourse is indeed significant and hence this phrasing might come across as too strong. We will update the text to reflect this better.

numbers [of individuals experiencing no recourse] are much much higher for the two ACS tasks in figure 8

Thank you for bringing this up. We assume you're referring specifically to the ACSMobility and ACSPublicCoverage tasks, where we observe notably higher rates of individuals experiencing no recourse. We provide some discussion on both tasks in the Appendix, but will revise the text to address and clarify this earlier in the paper.

Both tasks are characterized by high class imbalance, with a low prevalence of the positive class. Since the Rashomon set is constructed based on overall accuracy, it tends to favour models that prioritize the majority class. Especially pronounced for ACSMobility, the Rashomon set includes mainly models behaving highly similar to the constant majority-class predictor (see also the general model performance on ACSMobility), resulting in a high fraction of individuals receiving no recourse. This may be exacerbated by low predictive signal of the task, which makes it challenging for models to improve over trivial baselines.

When models are instead selected using balanced accuracy (see Appendix B.2.2), which accounts for class imbalance, the picture changes: fewer individuals experience no recourse, and more receive substantial or full recourse. A similar trend is seen for ACSPublicCoverage.

This suggests that the elevated rate of instances without recourse observed for these tasks may be driven by a combination of class imbalance, low signal, and accuracy-based model selection for the Rashomon set.

If you have any further questions or concerns, we will be happy to address them.

The authors

Dear Reviewer CF6P,

Thanks a lot for the time you spent on providing valuable feedback on our work and helpful suggestions on improving the paper. We appreciate your positive comments on the novelty of our evaluation and our sound experimental design.

To address remaining concerns regarding our manuscript, we have revised the text where suggested and, to improve the empirical evaluation, added additional results on the Survey of Income and Program Participation (SIPP). Please find below a detailed discussion of the points you have raised.

tasks used for evaluation are highly limited

To address your concerns regarding task diversity, we have added empirical results for one additional task defined on the longitudinal Survey of Income and Program Participation (SIPP). Here, the goal is to predict whether a person's income is significantly above the Official Poverty Measure (OPM). Summarizing the results we find similar patterns as observed for the other tasks. There is substantial predictive similarity across models. We observe a mean agreement rate of 0.84 that clearly exceeds the baseline agreement rates (63%). Still, agreement is generally below full agreement for all model pairs. In terms of recourse, 5% of individuals experience no recourse, 77% substantial recourse and 48% full recourse.

While we acknowledge your point that all ACS tasks are derived from the same underlying data source (US-wide ACS PUMS data), we want to point out that ACS, being derived from US Census data, is one of the most comprehensive surveys worldwide. As such, it provides a rich, diverse, and high-quality representation of the US population, making it a strong foundation for a wide range of prediction tasks such as employment, education or housing.

To ensure comparability with prior work, we adopt five tasks predefined by Ding et al. (2022). These tasks span a broad range of prediction challenges—from high predictive signal to more difficult low-signal settings (such as ACS Mobility)—and collectively make up a diverse and valuable benchmark suite. While some features (such as age, race, and sex) appear across tasks due to their relevance for fairness analysis, their overall feature sets differ, and none is a strict subset of another. Further, each task is constructed on a distinct subpopulation (e.g., adults, employed individuals), with no individual appearing in all five tasks and a maximum pairwise overlap of only 10.1% of individuals relative to the task’s size. Importantly, we view the partial overlap in individuals as an opportunity to study model multiplicity across different contexts (see Bommasani et al., 2022).

task modalities are also restricted

We appreciate the reviewer’s suggestion to include additional task modalities. We agree that extending the analysis to a broader range of modalities is a valuable direction. However, this falls outside the scope of the current paper as it would require a new model suite and task set. We want to note that our focus on tabular data reflects both practical and normative considerations: Tabular data is the primary modality for risk assessment in high-stake domains such as hiring, credit, criminal justice or other bureaucratic contexts. As a result, much of the normative discourse about the concerns of model multiplicity and algorithmic monoculture as well as recourse has centered on this modality. In this sense, our choice aligns with a broader body of research that has largely concentrated on tabular data. Nevertheless, we see expanding to other modalities as a promising direction, and we expect that our evaluation framework could be readily adapted for that purpose.

1. What are the possible factors that cause the significant differences in the sample proportions corresponding to rec(x) > 0.5 across different datasets, as shown in Figure 2?

We assume you're in particular referring to the ACSMobility and ACSPublicCoverage tasks, where we observe notably higher rates of no recourse affecting also the fraction of individuals that experience substantial recourse ($\text{rec}(x) > 0.5$). We provide some discussion on both tasks in the Appendix, but will revise the text to address and clarify this earlier in the paper.

Both tasks are characterized by high class imbalance, with a low prevalence of the positive class. Since the Rashomon set is constructed based on overall accuracy, it tends to favour models that prioritize the majority class. Especially pronounced for ACSMobility, the Rashomon set includes mainly models behaving highly similar to the constant majority-class predictor (see also the general model performance on ACSMobility), resulting in a high fraction of individuals receiving no recourse. This may be exacerbated by low predictive signal of the task, which makes it challenging for models to improve over trivial baselines.

When models are instead selected using balanced accuracy (see Appendix B.2.2, Figure 10), which accounts for class imbalance, the picture changes: fewer individuals experience no recourse, and more receive substantial or full recourse. A similar trend is seen for ACSPublicCoverage.

This suggests that factors such as class balance, the predictive signal of the prediction task, and the selection criterion for the Rashomon set can all influence the observed levels of recourse. We hope thas adequately addresses your question. If anything remains unclear, we’d be happy to clarify or discuss further.

2. Could the authors discuss more insights from their evaluation conclusions for the design of future LLMs, or provide recommendations on how to organize LLMs models to avoid harming individuals?

Thank you for raising this point. We will incorporate it more clearly into the discussion. Our work illustrates what a system-level analysis might look like to assess the degree of monoculture or multiplicity within a model ecosystem. We believe that conducting such system-level evaluations and increasing transparency about the extent to which individuals have recourse is both valuable and necessary. This kind of analysis could serve as a starting point for developing something like a "multiplicity index," as suggested by Reviewer 6KfP. Accordingly, we advocate for auditing entire model ecosystems, not just individual models. If homogeneity is identified as a significant concern, then concrete (policy) interventions, such as randomization, may be warranted. For a detailed discussion of the policy implications surrounding monoculture and multiplicity, see also Gur-Arieh et al. (2025).

With regard to individual harm, we caution against treating model multiplicity as a straightforward remedy for the risks posed by monoculture. Both extremes carry distinct concerns, and neither should be viewed as a solution concept for addressing harm in language model ecosystems. Simply increasing diversity among models does not guarantee better outcomes for individuals or society. Instead, our findings suggest that the kinds of harm individuals may experience are not adequately captured by focusing solely on either monoculture or multiplicity. A more nuanced framework is needed to understand and mitigate these risks effectively.

Thank you for your thoughtful comments. We believe the revisions have strengthened our work, and we would be happy to address any further questions or concerns.

The authors

Thank you for your response.

Regarding the insights from this paper, I understand that the intention is to conduct a system-level individual-recourse evaluation. My concern, however, lies in the fact that the recourse evaluation results appear to vary across tasks, making it hard to provide guidance for the subsequent design of model ecosystems. Furthermore, the tasks themselves are limited to binary classification and relatively small-scale risk assessments and predictions. Although the authors acknowledge these as existing limitations, I hope there can be at least one feasible discussion on how the findings of this paper can be applied to design model ecosystems — for instance, selecting models that maximize required outcome diversity, accuracy, or even fairness.

The models evaluated in this paper are open-source, and there may be overlapping base models and training data. I wonder whether these factors place the findings of this work in a somewhat underappreciated middle ground between monoculture and multiplicity. What would happen if the model sets were trained on disjoint datasets? For instance, when evaluating sensitive topics (highly national culture related) across models trained on different language corpus such as English, Chinese, Hindi, or Japanese, would we observe multiplicity? Alternatively, if the training datasets are highly identical, would this lead to monoculture? If such possibilities exist, it suggests that the conclusions drawn from the current experimental results may be limited and difficult to generalize broadly. I hope the authors have some thoughts on the observed “middle ground between monoculture and multiplicity” and the reasons behind it.

Given these points, I am currently inclined to retain the original score and look forward to hearing the authors’ thoughts on the insightful value of their work.

Dear Reviewer 6KfP,

Thanks a lot for the time you spent reviewing our work and for your valuable feedback! We appreciate your assessment of our work as accessible and our experiments to be comprehensive.

We address your questions in the following paragraphs:

1. What is a non-realizable task? Should we be concerned about the fact that this adjustment is necessary? Does it affect the conclusions of the paper?

A prediction task is non-realizable when no model, regardless of its complexity, capacity or training data can achieve perfect performance. This typically stems from inherent stochasticity in the data generating process, underdetermined problem formulations or a lack of ground truth. This is a relevant distincting from a body of literature that focuses on the evaluation of LLMs on realizable tasks where there is a unique correct label for each data point (such as factual question-answering).

Threshold adjustment is a standard component of classification tasks, not a cause for concern. Since models often output probabilities, adjusting the threshold allows alignment with trade-offs between sensitivity and specificity, particularly in imbalanced or cost-sensitive settings. In the main paper, we report results optimizing for balanced accuracy, which we consider appropriate given the class imbalance of some tasks.

To address your concerns, we also re-ran the analysis optimizing thresholds for overall accuracy. Results were largely consistent: observed recourse levels remained between strict monoculture and strong multiplicity. However, for highly imbalanced tasks (e.g., ACSMobility and ACSPublicCoverage), accuracy optimization favors models that resemble the majority-class predictor, resulting in high fractions of individuals with no recourse—models we consider unlikely to be deployed in practice. We will include those results in the Appendix of the paper for full transparency.

2. [Does the tolerance of $\epsilon=0.05$ for inclusion in the "Rashomon set" imply] we can't possibly expect to have full monoculture? Is there a way to quantify the highest level of monoculture given the 0.05 tolerance?

We appreciate the reviewer’s observation that differing accuracy levels already act as a force against monoculture. Indeed, two models with accuracies $a \neq a’$ cannot agree on more than $1-|a-a’|$ of the overall population. However, this bound applies at the population level.

Our analysis focuses on positive instances, where it does not directly apply without assumptions about the conditional agreement. Assuming the positive class constitutes at least an $\epsilon$-fraction of the dataset, perfect agreement on this subset (i.e., strict monoculture) remains theoretically possible—even when overall accuracies differ slightly. We will clarify this in the revised paper.

Even given this additional information, we maintain that strict monoculture provides a natural and interpretable baseline, capturing the limiting case of full predictive homogeneity.

3. Could you justify this particular value of epsilon? [...] 90% accuracy and 95% accuracy seems very substantial to me [...] wasn't clear what the absolute accuracy was for all tasks.

Thank you for this question. We fully agree that a 5% accuracy difference (e.g., 90% vs. 95%) is meaningful, as it reflects a substantial relative change in error. In prior work, $\epsilon$ values between 0.01 and 0.5 are commonly explored (see e.g. Marx et al., 2020, Hsu et al., 2023, Watson-Daniels et al., 2023).

Our particular choice of $\epsilon$ is not intended as a normative threshold, but reflects a practical trade-off: including plausible deployment candidates from multiple major LLM providers while avoiding overly narrow empirical Rashomon sets. In fact, for some tasks, lowering $\epsilon$ to 0.01, reduces the empirical Rashomon set to a single model..

That said, our key findings remain consistent even with more restrictive $\epsilon$ values. Currently, Figure 3 provides some insights into that, showing how the fraction of individuals with no recourse is affected by different $\epsilon$ values. In the revised version of the paper, we will further expand on this by adding a plot on the distribution of recourse for different ϵ values for greater clarity.

Finally, we agree that the Rashomon set abstracts away from absolute performance, which is why we report overall performance metrics in the Appendix. We will make this connection clearer in the revision.

hard to know what to make of the fact that the truth is somewhere in the middle [...] I wonder if there's a way to come up with a "multiplicity index"

Thank you for raising this important point. Our goal is to illustrate what a system-level analysis of model ecosystems might look like, highlighting the degree of monoculture or multiplicity and its implications for individual recourse. In light of our findings occupying the middle ground between strict monoculture and strong multiplicity, we recommend conducting such system-level analyses and create transparency by reporting the level to which individuals have recourse.

We appreciate your suggestion to develop a “multiplicity index” to compare different model ecosystems. This could be a valuable next step, and we hope our work provides a good starting point for such efforts. Importantly, we caution against viewing multiplicity as a solution concept to monoculture and vice versa. A more nuanced framework is needed – one that considers ecosystem dynamics and captures individual harms arising from it.

If you have any further questions or concerns, we will be happy to address them.

The authors

Dear Reviewer tNUx,

thanks a lot for the time you spent reviewing our work and your detailed and valuable feedback! We are pleased to read your positive comments on our empirical analysis and the value of our findings, contributing to a “holistic understanding of monoculture or multiplicity”. We address your remaining questions in the following paragraphs:

1. Why specifically 10-shot learning? Do you notice different results when reducing the amount of samples in few-shot learning?

We had performed a sensitivity analysis using 4-shot and 8-shot learning on ACSIncome and noticed that there was no difference—we’ll include these checks in the Appendix of the paper. We ended up showcasing the results for more samples to allow for considerations related to in-context variations and class balancing. Other than that, we acknowledge that the exact number of shots is somewhat arbitrary.

2. Does $x_i$ always represent a different individual or can there be overlap between individuals across the ACS tasks?

Thank you for the question. While all five ACS tasks adopted in his work are derived from the same data source, US Census data from 2018, each task is constructed on a distinct subpopulation (e.g., adults, employed individuals). As a result, individuals $x_i$ do not consistently appear across tasks: The maximum overlap in terms of individuals is at most 10.1% for any pair of tasks, 0.8% for any triplet, and 0.08% for any combination of four tasks—measured relative to each task’s size. No individual appears in all five tasks.

3. Why do you use base pre-trained versions of the model as well? I would imagine that most decision-makers would use the instruction tuned version.

We thank the reviewer for raising this point. As shown in Appendix B.1, we find that across many tasks, there is often no clear performance gap between base models and instruction-tuned variants. Including base models thus provides a more comprehensive picture. In line with your comment below, they also serve as a useful reference for analyzing the impact of instruction tuning on multiplicity and monoculture. We will update the paper to include results separated for base and instruction-tuned models.

Base models can be naturally more attractive to decision makers for several reasons. Unlike instruction-tuned models, they do not impose an instruction-response format, offering greater flexibility to be adapted in custom workflow or in deployment settings where a particular alignment may be undesirable. Moreover, while instruction-tuned models often show better alignment with human preferences on realizable tasks, they have also been observed to exhibit overconfident risk scores (Cruz et al, 2024) on the tasks we study.

It would be very interesting to know how [recourse] changes across model size and/or family and base pre-trained vs. instruction-tuned.

We appreciate the reviewer’s suggestions and have added additional results conditioned on either model developers or model type (base vs. instruction-tuned).

When grouping by model developer, we find no consistent trend across tasks for specific model families. Notably, all Rashomon sets contain models from at least four different developers, though for some tasks and families, the number of models is restrictively small (only one or two models), limiting interpretability. In terms of model agreement, the finer-grained results align with our aggregate-level findings. Among model families with at least three members in the Rashomon set, agreement rates consistently fall between the extremes of strict monoculture and strong multiplicity.

Extending the analysis to include agreement rates across all models—not just those in the empirical Rashomon sets—we observe that Qwen models exhibit slightly higher in-family agreement than others. In contrast, OLMo and Gemma models tend to show slightly lower agreement within their respective families.

When grouping by instruction tuning, we observe slightly higher agreement rates among instruction-tuned models across tasks. However, recourse levels remain comparable between base and instruction-tuned models.

We will update the revised paper to include these more fine-grained analyses and their implications.

it would really strengthen the paper if there is a discussion on whether there are specific instances [...] where the models perform rather monotonously versus this multiplicity.

Thanks for raising this point. We absolutely agree that this would strengthen the paper and will include a plot showing recourse levels by demographic attributes (e.g., age, sex, race) in the revised version to examine more fine-grained disparities.

Better understanding of which individual-level factors might contribute to whether an individual is more likely to have no recourse or many opportunities for recourse in a (language) model ecosystem would be an interesting endeavour for future work.

Introduction section could benefit from a bit more clarity and conciseness

Thank you for your detailed feedback and suggestions—we will revise the introduction accordingly. Regarding your comment on Figure 1, could you kindly elaborate on which aspects you find non-ideal? Any additional detail would be greatly appreciated and will help us improve this figure more effectively.

Line 217: Is it that bad that models yield highly similar outputs? For such high-consequential tasks, it might actually be a good thing that there is consistency in predictions across different models?

Thanks for bringing this up. Models agreeing on consequential tasks can be seen as an indicator of consistency, on the other hand this is exactly what literature on monoculture warns about. It may mean that individuals have no opportunity for recourse. In particular for unrealizable tasks this might be undesirable. Our point here is not to provide a normative framework, but to evaluate whether this occurs in practice in language model ecosystems. We will adjust the opening of Section 4 to clarify this.

Finally, we thank the reviewer for several concrete suggestions for improvements of the flow and layout of the manuscript (e.g. shortened introduction, link to risk assessment earlier, improved opening of Section 4). We agree with all points made by the reviewer and will update the manuscript accordingly.

If you have any further questions or concerns, we will be happy to address them.

The authors

Thank you for addressing my concerns, answering my questions, and adding the additional analysis!

A discussion on how to deal with this model multiplicity and/or monoculture based on what you see in this task extrapolated to general scenarios (task-agnostic) could be a nice addition as well.

With respect to Figure 1, by itself it is completely fine and clear. Personally, the effect of already showing a figure of the results in the introduction did not add much to the story there for me. It is actually discussed in the Results section and I feel that is a more fitting placement for it (but this might be a very personal opinion). A less detailed figure in the introduction would work better for me, e.g., where you highlight more the discourse surrounding model multiplicity and monoculture.

I will keep my score the same.