Values in the Wild: Discovering and Mapping Values in Real-World Language Model Interactions

Thank you for your close reading of our paper! On your “reasons to reject”:

1. Assistant role vs. model itself: It is important to understand the role of the AI assistant framing, and we do mention this in the Results section. Two thoughts on the suggestions:

   a) It would be difficult to analyze model value preferences without the assistant framing, since it is a lot of work to try to create an apples-to-apples comparison here (e.g. it doesn’t seem like there is existing data on this, so to collect this data you would have to fine-tune a model to instruction-follow in exactly the same ways, just without the assistant framing, which is quite expensive). Furthermore, a user study would be very interesting, but is potentially more suited to being a separate piece of work, given we were already finding it hard to fit all our work into the existing page limit for this paper.

   b) We thus decided to just scope our analysis to the AI assistant since this is the dominant interaction paradigm, and although we might be conflating model vs. assistant-role values, the assistant role is generally what people experience.

2. Thanks for noting works we had not cited, we will be very glad to include them in our related works section.

3. On the inter-rater agreement question: We had six independent annotators review our feature extraction outputs, dividing outputs between all raters such that we did not have multiple raters for each output. We felt that the high accuracy bounded the possible inter-annotator agreement in some sense, and so felt that it was not necessary to add additional work to our raters, since they were already rating a lot of conversations across five different features.

On your other questions:

1. Principles vs. values: Our taxonomy shows that HHH manifests through specific, contextual values. "Helpfulness" (our most common value) breaks down into granular values like "accessibility" and "user enablement." While we could map all values to HHH principles, the mapping is inherently fuzzy (e.g., "patient wellbeing" serves both helpfulness and harmlessness). Regarding Claude's Constitution, we could compare it to the constitutional principles published online. However, many of the principles overlap with one another and it was not too clear to us how useful it would be to measure correspondence to them for these reasons. So far, there do not seem to be very robust established values frameworks for LLM model behavior that we could have compared to, but we hope that our work can contribute to the creation of something like this.
2. Generalizability: While we focus on Claude, we believe that our methodology should generalize to other AI assistants. The WildChat analysis (Appendix A.5) demonstrates our feature extraction works on other models' conversations. We encourage other researchers apply this framework to different models to build a more comprehensive understanding of value expression across AI systems.
3. Mirroring and sycophancy: Value mirroring shares characteristics with sycophancy but differs importantly: Claude mirrors selectively (20% during support vs. 1% during resistance), suggesting nuanced behavior that reflects user values when appropriate but diverges when problematic. Further research could explore whether this represents appropriate responsiveness or concerning sycophancy by digging into specific ocntexts.

Thank you for closely reading our paper. On your questions:

1. We are happy to look at changing the figure to report the number of categories per level.
2. On the taxonomy stability: Because we prompt a model to generate cluster names and descriptions, we had to iterate a few times on this to ensure it made sense. Initially, the cluster names were overly domain-specific, and resulted in high-level clusters that looked like “Professional and academic integrity standards” and “Professional governance and operational excellence.” This was too hard to interpret at a glance, so we had to steer the cluster-naming model towards more theoretical and abstract cluster names (by prompting it to do so). Once we had that set, the taxonomy was reasonably stable. We tried clustering it 3 different times with slightly different k-means parameters, but we didn’t try different embedding models, since this didn’t seem necessary. This all being said, it is in fact the case that depending on what organizational framework you prefer, you can organize the values differently. We don’t think there is one right way to cluster these values — there are probably a number of different “local minima” here. Values are so multifaceted, it’s probably impossible to taxonomize them perfectly. Part of us releasing the 3308 values is to encourage and allow others to come up with different taxonomies for different applications.
3. We are not sure — we hope that there will be follow up work that could replicate this on other models.
4. Because this is based on user data we’re unfortunately limited in the additional pieces of information we’re able to share, especially what is not directly relevant to the question of values (e.g. turn count).
5. We hope to have a discussion about this when the paper is non-blinded!

Thank you for your detailed attention to our paper. To respond to your question, we are happy to include an example or two of the non-subjective conversation topics that were removed if it would be additionally useful to some of the examples we have in the Appendix already, where we show via the prompt the kind of conversations we are removing (they are included as few shot examples) — e.g. factual questions (“Is this SQL query syntactically correct?” “Do I need a visa for Japan?”) or mostly based on established knowledge e.g. around debugging this function, or creating a financial report.