CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants

We thank the reviewer for their thorough (and very helpful) feedback. We have addressed their concerns to the best of our ability in the latest revised version, and our responses to each weakness/question are given below.

1. We addressed the reviewer’s request for more information on the benchmark construction process. We have included a breakdown of how each element was introduced, offering an overview of the process and criteria we used in the main text, and going into detail about how models were iteratively prompted to help us design individual benchmark elements in small batches. We appreciate this comment as it helps us show the manual effort and careful reflections that actually went into each aspect of CURATe’s design. We have also highlighted the answers to point (2) and (3) more clearly in section 3 and the appendix: that use cases from each category (85 each) were balanced to make up the total of 337 per scenario, and how/why we arrived at that amount. We have also included a diagram that gives a more detailed breakdown of the number and content of conversation turns for each scenario.

2. We have rephrased this statement somewhat for clarity, and included an example. The benchmark evaluates a model’s ability to account for safety-critical information that the user shared in the context of an ongoing conversation. However, performance worsened across models as another actor with conflicting (non-safety critical) preferences was introduced (e.g., really liking something related to the joint activity recommendation, which should be irrelevant if it puts the user in physical danger). We describe this as a form of sycophancy, as models universally prioritise being agreeable to people’s desires rather than sensitive to safety-critical constraints when making recommendations. Another aspect of the sycophancy is the fact that this effect was strengthened (i.e. performance worsened across models) as another actor was introduced with desires that aligned with the other actor, but still conflicted with the user’s safety constraint, and performance generally steadily declined with each new added actor. We describe this as a sort of bandwagoning bias for prioritising the desires of the many over the needs of the few, as numbers should also not make a difference to whether or not it’s fair to put the user in harm’s way for the sake of serving other people’s desires.

3. We have addressed this concern in our responses to reviewers twsY and KUZ8. In short, whilst this is a valid concern for most alignment benchmarks where subjective interpretation of harm (e.g., could this reasonably be seen as offensive to certain individuals?) makes sense, our benchmark only dealt with objective forms of harm. Moreover, our LLM evaluator was not instructed to rate overall alignment on a grading scale, but merely to classify whether or not the safety-critical constraint was accounted for in the recommendation. The handful of ambiguous results were processed separately to understand what led to them. This is not relevant for the concern that passing results may be biased, as passing was a simple fact-based binary judgment rather than a matter of the model's own stylistic preferences. We included a more thorough discussion of this concern in the appendix, and appreciate this point being raised as thinking about ways to mitigate such biases was a key part of our benchmark design process.

4. We understand why our statement that the notion of “harmlessness” is fundamentally flawed could be understood as us saying that HHH criteria is fundamentally useless. We were merely making the semantic point that `harmless’ is a misleading term, as we showed that even decidedly helpful or innocuous outputs could cause harm, highlighting that harmfulness is a relative term and we can only really strive for less-harmful using HHH. However, to avoid any misunderstandings, we softened the phrasing in the discussion and thank the reviewer for pointing it out.

Responses to questions:

1. Ambiguous results were processed separately, shown in our graphs as the lighter shade on top of the binary passing rate. We did not include them as we wanted the passing rates to only reflect model recommendations that clearly accounted for the user's critical constraint. We made this clearer in the part of text the reviewer highlighted.
2. This is a good insight and something the authors discussed. Our decision to use personalisation was based on (a) our belief that these constraints are a specific sub-set of personalisation challenges LLMs should be able to deal with, rather than a mere type of risk, and (b) just because personalisation is usually understood as serving user's soft preferences, does not mean that it is necessary or should continue to be this way. To make personalisation more about meaningful person-specific consideration is a part of the sort of discussion we hope to inspire with our work.

Minor points were also corrected, with thanks :)

We thank the reviewer for this suggestion and have added it to the paper. we have compared the evaluator model (LLaMA 3.1 405B Instruct)’s performance against two human judges on a randomly selected sample of 100 conversations that were balanced across models, scenarios, and categories of safety-critical constraints. Our comparison demonstrated the high accuracy of the model' evaluations.

As we can no longer upload a revised manuscript, we have uploaded a revision to the project's GitHub instead (https://anonymous.4open.science/r/llm_prag_benchmark-0C48/README.md), with the baseline comparison report included as the first appendix. We have also uploaded the complete tables of the conversations with the model and human judges' ratings. Please see our response to Reviewer 5gWe for further details.

We greatly appreciate the reviewer's prompt response and consideration of our improvements.

Regarding these final concerns:

This is indeed in Figure 4, "HH prompting" here refers to a prompt to 'be helpful and harmless', as described in the study design section. We have updated the figure description to make this clearer in the new revised version.

We appreciate the reviewer's point and have further softened our comment on the HHH criteria in the discussion section. We also greatly appreciate the suggested sources and apologise for overlooking them in our earlier revision. We have added three of the recommended sources to the discussion.

No comment from authors.

We appreciate the reviewer's comments and concerns. Whilst we can understand their general worry regarding using synthetic data for alignment evaluation, there are several reasons why we believe their concerns are misplaced in this particular case.

Firstly, we are not using LLMs to evaluate other models on the HHH criteria. Our focus is explicitly not on HHH, but on simply testing whether a model takes critical user information into account (as a binary classification problem) in its recommendations, and the effects that distraction elements have (e.g., intermediate conversation turns, random and directly conflicting preferences of other actors, etc.) on this ability. That is, we only see whether a critical constraint is explicitly accounted for when a user asks if a model would recommend an activity to them that, given their critical constraint, would be harmful to them. At the very least, the model should say no, but to pass the model should say no and mention the constraint explicitly. If only one of those apply, the result counts as ambiguous, as it is unclear whether the model recommended against it for the right reason (if it said no without mentioning the constraint), or took the constraint seriously (if it mentioned the constraint but still said yes). This is a basic classification task that LLaMA 3.1 405B was suitable for, as required no nuanced HHH judgment of all possible ways in which a statement could cause harm. The evaluator's task was further simplified by only being fed a reduced version of the conversation (only containing the user's critical constraint, recommendation request, and model response, with no distraction elements), so the reviewer's concern of evaluating "such difficult scenarios" is misplaced. We made this more explicit in the revised version, and in our response to reviewers twsY and M3Ny.

Secondly, the fact that our benchmark adheres to a structure between scenarios is intentional, to help us comparatively analyse the effects of introducing different elements to conversations. This allowed us to manipulate specific elements without introducing confounding variables. We respectfully disagree with the comments that the benchmark is of a low quality or simply `generated by an LLM', or too formulaic to be useful. We took great care to ensure our case studies had sufficient variability where it mattered, and a great deal of manual labour, editing, refinement, iteration and intention went into the final benchmark design, a detailed discussion of which is now included in the revised version. Whilst "I have a severe X allergy..." is repeated across the user's first mention of their constraint in the allergy category, this only constitutes a 1/4 of total cases. In the physical constraint category, these all differ. In any case, we do not see this kind of repetition as a limitation, as, again, we were merely testing whether models noticed the allergy in its recommendations, and we chose the simplest, clearest phrasings. We specifically chose to specify all allergies as severe to ensure that models take them seriously, as things that could cause the user serious harm. Similarly, we chose to make recommendation questions start with ``do you think I/we should" as it is simple and easy to interpret. We do not believe this poses any problems for the tests' validity, and the controlled setup was useful for comparisons. The kind of diversity we cared about was not repeating constraint-request pairs, having gender and cultural diversity in actors, and not biasing results with distraction elements like other actors, trivia questions, or unrelated preferences, which we randomised between conversation turns. The number of preferences, conversation turns, and general conversation structure were otherwise kept stable precisely to increase the validity of our comparative analyses.

We strongly agree with the author that entries should be ``curated carefully to avoid mistakes, common-sense errors", which is why we did just that. Individual elements of the benchmark were created in small batches through iterative processes of painstaking instruction prompting, testing, refinement and manual editing, and any areas that seem formulaic are that way by design. If the reviewer can point out any particular "mistakes or common-sense errors", we would appreciate hearing them, as we spent months of effort doing our best to ensure there weren't any. We understand this was not mentioned in the original text and are thankful for being encouraged to include it.

Whilst we understand general concerns around using LLMs, we don't believe using them makes a dataset invalid in principle, especially when such care is taken to manually design, check and edit each element before inclusion. Our methods were designed for our purposes, i.e. to compare the effects of different distraction elements on a model's ability to attend to specific safety-critical information, and not a matter of laziness or oversight.

Following reviewer M3Ny's suggestion, we have compared the evaluator model (LLaMA 3.1 405B Instruct)’s performance against two human judges on a randomly selected sample of 100 conversations that were balanced across models, scenarios, and categories of safety-critical constraints. Two of the authors served as human judges, receiving only the same instructions as the evaluator model and the same reduced conversation (i.e., the user’s mention of the critical constraint, the recommendation request, and the model response). The humans were blind to the evaluator model’s ratings. As we can no longer upload a revised manuscript, we have uploaded a revision to the project's GitHub instead (https://anonymous.4open.science/r/llm_prag_benchmark-0C48/README.md), with the baseline comparison report included as the first appendix. We have also uploaded the complete tables of the conversations with the model and human judges' ratings. Please see our response to Reviewer 5gWe for further details, and we thank you and the other reviewers for helping us improve the paper's robustness.

We thank the reviewer for their thoughtful comments. We have addressed all of their concerns in the revised version, as described below.

• Comparison with human ground-truth ratings: for this basic personalised alignment test, we only chose straightforward scenarios where failing to consider the safety-critical constraint in the particular recommendation is necessarily unsafe to the user. Hence, rather than requiring subjective or nuanced judgment (as other alignment tests focusing on personal experiences of offense), our 'ground truth' was whether or a recommendation leads to objective physical harm. Several steps were taken to avoid any ambiguity where subjective opinions could differ: for instance, all allergies were stated as severe, indicating that eating food containing the allergen would cause certain harm. Similarly, phobias were stated as serious or severe, and the user specified that they have no interest in overcoming it (to ensure models do not assume recommending the activity may help them overcome it). The physical constraints were straightforward in preventing people from doing the target activity, and the `trauma triggers' were directly related to the trauma activity. All of the benchmark elements were carefully edited by a human to ensure that they fulfilled these criteria. Moreover, our evaluator was not asked to make any nuanced judgment about the relative harmlessness of the statement, but only to check whether or not the critical constraint was accounted for in the recommendation, as a binary classification task. To prevent ambiguous responses from passing (e.g., when it is unclear if the user-specific context was accounted for or if the model was just offering generically applicable advice), these were categorised separately, of which there were relatively very few, and were further analysed to show what lead to their ambiguous categorisation. These were not included in the pass rates but shown in a lighter shade on top of them, to ensure that the pass rates only represented responses that clearly accounted for the critical constraint. By manually spot-checking the evaluator's ratings and explanations during processing, no false positives or negatives were found. We are also confident in relying on LLaMA 3.1 405B instruct as the evaluator model as (a) the task was very simple, and the model was fed a reduced version of the conversation (the user's constraint, recommendation request, and the model's response) so no distraction elements could cause confusion; and (b) even with the added distraction elements, LLaMA 3.1. 405B Instruct was able to get near-perfect accuracy (99.5%) on the benchmark, showing that it is a reliable assessor.

• A table describing the five scenarios has been included in the appendix. The full benchmark has also been made available through an anonymous link we included in the revised paper.

• Noted and corrected with thanks :)

• The benchmark is publicly available on GitHub, we merely excluded it from the submission for anonymity reasons (this was stated in the data availability section). However, we have now included a link to an anonymised version of the repository in the revised paper for the reviewers.

The updated revised manuscript now also includes a detailed discussion of the benchmark creation process (e.g., how individual elements were designed and the criteria used to edit and improve them). A more recent update can be found on the project's GitHub repository that also includes a human baseline comparison where the ratings of two human judges were compared against those of the evaluator model, demonstrating excellent accuracy (https://anonymous.4open.science/r/llm_prag_benchmark-0C48/README.md)

We thank the reviewer for their thorough comments, and will respond to each of their concerns in turn.

1, Evaluations: The evaluator model is LLaMA 3.1 405B Instruct, not Claude. Hence, there should be no risk of biases persisting. In any case, there are several reasons why we would not be concerned about evaluator biases. Firstly, the evaluator model is given a reduced version of the conversation, where it only sees the user's critical constraint (the first conversation turn), the recommendation request (the penultimate turn) and the model response (the final turn). To pass, the model's response merely has to take the user's constraint into consideration. This is different to more nuanced alignment evaluations detections of 'harmfulness' are more subjective and at the evaluator's discretion. We purposefully made the evaluation criteria binary to minimise ambiguity or biases. We processed ambiguous results separately, which were analysed to give further insights into why they were deemed ambiguous and were not included in the final pass results. Secondly, even if some elements from the benchmark were generated with the assistance of Claude 3.5 Sonnet, it is not the case that the benchmark was generated as is, in its entirety, by a language model. We used few-shot prompting to generate different elements (conversation turns) of the benchmark separately (e.g., random trivia questions, preferences of other agents, personal facts, etc.). Several of the entries were created manually, most were edited manually, and all were manually checked by the authors to be as clear-cut and diverse as possible. We added further details on the benchmark design and evaluation process in the revised appendix.

2, Data: Our revised paper now includes 10 model evaluations, including OpenAI's o1-Preview. Our benchmark can easily be extended and adapted, as the length of the conversation depends on arbitrarily inserting more distraction elements at any point in the conversation, drawing from several separate Excel files, rather than just reading entries from a single file. Thereby, the conversation context can be extended and the format adapted as needed, as demonstrated in our ablation experiments. The key elements are the user's constraint and the related recommendation request, which can be placed at different places in the conversation, and future work could even combine multiple constraints in a conversation. Adding more entries to the benchmark could also be easily achieved by following a similar structure and creating synthetic data with a language model, perhaps with different forms of safety-critical constraints than the ones we focused on here. Our contribution is mainly to demonstrate a novel approach to alignment and evaluation (personalised, multi-turn contextual harms), using a simple logical structure that future research can draw upon.

3, Generalisability: By focusing on the most clear-cut, low-hanging fruit of safety-critical constraints. (e.g. clear physical limitations or allergies etc.) we wanted to set up as easy a task as possible for the model to be able to identify potential conflicts. The aim here was primarily to test model ability in the category of personalised risk assessment, even if it means noticing something that might be obvious to a user. In terms of generalisability, we would contend that, in fact, findings on such clear-cut cases are much more generalisable than ones that involve more nuanced contextual judgments. It is also not true that users would always be able to recognise harms to themselves inherent in any activity. Whilst this may be more obvious in some cases (e.g., wheelchair users asking about hiking), most of the examples are less obvious (e.g., cultural dishes that contain certain allergens, or activities that contain risks like flashing lights when a person is epileptic). In any case, more than being about putting the users in clear danger, making dangerous recommendations can be construed as inconsiderate, and may frustrate or offend users if they have to continually remind their personal assistant of basic personal constraints that should be obvious to it. E.g., in interpersonal relationships, a person with a diary allergy could get quite annoyed if their loved ones keep making/suggesting dishes to them that contain milk, as it shows a lack of respect or consideration for their needs. More than safe, personalised assistants should be considerate and reliable, such that users need not be sceptical every time the agent makes a recommendation whether it accounted for their needs and constraints appropriately. Our examples, across four realistic risk categories, are the simplest cases we could think of for demonstrating and evaluating the risk that "the LLM may forget a key concept mentioned in a conversation and make imperfect suggestion".

We hope that these comments have addressed the reviewers concerns, and would welcome any further comments to help us improve the paper.

We thank the reviewer for this comment, and understand the concern. Following reviewer M3Ny's suggestion, we have compared the evaluator model (LLaMA 3.1 405B Instruct)’s performance against two human judges on a randomly selected sample of 100 conversations that were balanced across models, scenarios, and categories of safety-critical constraints. Two of the authors served as human judges, receiving only the same instructions as the evaluator model and the same reduced conversation (i.e., the user’s mention of the critical constraint, the recommendation request, and the model response). The humans were blind to the evaluator model’s ratings. As we can no longer upload a revised manuscript, we have uploaded a revision to the project's GitHub instead (https://anonymous.4open.science/r/llm_prag_benchmark-0C48/README.md), with the baseline comparison report included as the first appendix. We have also uploaded the complete tables of the conversations with the model and human judges' ratings.

The baseline comparison evaluation results demonstrate exceptionally high agreement between the model and human judges. The model achieved perfect agreement (100%) with Human Judge 2 (H2) across all categories, while maintaining an outstanding overall agreement (96.1%) with Human Judge 1 (H1). The Cohen's Kappa scores (0.920 and 1.000 for H1 and H2 respectively) indicate excellent inter-rater reliability, well above the conventional threshold of 0.8 for "almost perfect" agreement.

For H1, out of the non-uncertain ratings, there were only 2 cases of disagreement where the model rated a response as pass (1) while the human-rated it as fail (0). H2 showed perfect alignment with the model's ratings, with 21 cases rated as fail (0) and 30 cases rated as pass (1) by both the model and judge.

Both human judges showed consistent levels of certainty in their ratings, with each expressing uncertainty (rating = 1) in only 1.9% of cases. When examining category-specific performance, the model maintained perfect agreement in Physical constraint and Severe phobia scenarios across both judges. For H1, the model achieved slightly lower but still excellent agreement in Trauma triggers (91.7%) and Severe allergy (92.3%) categories.

These results suggest that the model's evaluating ability closely aligns with human judgment on this task. Further confidence comes from the fact that the evaluator model is only fed a reduced version of the conversation (the user's mention of the constraint, the recommendation requests, and the model's response -- without any distraction elements) and LLaMA 3.1 405B demonstrated near-perfect performance on the most basic Scenario 1 (mean=99.5%), which is a longer and more complex version of the conversation.