Concept Alignment as a Prerequisite for Value Alignment

We thank the reviewer for their feedback. It is quite true that our experiment is run on a simplified setting and not open-world settings. We use a simplified setting intentionally for two main reasons: First, we intended to demonstrate in an unambiguous and interpretable manner the limitations of value alignment as it is currently framed in the literature (often in simplified settings—see, for example, see Hadfield-Menell et al. 2016; Sumers et al., 2022), and to show how incorporating concept alignment could address those limitations. This is a necessary prerequisite for scaling up to more complex settings and a standard research strategy in AI safety and alignment research (see previous citations). Second, a major component of our contribution is the comparison of our model with human judgments that were experimentally elicited. By focusing on a simplified, controlled set of scenarios, we can make more direct comparisons between the formalism and behavioral data.

Within this scope, we demonstrate the potential negative impact of concept-level misalignment if it is not taken into account, correct for it, and show human reasoning does take into account concept misalignment when inferring values. We know that it is necessary to both demonstrate the negative impact of concept misalignment and correct for it in real-world settings as well; in fact, this is the motivation for our work. Demonstrating this misalignment thoroughly in a simplified setting highlights the exact nature of the problem, and lays the groundwork for demonstrating it at scale and in the real world.

Correcting for the problem in a measurable way, which is afforded to us by the closed nature of the simplified setting, allows us to start the critically important discussion of how it can be corrected in larger-scale, open-world settings. It will take time and experimentation to demonstrate the impact of concept misalignment on value misalignment for these large-scale, real-world settings, and even more time and discussion to correct for the misalignment in real-world settings. Each real-world setting comes with significant contextual background that must be taken into account when defining what “alignment” means in that setting, and how much alignment is needed. For example, scaling up concept alignment to a healthcare diagnostic algorithm would not require agent-based RL at all, and would not be implemented in an IRL setting. Rather it would be about the medical imaging artifacts that lead to diagnostic decisions, the words human physicians use to describe those artifacts, and how to align those concepts to the diagnostic algorithm which analyzes the same medical image and suggests a course of action, like surgery. This type of concept alignment would be a necessary prerequisite to discussing surgery based on what the patient values most (mobility, lack of pain, effect on other organs, long-term psychological impact of the procedure). We have significantly revised our “Societal Impacts” and “Limitations and Future Work” sections (6.2-6.3) to discuss this as suggested in the reviewer’s feedback.

The code and data are indeed publicly available, linked here: https://osf.io/hd9vc/?view_only=967b0c2f981d4a87bf4d21ff818f1322

To answer the reviewer’s questions regarding the translation of Eqn 3 to code:

1. $\tilde{T}$, the demonstrator’s construed task dynamics, encodes blocks and notches as shown in Figure 4. If the demonstrator’s construal is that they are paying attention to notches, their MDP reflects this based on which blue tiles seem traversable to them (notches) and which don’t (blocks). If they are not paying attention to notches, then their MDP reflects this with all blue squares as blocks.
2. The prior $P(R, \tilde{T})$ is uniform. We have added text to the updated paper clarifying both of these points, thank you for the questions.

References

Hadfield-Menell, D., Russell, S. J., Abbeel, P., & Dragan, A. (2016). Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29.

Sumers, T., Hawkins, R., Ho, M. K., Griffiths, T., & Hadfield-Menell, D. (2022). How to talk so AI will learn: Instructions, descriptions, and autonomy. Advances in Neural Information Processing Systems, 35, 34762-34775.

We thank the reviewer for their feedback. It is quite true that this work is not intended to propose an improved IRL algorithm or framework. Rather the purpose of the work, as an AI safety contribution, is to use an existing IRL framework to experimentally demonstrate the consequences of concept-level misalignment to value-level misalignment between humans and machines, and also demonstrate how this misalignment can be remedied if concepts are taken into account.

To this end, our goal is to provide an empirical starting point for a discussion on how we can incorporate concept-level alignment across forms of AI. Many instances where concept alignment is required will not involve RL at all. For example, scaling up concept alignment to a healthcare diagnostic algorithm would not require agent-based RL at all, and would not be implemented in an IRL setting. Rather it would be about the medical imaging artifacts that lead to diagnostic decisions, the words human physicians use to describe those artifacts, and how to align those concepts to the diagnostic algorithm which analyzes the same medical image and suggests a course of action, like surgery. This type of concept alignment would be a necessary prerequisite to discussing surgery based on what the patient values most (mobility, lack of pain, effect on other organs, long-term psychological impact of the procedure). Due to the breadth of AI safety concerns and applications, it is not possible to exhaustively address concept alignment in a single paper; rather, it will require a larger discussion around how to achieve concept-level alignment between humans and AI.

We do of course have IRB approval, and a walkthrough of our study is included as an appendix to the original submission. The study was conducted through the commonly-used research participant recruitment platform Prolific, and participants were from the US and UK. Participants were not filtered by other demographics or through any other questions. We are happy to share deanonymized IRB documentation with ethics reviewers if appropriate.

Finally, we thank the reviewer for drawing our attention to this important body of work in automated planning on modeling human mental models. In the updated version of the paper, we now reference this literature. Our work builds on these important contributions in two ways that have not been previously approached in the literature (as far as we are aware). First, we benchmark discrepancies in mental models between standard IRL and real humans using controlled behavioral studies that builds on an established experimental paradigm from the psychology literature. Second, we propose a general inferential framework for approaching this problem grounded in recent work on computational cognitive science. As far as we are aware, this has not been previously explored in the literature. For example, although the workshop paper that the reviewer references by Gopalakrishnan et al (2021) does report comparisons with humans, it makes only coarse-grained predictions about policy imitation (e.g., in complex versus simple policies), whereas our account explicitly considers the role of mental models via inference about the process of construal. Our work takes a computational cognitive science perspective to ensure that discussion of value alignment in AI safety remains concretely grounded in human behavior.

I appreciate author's response to my concerns. [I am not able to see the revised pdf, as open review still shows that there are no revisions.]

After the author's response, I feel my understanding of the work was correct and I remain on my point on limited novelty as the need for mental modeling has been established in several prior works. I would appreciate if the authors could address my concerns on their assumptions on the transition dynamics function [see para 2 and 3 of my review].

I am happy to read that the authors followed an approved IRB protocol. They should follow HRI works to report human subject study information [for double-blind submissions] and provide details such as demographics, participant hiring information etc. for reproducibility. This becomes all the more important as the authors feel their primary contribution is to "benchmark discrepancies in metal models .... using controlled behavioral studies".

I will maintain my score based on our existing discussion and after reading comments by other reviewers. I feel the work can substantially benefit from making it explicit what their claims are and clearly contrast with prior work. The current study is definitely a first step towards establishing the need for mental modeling and the authors can explore its utility further in their setting of IRL or AI safety.

We thank the reviewer for their feedback. It is true that this is not intended to scale in its current form, but rather demonstrate concretely the limitations of value alignment as it is currently discussed in the literature, and to show how incorporating concept alignment could address those limitations. To understand the limitations of value alignment at scale, the AI safety community will need to study it across many different contexts and disciplines, of which this paper addresses only one.

What these limitations look like vary greatly depending on the context of the AI being used, and the degree of human-AI alignment needed in each of those contexts. For example, scaling up concept alignment to a healthcare diagnostic algorithm would not require agent-based RL at all, and would not be implemented in an IRL setting. Rather it would be about the medical imaging artifacts that lead to diagnostic decisions, the words human physicians use to describe those artifacts, and how to align those concepts to the diagnostic algorithm which analyzes the same medical image and suggests a course of action, like surgery. This type of concept alignment would be a necessary prerequisite to discussing surgery based on what the patient values most (mobility, lack of pain, effect on other organs, long-term psychological impact of the procedure). If the AI is not able to honestly and concretely align concepts with the patient or the doctor, and meaningfully explain why a diagnostic judgment was made, the resulting medical decision is likely to be value misaligned with the patient in critical ways.

In contrast, scaling up concept alignment to something like deep RL could be much more similar to our Bayesian IRL example, with the values explicitly tied to the reward functions. We anticipate and encourage this breadth of work studying concept alignment and the resulting effect on value alignment, or misalignment, as a critical step forward in AI safety discussions.

We have significantly revised our “Societal Implications of Concept Misalignment” section and “Limitations and Future Work” section (6.2-6.3) to discuss these implications, per the reviewer’s suggestion. We have also highlighted how our motivations differ from the previous works with similar mathematical models mentioned in the review, and the updated paper now references these works, thank you for bringing them to our notice.

Answers to questions:

1. All human subjects were shown the full set of trajectories, which were designed to have sufficient examples to rule out alternative hypotheses. The human subjects and the IRL models were of course shown the same set of trajectories.

2. Participants were not rejected/excluded for failing to correctly understand the "notch" concept.

We thank the reviewer for their feedback. The notion of a “concept” is indeed broad, and construals here are meant to be one potential instantiation of what a concept could look like in a concrete, contained example. Per the reviewer’s suggestion, we have added a clarifying sentence to the introduction explaining that different construals are used to encode different conceptual understandings of the world.

The reviewer is correct that we use a synthetic, simplified setting to study the problem of concept alignment. This was done intentionally for two main reasons. First, we intended to demonstrate in an unambiguous and interpretable manner the limitations of value alignment as it is currently framed in the literature (often in simplified settings—see, for example, see Hadfield-Menell et al. 2016; Sumers et al., 2022), and to show how incorporating concept alignment could address those limitations. This is a necessary prerequisite for scaling up to more complex settings and a standard research strategy in alignment research (see previous citations). Second, a major component of our contribution is the comparison of our model with human judgments that were experimentally elicited. By focusing on a simplified, controlled set of scenarios, we can make more direct comparisons between the formalism and behavioral data.

To understand the limits of value alignment at scale, the AI safety community must study it across many different contexts and disciplines, of which this paper addresses only one. “Values” of AI systems are tricky to define, and an IRL setting gives us one very concrete way to define them (in terms of reward functions). We agree that it is very important to study AI “values” outside of such a controlled setting, and even more important that we also study concepts in tandem with values (and we hope this paper demonstrates why studying both together is important). We hope that our empirical framework provides a starting point for designing such studies.

References

Hadfield-Menell, D., Russell, S. J., Abbeel, P., & Dragan, A. (2016). Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29.

Sumers, T., Hawkins, R., Ho, M. K., Griffiths, T., & Hadfield-Menell, D. (2022). How to talk so AI will learn: Instructions, descriptions, and autonomy. Advances in Neural Information Processing Systems, 35, 34762-34775.