Practical alignment requires more than learning from human feedback

We thank you for your appreciation of the paper. For your two questions, they are excellent for future exploration. We acknowledge the limitations of our framework in modeling such scenarios (lines 231–235) and have raised similar questions in the conclusion section. Addressing these questions may require insights from other fields, such as social sciences. Despite its current limitations, we believe our framework represents a significant improvement over existing ones and can inspire readers to think about important questions like these.

We appreciate your evaluation of the paper. While the presentation can be improved, we hope you consider that the problem we study is novel and may not immediately align with conventional thinking. Although readers may initially find the concepts challenging to grasp, the framework's impact could be enduring. In the next revision, we will include more examples to clarify the concepts and simplify the notation.

Below, we address your major concerns and questions:

The paper does not propose a method for 'practical' alignment; this is not strictly a weakness but lack of a method does weaken the contributions the paper is making a little.

We actually proposed a solution: that is to design AI systems that can detect human false beliefs and generate language utterances to correct those beliefs. Such systems differ dramatically from existing ones, which only passively learn from humans. Using LLMs to infer false beliefs and generate language utterances, we obtained higher results than a baseline that cannot influence humans (ostensible alignment). This paper is already dense so further improvement warrants an entire new paper.

1. Are the authors aware of inverse reward design (https://arxiv.org/abs/1711.02827)? It is probably the most related work to this paper that I can think of; but has not been cited in the paper. I would like authors to include discussion on novelty of this work relative to IRD. In general, the related works section is relatively thin for my liking with most citations after 2020. The 'teaching' aspect of the framework is also related to works on 'eliciting latent knowledge' (https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge), debate between agents (https://arxiv.org/abs/2402.06782) and eliciting human preferences better (https://arxiv.org/abs/2310.11589).

We thank you for your great suggestions. We did cite Li et al.'s work on eliciting human preferences. We will add more works to the related work section. Nevertheless, we emphasize that they are remotely related to our work because we focus on a theoretical formulation of the problem rather than empirical approaches.

IRD highly related, thanks for pointing it out! The fundamental differences between our work and IRD are:

• IRD assumes that the AI knows the human’s world model, so its job is only to infer their desire function. In their experiments, the AI is trained in the environment (world model) from which the human observes and derives their proxy reward. This is an unrealistic model of communication, in which the mental representations (belief, intention) of one agent is hidden from the other. Our framework captures this aspect of communication. In our experiments, the AI has to infer the world model of the human.
• IRD does NOT model world-model change, while our framework does.
• IRD’s solution focuses on uncertainty, while we propose a communication-based solution in which the AI actively informs the human about the world.

I don't think 'practical alignment' is a great name for the concept/framework being introduced in the paper. In fact, it seems to be harder to implement and thus seems a bit impractical. Authors should rename it to something like 'idealized alignment'.

“Practical alignment” is actually a shorthand for “alignment for practical outcomes”, i.e. outcomes that succeed in the real world. It describes what is encoded in the objective of the framework. It contrasts with the existing alignment frameworks which may produce solutions that fail in the real world. We will think of a different term to avoid this confusion.

In practical alignment, does the agent observe ω∗? This is not explicitly stated but based on the description of the experiments, seems to be assumed. This seems like a very strong assumption and should be made explicit and acknowledged as such.

The AI does observe omega^* in our experiments but not in a general practical alignment problem. It was clearly stated in lines 431-432.

In equation 4 the meaning of θ^A is not obvious to me. Similarly, the meaning of subscript of T in θ_T^Z is also not clear.

We couldn’t find θ^A in equation 4. We have defined θ_t^A and θ_t^H in lines 213-221. The meaning θ^T_Z can be inferred from those definitions but we will add a reminder.

Can you give a scaling curve for this? My guess is that performance would change abruptly past some threshold.

Could you clarify the scale you are referring to (number of examples or model size)? In table 1, we have ordered the models roughly by sizes and presented results with different numbers of examples.

Below are the answers to your questions:

1. In the practical alignment framework, it is assumed that the humans know ψ^H, the parameters for their desire function. If this is the case, then what prevents the AI agent from directly learning the desire function and optimizing that. In other words, what is the difference between the descriptive reward function of the ostensible framework and the desire reward function of the practical framework? I think giving more concrete examples here can be very helpful.

The desire function is a parameter of the reward function. For example, as you evaluate my paper, your reward function determines the score you assign to it. But to decide that score, you could be simulating how impactful my paper is going to be in the future. The impact of my paper is measured by what we call a desire function. The word "desire" suggests that is a deeper, more latent motivation that influences your evaluation.

The desire function alone does not fully define the reward function. The reward function is also dependent on a world model. If this world model is the real world, the reward function is normative. If the world model is imagined by the human, the reward function is descriptive.

Even if the AI perfectly learns the desire function, it may still have imperfect knowledge of the reward function. Optimizing the right desire function in a wrong world model can still result in a suboptimal plan or policy. This is illustrated in Figure 1a: if the robot chooses Plan A, it optimizes the human's desire function in their (incorrect) model of the environment. That plan is suboptimal in the real environment.

2. The presented framework assumes that human preferences might change with their belief about the world, but their desires stay constant. Although this is very abstract, it is possible for desires to change with beliefs about the world. I think it comes down to the modeling assumptions and there might not be one right answer, but I am curious to know the authors’ thoughts on this.

This is a great question. This work presents a two-level hierarchy. You have preference over plans being influenced by beliefs and preference over world state-actions (the desire function). We assume a static desire function so that the objective is well-defined. A generalization of this framework would have more levels of abstraction. A desire can be driven by more abstract beliefs and desires. For example, you desire to be rich because you believe it would make you happy. Here, wanting to be happy is a more abstract desire than wanting to be famous. The most abstract desires, like wanting to be happy or to survive, may rarely change.

3. I see this framework as being more applicable to embodied AI systems that are interacting with the real world. These settings would naturally have many instances where a human’s internal world model is different from the real world. However, is this framework also applicable in the digital setting (such as AI chatbots)? What do the authors consider to be a world model here, what are concrete instances where the human’s model is incorrect?

The world model represents any assumptions that a human makes about the world. Humans make false assumptions all the time. For example, you chat with your virtual assistant to figure out the best gift for your loved one. You tell the assistant to buy an iPhone without knowing that they already ordered one yesterday. Rather than blindly executing the order, we want the assistant to correct your false beliefs by saying: “Your loved one has already purchased an iPhone”

We will first answer your two major questions:

1. How could this work help to solve the proposed alignment problems?

We want to emphasize that this is not a typical paper that proposes solutions to a well-established problem. Instead, our goal is to introduce readers to a new problem and provide the theoretical and empirical tools needed to begin researching it. Specifically, we:

• Mathematically formulate the problem. This is a prerequisite for developing theoretically grounded approaches. Without a mathematical formulation, the development of solutions is limited to relying on crude intuitions.
• Explain why this problem is significant by formally proving approaches that do not recognize the problem leads to undesirable outcomes (section 5.2 & 5.3).
• Characterize the general properties of the solution: it must involve teaching humans in addition to learning from them. This brings to attention a new set of problems orthogonal to learning from human feedback, opening new research opportunities.
• Build a toolkit to empirically study this problem (MindGrid). It might be deceptively simple but it is the first of its kind. Simulating communication with humans is extremely challenging. It is not just a matter of generating good-looking conversations; one needs to simulate the derivation of the communication intentions, which depends on various mental representations. On the other hand, if the simulation is too complex, the problem might be impossible to solve with the current methods. MindGrid achieves the balance: it is challenging enough for making research progress, but is simple enough to solve.
• Propose a solution to the problem: we design AI that can teach humans. We use LLMs to enable such capability. Our experiment results clearly suggest that AI that can also teach humans outperform any AI that only learns from them.

2. What kind of results does your framework allow you to derive that prior work cannot study?

Below we list our results that are unattainable by previous frameworks:

• Alignment requires AI that can teach humans. Previous frameworks implicitly assume humans have perfect world models, so teaching is unnecessary. We believe that this is an important result, as it points to a set of problems that are currently overlooked.
• The relationship between the human’s knowledge gap and the suboptimality of the performance of the AI. Concretely, the AI performance suboptimality gap scales with a rate between Omega(H*D) and O((H^2)*D) where H = 1/(1 - gamma) is the effective horizon and D is the divergence between the human’s world model and the real world (theorems 4.1 and 5.2). Previous frameworks do not model the human’s world model, so such a result was not possible.
• AI systems trained with previous frameworks are incentivized to manipulate humans (theorem 5.3). This problem does not exist if humans have perfect world models, which is assumed in previous frameworks. If a human knows everything about the world, the optimal behavior for an AI is to respect their beliefs and solely focus on learning.

We note that while some of these results may not be surprising to you, writing them down in precise mathematical terms takes effort and was not done before to our best knowledge.

We appreciate the time and effort you spent on evaluating our paper. We would like to reply to some of your points. Please understand that we are not pressuring to change your evaluation, but we think these clarifications might be helpful for the AC to make the final decision.

The paper correctly identifies there is a problem in learning from human feedback that human's can be mistaken about the world. However, this is well known, and not a novel contribution (eg. see [1] and citations therein).

We believe that the familiarity of a natural phenomenon does not diminish the value of a mathematical framework proposed to model it. For example, while everyone knows the sun rises in the east, writing an equation to explain and predict the sun's morning movement would be considered a marvelous scientific achivement. We think that it is a bit unfair to compare our mathematical framework with [1], a paper that only describes natural phenomena but provides no mathematical formulations. It would be more convincing if you presented a paper that proposed a mathematical formulation of the problem we study. There are indeed a few and we have compared with them in the related work section.

Of couse, novelty is not enough. As you said, we also need to convince readers why our formulation is useful. We reply to this point below.

[the formulation is] analogous to what's typically done when formalizing model-based RL.

We think that this should be regarded as a strength rather a weakness of the proposed formulation. The formulation enables us to borrow theoretical tools from model-based RL to derive theoretical insights. The fact that the results appear intuitive and natural is evidence that the problem is formulated in the right way.

Note that our problem is different from model-based RL. Model-based RL concerns how the error of an AI's world model affects its performance. There is no interaction with humans. We extend this framework to relate the error of the human's world model with the performance of an AI that learns from them.

However, the paper, in my opinion, fails to convincingly show that this formalization is useful or insightful. None of the results are surprising or provides a new angle for solving the proposed problem -- or at least the discussion doesn't make the new insights clear enough.

In addition to connecting with model-based RL, our formulation leads to these insights:

• Alignment requires AI capable of teaching humans. This idea of building AI to influence humans initially seems taboo, as current approaches treat the human reward function as “sacred”, untouchable information. Our formulation recognizes that humans have multiple layers of will: while deeper desires should be respected, preferences shaped by beliefs can—and must—be corrected to avoid undesirable consequences. If you look at the list of solutions proposed by [1] (section 4.2), you will not find any similar ideas. Our solution idea goes beyond conventional thinking.
• A simple but important result (theorem 5.3), which states that AI can be unintentionally trained to manipulate human's beliefs about the world. Recently several papers have provide empirical evidence of this phenomenon (e.g., https://arxiv.org/abs/2411.02306, https://arxiv.org/abs/2409.12822). Our result formally explains (or predicts) these empirical phenomenon. The result is possible because we incorporate the human's world model in our formulation.