Thank you for the detailed review of our work!

Given the formalization, the reader would expect the authors to (1)explain the question within the formalization, (2)give an experiment to empirically reveal the question or justify the proposed solution, and (3)give metrics to judge the effectiveness of methods that aim to solve this question. And the paper only did (1). Though this paper gives proof of specific propositions, it's not reasonable for a formalization lacking (2) or (3).

Our work is indeed theoretical. For our general philosophy justifying this choice, we refer to our global rebuttal.

With respect to empirical justifications of proposed solutions, we want to additionally highlight that in Section 6, we sketch the beginnings of a method that is theoretically justified. In particular, if $B$ is known, then in Proposition C.5 we prove that there is a differentiable loss function whose minimum contains only feedback-compatible reward functions, which are safe if the ambiguity $\ker(B \circ \Gamma)$ disappears.

Despite the clear figures, some logic does not match. The purposes stated ahead of sub-sections are lost, for example, due to the lack of translation from proven Propositions to the answers to proposed questions.

Thank you for this comment. We will extend the first paragraph in Section 4 to foreshadow Proposition 4.1 in Section 4.1, which, so far, was unmentioned in the intro of Section 4. Our new intro paragraph to Sec. 4 will read:

“We now analyze failure modes of a naive application of RLHF from partial observations, both theoretically and with examples. In Proposition 4.1, we show that under partial observations, RLHF incentives policies that maximize what we call $J_\text{obs}$, a policy evaluation function that evaluates how good the state sequences “look to the human”. The resulting policies can show two distinct failure modes that we formally define and call deceptive inflation and overjustification. In Theorem 4.5 we prove that at least one of them is present for $J_\text{obs}$-maximizing policies. Later, in Sections 5 and 6, we will see that an adaptation of the usual RLHF process might sometimes be able to avoid these problems.”

Furthermore, we think that the start of Section 5 might be confusing in its current form, where we ask “Assuming the human’s partial observability is known, could one do better?”. This question is only directly addressed in Section 6. To improve the logic, we will make the following changes: We will merge Sections 5 and 6 (i.e., Section 6 becomes the new Section 5.3), and change the second paragraph of the intro to Section 5 as follows:

“We start in Section 5.1 by analyzing how much information the feedback process provides about the return function when the human’s choice model under partial observations is known precisely. We show that the feedback determines the correct return function up to an additive constant and a linear subspace we call the ambiguity (see Theorem 5.2). If the human had a return function that differed from the true return function by an element in the ambiguity, they would give the exact same feedback — such return functions are thus feedback-compatible. In Section 5.2, we show an example where the ambiguity vanishes, and another where it doesn’t, leading to feedback-compatible return functions that have optimal policies with high regret under the true return function. Finally, in Section 5.3 we explore how one could in theory use Theorem 5.2 as a starting point to design reward learning techniques that work under partial observability.”

Please let us know if you see further problems in the clarity of our writing, and we are happy to address them!

The authors admitted that some of the assumptions that this method is based on may not hold, thus need further improvement.

It is true that some assumptions on the human model may not hold. Nevertheless, see our answer to reviewer LhS3: we think a more realistic feature-map formulation of the human model (replacing the belief $B$) would effortlessly connect with our formalization. Thus, we think there is great potential for our formalization to remain relevant even when taking into account more realistic human models. We will add a discussion on this viewpoint in the final version.

Additionally, it is a priori our belief that the results in Section 4 are robust to more realistic human models, since they are about failure modes that we show to be present even when the human has excellent rational expected-value thinking.

Please let us know about any remaining concerns, and we are happy to discuss them.

Thank you for your review!

The proofs are based on assumptions of a specific MDP structure and a particular human belief function, which might not be easily generalized to realistic, complex environments.

It is not true that we assume a specific MDP structure. Our work applies to any MDP, encompassing all reinforcement learning problems. Our human belief function is also assumed to be general: The only restriction is that it sums to 1 over all state-sequences, for a given observation sequence. Future work could think about relaxing the human belief function to allow for "feature maps" since humans cannot think about the whole environment state of complex environments. We detail this “feature map” idea in our answer to reviewer LhS3.

No empirical evidence is included in this paper, though real-world examples are given.

Our work is indeed theoretical. For our general philosophy, we refer to our global rebuttal.

Please let us know if you have remaining concerns. We will be happy to address them!

Thank you for your detailed review!

This work discusses the partial observability mostly in the theoretical sense, with a few hypothetical examples. It would be nice to see or verify the impact of partial observability in real experiments.

Our work is indeed theoretical. For our general philosophy, we refer to our global rebuttal.

Feature count formulation of the human model

As also mentioned in the limitations listed in Section 7, the formulation of the belief matrix is a bit strong in my mind.

This is indeed appears to be a strong assumption. We think further work is needed to evaluate the strengths and shortcomings of this formalization. We want to explain one implicit viewpoint of ours that we think means this formalization might reach quite far. Namely, we think it is compatible with how humans intuitively form judgments over events.

Concretely, if you are a human tasked with judging the quality of an observation sequence $\vec{o}$, what you will do is admittedly not to compute an explicit posterior belief $B(\vec{s} \mid \vec{o})$ over $\vec{s}$ – indeed, if the environment is very complex, it may already be impossible to even think about entire state sequences $\vec{s}$. More likely, you would think about the presence of certain features in the state sequence; e.g., “how often was there a bug”, or “has the software been installed”, or “how efficient is the code”. One would then implicitly assign a reward to each such feature and then add up the rewards.

This feature viewpoint could a priori be compatible with our formulation, namely if the feature-based returns implicitly come from a belief over state sequences. This could work as follows: Assume there is a feature map $\phi(s)$ that maps high-dimensional states $s$ to low-dimensional feature vectors $f$. Assume the reward only depends on feature vectors: $R(s) = R’(\phi(s))$ for some function $R’$. We obtain:

$\sum_{\vec{s}} B(\vec{s} \mid \vec{o}) G(\vec{s}) = \sum_{\vec{s}} B(\vec{s} \mid \vec{o}) \sum_{t = 0}^{T} \gamma^t R(s_t) = \sum_{f} \left( \sum_{\vec{s}} B(\vec{s} \mid \vec{o}) \sum_{t = 0}^{T} \gamma^t \delta_f(\phi(s_t)) \right) R’(f) \eqqcolon \sum_{f} N(f \mid \vec{o}) R'(f),$

Where the outer sum runs over possible features, and where $\delta_f(\phi(s_t))$ evaluates to $1$ if and only if $\phi(s_t) = f$. In the last step, we denoted the big coefficient of $R’(f)$ by $N(f \mid \vec{o})$, which is the expected discounted feature-vector count of $f$ upon observing the whole sequence $\vec{o}$.

Then instead of modeling the human as coming with a reward function $R$ and a belief matrix $B$, we have found an alternative model: we can model the human as coming with a feature-based reward function $R’$ and an expected feature-count $N(f \mid \vec{o})$. This highlights that the whole formalism could also be built upon a human model where the human “counts the presence of features”, as long as these feature counts come with consistency properties that allow them to be compatible with a belief over state sequences. We think this viewpoint makes it natural to extend our work to more realistic human models that effortlessly connect with our work.

We will add this additional discussion to our paper to explain the reach of our belief matrix formulation.

Please let us know if you have remaining concerns, and we are happy to address them.

Thank you for your thorough review!

The paper would greatly benefit from carefully designed experiments that unambiguously demonstrate the existence of such problems in practice.

Our work is theoretical and we advocate for judging it on a theoretical standard. We explain our viewpoint in more detail in our global rebuttal.

The paper does not propose any concrete solutions to the problem [...] adding some concrete attempts at solving this problem, either by presenting algorithms with new theoretical guarantees [emphasis ours] in this context, practical experiments showing a quantifiable improvement in meaninigful metrics associated to the problem, or both.

We want to highlight that Section 6 presents exploratory ideas for practical solutions, and in particular, that the discussed Appendix C.3 with Proposition C.5 provides a first theoretical guarantee: If the belief matrix B is known explicitly, one can design a differentiable loss function whose minima are feedback-compatible. In particular, this proves that if the ambiguity $\text{ker}(\text{B} \circ \Gamma)$ vanishes, then the minima of the loss function are guaranteed to be safe, in the sense that they have return functions that differ up to an additive constant from the true return function. Thus, future work has a theoretical guide for designing practical algorithms, which requires carefully designing the type of partial observability so that the ambiguity is “benign”, and potentially specifying an approximation of the belief matrix B.

The paper does not discuss other work in RLHF that deals with partial observability and heterogeneity (albeit in a subtly different context), for example [1, 2, 3].

Thank you for making us aware of these recent works. We will add the following paragraph to the related work:

“The literature also discusses other cases of partial observability. [1] and [3] deal with the situation that different human evaluators can have different unobserved preference types. In contrast, we assume a single human evaluator with fixed reward function, which can be motivated by cases where the human choices are guided by a behavior policy, similar to a constitution or a model spec. [2] assumes that the choices of the human evaluator depend on an unobserved reward-state with its own transition dynamics, similar to an emotional state in a real human. In contrast, we assume the human to be stateless.”

We also add paragraphs detailing related work on the modeling choices for human preference selection, truthful AI, and other works that deal with missing information. Please let us know if you would like to see these paragraphs, then we will promptly add them in a subsequent answer.

Are there easy ways to force the ML model to"expose the underlying state a bit more" from time to time? A naive example is forcing verbosity in the case of the deceptive inflation example, but of course one needs to design a more general and sustainable fix.

We think about this in terms of trade-offs. In principle, the environment state could be fully observed by the human: We could just send all the information that the AI receives also to the human evaluators. The reason why we still think our work is necessary is that we expect it to increasingly become prohibitively expensive to show human evaluators all content; after all, human evaluators have limited time, and so if they always fully observe a state sequence, that trades off against the number of labels they can provide within a given timeframe.

Thus, the question becomes how to design the partial observability in exactly such a way that despite limited information, the feedback-process still leads to correct reward functions being learned. Your idea to “Expose the state from time to time” (e.g. randomly) is interesting, and we are indeed interested in further research that deeply analyzes such settings.

Please let us know if you have remaining concerns. We will be happy to discuss them.

While I appreciate the response and acknowledge that I am no stranger to an emphasis on theory, I continue to have qualms.

I believe that there is a difference between a theoretical work that introduces a new model and theoretical work that identifies a problem with existing work. Since your work falls into the latter, it needs more than just hypothetical justification to demonstrate the severity and impact of the problem. The experiments demonstrating the existence of this problem need not be your own, but they need to exist and be referred to.

An example I would like to give is the work of Alon and Yahav introducing oversquashing in GNNs. Their paper lacked substantive theory, but it empirically demonstrated the existence of the problem they were postulating. I think if we identify a potential problem in our minds, it is our duty to make sure that we haven’t made it up.

Further, I apologize for not acknowledging the theoretical result they had. The reason it didn’t satisfy me was because assuming that the belief matrix is known is too strong an assumption. This made the result too weak for me to appreciate, despite it seeming technically non-trivial to show.

If either of these two concerns were addressed (strength of theoretical result or experiments demonstrating problem), then the other issue would be less major to me. Since neither has been addressed, I am afraid I cannot raise my score.

I must emphasize that this is an important problem and a paper with potential, it just needs to be strengthened on at least one of these ends. While I know that it can be tempting to work on exciting theory, in this case I implore you to design an experiment that can demonstrate the potential existence of this problem. Work like that of Alon and Yahav is in the unrelated domain of GNNs, but nevertheless it should help you understand the broad flavor of experiments that could help you convince readers.

Thank you for the trust in our work! For the camera-ready version, we included simple empirical validations of our concerns in the toy environments that we also study conceptually. We hope that makes our claims more convincing to some readers.