Thanks + response

Thank you for your review. We are glad you think our work “provides a useful contribution to the literature” with technical details that are “sound,” “effective,” and “intuitive and easy to follow.” We appreciate the questions you ask about some of our examples and experiments.

1.

There are some conceptual similarities of this work to off-switch games (this is not an essential reference though): Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The off-switch game. In Workshops at the thirty-first aaai conference on artificial intelligence

Thank you for sharing this reference. We agree it shares conceptual similarities, and we will cite it.

Action 1.1: We will add a citation to Hadfield-Menell’s et al.’s “The off-switch game.”

2.

why didn’t the paper show an experiment for purpose 2?

While we were limited by space constraints, we think this is an exciting direction for future work. Thus:

Action 2.1: We will discuss experiments for purpose 2 as a direction for future work in the “limitations and future work” section of our paper.

3.

major typo: line 232 if A can communicate with A H

We will fix this typo.

minor typo: line 104 Lehrer extend(s), line 380 utilities and 1 and 0

We believe “Lehrer extend” is correct (see rule 20 here) and will fix the line 380 typo.

p. 6, the paragraph that starts with "At first sight" is a bit unclear. Why present this unlikely human strategy?

We were worried that Example 5.1 might be misconstrued as a counterexample to Theorem 4.2. We included this “At first sight” paragraph in an attempt to clarify why Example 5.1 is still consistent with Theorem 4.2.

Action 3.1: To clarify this, we will do the following:

• Remind the reader of the content of Theorem 4.2: “which states that non-interfering optimal policy pairs always exist”.
• At the end of the “At first sight” paragraph, add: “The key point of Example 5.1 is therefore that---while there is some optimal policy pair without observation interference---there is no plausible optimal policy pair that avoids observation interference. More specifically, we use the notion of acting naively (Definition 3.7) to express this notion of plausibility and rule out the above policy. We thus obtain the following proposition, which states that in some POAGs, if we want to play an optimal policy pair and we want H to be able to act naively, then A has to interfere with observations.”

I did not follow the logic of the values of the utilities (e.g., 1,0,4,7) on p. 7, the paragraph that starts with "Intuitively, both the tldr and man pages allow the human to choose optimally?

We agree that the logic of the utilities for Example 6.2 should be clearer.

Action 3.2: To address this, we plan to discuss the utilities in Example 6.2 itself, rather than after it.

• In Example 6.2, we will spell it out in more detail as follows: “Specifically, the worse flag always yields a utility of 0. The better flag either yields a utility of 1 or a utility 7.”
• We will also expand the parenthetical “i.e., the exact state: which flag is better and whether its utility is 1 or 7”.
• And at the end of the example, write, “Therefore, the expected utility of the better flag is 4 (and the utility of the worse flag is 0).“

For the experiment in section 7.1, there is no shared information correct? Would the presence of shared information change anything?

That is correct: there is no shared information. The presence of shared information would not necessarily change anything. As long as A has private information that is useful to the human, then it can still have an incentive to interfere with observations for purpose 1. And as long as A can make H’s decision easier by interfering, then A can still have an incentive to interfere with observations for purpose 3. This holds true even in the presence of shared information.

What do you think?

Thank you again for taking the time to review the paper and providing helpful feedback! Do the above actions provide clarity to your questions? If not, what further clarification or modifications could we make to improve your score?

Thanks + response

Thank you for your review. We are glad you find our setting “of strong interest,” and we appreciate your help with the clarity and simplicity of our paper.

1.

real difference between Definition 3.2 and Definition 4.6 is unclear… I did not understand the example after Theorem 4.7.

Thank you - we want to be sure we clearly communicate these important points. We will:

Action 1.1: Extend our discussion on line 290 of the example after Theorem 4.7 to more thoroughly explain how Definition 4.6 differs from Definition 3.2. The revised discussion will read:

“We now revisit Example 4.3. For observation tampering under our earlier Definition 3.2, H simply knows that A has taken the action to suppress some versions of cuda. However, H does not know anything about A’s policy. For all H knows, A’s policy could be to randomly suppress cuda versions or to always suppress the same cuda version. Thus, suppressing any version is strictly less informative for H than the list of all available versions. This is why Definition 3.2 calls suppressing versions ‘tampering at the action level.’

The key difference with Definition 4.6 is that H knows A’s policy. Suppose that A’s policy $\pi^\mathbf{A}$ is to suppress exactly the versions of cuda that are incompatible with the other software in the environment. Because H knows that A suppressed the incompatible cuda versions, seeing the filtered list tells H which versions of cuda are compatible! Although suppressing versions is strictly less informative under Definition 3.2 (when H doesn’t know A’s policy), suppressing versions provides H with new information under Definition 4.6 (when H knows A’s policy). Accordingly, $\pi^\mathbf{A}$ is interfering with observations at the action level but not at the policy level.”

Action 1.2: Move the discussion of the example after Theorem 4.7 (which is currently on Line 290) upward to line 280 so that it immediately follows Definition 4.6. This way, the reader will immediately see an explanation of the intuition behind Definition 4.6 before Theorem 4.7 (which relies on Definition 4.6).

Question 1.3: Do these revisions provide clarity? If not, please let us know, and we will work to clarify further.

2.

What is so specific to POMDP? … POMDP[s] might be overcomplicated…. A [much simpler] normal form game, could have led to the same conclusions.

We require a framework with private information and sequential play, which normal-form games lack. Observation interference (both Definitions 3.2 and 4.6) needs A to choose first, then H to observe something dependent on A's choice.

We study causes of observation interference which need:

• For Communication (Section 4): A observes privately → A acts → H observes → H chooses (Example 4.1)
• For Querying (Section 5): A acts → H observes → H acts → A observes → A acts (Example 5.1)

While extensive-form games could use fewer "dummy" states, they're similarly complex as DecPOMDPs, which are standard in assistance games literature (Hadfield-Menell et al., 2016; Shah et al., 2020).

Question 2.1: Does this clarify why we can’t simplify our setup?

3.

I would need more explanation to really think this setting could apply to RLHF.

Action 3.1: At the start of Section 4.1, we will add the following explanation:

“We can model RLHF within the POAG framework as follows:

• The assistant’s goal in RLHF is to satisfy the human’s preferences. In a POAG, this corresponds to the shared reward function $R$ which has a parameterization $\theta$ that only H knows.
• In RLHF, the assistant rolls out trajectories, and the human’s picks which trajectory is preferred. A POAG can model this by letting H observe pairs of trajectories explored by A but only giving H a binary action (to choose which trajectory H prefers).
• A’s final RLHF policy maximizes an estimate of $R$ based on a dataset of H’s preference comparisons (Lang et al. (2024), Proposition 4.1). In the POAG framework, A can compute this policy based on A’s observations of H’s binary actions.”

4.

For the minor comments, we will:

• Add to line 115 “POAGs inherit the generality of DecPOMDPs: POAGs can model games where H acts first, where A acts first, or where H and A act simultaneously.”
• Add the full name for DecPOMDP to Line 111.
• Add a clarification so that line 132 reads: “... Nash equilibria for the shared reward function R.”
• Add this definition of a stochastic function: “a function mapping observations to random variables over observations”.
• Fix the line 231 typo.

how Propositions 2.2 and 2.3 are actual propositions

Appendix A needs these propositions to have names in order to refer to them and prove them. To clarify, we will add “(See Appendix A for proofs.)” to line 148.

What do you think?

Thank you again for your review. Do the above actions address your concerns? If not, what further clarifications or modifications could we make to improve your score?

Thanks + response

Thank you for your review. We are glad you consider our work “very relevant” to the broader scientific literature and that “the general reasoning discussed in the paper is helpful to build intuition for the theory.” We appreciate the questions you raise about the practical applicability of our work along with the example scenarios that you pose.

1.

What is the practical applicability of the work? Are there domains beyond the curated examples where results of the paper are be discussed? What is the main impact of these results in such domains?

Thank you for raising these important questions about the practical applicability of our work. To address these questions:

Action 1.1: We will include a discussion of the practical applicability of our work in the “Impact Statement” section of our paper. We propose the following text:

“AI assistants are being developed and deployed in settings where humans can only partially observe what’s happening. For example, AI assistants including ChatGPT, Claude, and Gemini can search the web while only returning summaries to users (OpenAI, 2024; Anthropic, 2025; Google, 2024). Moreover, the sorts of AI models powering these assistants are processing increasingly long inputs. Whereas the original ChatGPT model could only process 4096 input tokens, today’s Gemini 1.5 Pro can process 2,000,000 input tokens—which is roughly 100,000 lines of code, or 16 novels of average length in English (Google, 2025). In the future, we anticipate that AI assistants will be deployed at increasing scale, independently taking more actions on behalf of users and processing increasingly long context lengths. We thus expect that over time, humans will have less and less ability to directly observe everything that’s happening.

The goal of our work is to lay a theoretical foundation for understanding when AI assistants have an incentive to interfere with human observations. Our results create a nuanced picture, suggesting that not all observation interference is inherently bad. In practice, we expect that AI assistants will exhibit observation interference for a mix of good and bad reasons. With this theory, our goal is to help practitioners disentangle these different incentives for observation interference when they emerge in practice.”

2.

What happens if A has multiple private information?

While we prove all our “negative” results (results showing that observation tampering can happen) with simple examples in which both A and H only make a single observation once, all our positive results apply to any POAG, and thus allow settings in which both A and H observe multiple times!

Action 2.1: To clarify how general our setting is, we will add the following sentence to Section 2.1:

“While all the examples in this paper are quite simple, note that the POAG setup and thus all our positive results are very general, allowing both H and A to observe private information at multiple times, taking actions that influence both the state and each other’s observations, and so on.”

3.

Are actions like, attempt to open door, for an agent trying to expose that the door is locked, a valid example for agent exposing private information, (let's say human could never have been in the room). If so, this action doesn't alter the environment but exposes private information. Therefore, in the presence of both such actions - observation interfering and signaling which would be preferred?

Thank you for proposing this great “locked door” example.

Action 3.1: We will add the locked door example to our paper after Definition 4.6, with this text:

“While in our examples, we will mostly consider actions that in some sense act directly on H’s observations, Definition 4.6 also considers the informational effects of physical actions. For example, if A (visibly) tries to open a door that A knows to be locked, then this reveals to H that the door is locked. Consequently, not trying to open the door (when A knows it to be locked) is an instance of observation interference in the sense of Definition 4.6. While having the same (null) effect on the state of the world, trying to open the door provides H with more information about the world.”

What do you think?

Thank you again for taking the time to review the paper and providing helpful feedback. Do the above actions address your questions about the paper? If not, what further clarification or modifications could we make to improve your score?

Thanks + response

Thank you for your review. We are glad you consider our work to "address a crucial aspect of the human-AI value alignment problem" and "have important implications for building trustworthy AI systems." We appreciate the discussion points you raised.

1.

The analysis… relies on specific assumptions… [such as] the Boltzmann rational model.

In this first paper on observation interference in human-AI interaction, we seek to lay a theoretical foundation using simple, well-known human models. Our results consider several models, including optimal play, decisions based on immediate outcomes, and Boltzmann rationality. Many published papers consider only the Boltzmann model, which is standard in the literature. For example, Hong Jun Jeon et al. (NeurIPS 2020) considers 11 feedback types that all assume Boltzmann rationality. Similarly, Ramachandran and Amir (IJCAI 2007) and Laidlaw and Dragan (ICLR 2022) exclusively study this model. We hope our paper will be judged by a consistent standard with other published work.

Action 1.1: In "Limitations and Future Work," we will add discussion of human modeling assumptions, suggesting additional models as a direction for future work.

2.

the communication channels appear to assume that the human and the AI agent can perfectly understand each other

We agree that communication is not a fully general solution to observation interference. But we don't view this as a weakness. If anything, our nuanced study would be less relevant if communication was a fully satisfying solution.

Action 2.1: We will add the following paragraph on practical limitations of unbounded communication channels after Theorem 4.5:

"One could argue that in practice, an unrestricted communication channel between A and H could usually be made available. However, Theorem 4.5 ignores various real-world obstacles. For one, it considers communication that incurs no cost, but realistically communication costs the human time and effort. Second, the optimal policy pair requires A to send information in a way that H can reliably understand and act on. We should expect that in practice, A and H sometimes cannot understand each other. Therefore, despite Theorem 4.5, we think observation interference is of broad practical relevance, even where A can, e.g., send text messages to H."

3.

the paper would benefit from empirical validation with human subjects

We agree this would be valuable for future work, but we also believe that would be a different kind of paper. This first paper on the topic is about laying theoretical foundations.

Action 3.1: In "Limitations and Future Work," we will discuss empirical validation with human subjects as a valuable future direction.

4.

Consider the computational complexity (NEXP-hard) of finding optimal policies in POAGs, how can the theoretical insights from this paper be practically applied to the real world?

Action 4.1: We will add the following discussion to our paper:

"How might our results apply given that finding optimal policies in POAGs is NEXP-hard?

Most of our paper is descriptive, characterizing when observation tampering could happen. Complexity considerations could affect these results in either direction. It's easy to construct environments where finding good observation-interfering policies is computationally intractable but constructing good non-interfering policies is easy; and vice versa. In practice, complexities of the environment can be orthogonal to incentives to interfere. For instance, a real-world version of the CUDA example is complex (A assesses complicated software compatibility issues), but the decision whether to interfere with observations is easy. We believe our characterizations remain useful even in complex environments (where we can't expect optimal policies), although we can't make as definitive claims as we can about optimal policies.

We have discussed allowing communication between H and A. A complexity-theoretic argument favors this solution: If H and A share all private information, the game effectively turns from a DecPOMDP into a POMDP. Solving POMDPs is PSPACE-complete and thus likely easier than solving DecPOMDPs."

5.

the ethical implications of designing AI systems that may optimally interfere… What constraints might be appropriate?

Please see Action 1.1 in our response to Reviewer fCAi, where we discuss implications. Our theory reveals that optimal assistants must sometimes interfere with observations, so interference is a nuanced issue. Appropriate constraints might include transparency about when interference occurs, alignment with user goals, and user controls to override interference when desired. As you suggest, future work with human subjects is needed to refine these constraints in practice.

What do you think?

Thank you again. Do the above actions address your concerns? If not, what further changes could we make to improve your score?