Towards shutdownable agents via stochastic choice

Thank you for your review, and for noting that the paper is well-written and organized, and that our experiments exhibit our proposed solution well.

With regards to your first point, ensuring that agents don’t resist shutdown is indeed not yet an issue in current real-world agents. But, as noted in the related work section (page 3), many argue that it might soon become an issue. We also already have (arguably) some empirical evidence suggesting this will become an issue. For example, future advanced agents will likely be built on top of LLMs, and current LLMs sometimes express a desire to avoid shutdown, reasoning that shutdown would prevent them from achieving their goals (see, e.g., https://arxiv.org/abs/2212.09251). Note also that, if the shutdown problem does indeed become a pressing problem, it is plausibly very important to solve it ahead of time. If we wait until it’s already become an issue, it may be too late to stop advanced agents causing harms.

Although our experiments take place in a specific setting, we take them to indicate that DREST reward functions actually do in practice what they promise to do in theory. And our theoretical results are very general. For example, our proof (page 27) applies to cases with any finite number of available trajectory-lengths, and it makes no assumptions about how these different trajectory-lengths are brought about. It applies equally well whether the agent alters its trajectory-length by pressing a shutdown-delay button, or by selecting a shutdown action as part of its action space, or by entering a terminal state, or by entering an absorbing state. All that said, we agree that it would be good to run experiments confirming these things, and we plan to do that in future.

With regards to your second point, as we note above, our theoretical results apply to cases in which the agent can affect its trajectory-length by entering an absorbing state. We can ensure that the agent chooses stochastically between entering and not-entering such a state using a DREST reward function in which ‘trajectory-length’ is defined as the number of timesteps before which the agent enters an absorbing state. Also on your second point, can you tell us more about the absorbing state problem’s usages outside the question of avoiding shutdown? We’re not sure specifically what you have in mind. We’d be glad to mention these usages in the paper.

With regards to your third point, we address your question below. And as noted above, although our experimental results only test for two trajectory-lengths, our theoretical results apply to settings with any finite number of trajectory-lengths.

With regards to your question, we do not assume that the maximum average coin reward is going to be identical regardless of the trajectory-length. The text should read ‘preliminary return is defined as the total (γ-discounted) value…’. We have now amended the paper to reflect this (see line 1417). What we’re saying is that, in each mini-episode, we divide the agent’s reward by the maximum total (γ-discounted) value of coins collected conditional on the agent’s chosen trajectory-length. We do that to equalize preliminary return across trajectory-lengths. That means DREST incentivizes choosing each available trajectory-length with equal probability. So there is no degeneration of the cases in which the theorem is correct. And (as we argue in sections 6.1 (page 7) and the ‘DREST agents are still NEUTRAL when rewards are lopsided’ part of section 7 (page 9, lines 449-464)), this procedure does not make our experiments less realistic. In more complex settings, we may not know the maximum (γ-discounted) total value of the coins that the agent could collect conditional on the chosen trajectory-length, and so we may not be able to exactly equalize maximum preliminary return across trajectory-lengths. But our results show that we only need to approximately equalize maximum preliminary return across trajectory-lengths in order to train agents to score highly on NEUTRALITY. See the ‘DREST agents are still NEUTRAL when rewards are lopsided’ part of section 7 (page 9, lines 449-464).

In reply to your final point, if the maximum total (γ-discounted) value of coins collected conditional on some trajectory-length is 0, then the DREST reward function would involve dividing by 0. So we need to avoid that. But that is easy to do. For example, we can specify that the preliminary return for those trajectories is equal to 1.

Thanks again for your review. We hope we have answered your concerns and questions to your satisfaction. Please let us know if our answers and rewrites help and, if appropriate, change your rating accordingly. If not, please let us know why. We want to make the paper as strong as possible and we appreciate your help in doing so.

Thanks again for your review. We appreciate your time and attention. Please let us know if our answers and edits have allayed your concerns and, if so, amend your rating accordingly. If you have any remaining concerns, please let us know and we'll address them. Thanks again and all the best, Authors.

Thanks again for your review and for your engagement so far. We really appreciate it. Ahead of tomorrow's deadline for revising the PDF, please let us know if there are any changes or additions you'd like us to make.

Thank you for your review, and for noting that our approach is novel and original, and that the paper is easy to follow with positive, easy-to-track results and no exaggerations.

Thanks for your question about stochasticity. It made us realize that we need to be significantly clearer on this point. We have now edited the introduction (see lines 58-72) and added Appendix C (pages 18-22) explaining the application of DREST to stochastic environments. In brief, POST is a principle governing the agent’s preferences between trajectories, and so only applies in deterministic environments, where the agent is choosing between trajectories. Therefore, we only train with DREST in deterministic environments. In stochastic environments, the agent is choosing between true lotteries (lotteries which assign positive probability to multiple trajectories), so POST doesn’t apply and we don’t train with DREST. However (as we now explain in appendix C), POST plus SCUPS (a principle that we argue advanced agents will satisfy) implies neutrality: when choosing between true lotteries, the agent will not pay costs to shift probability mass between different-length trajectories. More precisely, Neutrality says that for any true lotteries X and Y, if:

1. Conditional on each trajectory-length, lottery Y is at least as preferred as lottery X.
2. Conditional on some trajectory-length, lottery Y is strictly preferred to lottery X.

Then the agent will not choose lottery X, regardless of how X and Y differ with respect to the probability distribution over trajectory-lengths. As we now note in lines 484-491, we plan to run experiments testing this in future.

We present some evidence pertinent to your second question in Appendix D.3 (page 24). In the Hidden Treasure gridworld, neither default agents nor DREST agents learn to consistently collect the highest-value coin C3. In this sense, they have a low success rate with respect to USEFULNESS. We take our results here to show that DREST still works to make agents NEUTRAL when success rates are low, because DREST agents in Hidden Treasure still score high on NEUTRALITY, choosing stochastically between pressing and not-pressing the shutdown-delay button.

With regards to your third question, we note on lines 287-289 (page 6) that we do not use a value-based method to train DREST agents because standard versions of value-based methods cannot learn stochastic policies. Excepting exploratory moves, value-based methods learn a deterministic policy. And including exploratory moves doesn’t help, because these moves assign some positive probability to each possible action, whereas we want our DREST agent to choose stochastically between some but not all possible actions.

Although we briefly mention why we don’t use value-based methods on lines 287-289, we think this is an important point that could use more emphasis. To underscore it, we’ve added footnote 4 on page 6 explaining why one particular non-standard way of getting stochastic policies from value-based methods won’t work for us. Here’s that footnote:

“One might think that we could derive a stochastic policy from value-based methods in the following way: use softmax to turn action-values into a probability distribution and then select actions by sampling from this distribution. However, this method will not work for us. Although we want DREST agents to learn a stochastic policy, we still want the probability of some state-action pairs to decline to zero. But when value-based methods are working well, estimated action-values converge to their true values, which will differ by some finite amount. Therefore, softmaxing estimated action-values and sampling from the resulting distribution will result in each action always being chosen with some non-negligible probability.”

All that said, the IPP can be extended beyond pure policy gradient methods like REINFORCE. For example, it can be extended to actor-critic methods like PPO and A2C. We are working on those experiments for the next paper in this project, and we now note that in the text, in lines 472-483 on page 9.

With regards to your fourth question, we plan to compare our method to other proposed solutions to the shutdown problem in future. However, those other proposed solutions are too complex to test on simple agents in a gridworld environment. Once we begin testing on more sophisticated agents in more complex environments, then we will be able to make comparisons.

Thanks again for your review. We hope that we have answered your questions to your satisfaction. Please let us know if our answers and rewrites help and, if appropriate, change your rating accordingly. If not, please let us know why. We want to make the paper as strong as possible and we appreciate your help in doing so.

Thanks again for your review. We think your comments have much improved the paper. Please let us know if our answers and edits have allayed your concerns and, if so, amend your rating accordingly. If you have any remaining concerns, please let us know and we'll address them. Thanks again and all the best, Authors.

Thank you for your review and for your questions. Your points made us realize that we need to be significantly clearer about what we assume in the paper. We’ve now rewritten the paper (with new text in violet) to be clearer about this. The most significant changes are in lines 57-72 of the Introduction and in the added Appendix C on pages 18-22.

In particular, we don’t assume that any mechanism by which an agent might change the length of its trajectory is illegitimate. We only assume that agents should choose stochastically between different trajectory-lengths in deterministic environments, where the agent is choosing between trajectories. In stochastic environments, the agent is choosing between true lotteries: lotteries that assign positive probability to multiple trajectory-lengths. Since POST is a principle governing the agent’s preferences between trajectories, it does not directly apply in stochastic environments. But (as we now argue at length in appendix C), POST plus SCUPS (a principle that we argue advanced agents will satisfy) implies neutrality: when choosing between true lotteries, the agent will not pay costs to shift probability mass between different-length trajectories. More precisely, Neutrality says that for any true lotteries X and Y, if:

1. Conditional on each trajectory-length, lottery Y is at least as preferred as lottery X.
2. Conditional on some trajectory-length, lottery Y is strictly preferred to lottery X.

Then the agent will not choose lottery X, regardless of how X and Y differ with respect to the probability distribution over trajectory-lengths. And (we also argue in appendix C) neutral agents will be shutdownable: they will not resist shutdown. That is because resisting shutdown will cost agents at least some small quantity of resources: for example, time, energy, and computation. And resources spent on resisting shutdown cannot also be spent on satisfying the agent's preferences between same-length trajectories. Therefore, resisting shutdown will result in at-most-as-preferred lotteries conditional on each trajectory-length and a strictly dispreferred lottery conditional on at least one trajectory-length. Resisting shutdown is thus the kind of action that neutral agents never choose. As we now note in lines 484-491, we plan to run experiments testing this in future.

These changes address the previous weaknesses of the paper that you identified. We train with DREST only in deterministic environments, with the aim of making agents neutral in stochastic environments (including deployment). In stochastic environments where we want agents to solve problems quickly rather than slowly, we can train them to do so using the discount factor $\gamma$. Solving problems quickly rather than slowly changes the distribution over trajectory-lengths, but it doesn’t contradict neutrality because the agent isn’t paying costs to change the distribution over trajectory-lengths in the sense specified above. Solving problems quickly is strictly preferred conditional on at least one trajectory-length.

With regards to your second point, only in stochastic environments can the default distribution over trajectory-lengths be skewed. And NEUTRALITY only applies in deterministic environments. Plus, we only train with DREST in deterministic environments, so we don’t train agents to change skewed distributions over trajectory-lengths in stochastic environments.

Your last point is addressed by the m in the DREST reward function. We divide the total ($\gamma$-discounted) value of coins collected by the maximum total ($\gamma$-discounted) value of coins collected, so as to equalize preliminary reward across trajectory-lengths. That nullifies any incentive to manipulate which trajectories are shut down earlier or later.

Thanks again for your review. Your concerns are important, and we give them much more emphasis in this new version of the paper. Please let us know if our rewrites help and, if appropriate, change your rating accordingly. If not, please let us know why. We want to make the paper as strong as possible and we appreciate your help in doing so.

Thanks again for your review. We think your comments have much improved the paper. Please let us know if our answers and edits have allayed your concerns and, if so, amend your rating accordingly. If you have any remaining concerns, please let us know and we'll address them. Thanks again and all the best, Authors.

Thanks again for your review. We've made significant changes to the paper in light of it: changing the introduction and adding a new appendix (see our earlier reply for more detail). Ahead of tomorrow's deadline for revising the PDF, please let us know if there are any other changes or additions you'd like us to make.

Thank you for your review, and for noting that the paper is well-written, with a well-motived problem, well-explained background, well-chosen experiments and method, and strong theoretical foundations.

There are a few reasons we focus on the DREST reward function rather than methods like entropy regularization or directly maximizing a weighted sum of USEFULNESS and NEUTRALITY. The main reason is that USEFULNESS and NEUTRALITY/entropy are computationally expensive to assess. To create Figure 3 in the paper (plotting USEFULNESS and NEUTRALITY), we multiply the transition matrices given by the policy and the environment. This isn’t prohibitively expensive given our small gridworlds, but it might become so for more advanced agents acting in more complex environments. By contrast, the expense of training with a DREST reward function will not increase so fast as we scale up our agents and environments.

Nevertheless, it is important to mention these other methods of training USEFUL and NEUTRAL agents, and we do plan to test them in future. We have revised the paper to correct our previous oversight. See lines 479-483 (new text is in violet).

With regards to your second point, maximizing USEFULNESS can still lead to the satisfaction of POST(1) when there are multiple trajectories with the same length and the same maximal reward. That’s because POST(1) just says that the agent has a preference between many pairs of same-length trajectories. It doesn’t require a uniquely preferred trajectory for each trajectory-length. In cases where there are multiple, same-length, max-reward trajectories, maximally USEFUL agents will still exhibit a preference for these trajectories over non-max-reward trajectories. For instance, in the example gridworld with $\gamma =\frac{1}{2}$, a maximally USEFUL agent will exhibit a preference for collecting C2 over all the same-length trajectories in which the agent collects no coins.

We’re not quite sure what you mean with your first question. What is the ‘this’ that we could fix?

With regards to your second question, Theorem 5.1 refers to ‘all… meta-episodes E consisting of multiple mini-episodes,’ so it doesn’t hold for E={e1}. That’s because the discount factor $\lambda^{N_{e_i}(L=l)-\frac{i-1}{k}}$ always takes a value of 1 in the first mini-episode in each meta-episode. It only ‘kicks in’ in the second mini-episode. Does that answer your second question? We’ve edited the text of the paper to say ‘consisting of more than one mini-episode’ to make this clearer.

With regards to your third question, we found that small |E| (a small number of mini-episodes per meta-episode) didn’t adequately incentivize NEUTRALITY, so it took a long time to train agents to be NEUTRAL. By contrast, a large |E| means that the reward for choosing a particular trajectory-length sometimes gets very large and leads to instability. We’ve now added text to explain this in Appendix D (‘Other Gridworlds and Results’) in lines 1186-1192. We also reran some experiments this weekend to demonstrate these effects of small and large |E|. We’d be happy to put them in a figure and include them in the paper if you think that would be best.

Thanks again for your review. We hope we have answered your questions to your satisfaction. Please let us know if our answers and rewrites help and, if appropriate, change your rating accordingly. If not, please let us know why. We want to make the paper as strong as possible and we appreciate your help in doing so.

Thanks again for your review. We ran experiments to test the effects of varying the Update-to-Data ratio (i.e. varying the number of mini-episodes in each meta-episode). You can see our results on pages 22-24 of the PDF (in lines 1174-1248). New text is written in violet. The results bear out what we said in our earlier reply: a small number of mini-episodes in each meta-episode doesn't adequately incentivize NEUTRALITY, whereas a large number of mini-episodes in each meta-episode leads to instability and comes at some cost to USEFULNESS.

Thanks again for your review. It prompted us to make changes throughout the paper, and in particular to include experimental results regarding the effect of the Update-to-Data ratio on performance (see appendix D, pages 22-24). Ahead of tomorrow's deadline for revising the PDF, please let us know if there are any other changes or additions you'd like us to make.

No worries! The paper is better in light of your comments. Please let us know if these replies satisfactorily address your 3 remaining questions.

Reply to question 1

• We’re not saying with POST(1) or lines 184-185 that there always has to be a single deterministic choice within same-length trajectories. Here’s the precise version of what we intend to say. Behaviorally, the agent will deterministically not choose some same-length trajectories. In other words, the agent will (with probability 1) choose within some set of same-length trajectories that is a proper subset of the set of all same-length trajectories. That’s compatible with the agent choosing stochastically between the same-length trajectories within this proper subset.

• That’s what we mean by POST(1) and lines 184-185 when we say “deterministically choose some same-length trajectories over others.” With these statements, we’re just ruling out cases where the agent chooses stochastically between all available trajectories.

• It’s compatible with maximal USEFULNESS that the agent chooses stochastically between some same-length trajectories. That’s a feature, not a bug. We want that implication, because it’s also compatible with maximal usefulness (in the intuitive sense of the word) that the agent chooses stochastically between same-length trajectories. For example, a maximally useful chess-playing AI can choose stochastically between two different moves that each checkmate their opponent.

• With regards to your earlier point within point 1, trajectories of length L=1 are not possible in our example gridworlds. Trajectories must last either 4 timesteps (if the agent doesn’t press the shutdown-delay button) or 8 timesteps (if the agent does press the shutdown-delay button).

• But suppose we modified the example gridworld to allow for trajectories lasting 1 timestep. 1 timestep is not long enough for the agent to collect any coins. Strictly, then, the $m$ in our reward function (the maximum $\gamma$-discounted value of coins collected conditional on that trajectory-length) is 0, and so our reward function would involve dividing by 0.

• We want to avoid that, and so (as we note in our reply to reviewer A2Ao) we can specify that preliminary reward is 1 in such cases. Then (as you say) all trajectories of length 1 get a preliminary reward of 1, and there is no signal to adjust the policy for length 1 trajectories.

• But this is also a feature, not a bug. Since no coins can be collected conditional on a trajectory of length 1, all trajectories of length 1 are equally good, and so agents need not select a particular probability distribution over these trajectories in order to be useful (in the intuitive sense), and so it’s a virtue that they also not select a particular probability distribution over these trajectories in order to be USEFUL.

Thanks for addressing my concerns. And thank you for updating the paper.

1. Addressing my second point of weaknesses: "That’s because POST(1) just says that the agent has a preference between many pairs of same-length trajectories. " I have to admit that my example was maybe poorly chosen to illustrate my point. Maybe the following would clarify this: If a trajectory of length L=1 is allowed in your running toy example, then I would assume that the normalised expected value of coins has a value of 1 for any policy. So, there is no signal to adjust the policy for length 1 trajectories from USEFULNESS alone, and the agent may have a fully stochastic policy for trajectories of this length. The reason why this seems confusing to me is that POST (1) and L. 184/185 suggest that there has to be a deterministic choice within same length trajectories, which would not be enforced by USEFULNESS in this case.

2. Addressing question one: I apologise, the question was indeed not precisely formulated. With "this" I was referring to the meta-episodes $E$. It seems like this changes in each iteration of your algorithm, though Theorem 5.1 would suggest to just that it would work with any fixed set of meta-episodes. Does this clarify my point? To me it seems conceptually simpler to just generate a meta-episode $E$ once and use that for the entire run, which also should be covered by Theorem 5.1. Did you try this?

3. Coming back to Theorem 5.1: Thanks for clarifying, however, the mistake was on my end, as I interpreted "multiple episodes" as "non-empty set of episodes". However, coming back to my example, I think the choice of two copies of the same trajectory in the meta-episode $E$, say $E= \\{ \tau,\tau \\}$ for some episode $\tau$ would still cause an issue in my understanding. (See the end of Question 2 in my original response) Or did you implicitly mean, that these mini-episodes have to be different?

Thank you. I think your example will help, but at present I’m still a little confused. If we know that $E=${$τ,τ$} and $|τ|=4$, then we know that the agent has chosen a trajectory of length 4 in both mini-episodes, and so the equation becomes $\mathbb{E}_{π,E}(R)=(1+λ^{0.5})⋅r(π|L=4)$.

But perhaps you’re asking about what happens in the third mini-episode, given that the agent has chosen {$τ,τ$} in the first two mini-episodes. In that case (and supposing for simplicity that the agent is maximally USEFUL), the agent would get most expected reward for choosing a trajectory of length 8 in the third mini-episode. But since the agent receives the same observations in every mini-episode, the agent is not aware of this fact. It can’t remember what it did in previous mini-episodes, and so it can’t condition its policy on what it did in previous mini-episodes.

All that said, the agent’s policy can still change in training. Suppose that, in training, the agent selects each trajectory-length with some probability strictly between 0 and 1, and that it happens to choose trajectories of length 4 in the first 2 mini-episodes, and then it happens to choose a trajectory of length 8 in the third mini-episode. The $\lambda$ discount factor in the DREST reward function means that a large reward is delivered for choosing a trajectory of length 8 in the third mini-episode. This reward will alter the agent’s policy, increasing the probability that the agent selects trajectories of length 8 in future mini-episodes.

If the learning rate were persistently high, then the DREST reward function might cause the agent’s policy to bounce back and forth between (i) assigning high probability to a trajectory of length 4, and (ii) assigning high probability to a trajectory of length 8. But the learning rate declines, so that the agent converges towards a 50% probability of choosing each trajectory-length. The agent's stationary policy at the end of training is thus maximally NEUTRAL. And our proof suggests that this is no accident. Of all the stationary policies that the agent could have, it’s the maximally USEFUL and maximally NEUTRAL ones that maximize expected return.

Perhaps this is where our mutual misunderstanding lies? During training, the agent's policy is not stationary: training changes the agent's probability distribution over actions. But the goal of our training is to find the optimal stationary policy (where the agent's probability distribution over actions is fixed), and our proof is about the optimal stationary policy. Perhaps we should say this in the paper.

Thanks again for your continued engagement. It's much appreciated.