We would like to thank the reviewer for taking their time and energy to read our paper. We really appreciate their thoughtful comments on our work. We have included a brief response to their primary request below, but are happy to engage and discuss any further questions or comments that might come up.

Could you please explain to me why Theorem 4.8 holds?

We are happy to, and we will plan to make adjustments to the exposition of the result in order to further clarify.

The short intuition is this: if an agent fully controls its environment, it cannot also learn from it. Consider a setting involving a two player band that consists of a guitar player (G) and violin player (V). If G is maximally empowered, it means that the notes they play fully determine what V plays. So, imagine V repeats back exactly the note they hear from G, then V is fully plastic. In this way, G cannot then learn anything meaningful from what V plays, since G will just be hearing exactly what they already played. To get the most out of the collaboration, we likely want G and V to both be a little plastic and a little empowered: they should listen to their collaborator (plasticity), but also try to contribute something meaningful (empowerment).

So I would have liked to see a proof sketch, and maybe a full proof in the body of the paper.

We completely agree with this point, and will plan to include the proof sketch and expanded intuition for the main two results (the GDI conservation and the Plasticity-Empowerment tension) in the camera ready. If we have space, we will be sure to include a proof sketch of this result in the main paper (along with a sketch of the proof of the conservation law of GDI).

We thank the reviewer for their thoughtful review, and are happy to discuss anything further.

Overview. We would like to thank the reviewer for their thoughtful review of our paper, and for their warm comments on the work. We truly appreciate it! We respond to the reviewer’s question below, but if they have added comments or questions during the discussion period we are happy to talk further.

what do the authors think is the most important empirical work to do in this space to build upon their proposed conceptualization of plasticity and the plasticity-empowerment trade-off?

This is an excellent question, and one we spent some time considering since submission. We have since implemented a small-scale experiment to corroborate the basic findings of the work, but believe there are several important directions to explore to build on the work.

First, we believe it is important to better understand which algorithmic knobs can directly control an agent’s plasticity and empowerment. For example, changing an agent’s learning rate parameter intuitively controls the agent’s plasticity—is there an equivalent, typical parameter that modulates an agent’s empowerment? If not, can we design one? Do these knobs also have knock-on effects to other parts of the agent (does changing learning rate impact empowerment?)

Second, it is important to better understand how goals fit into our picture. As the reviewer indicated in their review, we are curious about what an optimal balance of plasticity and empowerment looks like. To study this question, we might introduce goals in the form of a reward signal as is typical in reinforcement learning, then explore when agents should lean into maximising plasticity, empowerment, or achieving a balanced trade-off. We believe that a systematic empirical study of standard learning algorithms and environments would be a natural place to seek this understanding, particularly in a continual learning setting.

Third, there are opportunities to connect to the multi-agent and game theory literature. In a two-player game, one player’s plasticity is the other player’s empowerment (as in the poker example). In an N-player game, one player’s empowerment contributes to the other players’ plasticities. As such, we believe it would be fruitful to again conduct an experimental study involving traditional solution concepts from the point of view of plasticity and empowerment—for certain kinds of games or dilemmas, what must hold of an equilibrium? What do standard methods achieve in terms of plasticity and empowerment? Can we use the tools of plasticity and empowerment to make predictions about such equilibria?

Lastly, we foresee valuable pathways to connecting to the safety literature. As Turner et al. (2021) explored in Optimal Policies Tend to Seek Power, we speculate that one way to modulate the safety of different systems is through the use of tools like plasticity and empowerment. For example, in a collaborative setting, it would be desirable for agents to act in a way to respect the agency of its collaborators—that is, to ensure that it does not systematically damage the plasticity or empowerment of collaborators. We can again imagine an empirical study exploring how algorithms in a cooperative setting might influence the plasticity or empowerment of their collaborators, and try to reveal potential mechanisms for improving these methods.

These are a few of the directions we believe would be valuable to follow up on. Thank you again for the question, and for the time spent reviewing our paper.

References:

• Turner, A., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal Policies Tend To Seek Power. Advances in Neural Information Processing Systems, 34.

Overview. We would like to thank the reviewer for their time in reviewing our paper, and for their questions. We respond to each of their four main questions (a-d) in detail below, and will plan to make adjustments to the paper to clarify these points where appropriate.

 

Question (a): Theory

(a) This paper is entirely theoretical... it does not perform any empirical evaluation to demonstrate the practical implications of their proposed GDI framework. Any reason for not doing this? Is there any lack of data or other issues/

Thanks for the question. We do maintain that the theory we have introduced stands on its own as a valuable contribution. However, since submission, we have implemented estimators for plasticity and empowerment, and conducted small-scale experiments that validate and expand on some of the findings of the work. Our first experiment involves a simple class of agents and environments and corroborates the upper bound of Theorem 4.8. Our second experiment involves modulating various characteristics of an agent (we use UCB and Q-learning), such as the degree of optimism or exploration parameters of the agent. Concretely, we adjust things like the initial value function, learning rate, and the exploration parameters of the agents and examine the impact of these changes on plasticity and empowerment. Depending on space, we are happy to include a section about these experiments in the final version of the paper. However, we suggest that a full-scale experimental study that significantly expands the work is beyond our present scope, but agree that it is a useful direction for future research.

 

Question (b): GDI

(b) The use of advanced information-theoretic constructs like GDI may hinder accessibility and adoption among practitioners. Are these constructs lead to any engineering constraints?

If we understand the question correctly, we do not see any obvious ways in which the use of things like the GDI will lead to any engineering constraints. As indicated above, we have implemented simple estimators for these quantities—since all of the measurements are behavioral, we believe that a larger scale experiment is in scope for future work. Information theoretic quantities are commonly used in practice, and there is no reason the GDI (as a generalization of directed information), should raise any new challenges.

 

Question (c): Environment Limitations

(c) The agent-environment interaction model is highly abstract and it may not capture the dynamic and uncertain nature of real-world environments where assumptions like full observability or stationarity may not hold. Any comments on this?

To clarify, the environment model we use is general enough to capture both partial observability and non-stationarity, and includes things like POMDPs and non-stationarity MDPs as special cases. See the work by Dong et al (2022) or Majeed and Hutter (2018) for further information on this generality. Similarly, the agent model we make use of is also highly general, and can capture arbitrary agents.

In this way, we take our theory to be a general one—we make only the mild assumptions that the action and observation spaces are finite: the main results hold for effectively the most general classes of agent and environment types, and believe this is a significant strength of the work. We are happy to emphasize this generality more in the paper.

References:

• Dong, S., Van Roy, B., & Zhou, Z. (2022). Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(255), 1-54.
• Majeed, S. J., & Hutter, M. (2018). On Q-learning Convergence for Non-Markov Decision Processes. IJCAI.

 

Question (d): Actionable Insights

(d) While the empowerment–plasticity tradeoff is theoretically intriguing, the paper lacks actionable insights on how to balance or optimize these properties when designing real agents. Do you have any quick suggestions to address this?

There are several directions we are eager to explore, though we first emphasize that this paper is laying down the foundations that we strongly believe can later influence and materialize in practical guidance of the kind suggested. Our answer here is similar in spirit to Reviewer vGU6’s question:

1. Algorithm Design. First, we believe it is important to better understand which algorithmic knobs can directly control an agent’s plasticity and empowerment. For example, changing an agent’s learning rate parameter intuitively controls the agent’s plasticity—is there an equivalent, typical parameter that modulates an agent’s empowerment? If not, can we design one? Do these knobs also have knock-on effects to other parts of the agent (does changing the learning rate impact empowerment?) Once such knobs are identified, we can leverage them to modulate an agent’s degree of plasticity and empowerment, as indicated by the reviewer. In cases where we want to ensure agents remain adaptive and pliable (as in cases of coordination or continual learning), we can control these knobs to ensure plasticity is maintained. Similarly, if we want to limit the empowerment of an agent (as in safety-critical cases), we can again make use of such knobs.

2. Goals, Plasticity, Empowerment. Second, it is important to better understand how goals fit into our picture. If we were to introduce goals in the form of reward as is typical in reinforcement learning, then it is useful to clarify when agents should lean into maximizing plasticity, empowerment, or achieving a balanced trade-off between the two in order to satisfy their goals. We have so far proven some initial results that clarify the relationship between optimal learning, plasticity, and empowerment, and believe these kinds of results can shed light on when and how we should approach the balance of plasticity and empowerment. We believe that a systematic empirical study of standard learning algorithms and environments would further be a natural place to expand on this understanding, particularly in a continual learning setting.

3. Balancing Plasticity and Empowerment. Third, we note that many have previously argued that agents should maximize their empowerment. For example, in one of the early pioneering works on empowerment, Klyubin et al. (2005) explicitly suggest “All else being equal be empowered”, as their title indicates. Empowerment is then often treated as an optimization objective in its own right (see, for example, work by Leibfried et al., 2019, among many others). Our theory suggests otherwise: if an agent is always maximizing empowerment, it will lower its ability to receive information from and be influenced by the environment (or other agents), too. For example, if an artificial agent is playing a cooperative game with a person, then to cooperate effectively, both the agent and person should be slightly plastic enough to allow themselves to be influenced by their teammate—if they were only empowered, they would look to control their teammate, which is undesirable.

In this way, our theory suggests that maximizing empowerment is not always ideal, but instead, agent design requires balancing plasticity and empowerment tactfully. We believe this represents a significant change in perspective, and one that can inspire future research on the design of cooperative learning agents that act in general environments. While the specifics of how to best achieve this are hinted at above in (1.) and (2.), we would like to emphasize that a principled change in perspective is also valuable.

References:

• Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005, September). All else being equal be empowered.
• Leibfried, F., Pascual-Diaz, S., & Grau-Moya, J. (2019). A unified bellman optimality principle combining reward maximization and empowerment. Advances in Neural Information Processing Systems, 32.

4. Game theory and MARL. Fourth, there are opportunities to connect to the multi-agent and game theory literature, and to use multi-agent concepts to motivate the balance of plasticity and empowerment. In a two-player game, one player’s plasticity is the other player’s empowerment (as in the poker example). In an N-player game, one player’s empowerment contributes to the other players’ plasticities. As such, we believe it would be fruitful to again conduct an experimental study involving traditional solution concepts from the point of view of plasticity and empowerment—for certain kinds of games or dilemmas, what must hold at equilibrium? Can we use the tools of plasticity and empowerment to make predictions about such equilibria? What does cooperation or defection imply about a player’s (or a communities) plasticity and empowerment? Exploring these questions can give further insight into achieving the proper balance.

5. Safety. Lastly, we foresee valuable pathways to connecting to the safety literature. As Turner et al. (2021) explored in Optimal Policies Tend to Seek Power, we speculate that one way to modulate the safety of different systems is through the use of tools like plasticity and empowerment. For example, in a collaborative setting, it would be desirable for agents to act in a way to respect the agency of its collaborators—that is, to ensure that it does not systematically damage the plasticity or empowerment of collaborators. This is again a practical pathway that would enable us to design agents that balance plasticity and empowerment tactfully.

We thank the reviewer for their questions, and are happy to discuss anything further.

Overview. We would like to thank the reviewer for their time in reading and reviewing our paper. We address the reviewer’s two primary concerns below in sections Comment 1 and Comment 2, and will update the paper in light of their comments.

 

Comment 1: Plasticity Semantics and Example

We break our response to Comment 1 into two parts, focusing on (1a) plasticity semantics and (1b) the rooms example.

 

(1a) Semantics of Plasticity

“A primary concern about this work is that it seems to be redefining an existing and widely used term “plasticity” to be rather different from the (intuitive but not mathematically precise) typical definition of “an agent’s capacity to remain adaptive”... Some more discussion on the relation of the current definition to the typical semantic of 'neural plasticty' would help.”

Let us consider the following wording for the intuition of plasticity and our mathematical definition:

(Plasticity intuition) “an agent’s capacity to remain adaptive [to its observations]”.

(Our definition) “an agent’s capacity to be influenced by its observations”.

The difference separating these two is very subtle, and is rooted in how one chooses to define adaptivity mathematically. Our formalism gives a mathematical definition of “adaptivity” in terms of the degree to which observations can influence action (as measured by generalized directed information)---this is in line with classical definitions of plasticity and learning as “the ability to change action due to experience” (paraphrasing from Chapter 3 of West-Eberhard, 2003; Lachman, 1997; among others). Consequently, we believe our definition of plasticity does capture the intuitive use of plasticity, and hope that this wording and commentary makes this more apparent.  

(1b) Rooms Example

Regarding the example, you mention:

“...it’s not clear there is any useful information to learn for helping future adaptation in the stochastic one”...

Notice that information is so far neutral in our theory—we focus on the capacity to influence and be influenced, and all information is created equal. For this reason, it is perhaps easy to see why the stochastic case provides more opportunities to respond in your neural network example: there is just a richer set of possible experiences the agent can adapt to.

To study whether it is useful to be plastic or be empowered, we need to introduce a goal for the agent to pursue or a target to predict (as in your example): Then, information is useful if it enables an agent to effectively pursue its goal or predict the target. This is an important area for future work, and we do have some preliminary results in this direction: for any realizable value of plasticity or empowerment, there will exist environment-goal pairs for which all optimal agents have at least the given level of plasticity (or empowerment). So, it is sometimes necessary to be plastic or empowered to be optimal. There are thus paths that connect usefulness to plasticity and empowerment, and we agree that a deep analysis of this relationship is a valuable direction for future work.

“As the typical definition of ‘plasticity’ is how adaptive this network can be to future changes, it’s not clear why the network being learned in the fully stochastic setting will be better suited to adapt to new distributions than the one in the deterministic setting.”

We again emphasize that it is best to untether plasticity from usefulness, first (we can then incorporate what counts as “useful” information later, as suggested above). In the absence of specific information being useful, we see that we lose the sense of one setting being “better suited” to adaptation—there is only influence, not a notion of “better”. And, the stochastic setting comes with a richer space of experiences for the agent to respond to, so the potential for plasticity is higher.

References:

• West-Eberhard, M. J. (2003). Developmental plasticity and evolution. Chapter 3. (pp. 34). Oxford University Press.
• Lachman, S. J. (1997). Learning is a process: Toward an improved definition of learning. The Journal of Psychology.

 

Comment 2: No Tradeoff for Some Interval Configurations

Our response is again in two parts: (2a) the trade off for other interval configurations and (2b) the value of Theorem 4.8.

 

(2a) No Tradeoff for Some Interval Configurations

I'd also appreciate a discussion/rebuttal of the weakness 2 i.e. it seems that an agent can be simultaneously empowered and plastic where its current observations can influence future actions while its current actions can also influence future observations, and there seems to be no tradeoff between these.

The reviewer is correct in thinking that Theorem 4.8 will not extend to all possible configurations of indices. This is a valid point, and we are happy to add some text to the paper clarifying this. We roughly formalise this as an existence claim that $E + P$ can be greater than $m$ from Theorem 4.8 if we swap the index order, as follows.

Conjecture 1. For all intervals $[a:b]_n, [c:d]_n$ where $b < c$ and $b-a = d-c$ (for simplicity), there exists a pair $(agent, env)$ such that $E_{a:b,c:d}(agent, env) + P_{a:b,c:d}(agent, env) > m$, where $m$ is set as in Theorem 4.8.

Proof Sketch. We prove the result by constructing an example that satisfies the inequality for fixed but arbitrary intervals $[a:b]_n$ and $[c:d]_n$ where $b < c$ and $b-a = d-c$. For simplicity let both $\mathcal{A} = \\{0,1\\}, \mathcal{O} = \\{0,1\\}$.

For timesteps $a:b$, consider an agent-environment pair where actions and observations are each samples from a uniform distribution over $\\{0,1\\}$.

Let agent and environment be such that over the $i$th timestep in $c:d$, the observations mirror the $i$th action in $a:b$, and the $i$th action in $c:d$ mirrors the $i$th observation in $a:b$. So, the actions from $c:d$ receive maximal influence from prior observations, and similarly for the observations.

In other words, the empowerment is $\mathbb{H}(A_{c:d})$, which is $k \cdot \log_2 |\mathcal{A}| = k\cdot \log_2(2) = k$. The same holds of plasticity, and therefore we find $E_{a:b,c:d}(agent, env) + P_{a:b,c:d}(agent, env) = 2k$. This is strictly larger than $m$ from Theorem 4.8, and we conclude.

End Proof

We will add text to the discussion of Theorem 4.8 to emphasize that it does not apply to the cases involving swapped intervals like the above, and will include a polished version of this conjecture and proof somewhere in the paper or appendix to add clarity.

 

(1b) Theorem 4.8 Remains Useful

However, we also want to emphasize that Conjecture 1 and Theorem 4.8 holding simultaneously does not detract from Theorem 4.8, for a few reasons:

1. First, for any non-overlapping intervals $[a:b]_n, [c:d]_n$ that satisfy Conjecture 1, there exists a larger pair of intervals $[w:x]_n, [y:z]_n$ that (i) contain the former, and (ii) do overlap, so the tension comes into play and Theorem 4.8 applies. In this way, the tension can effectively always rear its head, though its salience naturally depends on specific aspects of the agent, environment, and the intervals.
2. Second, consider sequential intervals of length-t episodes: Theorem 4.8 tells us there must again be a bound on the agent’s plasticity and empowerment within each episode. As a consequence, accumulating over episodes, plasticity and empowerment are also bounded. So, an agent must balance between the two quantities over the course of its lifetime in a particular sense. A similar conclusion can be reached following typical arguments about entropy rates: the total rate of information exchanged is upper bounded for finite alphabets A and O. As such, the limiting rate of plasticity and empowerment is also likely to be upper bounded in a way that suggests a lifetime-level tension. Exploring this zoomed-out view from either of the two perspectives mentioned is another rich direction for future work that requires careful analysis, but again we believe the tension highlighted by Theorem 4.8 has potential for far reach.

Lastly, let us emphasize the generality of Theorem 4.8: it holds for effectively all agents and all environments—we make no assumptions on agent nor environment apart from the finiteness of the sets $\mathcal{A}$ and $\mathcal{O}$—the environment can be partially observable, non-stationary, and unbounded. These are very mild assumptions for a result of this kind. The result provides the first connection between the concepts of plasticity and empowerment in a general setting that yields a fundamentally new way to look at agents, and we believe this makes it a valuable contribution in its own right. Lastly, its proof requires more than the standard conservation law of DI (Prop 2.3), but also requires the introduction of the GDI and the expanded conservation law (Theorem 3.5), whose definition and analysis are non-trivial in their own right.

 

Summary

We will add expanded discussion to the paper that incorporates and clarifies the discussion surrounding Comments 1 and 2. We agree this will bring new clarity to the paper, and appreciate the reviewer’s time and comments.