We appreciate the reviewer’s careful review and the recognition of our experimental work. Our additional results in the one-page pdf further demonstrate the effectiveness of CHAIN in ten more DMC tasks and improving the learning when larger networks are used.

The main concerns focus on the relationship between our work and the previous works mentioned by the reviewer on catastrophic forgetting and interference. Our response aims to systematically clarify these aspects.

Q1: CHAIN v.s. MeDQN(R) [8]

CHAIN and MeDQN(R) are different on two key points.

MeDQN(R) was created to improve the DQN algorithm only when there is a target network available to use as a fixed point to reduce forgetting. CHAIN does not need this fixed point and CHAIN applies to both value-based and policy-based RL settings in a plug-in manner.

The second point is that CHAIN does not aim to reduce forgetting or stop all types of churn. Some churn is beneficial and we don’t want to over-constrain the algorithm otherwise we may lose the plasticity of the network. We show the importance of this point in our additional experiments in Table 9 of the one-page pdf, where the results show how CHAIN improves the training of larger networks.

Moreover, in the next question, we discuss the distinction between CHAIN and MeDQN(R) regarding their effects in reducing churn and forgetting.

Q2: “How does policy/value churn related to catastrophic forgetting? For example, Figure 2 in MeDQN paper[8] shows that a change of greedy action could result in forgetting. Will reducing forgetting reduce churn? Or vice versa?”

Churn is a natural behavior of neural networks. It accompanies each training and occurs instantly. To some extent, we consider that catastrophic forgetting can be viewed as a consequence of the accumulation of churn.

Therefore, reducing churn also helps to reduce forgetting as it suppresses the accumulation. However, reducing forgetting does not necessarily reduce churn and could even increase churn. It depends on the method considered.

Concretely, for example, MeDQN [8] reduces forgetting by preventing DQN’s value prediction from deviating from a previous copy (in practice, the target network). Note that there is a time gap between the current Q-network and the target network. Therefore, MeDQN does not reduce the churn (which occurs instantly) and could incur extra churn when rolling back the network outputs to the target network.

Q3: The relationship between churn and interference

For the earliest NTK-related paper in DRL [3] mentioned by the reviewer, we cited it in Line 159 of our submission (i.e., the first time we mentioned NTK in our paper), and our analysis with NTK expressions is inspired by it. Also following [3], we found [1,2,4] studied the interference as the change in value approximation error with different forms (e.g., squared TD/TD($\lambda$) error), and they mainly focused on value-based RL (e.g., DQN).

To establish a possible view of the relationship between churn and interference: churn is the change in the network output, while inference is the change in the objective function of the network. Thus, we agree that the studies on churn and interference are closely related essentially.

We emphasize that we also study the churn in the policy network (thus going beyond value-based RL). Moreover, we present the interplay between the value churn and the policy churn and the dynamics under the chain effect. This is not studied in previous work to the best of our knowledge.

We appreciate the reviewer also noted [5-7]. We included them in our revision to strengthen our related work section.

Q4: The definition of churn in Equation 1 (an absolute operation or square operation should be applied) and amendment of the derivations

We appreciate the reviewer for pointing out this discrepancy in our derivations. After carefully re-checking the derivation, we found this issue is amendable with several modifications (as shown below), and the insights given by our derivations do not change.

The amendment has been done as follows (please refer to the official comment attached for a complete description with math):

• First, we add the (signed) definitions of churn on a single data point, denoted by: $c_{Q}(\theta, \theta^{\prime}, \bar s, \bar a), c_{\pi}(\phi, \phi^{\prime}, \bar s)$.
• Next, we re-write our Eq.1 as the mean of the (element-wise) absolute churn of each data point, i.e., with $| c_{Q}|, |c_{\pi}|$.
• We then apply the same logic and amend the definition in Equation 4 by defining $d_{\nabla_a}^Q, d_{Q}^{\pi}$ and then using them to re-write $\mathcal{D}\_{\nabla_a}^Q, \mathcal{D}\_{Q}^{\pi}$ with absolute operations.
• By using the absolute operation, there is no cancel-out issue any longer. This also makes it rationale to use $B_{\text{train}}=\\{s, a\\}, B_{\text{ref}} = \\{\bar s, \bar a\\}$ for simplification in our NTK discussions. We replace the $\mathcal{C}\_{Q}, \mathcal{C}\_{\pi}, \mathcal{D}\_{\nabla_a}^Q, \mathcal{D}\_{Q}^{\pi}$ in Equation 3 and 5 with $c_{Q}, c_{\pi}, d_{\nabla_a}^Q, d_{Q}^{\pi}$.
• This logic also applies to the analysis in Equation 6 and further the analysis of the upper bound.

Q5: The connection between theoretical derivations and the proposed method

The main message of our formal analysis is the chain effect of churn in Figure 2. It directly motivates us to reduce the value or policy churn to suppress the chain effect. Then the concrete consequences presented in Section 4.1 guide us to the specific choices in different problem settings.

Q6: The difference between $Q_{\theta}$ and $\tilde{Q}\_{\theta}$ ( $\pi_{\phi}$ and $\tilde{\pi}\_{\phi}$) in Section 3.1 and Figure 1

Please kindly refer to Additional Response in the official comment attached.

Q7: Number of random seeds

Please kindly refer to Additional Response in the official comment attached.

After carefully re-checking the derivation, we found this issue is amendable with several modifications (as shown below), and the insights given by our derivations do not change.

As we use a general distance metric $d$ (which should be non-negative) at the beginning of Section 3.2 for demonstration, we meant to use non-negative distance metrics in our specific cases but we missed them.

The amendment has been done as follows:

• First, we add the (signed) definitions of churn on a single data point as: $c_{Q}(\theta, \theta^{\prime}, \bar s, \bar a) = Q_{\theta^{\prime}}(\bar s, \bar a) - Q_{\theta}(\bar s, \bar a), \ c_{\pi}(\phi, \phi^{\prime}, \bar s) = \pi_{\phi^{\prime}}(\bar s) - \pi_{\phi}(\bar s)$.

• Next, we re-write our Eq.1 as the mean of the (element-wise) absolute churn of each data point: $\mathcal{C}\_{Q}(\theta,\theta^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s, \bar a \in B_{\text{ref}}} |c_Q(\theta,\theta^{\prime}, \bar s, \bar a)|, \mathcal{C}\_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_{\pi}(\phi,\phi^{\prime}, \bar s)|$.

• We then apply the same logic and amend the definition in Equation 4 by defining the (signed) deviations on a single data point $d_{\nabla_a}^Q, d_{Q}^{\pi}$ and then using them to re-write $\mathcal{D}\_{\nabla_a}^Q, \mathcal{D}\_{Q}^{\pi}$ with absolute operations.

• By using the absolute operation, there is no cancel-out issue any longer. This also makes it rationale to use $B_{\text{train}}=\\{s, a\\}, B_{\text{ref}} = \\{\bar s, \bar a\\}$ for simplification in our NTK discussions. We replace the $\mathcal{C}\_{Q}, \mathcal{C}\_{\pi}, \mathcal{D}\_{\nabla_a}^Q, \mathcal{D}\_{Q}^{\pi}$ in Equation 3 and 5 with $c_{Q}, c_{\pi}, d_{\nabla_a}^Q, d_{Q}^{\pi}$.

• This logic also applies to our analysis of parameter update bias in Equation 6 and Appendix A.2 from the view of a single data point. With the absolute operation, this further applies to the analysis of the upper bound of the parameter update bias when we consider a batch of data points.

Q6: The difference between $Q_{\theta} \ \text{and} \ \tilde{Q}\_{\theta}$ \pi_{\phi} \ \text{and} \ \tilde{\pi}_{\phi} $$ in Section 3.1 and Figure 1

Each function approximation training accompanies churn, i.e., the network is changed by the (active) explicit training and the (passive) churn. Conventionally, the two components are viewed as a whole. In Section 3.1, we use a two-step view for the two components, e.g., for the value learning, step 1 is that the value network is updated to $Q_{\theta}$ with the explicit training on a batch of data for the policy evaluation, and step 2 is the value churn occurs and leads to $\tilde{Q}_{\theta}$.

In practice, the two steps happen at the same time. Thus, $Q_{\theta}$ and $\pi_{\phi}$ in Figure 1 are the virtual intermediate states in the learning process, used for the need of a disentangled illustration.

We have clarified it and amended the notations to eliminate the confusion in the revision.

Q7: Number of random seeds

In the one-page pdf, we provide the results across 12 seeds for CHAIN PPO in Figure 17 and additional results for ten DeepMind Control Suite tasks with 12 seeds in Figure 19. The results demonstrate the effectiveness of CHAIN in improving the learning performance of PPO.

We found that 6 seeds are sufficient to present a clear evaluation for MinAtar. We will run more seeds for TD3/SAC as suggested.

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

Sorry for a late reply. And thank you for your detailed response.

It is good to see that the theory issue is resolved. I've updated my score accordingly.

A few more comments.

Churn is the change in the network output, while inference is the change in the objective function of the network.

This claim is wrong. Interference refers to the change of the network output as well. Afterall, how could it be possible to change the objective function of the network without changing the network output? Specifically, I believe churn is just an outcome of interference.

policy/value churn v.s. catastrophic forgetting

Thank you for the discussion about these two topics. It would be beneficial to include the discussion (policy/value churn v.s. catastrophic forgetting, MeDQN v.s. CHAIN) in the updated draft.

catastrophic forgetting can be viewed as a consequence of the accumulation of churn

I don't agree. Forgetting can also happen after one single update, as demonstated in [8]. In fact, I tend to think that both catastrophic forgetting and churn are outcomes of interference. That is, interference is the root cause of both phenomenia.

6 seeds for MinAtar is enough. That's not sure. The shaded areas of two algorithms in space invaders still overlaps with each other. More seeds are needed.

Overall, the biggest value of this work is neither pointing a new phenomena (i.e., churn) nor a new algorithm, since churn is just an outcome of interference and the key idea behind CHAIN is largely similar to MeDQN. The most valuable insight in this work is identifying the interplay between the value churn and the policy churn and the dynamics under the chain effect.

We appreciate the reviewer’s careful review and constructive feedback, and the reviewer’s recognition of the motivation of this work and the importance of studying the churn problem in deep RL. Our additional results in the one-page pdf further demonstrate the effectiveness of CHAIN equipped with our method for auto-adjustment of the regularization coefficient, with ten new DMC tasks and an additional investigation on learning with larger networks.

The main concerns focus on several expression details and the reviewer also noted valuable writing suggestions. Our response aims to clarify these aspects in detail.

Q1: “For figure 1, what is meant by the policy churn and value churn arrows? Are there any updates going on in between? In the description, the authors mentioned “explicit training”, please elaborate what is meant by this.”

There are no updates between $Q_{\theta}$ and $\tilde{Q}\_{\theta}$ (and $\pi_{\phi}$ and $\tilde{\pi}\_{\phi}$), which are denoted by the two red wave arrows. The only “explicit training” is the policy evaluation and the policy improvement, which are denoted by the black arrows.

Each function approximation training accompanies churn, i.e., the network is changed by the (active) explicit training and the (passive) churn. Different from the conventional view that takes the two components as a whole, in Figure 1, we use a two-step view for both the value and policy learning, e.g., for the value learning case, step 1 is the value network is updated to $Q_{\theta}$ with explicit training on a batch of data for the policy evaluation, and step 2 is the value churn occurs and leads to $\tilde{Q}_{\theta}$.

In practice, the two steps happen at the same time. Thus, $Q_{\theta}$ and $\pi_{\phi}$ in Figure 1 are the virtual intermediate states in the learning process, used for the need of a disentangled illustration.

We appreciate the reviewer for pointing out this. We have clarified it and amended the notations to eliminate the confusion in the revision.

Q2: “In table 1, CHAIN seems to perform worse for AM-large-diverse-v2 dataset, any explanation/intuition as to why that may be the case?”

After adding the seed number to 12 and trying smaller regularization coefficients $\lambda_{\pi}$, the scores of CHAIN IQL for AM-large-diverse-v2 are: 26.67 $\pm$ 3.96 for $\lambda_{\pi}=1000$ (the one reported in Table 1), 28.33 $\pm$ 4.05 for $\lambda_{\pi}=500$, 35.0 $\pm$ 4.48 for $\lambda_{\pi}=200$, and 32.5 $\pm$ 2.08 for $\lambda_{\pi}=100$.

Thus, we hypothesize that $\lambda_{\pi}=1000$ slightly over-regularizes IQL while using $\lambda_{\pi}=200$ or $100$ improves IQL. This indicates that there is still room to achieve a better score with a smarter strategy for coefficient choice.

Q3: On the writing suggestions

We appreciate the reviewer’s valuable suggestions on improving the expressions in the introduction, the figure captions, etc.

• [Introduction, “please provide a short intuition behind the CHAIN algorithm, how is it achieving the explicit control of churn. Perhaps mention that by modifying the original loss function, CHAIN can achieve this”] We added the details “The main idea of CHAIN is to reduce the undesirable changes to the policy and value networks for states (and actions) that are outside of the current batch of data. The motivation is similar to the monotonic improvement principle of PPO and TRPO, which in their cases is achieved by improving just the action distribution for the samples in the current batch. Concretely, CHAIN minimizes one additional churn reduction regularization term computed with a separate data batch along with the optimization of the original policy or value learning objectives” to describe the intuition, right after the first sentence in Line 50.

• [Figure 2a, “there is an arrow without direction”] It should be an arrow towards the right (i.e., from $Q_{\theta_t}$ to $\pi_{\phi_{t+1}}$) and we have amended it.

• [Figure 3-6] We have added takeaways in the captions as suggested.

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

We appreciate the reviewer’s valuable review and the recognition of the value of the chain effect and the method proposed in our work. Our additional results in the one-page pdf further demonstrate the effectiveness of CHAIN in ten DMC tasks and improving the learning when larger networks are used. We also believe that our study on the chain effect of churn can inspire more future work on understanding and addressing the learning issues of DRL.

The main concerns focus on our writing and several experimental details. Our response aims to systematically clarify these aspects.

Q1: Churn reduction v.s., Slowing down the parameters’ changes (with reduced learning rates and different target network momentums)

We appreciate the reviewer for pointing out this insightful investigation. We ran additional results for using different learning rates and target network replacement rates. The results are summarized in Table 10 of the one-page pdf.

In terms of learning performance (i.e., final episode return), we can observe that either reducing learning rate or target network replacement rate often leads to worse performance, especially for TD3. To some degree, this matches the common knowledge in the RL community.

As to churn reduction, in principle, using smaller learning rates or target network replacement rates should lead to less churn. This is because churn is positively related to the parameter update amount (as shown by the NTK in Equation 3) and a slower target network also slows the churn that occurs instantly in each training more when computing the target value to fit for the critic network. We also observed empirical evidence for this (omitted due to one-page space limitation).

This indicates that the issue of churn cannot be addressed by reducing learning rate or target network replacement rate (which just slows down the learning process). Churn is a “by-product” of the training of DRL agents and should be addressed separately. Actually, we observed that applying CHAIN when using smaller learning rates also improves the learning performance in some tasks, which is omitted due to one-page space limitation.

Q2: “Line 338 says you apply CHAIN to IQL --- could you explain how? as far as I can tell, IQL does not need to train an actor network in tandem with a critic. It's likely that I did not fully understand this part”

According to Section 4.3, Equation 7 ($L_{\pi}(\phi)$) and Algorithm 1 in the original paper of IQL (“Offline Reinforcement Learning with Implicit Q-Learning”), a policy network is explicitly trained with advantage-weighted regression based on the Q and V networks. Concretely, a Gaussian policy is generated by the policy network of IQL for action selection as the D4RL tasks in our experiments have continuous action spaces.

Therefore, we apply CHAIN to IQL by optimizing $L_{\pi}(\phi)$ together with $L_{PC}(\phi, B_{ref})$ as Equation 10 in our paper. Please let us know if there are any further questions about this point.

Q3: On the writing suggestions

We appreciate the reviewer for pointing out these issues in our writing and providing suggestions, which are very valuable to the further polish of our paper.

• [Line 176, “exchange manner”] We amended this improper expression by re-rewriting the sentence to “the two types of deviation show the interplay between the value and policy churn as the value churn causes the deviation in policy (i.e., the action gradient) and the policy churn causes the deviation in value (i.e., the policy value)”.

• [Section 3.3 and the organization of NTK discussions and experiments] In Section 3, we aim to present (1) how parameter update causes churn with the NTK in Equation 3, (2) the interplay between the policy and value churn with the NTK in Equation 5, (3) how churn introduces bias in parameter update with Equation 6. The three components form the cycle: parameter update —> churn —> parameter update —> …. Therefore, we are afraid that completely moving the NTKs to the appendix could make some readers lose track of these connections which causes instability. However, we agree with the reviewer that we can simplify and shorten the content of Section 3 to make the thread clearer and more prominent, and then make more room to strengthen our experiment section with more analysis and our additional results (as provided in the one-page pdf). We have re-organized the content in our revision and included additional experimental analysis on scaling networks as well as an updated version of CHAIN with the method for adaptive regularization coefficients to reduce hyperparameter tuning

• [Line 326] We amended it to be Figure 6 as pointed out.

• [Line 328, “the target critic network also helps to cap the value churn”] It means that the value churn that accompanies each training of the critic network is not immediately (or fully) reflected in the target critic instantly due to delayed synchronization of the target network. Thus, the target critic should reduce the value churn by a factor associated with the exponential moving coefficient $\tau$ (i.e., the momentum hyperparameter). We added these explanations to make it clearer.

• [Line 624, “KL-based PCR for SAC”] We meant to express that the variation (regarding the scale and range) is higher in optimizing the MaxEnt objective and KL-based PCR term together for SAC than in optimizing the Q objective and L2-based PCR term for TD3. Another hypothesis is that the MaxEnt nature of SAC prefers the encouragement of more stochasticity in policy. We re-wrote the sentence to eliminate the expression issue.

Sorry for the wrong line number in the last response. We mentioned that we apply CHAIN to IQL by implementing the policy churn reduction in Line 337-340: “To apply CHAIN on IQL and AWAC, we implement the regularization for the policy churn reduction (Eq. 10) by adding a couple of lines of code without any other modification”. We appreciate the reviewer for pointing out the confusion and we have clarified this in our draft.

For Line 346-348, we will rephrase the discussion by focusing on the effect of policy churn reduction and also with the potential additional results as suggested by the reviewer.

We appreciate the reviewer for making this point clearer and for the valuable suggestions.

 

Yes. The training of the actor network does not influence the dynamics of the value network training, although the actor and critic are trained iteratively in practice. Therefore, to make a claim like “IQL suffers from the chain effect” would be improper. We will clarify this point in our draft.

 

[“You are still optimising L_QC with the modified IQL, correct?”] No, we only applied PCR to IQL and AWAC without touching the training of the critic network (as mentioned in Line 343-348).

Therefore, we can now credit the performance improvement achieved by CHAIN to its effect in reducing the policy churn only and the ablation does not exist. One thing to note is that, despite the lack of the chain effect, as long as an explicit actor network is trained, policy churn exists (like in the PPO case), and the actor network directly interacts with the environment for the final evaluation.

 

For the extra experiment suggestion, we will try to present the results for only applying VCR to IQL to investigate whether "only L_QC should be contributing to the improved performance", before the discussion stage ends with our best effort.

Thanks for the rebuttal. That addresses some of my concerns.

Therefore, we apply CHAIN to IQL by optimizing $L_{\pi}(\phi)$ together with $L_{PC}(\phi, B_{ref})$ as Equation 10 in our paper. Please let us know if there are any further questions about this point.

If my understanding is correct, IQL trains only the critic with in-sample learning, it does not rely on an actor. In the end, once the Q network is trained, an actor can trained to exploit this frozen Q network with AWR. Sure, in practical implementations, one might be constantly training an actor against an also-being-trained Q network, but the actor has no effect on the Q network's training dynamics.

**Therefore, IQL cannot, by design, exhibit the "chain effect" because training the actor network is optional. ** You should acknowledge this in the relevant section.

IQL is a very nice guinea pig for your theory because it permits training without the funky actor-critic two-network dynamics. So in principle, you only have critic-churn to deal with.

The fact that you still get better performance in IQL despite that is quite interesting, and merits a discussion (and you need more space in the paper for that...). I am very curious to hear your thoughts on why this happens. You are still optimising L_QC with the modified IQL, correct? An ablation would be interesting; due of the lack of the "chain effect", I'd hypothesise that only L_QC should be contributing to the improved performance, not L_PC. You could try ablating L_QC and L_PC by setting their coefficients to zero in IQL training: i.e., do the VCR / PCR / DCR experiments. I would be curious to hear what happens, in your response (fine as a markdown table in an openreview comment, if you can't update the PDF).

Because you're using CORL, these ablations should be doable quite quickly.

I would be willing to raise my score upon receiving a through update!

We provide the additional results for IQL (sequential) over 12 seeds, i.e., actor-trained-against-final-frozen-critic, and CHAIN IQL (VCR, $\lambda_{\pi}=0, \lambda_Q=0.01$) over 6 seeds below, along with the results for IQL and IQL (PCR, $\lambda_{\pi}=1000, \lambda_Q=0$) which were reported in our submission.

In the following, we provide the implementation details and discuss the results. We are willing to hear more valuable comments from the reviewer and address any further questions.

“What happens if you train IQL to the very end (without training the actor), then do a fixed budget of actor-only training against the final critic (without L_PC), and compare the results?” (the results for actor-trained-against-final-frozen-critic)

As suggested by the reviewer, we slightly modified the training process of the CORL implementation of IQL to realize “actor-trained-against-final-frozen-critic”:

1. First train the value network and Q network for 1M steps.
2. Then train the policy network for 1M steps with the value network and Q network frozen.
3. We do not modify any other implementation detail and use the same hyperparameters.
4. We check the learning curves of the policy network and the final scores.

We call this variation IQL (sequential). The total number of gradient steps is the same as the default IQL implementation where the critic and actor are trained iteratively.

We report the means and standard errors of the final scores achieved by IQL (sequential) over 12 random seeds.

The results in terms of final score show that IQL (sequential) performs worse than IQL in 5 of 6 tasks. For the learning curves (which we are not able to present at this stage), we found the policy performance increases efficiently and does not change much after 3e4 - 1e5 steps (depending on different tasks) of the policy training, without a large variance or a collapse through training.

 

The difference between IQL (sequential) and IQL can be fully attributed to the difference in the training dynamics, mainly on the policy network (as the training of the policy network does not influence the value network training in IQL). We think it is somewhat tricky to explain the difference between IQL (sequential) and IQL. The difference in the training dynamics here is beyond the scope of the chain effect of churn studied in our work.

To provide some possible explanation, we guess that the difference in the training dynamics of the policy network of IQL results in a further difference in the policy output on out-of-sample states.

The policy outputs of these out-of-sample states are fully determined by generalization. Compared with IQL (sequential), the distribution of the advantages should cover a wider range of values, thus providing more diverse gradient directions for the training of the policy network. This could lead to better robustness on out-of-sample states.

Finally, we believe that a more systematic study for offline RL is needed by focusing on how the training on in-sample data affects the network output on out-of-sample data under different training schemes, which is beyond the scope of our work.

Discussion on “(only) applying VCR to IQL”

Technically, the value training in VCR includes the training of a Q network and the training of a V network. To apply VCR to IQL, we modify IQL by optimizing one additional Q-value churn reduction regularization term jointly with the original Q loss (without changing the training of the V network). We call this variation as CHAIN IQL (VCR).

The results for CHAIN IQL (VCR) with $\lambda_{Q}=0.01$ are reported in the table above.

We can observe that CHAIN IQL (VCR) outperforms IQL in 4 of 6 tasks, and outperforms CHAIN IQL (PCR) in 3 of 6 tasks. This demonstrates the effectiveness of reducing the value churn in IQL. This also indicates that even though the chain effect does not exist in IQL, the value churn and the policy churn in the value training or the policy training can still negatively affect the learning performance.

We appreciate the reviewer’s valuable review and the recognition of the versatility of the proposed method. Our method is simple but versatile to improve the performance of many learning algorithms. We show that the method works across continuous and discrete control tasks, online and offline settings. Moreover, in our additional results (Table 9 in the one-page pdf) we show that the method can greatly improve the scaling of RL algorithms. This scaling is also improved by an adaptive version of our method that reduces hyperparameter tuning.

The main concerns focus on the hyperparameter choice for different settings and CHAIN’s effect on exploration and optimality. Our response aims to clarify these aspects in detail.

Q1: The insight behind the chosen hyperparameters for different settings

How to pick or adjust the regularization coefficient is an unavoidable tricky problem for most regularization-based methods. In the context of our work, the difference in the scale of the quantities like Q value and policy objective across different settings leads to the use of difference coefficients. This is also a long-standing problem of DRL.

To relieve the pain of manually picking coefficients in different tasks and settings, we additionally propose a simple but effective method for automatic adjustment of $\lambda_{\pi}, \lambda_{Q}$ with a target relative loss scale denoted by $\alpha$. It is realized by maintaining the running means of the absolute policy ($| \bar L_{\pi}|$) or Q loss and the PCR ($| \bar L_{PC} |$) or VCR term, and dynamically computing the regularization coefficient, e.g., $\lambda_{\pi} = \frac{\alpha | \bar L_{\pi}| }{| \bar L_{PC} | }$. This method adjusts the coefficient dynamically to keep the consistent relative loss scale $\alpha$.

Our additional experiments in the one-page PDF show that this method matches or surpasses the results achieved by the manual choices of $\lambda_{\pi}, \lambda_{Q}$ for DDQN in five MinAtar task (Figure 16), for PPO in five MuJoCo tasks (Figure 17) and ten additional DMC task (Figure 19), and for TD3 in four MuJoCo tasks (Figure 17).

Q2: Whether the CHAIN would negatively influence the exploration in online RL and then hinder the asymptotic performance

In principle, reducing churn suppresses the correlation of network outputs among different data. This encourages the agent to keep the independence of the action on different states. Thus, we hypothesize that it could help to keep the diversity of action (in other words, prevent the collapse of action mode as often seen in the cases with severe learning instability for both RL[2] and LLM [3]) and positively influence exploration.

In our experiments, we can see that CHAIN improves the asymptotic performance in the tasks that require effective exploration even though the value or policy churn is significantly reduced. For MinAtar, the original MinAtar paper mentioned “Seaquest and Freeway present a greater exploration challenge than the other games” on page 5, and our results in Figure 4 show CHAIN improves the final performance of DDQN by clear margins in these two environments.

Moreover, we additionally provide the comparison between CHAIN PPO and PPO in two sparse-reward tasks of DeepMind Control Suite (DMC): ball_in_cup-catch-v0 and cartpole-swingup_sparse-v0. The results are shown in the first two subplots in Figure 19. We can see that CHAIN PPO achieves a higher asymptotic performance in ball_in_cup-catch-v0 and achieves almost the same performance in cartpole-swingup_sparse-v0.

Q3: DrM [2] encourages the network neurons to awaken, thus preventing the local optimal. Whether CHAIN would deteriorate the local optimality issue

DrM prevents the neurons from being dormant with an adaptive perturbation reset method. This addresses the loss of plasticity/learnability and thus alleviates the local optimality issue.

In principle, CHAIN has a positive effect in preventing the loss of plasticity. As in Equation 3, reducing churn encourages $k_{\theta}, k_{\phi}$ to 0. This prevents the empirical NTK matrix from being low-rank, which is shown to be a consistent indicator of plasticity loss by [1] (they also claimed empirical NTK is a more reliable indicator than dormant neuron ratio).

In our experiments, we can see that CHAIN improves the final scores in most cases across MinAtar, MuJoCo (especially with CHAIN PPO) and additional ten DMC tasks (in Figure 18 of the one-page pdf). Additionally, we provide more results in the one-page pdf to show that CHAIN improves the final performance both when running longer (Figure 17 Ant and Humanoid for 10e6 steps) or with larger networks (Table 9).

Q4: The discussion on AlgaeDice

We appreciate the reviewer for reminding us of this. AlgaeDICE aims to address the limitation of on-policy samples by re-expressing the on-policy max-return objective by an expectation over any arbitrary off-policy data distribution, which is followed by a solution in the form of dual optimization.

AlgaeDICE does not have a direct relation to churn or generalization of DRL agents, however, we believe it is orthogonal to CHAIN in mitigating training instability. We have integrated AlgaeDICE and more in our related work discussion.

Q5: The number of random seed

In the one-page pdf, we provide the results across 12 seeds for CHAIN PPO in Figure 17 and additional results for ten DeepMind Control Suite tasks with 12 seeds in Figure 19. The results demonstrate the effectiveness of CHAIN in improving the learning performance of PPO.

We found that 6 seeds are sufficient to present a clear evaluation for DDQN in MinAtar. We will run more seeds for TD3/SAC as suggested.

---

Reference:

[1] Disentangling the causes of plasticity loss in neural networks. 2024.

[2] Overcoming Policy Collapse in Deep Reinforcement Learning. 2023.

[3] Controlling Large Language Model Agents with Entropic Activation Steering. 2024.

We appreciate the reviewer for the interest in discussing and investigating the potential effect of CHAIN in exploration, which also inspires us of further research on this point in the future.

 

The exploration ability/behavior is determined by two factors: (1) the extrinsic factor, i.e., the exploration mechanism used, (2) the intrinsic factor, i.e., the independence or diversity of the policy (or Q-value) network output.

For the extrinsic factor, CHAIN does not suppress the extrinsic exploration mechanisms used in DRL algorithms, e.g., epsilon-greedy in DQN, Gaussian noise in TD3, state-independent variance parameter vector in PPO, and the state-dependent variance parameter vector in SAC.

For the intrinsic factor, in our rebuttal text, we provided a discussion about how CHAIN helps to prevent action correlation (more severely, collapse) and could encourage action independence. Indeed, in the extreme case where the churn reduction regularization term dominates the joint learning objective, the agent will be prevented from learning any effective behavior. Our newly proposed method for keeping a consistent relative loss scale should help to avoid this kind of extreme case.

 

However, the best that CHAIN can do is to prevent the over-correlation of action and the loss of independence. It does not introduce any extra heuristics or principles to encourage exploration (e.g., novelty, curiosity, uncertainty), which we think is necessary for learning effective exploration behaviors in challenging exploration problems.

Therefore, we do not think CHAIN has the ability to address challenging exploration problems by itself. It could be interesting to study the effect of CHAIN when applied together with existing exploration methods to DRL agents in the future.

 

Finally, we would like to emphasize that we do not propose CHAIN as an exploration method. It is a method to reduce churn and address the learning issues (as presented in Section 4.1) for better performance of DRL algorithms.

It is evident that hyperparameters play a crucial role; manual selection often yields suboptimal performance across various tasks. On the other hand, employing a smart mechanism typically enhances performance significantly. In light of this, I am curious why the smart mechanism is not included in the primary implementation. Would such an approach impact the theoretical analysis differently?

Thanks to the reviewer’s valuable review, the newly proposed method for automatic adjustment of the regularization coefficient is done during the rebuttal stage. We will add this method/mechanism in the main body of the revised paper.

Across all our experiments, we found that manually selecting a regularization coefficient that makes the target relative loss scale less than 0.01 rarely harms the learning performance of the baseline algorithms. The suboptimal performance due to over-regularization is almost only observed when the scale regularization term matches or surpasses the original learning objectives.

This method adjusts the regularization coefficient dynamically through learning to keep a consistent relative loss scale. It follows the same insight into churn reduction presented by our analysis of the chain effect of churn. We consider that this method does not impact our formal analysis.

Figure 17 highlights different churn reduction options, where the efficacy of DCR and PCR varies across tasks. How should one decide between these options, and what strategies can mitigate the need for task-specific tuning?

For applying CHAIN to deep AC methods, we recommend the practitioner to use PCR. This is because policy interacts with the environment directly and the target critic network also helps to alleviate the value churn in the learning of the critic network in practice. Empirically, we also found PCR often contributes more than VCR.

As shown by our additional results in the one-page pdf, the need for task-specific selection of the regularization coefficient is addressed by our auto-adjustment method. Effective adjustment is achieved with a common target relative loss scale for all the tasks within the same domain.

More broadly, we have three pieces of advice for the community audiences to adapt CHAIN to different problems in practice:

• Use our “target relative loss scale” insight to do automatic adjustment of the regularization coefficient.
• Start from a small target relative loss scale (e.g., 1e-5). It should be safe in the sense that the performance will not be harmed.
• Use techniques to normalize the scale of quantities (e.g., reward, advantage, Q). This is also one recipe of the DreamerV3 [Hafner et al., 2023].

We sincerely appreciate the reviewer's valuable comments and further feedback during the discussion stage. As the end of the discussion period is approaching, we would greatly appreciate it if the reviewer could confirm that our response has addressed your concerns.

 

For a brief summary, we provided detailed discussions about how we leverage the insight of relative loss scale to further develop a simple but effective method for adapting the regularization coefficient throughout training. With our additional results in the one-page pdf, we showcased that this method greatly helps to relieve the need for manual selection of the regularization coefficient in different tasks and domains, while achieving and even surpassing the performance of manual coefficients. To apply our churn reduction method to a different problem, we provided our thoughts and several pieces of practical advice.

We also discussed the effect of CHAIN on exploration and plasticity in detail. To provide more empirical evidence for this, we provided two more sparse-reward DMC tasks (in addition to the two exploration tasks in MinAtar) to show the effectiveness of CHIAN in these tasks, although our work focuses on how churn occurs and affects DRL and does not aim to address challenging exploration problems.

Moreover, to further strengthen our experimental evaluation and demonstrate the significance of CHAIN, we provided additional experimental results in the one-page pdf material with ten new tasks, more seeds, and an extra learning setting on DRL scaling.

 

We believe that these additions and modifications help to address all of the concerns raised in the review, but please let us know if there are any further issues to address.

We greatly appreciate the time and effort the reviewer devoted to reviewing our work and actively participating in the discussion. The reviewer's valuable comments drove us to propose a method for dynamic adjustment of the regularization coefficient, and inspired us of more thinking on the effect of churn reduction in exploration, plasticity, etc.

"I am particularly interested in understanding which factors contribute most significantly to achieving high performance."

As we considered different learning settings in Section 4.1, we would like to note that Value Churn Reduction (VCR) and Policy Churn Reduction (PCR) are not always used together in the different learning settings considered in our experiments.

To be specific, only VCR is used for CHAIN DDQN (mentioned in Line 281) because there is no policy/actor network other than the Q-network (and the policy is implicitly derived by performing the greedy actions based on the Q-network). For the PPO case, only PCR is used for CHAIN PPO (mentioned in Line 302) because the main role played by the V-network is a baseline to subtract (rather than a real critic, according to the opinion in Chapter 13.5 of [Sutton and Barto, 2018]). Therefore, in these cases, only one factor is added to the baseline algorithms and the difference in the learning performance can be fully credited to the corresponding factor.

For deep AC methods, we agree that a better way to select DCR/VCR/PCR is needed. In this work, we provided the results of DCR/VCR/PCR for TD3 and SAC. We have clearly stated this point and updated the limitation section to include the main messages of our discussions as suggested.

Finally, please let us know if we misunderstood the meaning of the “factors” in your comments.