Decoupled Actor-Critic

We thank the Reviewer for their time writing the review. We are happy that the Reviewer finds the proposed approach “well-motivated” and “solid”, the performance gains “significant” and the experiments “extensive”. We would like to point the Reviewers attention to new experimental results described in our meta-comment. The new results further indicate that DAC achieves very strong performance on locomotion tasks, significantly outperforming both model-based (TD-MPC) and model-free (SR-SAC) SOTA on the dog domain. Below, we respond to each question/weakness listed by the Reviewer.

The paper does not compare against the ensemble-based methods [e.g., REDQ], which I believe is relevant and necessary.

We kindly ask the Reviewer to investigate Figures 1, 9 & 10 of the original manuscripts to find the evaluation of DAC against exactly REDQ. As follows, REDQ achieves significantly worse results than other considered algorithms in the high replay benchmark. This result is consistent with [1], where SR-SAC (high replay SAC with resets) massively outperforms REDQ across various replay regimes. For completeness, we included both REDQ and SR-SAC in the evaluation presented in Figure 1, but dropped it in most of the latter evaluations because of substantial performance differences between REDQ and other algorithms.

On the general applicability of the idea, will this idea work with different value-to-policy generation rules?

Our approach is trivially applicable to other value-based continuous actions algorithms like DDPG, TD3 or Dreamer. It would require some accommodations to be applied to discrete actions algorithms like Rainbow or BBF. We are happy to expand on the answer if the Reviewer shares which particular algorithm the Reviewer is interested in.

There is no text pointing to Figure 1 and Figure 2. Also, many of the abbreviations are used since the beginning of the paper but are introduced at very late stages. The overall presentation of the paper can be improved.

We thank the Reviewer for this suggestion! We added text in the main body that describes why Figures 1 & 2 are important for our narrative. We make some additional changes to make sure that abbreviations are explained earlier in the text.

The authors do not disclose any pitfalls of the DAC algorithm.

We ask the Reviewer to note that our original manuscript contains a “Limitations” section, where we describe pitfalls of our proposed DAC algorithm. Those limitations include:

1. Complexities of the tandem setting - “DAC divergence minimization presents unique optimization challenges. Unlike typical uses of KL divergence, where the target distribution remains fixed (eg. Variational Autoencoders (VAE)), DAC deals with a constantly evolving policy that is continually improving. Consequently, the optimistic actor needs to keep up with the conservative actor’s changes. As depicted in Figure 4, DAC heavily relies on maintaining a low divergence between the actors. While DAC adjustment mechanisms proved effective in the tested environments, there is no guarantee that they will suffice in more complex ones.”
2. Optimistic actor architecture - “The second drawback of DAC lies in its inherent use of two actor networks, which results in slightly increased memory and computational demands compared to the standard Soft Actor-Critic (SAC) approach. In practice, the wall-clock time of DAC is around 10% greater than that of SAC and is indistinguishable from the overhead induced by OAC, which requires additional backpropagation through the critic ensemble. Moreover, since DAC initializes both actors with identical parameters, they must share the same network architecture. However, as indicated by Figure 4, simply copying parameters between them offers only minimal performance enhancement. In light of this, we believe that the necessity for identical architectures can be mitigated by employing techniques like delayed policy updates or by learning rate scheduling.”

As DAC builds on SAC, we focus on limitations stemming from components specific to DAC. We add a line in the limitations section mentioning that DAC inherits limitations of SAC.

We thank the Reviewer for their time writing the review. We are happy that the Reviewer finds our approach “well-motivated” and appreciated the proposed adaptive mechanisms that address OAC shortcomings. We would like to point the Reviewer’s attention to the new results added to our experiments (described in our meta-comment). In those results, DAC appears to significantly outperform model-based DMC SOTA (TD-MPC) and to the best of our knowledge represents the best recorded sample efficiency on the dog/humanoid domains of a model-free RL algorithm . Please find our responses to the Reviewer’s questions below.

It is unclear how does Figure 2b and 2c shows the critic disagreement for different actors?

Figure 2b & 2c show conservative and optimistic policies and related sampled state-action samples (denoted by different dot colors - red for optimistic and black for conservative). As follows, the policies are not massively different which is due to a small KL divergence between the two policies which is enforced by DAC. Despite similarity of the policies, the optimistic policy allocates some of the allowed KL budget to achieve bigger variance than the baseline conservative policy (indicated by the smaller likelihoods of the optimistic policy); secondly, the optimistic policy leads the agent to the corners of the state-action space which as shown in Figure 2a yields most critic disagreement. If the Reviewer wishes, we are happy to create a new graph where the optimistic policy is less penalized for the divergence and should thus lead to more contrasting policies.

If it is possible to use non-linear approximation of Q-value in Eqn. (6)

Eqn. 6 describes the optimistic policy rule used in OAC (not in our proposed DAC), which as described by the authors of OAC used a linear approximation. To the best of our knowledge, some of the closed-form solutions used in OAC require the model to be linear. In fact, one of the key novelties of DAC that contrast to previous work is that it uses a non-linear model of Q-value upper-bound.

What is the impact of the ensemble size as in Eqn. (4)

With more than two critics, the minimum of the ensemble stops being equivalent to Eqn. 5 with $\beta^{lb} = -1$. However, it can be easily calculated what $\beta^{lb}$ corresponds to a clipped double Q-learning rule with more than two critics (this is discussed in [1]). We believe that extending DAC to a bigger ensemble is an interesting direction. Approaches like REDQ take a subset of the ensemble for every TD calculation and we think that this is a design choice that would have to be thoroughly ablated in the context of DAC. As such, we believe that it mandates an independent research endeavor.

Some notations can be confusing, i.e., |A| is used to denote the action dimensionality where normally it is used as the cardinality of the action space.

We used the notation inherited from the SAC manuscript, which uses |A| to denote control dimensions. We though that such notation is fitting, as we consider control over continuous space. We are happy to change the notation according to the Reviewer's proposition.

We hope that the above responses anwer the Reviewers' doubts about our proposed method. Furthermore, we hope that the new experimental results presenting DAC performance on complex DMC environments, as well as changes done to the manuscript increase the Reviewers confidence in DAC. If so, we kindly ask the Reviewer to consider adjusting the score of our work.

[1] - Ciosek, Kamil, et al. "Better exploration with optimistic actor critic." Advances in Neural Information Processing Systems. 2019.

We thank the Reviewer for a thorough and insightful review. We are excited that the Reviewer finds our approach “neat”, the paper “well written” and the results “promising”. We extended our experiments by a setting suggested by the Reviewer - 3mln steps on humanoid and dog tasks. Furthermore, we added an experiment verifying our claim regarding overestimation and plan on number of changes in the manuscript. We are excited to share that the new results further indicate that DAC is a substantial improvement over the existing model-free approaches. We described the new additions in detail in our meta-response. Below, we directly respond to the Reviewers’ questions.

Dreamer results in Figure 11 are looking good, while harder tasks such as Quadruped Run and Humanoid Walk are missing – Dreamer-v2 is able to solve these tasks well even for visual-control, why not compare on tasks like Quadruped/Humanoid/etc.?

We used the results provided by the authors of DreamerV3 in their official repository. As such, we restricted evaluation to the 18 DMC tasks considered in DreamerV3 prio evaluation. DreamerV3 has pretty substantial compute requirements as compared to model-free algorithms. To that end, we are unsure if the potential small changes to the evaluation (there are already 18 tasks in the benchmark) mandate the cost of running Dreamer on additional environments.

(or even extend to visual control)?

We agree that prio-based control is not where Dreamer shines. However, we also think that using DAC in visual tasks would require non-trivial architectural choices. Such non-trivial choices stem from the intersection of using two actor networks, the common practice in visual control to use a shared encoder for actor-critic structure, and the recently researched impact of gradient weighting between actor-critic in visual tasks [1]. As such, we would have to ablate on various design choices regarding architecture and gradient passing in our extended actor-critic context. As such, we think that extending DAC to visual control validates a separate research endeavor. We kindly ask the Reviewer to note that it is a common practice to evaluate a continuous-control RL algorithm on a prio-based benchmark alone [2, 3, 4].

More complex Control Suite tasks (Figures 12 & 13) like Humanoid Walk/Run should be run for longer as they have not converged yet, while recent papers have also evaluated on the Dog domain

DMC 500k/1mln has become a popular evaluation protocol used in recent works [1,5,6]. We think that running many tasks of varying difficulty for a consistent amount of timesteps makes sense from the aggregate evaluation perspective. However, we also agree that harder, unsolved tasks are more interesting than well researched propositions like cheetah. To this end, we added a new experiment: dog {walk, run, trot} and humanoid {walk, run} on 3mln environment steps - following the Reviewer’s expectations. We describe the experiment in our meta-response. There, we compare SR-SAC, DAC and a recent baseline pointed by the Reviewer (TD-MPC). Please find a summary table below:

In particular, SR-SAC does not seem to work on dog domain. This result is consistent with the evaluation performed in the TD-MPC manuscript.

SAC is in general a good baseline, however, it would be nice to also compare performance to a more “DMC-native” baseline such as D4PG or (D)MPO to provide another reference point

We are happy to share that we added D4PG and MPO to the Dreamer evaluation. The results are consistent with the Dreamer manuscript (ie. DAC > DreamerV3 > D4PG > MPO)

There are quite a few missing articles / words + typos that should be fixed

We made a sweep through the manuscript, we will make sure that the camera-ready version is fixed!

It would be good to extend the discussion to model-based exploration agents (...)

Thank you for pointing us to those works! We have added a paragraph to the Related Work section detailing how DAC related to this literature.

The algorithm introduces many more hyper parameters comparing with commonly seen SAC or TD3 due to the added components in the losses, I would not consider the comparison to be fair with other baselines, unless evidence of similar efforts have been made to thoroughly sweep baseline’s hyper-parameters is provided.

We would like to respond to the above assertion in three points:

1. SAC/TD3 are arguably the most popular RL algorithms at the time of writing this response. Both were evaluated numerous times on DMC locomotion, which is one of the most popular RL continuous-control benchmarks. As such, we believe that the hyperparameter settings used for SAC/TD3 were arguably more “sweeped” than those of DAC. The hyperparameter choices used for our implementations of SAC and SR-SAC are used in the current DMC locomotion SOTA (SR-SAC) [9].

2. We kindly ask the Reviewer to note that reusing hyperparameters proposed in the earlier works is a common practice, especially in the case of such well-known benchmarks as DMC/gym [1,2,4,5,6,7,8,9,10,11].

3. Finally, we would like to point towards experiments presented in Figure 5 of our original manuscript. There, we evaluate 14 hyperparameter configurations of DAC (our entire sweep), as well as the reference SAC performance. According to the results, all 14/14 DAC configurations significantly outperform SAC.

Considering the above arguments, we kindly ask the Reviewer to reconsider his “unfair comparison” assertion.

Eq 7, how do you calculate the gradient w.r.t optimistic actor parameters, it appears the gradient should be also propagated through/to the first two Q functions and these Qs are interdependent with actor.

The Eq 7 is implemented the same way a standard backprob through critic is implemented in DDPG, D4PG, SAC, TD3, OAC, TOP etc. It is not required to update the critics parameters as a result of this update, we can just use the gradient calculation to propagate it further to the actor. If the Reviewer wishes, we can further expand on how it is usually implemented.

The motivation of the paper is to avoid overestimation while keep good exploration. Isn’t it quite intuitive to combine some exploration method with methods that mitigate overestimation? it shouldn’t be difficult to design such a baseline as mitigating overestimation typically require multiple critics and uncertainty estimate could be derived from the ensemble for exploration purpose. What is the proposed algorithm’s advantage?

We evaluate against OAC, TOP, SAC, REDQ and TD3 which all “combine some exploration method with methods that mitigate overestimation” and all use multiple critics to derive an uncertainty estimate “for exploration purpose”. Furthermore, in Figure 4 we consider 10 DAC simplifications, which all as well fit the Reviewer’s description. We discuss the advantage of our approach in Section 3.

it shouldn’t be difficult to design such a baseline

We design and evaluate a variety of baselines common to DAC. The results are presented in Figures 4 & 6. We are happy to add more baseline designs to our evaluations, if the Reviewer shares more details about the Reviewer’s envisioned baseline.

The experimental design of studying replay ratio appears to disconnect with the primary motivation of this paper; this should be put in the appendix.

We argument why we study two replay ratio regimes in Section 1 & 4. In our view it is an important aspect in the light of modern RL studies, which was repeatedly shown to play a substantial role in the performance of RL algorithms. Also, as Reviewer XQ6P wanted us to expand the discussion of replay ratios, we would like to keep the analysis in the main body of the paper.

We thank the Reviewer for their time and valuable input. In the light of new experimental results, we would like to ask the Reviewer to reconsider his score of the manuscript.

We thank the Reviewer for their time. We are happy that the Reviewer finds our algorithm presentation clear, the considered problem interesting and that the Reviewer appreciates the amount of experiments performed within our work. We particularly thank the Reviewer for the suggestion regarding the overestimation experiment. Furthermore, we perform additional experiments on the dog domain. There, we record that DAC achieves significantly better performance than model-based and model-free DMC SOTA. We describe the experiments in our meta-response. Below, we respond in detail to each of the Reviewer’s questions.

The paper claims that there seems to be a conflicting demand in actor-critic architecture

This conflict is not something that we discuss first - it has been thoroughly discussed in [1, 2]. We hope that our writing does not imply otherwise.

The proposed algorithm is mostly designed by heuristics and the implementation details are not theoretically justified.

We would like to note that DAC follows fundamental concepts in RL theory, such as the actor-critic framework (stemming from the Policy Gradient Theorem), optimism in the face of uncertainty and off-policy learning. Its design incorporates widely researched principles of policy and value iteration in the context of maximum entropy formulation, which all are foundational for the field of RL. Even the automatic optimization of optimism and KL regularization can easily be linked to dual optimization [3], used in similar contexts in many other RL algorithms [4, 5]. However, we realize that our manuscript focuses on evaluating implementation of theoretically founded principles in practice, and as such could be considered an empirical work.

I expect empirical evidence to verify the critical claims/algorithmic designs of this paper

Figures 4 & 5 extensively evaluate the performance impact of many design decisions (10 DAC simplifications & 14 hyperparameter configurations). We note that 7/10 DAC simplifications and 14/14 hyperparameter configurations perform better than baseline SAC.

Furthermore, since the proposed algorithm is basically a synthesis of different intuitive designs, a discussion of where the algorithm would converge to should be provided.

In Section 3.2 we write: “This mechanism allows for separate entropy for TD learning and exploration while retaining standard convergence guarantees of AC algorithms. In fact, (...) it follows that in the limit both actors recover a policy that differs only by the level of entropy”. Furthermore, since SAC is an off-policy algorithm and DAC differs only by samples that land in the experience buffer, we believe it trivially follows that DAC retains convergence guarantees of SAC.

Currently the paper is written in a way that different updating rules are introduced; I expect to see a clear objective function (maybe with constraints) of Algorithm 1, so readers can easily see what it is optimizing, can the authors write it down in the rebuttal?

The current version of the paper features a pseudo-code with links to every equation used in the calculation. Furthermore, Sections 3 & A2 detail what and why DAC is optimizing. If the Reviewer points us to what particularly the Reviewer finds confusing, we are happy to adjust the text.

When simultaneously maximizing both lower and upper bound of the Q values, would it squash all action values higher and still result in overestimation?

We think that there is a misunderstanding. Overestimation is a property of the critic and is believed to stem from TD learning [2,5,6]. DAC features no “simultaneous maximization of lower and upper bound of the Q-values” and the critic used in DAC learns using a traditional clipped-double Q-learning [4,6]. The lower and upper bounds policies are optimized by conservative and optimistic actors respectively (not ‘simultaneously’), and the optimistic actor does not interact with TD learning in DAC design. As such, there is no reason for overestimation in DAC beyond overestimation that occurs in SAC. This is underlined in Section 1 of our manuscript: “DAC employs two actors, each independently optimized using gradient backpropagation with different objectives. The optimistic actor is trained to maximize an optimistic Q-value upper-bound while adjusting optimism levels automatically. This actor is responsible for exploration (sampling transitions added to the experience buffer). In contrast, the conservative actor is trained using standard lower-bound soft policy learning and is used for sampling temporal-difference (TD) targets and evaluation.”

We thank the Reviewers again for their time and suggestions on how to improve the quality of our manuscript. Following our commitment, we uploaded a revised version of the manuscript. In the revised version, we colour major changes red for an easy read. We summarize the implemented changes below.

ADDITIONAL EXPERIMENTS - we add a paragraph describing the experiments on the dog domain, as well as extended runs on the humanoid domain. We add the two evaluations on this domain:

1. 1mln environment steps - where we run the algorithms in a setting consistent with the high replay ratio regime
2. 3mln environment steps - where we make adjustments to DAC such that it is comparable to the model-based approach that yields state-of-the-art performance [1]

We detail results for the hard DMC tasks suggested by the Reviewer XQ6P. To the best of our knowledge, DAC achieves the highest recorded sample efficiency and final performance achieved by an RL algorithm on the “Run” tasks of dog and humanoid domains. The results are presented in Section 4 and Appendix D.3 of the revised manuscript. We summarize the results in the table below.

OVERESTIMATION TESTS - we added an Appendix section describing the overestimation experiments suggested by the Reviewer SCgY. There we evaluate Q-value overestimation implied by a critic updated in three regimes:

1. Conservative - exploration and TD learning are both conservative
2. Optimistic - exploration and TD learning are both optimistic
3. Decoupled - optimistic exploration and conservative TD learning

We find that the decoupled architecture yields the least overestimation from the compared methods. We hypothesize that the slightly lower overestimation than the conservative architecture stems from better state-action coverage of the decoupled architecture which we discuss in Section 3. We describe the results in Appendix D.1 of the revised manuscript.

NEW BASELINES - we added new baseline algorithms suggested by the Reviewer XQ6P: MPO; D4PG and TD-MPC. The comparison is presented in Figure 1 and Appendix H of the revised manuscript.

OTHER MINOR CHANGES - we expanded related work; added new information to the hyperparameters and experiments descriptions; added information on the newly added baseline algorithms; added a section on replay ratio and RL; added some minor changes, fixed some typos and missing words.

We hope that the newly added content reinforces the confidence in our proposed method and answers the weaknesses discussed by the Reviewers. If any of the Reviewer has any further suggestions, we are happy to implement them.

[1] Hansen, N., X. Wang, and H. Su. "Temporal Difference Learning for Model Predictive Control." International Conference on Machine Learning, PMLR. 2022