Rethinking Actor-Critic: Successive Actors for Critic Maximization

We thank you for your constructive feedback. We appreciate your positive comments on the approach and experimental performance. We address your concerns below:

FiLM, DeepSet Design Choices

• In Appendix C.3 and C.4, we added the descriptions of Deepset along with other design choices of list-summarizers as well as Feature-wise Linear Modulation (FiLM). In short, Deepset effectively aggregates the information in the list input while FiLM helps to condition on preceding list-input. Those modules contribute to better training the list of actor-critics in SAVO.
• We also added ablations in Appendix.D.2 & D.3 to compare (i) FiLM v/s no-FiLM and (ii) the choice of DeepSet v/s LSTM v/s Transformers to aggregate the information from previous selected actions. The results validated our choice of DeepSet as it is more scalable to an increasing number of actor-critic pairs. Also, we acknowledge that this design choice depends on the environment and the use-case as some tasks in continuous domains show that LSTM / Transformers outperform Deepset.

Visual insights on central claim

[New Figure.3] The optimization landscape of Q-value concretely represents the landscape where the N-D action space represents the independent variable and the 1-D Q-value represents the dependent variable. For visualization, we map the action space to 2-D when N > 1. Fig.3 demonstrates that the Q-function becomes more easy to globally optmize with every iteration of our successive actor-critics. This is because the locally optimal peaks below past actions’ Q-values are pruned away from the optimization landscape. As we can observe: The number of local optima are reduced in $Q_0 \to Q_1 \to Q_2$, as most action values are shifted up in the Q-value space. This enables successive actors $\pi_0 \to \pi_1 \to \pi_2$ to focus on more optimal regions of the true Q-value space. The actions $a_1$ and $a_2$ found by $\pi_1$ and $\pi_2$ are indeed more optimal than $\pi_0$ whose actions were only locally optimal peaks, but globally suboptimal

Clarifications of Baseline Selection

We choose TD3 because of its unified applicability to both action-representation-based discrete action spaces and continuous action space. Particularly, the Wolpertinger algorithm [Dulac et al 2015] is based on the DDPG algorithm and is the most commonly used base algorithm for large discrete action spaces. While we agree with the reviewer’s suggestion that combining SAVO with SAC would be good to show wider applicability of our method, we think this extension might offer more flexibility in the design choices. Thus, for this project, we decided to stick to the simple architecture of TD3 as described above.

Notational fixes

We have fixed all the suggested typos, missing legends, and source code naming in our revision. Thank you for your detailed suggestions.

Please do let us know if you have any leftover concerns or questions.

Technical explanation and analysis of FiLM, DeepSet

• In Appendix C.3 and C.4, we added the descriptions of Deepset along with other design choices of list-summarizers as well as Feature-wise Linear Modulation (FiLM). In short, Deepset effectively aggregates the information in the list input while FiLM helps to condition on preceding list-input. Those modules contribute to better training the list of actor-critics in SAVO.
• We also added ablations in Appendix.D.2 & D.3 to compare (i) FiLM v/s no-FiLM and (ii) the choice of DeepSet v/s LSTM v/s Transformers to aggregate the information from previous selected actions. The results validated our choice of DeepSet as it is more scalable to an increasing number of actor-critic pairs. Also, we acknowledge that this design choice depends on the environment and the use-case as some tasks in continuous domains show that LSTM / Transformers outperform Deepset.

Clarifications

Baseline Selection of TD3 We choose TD3 because of its unified applicability to both action-representation-based discrete action spaces and continuous action space. Particularly, the Wolpertinger algorithm [Dulac et al 2015] is based on the DDPG algorithm and is the most commonly used base algorithm for large discrete action spaces. While we agree with the reviewer’s suggestion that combining SAVO with SAC would be good to show wider applicability of our method, we think this extension might offer more flexibility in the design choices. Thus, for this project, we decided to stick to the simple architecture of TD3 as described above.

How to find the optimal action a' based on the primary critic We improved the clarity of writing in the approach section and added the following qualitative result to support our claim; [New Figure.3] The optimization landscape of Q-value concretely represents the landscape where the N-D action space represents the independent variable and the 1-D Q-value represents the dependent variable. For visualization, we map the action space to 2-D when N > 1. Fig.3 demonstrates that the Q-function becomes more easy to globally optmize with every iteration of our successive actor-critics. This is because the locally optimal peaks below past actions’ Q-values are pruned away from the optimization landscape. As we can observe: The number of local optima are reduced in $Q_0 \to Q_1 \to Q_2$, as most action values are shifted up in the Q-value space. This enables successive actors $\pi_0 \to \pi_1 \to \pi_2$ to focus on more optimal regions of the true Q-value space. The actions $a_1$ and $a_2$ found by $\pi_1$ and $\pi_2$ are indeed more optimal than $\pi_0$ whose actions were only locally optimal peaks, but globally suboptimal.

Language clarifications As per the reviewer’s suggestion, we have improved the language-ambiguous stataments: "navigate the action-value landscape more proficiently" → “globally optimize the Q-value landscape” An actor is trained with gradient ascent over the Q-value landscape and can get stuck in local optima, which we resolve. "distribute the optimization load over to the critic" → “utilize the critic to evaluate multiple actions”. "This can make the learning procedure even slower than the original inefficiencies of the actor" → “A larger action space in the joint-actor-critic model present another optimization challenge, making this solution infeasible”.

We express gratitude for your valuable feedback and constructive input. Your positive remarks regarding the approach and experimental performance are acknowledged and appreciated. Below, we provide responses to the concerns you raised.

New Ensemble Baseline

Novelty over ensemble methods: Ours is not a typical ensemble method, where usually the constituents of the ensemble are symmetric. In fact, ours is a sequential ensemble, where successive actor-critic pairs optimize different optimization problems, building up on the past actor-critic pairs. Our added visual result in Fig.3 shows how this successive optimization eases the Q-function optimization landscape, which indeed helps the successive actors to find more globally optimal actions than before, with respect to the current Q-function

Visual demonstration of improving optimization landscape

• [New Figure.3] The optimization landscape of Q-value concretely represents the landscape where the N-D action space represents the independent variable and the 1-D Q-value represents the dependent variable. For visualization, we map the action space to 2-D when N > 1. Fig .3 demonstrates that the Q-function becomes more easy to globally optmize with every iteration of our successive actor-critics. This is because the locally optimal peaks below past actions’ Q-values are pruned away from the optimization landscape. As we can observe:
• The number of local optima are reduced in $Q_0 \to Q_1 \to Q_2$, as most action values are shifted up in the Q-value space. This enables successive actors $\pi_0 \to \pi_1 \to \pi_2$ to focus on more optimal regions of the true Q-value space. The actions $a_1$ and $a_2$ found by $\pi_1$ and $\pi_2$ are indeed more optimal than $\pi_0$ whose actions were only locally optimal peaks, but globally suboptimal.

Learning stability

Action selection decision This is true for all actor-critic methods, especially those learned with deep learning. However, the key goal of any RL agent is to take actions that maximize its expected return in the environment, which is why actor-critic algorithms learn actors that can find actions to maximize the Q-values. This does not create any bias in learning of the actors, because their true objective is to maximize the critic. While the Q-function itself can be biased during deep RL training, it is still the best estimate of the objective for the actor. Our method does not degrade the primary critic’s function which is already based on stabilizing tricks adopted from TD3, like twin critics, delayed updates, and soft-action evaluation. Thus, stability of training of the primary critic is not negatively affected at all by our method. And the actor learned by our method is guaranteed to be better than a single actor, because of the max operator in Line 13 of Algorithm 1, which ensures the action is always at least as good as the one suggested by a single actor. Please also refer to combined response for more details.

Exploration

• Exploration for each successive actor $\pi_i$ still follows the standard OU Noise based exploration from DDPG, TD3. The successive actors provide an action that has a higher Q-value than a single actor would have provided, but the exploration term is still added over the actions while acting in the environment. We have modified the Algorithm 1 to clarify this important detail.
• Furthermore, the sample efficiency improvements across 14 tasks (Fig.4 using RLiable as AC suggested) empirically justifies that our method never hurts exploration in the environment.

Validity of policy gradient objective

Each successive actor-critic pair solves its own deterministic policy gradient objective, where only the effective state space is modified. Conventional actor-critics take the form $\pi(s, a)$ and $Q(s, a)$. Whereas, each of our successive actor-critics take the form $\pi_i(\{s, a_{0:i-1}\}, a)$ and $Q_i(\{s, a_{0:i-1}\}, a, a)$. Thus, only the effective state space of the MDP is modified, and we can directly apply any actor-critic algorithm to train these actor-critics (we use TD3).

Please do let us know if you have any leftover concerns or questions.

We thank you for your constructive feedback that helped improve our paper significantly. We address your concerns below:

Technical explanation and analysis of FiLM, DeepSet

Please refer to the combined response

Added latest baselines

Thanks for your suggestion. Please refer to the combined response. We have added 4 new baselines: CEM, Greedy-AC, Greedy-AC-TD3, Ensemble.

Added more experimental benchmarks

“The experimental results are not sufficiently reliable.” "The experiment in the Appendix only has the Easy environments, what about the other Hard environments?”

[Figure 2] We added more continuous control environments in the paper, and report results across 11 environments. [Figure 4(c)] We also report the results using the RLiable library (also suggested by AC) to demonstrate that our method outperforms the baselines and ablations reliably across the environments.

Computational Efficiency

“I am concerned about the computational efficiency of the algorithm. Please supplement the experiments or provide an analysis.” [Figure 4 (d)] Thank you for this suggestion. We have added a complexity v/s performance analysis that shows the computational requirement increase is negligible. Please refer to the combined response.

Supplemented discussion in Related Work

[Section 2] Added discussion on evolutionary methods and ensemble methods, and also restructured the related work section significantly.

Clarifications

“There is no mention of hyperparameter sensitivity or setting experiments.”

• Appendix E.2 and E.3 already provides a discussion of hyperparameter sensitivity and setting details across all baselines.
• To reiterate, the sensitive hyperparameters were found to be learning rates and network sizes of actors and critics, and were searched for each baseline.
• [Figure 4 (c)] A key hyperparameter of our approach is $K$, the number of actor-critic pairs. Figure.4 shows the improvement in performance as $K$ increases in continuous control tasks. However, we chose $K$=3 across environments as that is enough to show significant gain with our method.
• [Appendix D2, D3] We also add experiments to show different design choices are viable: FiLM v/s No-FiLM for conditioning, and DeepSet v/s LSTM v/s Transformer for past action summarization.

“In Algorithm 1, the "state s" in lines 8 and 10 need clarification.”

• We correct the typo in lines 8-10, as $s$ → $s_t$. $s_t$ is the state observed in Line 5.

"Does the proposed method in this paper suffer from the problem of action values overestimation?”

• Since our algorithm is based on TD3, the problem of overestimation is reduced by the presence of twin critics and delayed updates. We clarify this in Algorithm 1 to alleviate the concern of overstimation. To further justify, in the Q-learning update of primary critic in Line 19 of Algorithm 1, our method only modifies the search of the Q-value maximizing action. This is a key requirement of the DPG algorithm and the training objective of DPG, DDPG, TD3, etc. Works like DDPG and TD3 alleviate the issues of overestimation of Q-functions, and our implementation being based on TD3, also benefits from those solutions. All the baselines and ablations also equivalently benefit from this, and do not suffer from overestimation any more than TD3.

“Why only select 3 seeds in some experiments, such as Figure 4?”

• All analysis results have been updated with 5 seeds. No trend is affected.

“Is the modification of the experimental setup fair? Other baselines may not have been designed specifically to address the problem presented in this paper.

• To exemplify the problem of challenging Q-function landscapes, we curate hard benchmarks of continuous control tasks, but we also reported results on the original easy benchmarks with relatively simple optimization landscapes, in Figure 2. SAVO outperforms baselines in both the easy and hard versions of the tasks, but more significantly in the hard versions. This shows that our algorithm’s contribution is more robust to challenging optimization landscapes, that would be more common as RL scales up to harder problems. This is already evident in our results in the discrete action space environments, where the optimization landscape is already challenging because of the presence of only few valid points in the action representation space, that correspond to actual discrete actions.

• The baselines from various prior works (Figure 2) are indeed incapable of addressing the problems of Q-function optimization presented in this paper. That is the key claim that we are making. All the implementations of the baselines are fair and based on the same code skeleton (code is attached in supplementary), and we clearly demonstrate that it is because of the better optimization of the Q-function by SAVO’s successive actors than baseline actors like single actors, Sampling-augmented actors, CEM Actors, and Ensemble of Actors.

We thank you for your constructive feedback that helped improve our paper significantly. We address your concerns below:

Important Baselines from Related Work added:

We appreciate the reviewer’s suggestions and pointing out the relevant work and baselines from prior work. We have added a discussion and comparison to all the suggested baselines:

Ensemble actors

[Figure 2, Section 2.3] As per your suggestion, we implement an ensemble-actors approach where we train multiple actors with the same critic, but explore using the critic maximizing action found by the actors. This approach helps disentangle SAVO’s ability to prune the optimization landscape from the improved exploration due to the presence of multiple actors. Please refer to combined response for further details, and Section 2.3 for a discussion and references of past work.

Evolutionary Actors

[Figure 2, Section 2.4] As per your suggestion, we add:

• CEM-actor baseline: Algorithms like QT-Opt, CGP, CEM-RL, and GRAC employ CEM as the actor in online RL training. We implement QT-Opt style CEM, where the critic is still trained with our TD3-augmented tricks. These algorithms involving CEM require infeasible amounts of repetitive evaluations of the Q-function and do not scale well to high-dimensional action spaces (Yan et al., 2019). Note that CGP, CEM-RL and GRAC can be considered as suboptimal versions of a good CEM actor in the QT-Opt, because they already use CEM as a guide to train their actors. Thus, we only implement and compare against the QT-Opt style CEM baseline.
• Greedy-actor critic baseline: emulates CEM by sampling from a high-entropy action proposal policy, evaluating these actions with the Q-function, and training the actor to greedily follow the best actions. However, since this approach also depends on gradient ascent and samples actions around the mean actions of a single-actor, thus limiting its ability to find the globally optimal action.
• Greedy-TD3: We implement another version of Greedy-actor-critic in TD3 style training, where the critic now benefits from TD3 update tricks, and the exploration is performed with OUNoise added to the mean action of the stochastic greedy policy.

Computational Cost Analysis

[Figure 4(d)] Please refer to combined response.

Clarifications

“The connection between finding the maximum action and improved sample efficiency is not supported”

• We bring attention to Figure 4(a) and Figure(b) where we already show that SAVO results in an improved Q-value of its actions, which translates into a sample efficiency improvement when evaluating SAVO with all its actors v/s only a single actor.

“Would it introduce overestimation?”

• No. SAVO only improves the actor’s capability to find the Q-value maximizing action, which is the key goal of the base algorithm of DPG / DDPG / TD3.
• There is still exploration on top of the action found by SAVO, so the exploration is not hindered.

“It seems at each time step, the algorithm update both policy and critic parameters K times?”

• No. We clarify that, as shown in Lines 16 and 17 of Algorithm 1, each individual $Q_i$ and $\pi_i$ is only updated once every iteration. $K$ denotes the number of actor-critic pairs, as written in the first line of Algorithm 1. So, the for loop in line 15 is to denote all the actor-critic pairs. To further clarify, all the baselines and ablations are based on the same algorithm design and implemented in the same code file, for fairness.

[References]

• Yan et al. Learning probabilistic multi-modal actor models for vision-based robotic grasping. ICRA 2019.

Thank you for your insightful questions. We have incorporated your suggestions in the revision, and clarify your questions below:

1. How nearest-neighbor mapping works in the discrete action setting?

A common strategy used in large discrete action spaces is to represent discrete actions with continuous representations, and solve the task like continuous action space RL. We follow the strategy of Wolpertinger [1], where a Q-function is learned over the continuous space. The actor’s continuous output is used to obtain k nearest-neighbor true action representations. Finally, the Q-value is evaluated over these “true” actions and the best Q-valued action is chosen.

However, as you correctly point out, while training the actor with gradient ascent over the Q-function, there will be times when the critic is queried for “fake” action representations, not associated with any true discrete action. The critic would be undefined on such intermediate actions — a neural network would at-best interpolate or smooth the Q-values when queried on such samples. This creates a smoothed piecewise-like critic with a challenging optimization landscape with many locally optima, as there are peaks on the true action representations, and the in-between landscape is hard to optimize.

[1] runs gradient ascent on this critic landscape completely ignoring the problem and thus underperforms empirically (Figure 2). In contrast, SAVO learns a sequence of actor-critics that successively simplify Q-value landscapes. This results in finding the action that better finds the optimal “true” peak of the Q-function.

2. Twin-critics of TD3 in Algorithm 1

[Algorithm 1] We follow TD3 and do use twin critics for primary critic to avoid overestimation (along with other improvements like clipped double Q-learning, delayed policy updates, and target policy smoothing). As per your suggestion, we modify Algorithm 1 to denote that the update follows TD3.

3. Direct evidence showing reduced local optima

[Figure 3, 11] As per your suggestion, we visualize the Q-value landscapes resulting in the successive critics of SAVO for all continuous control environments. For high-dimensional action spaces, we project the action space to 2D and generate a plot of Q-value over the action space. We observe:

• successive critics $Q_0 \to Q_1 \to Q_2$ have optimization landscapes with fewer local optima, as most optima below previously selected actions are pruned away. This is visually seen as the warm (high Q-value) regions of the contour plot increase successively.
• Better Q-valued actions are found than just a single actor $\pi_0$ that maximizes $Q_0$. Since all the peaks below the Q-value of previous actions vanish in successive critics, successive actors can focus on optimizing over better peaks.

Figure 4 (a) also demonstrates our actions have 40-50% better Q-values than the baseline over the course of training, and result in better returns than if we evaluated just using a single actor.

4. RLiable to summarize results

[Figure 4 (c)] As per your recommendation, we provide a summarized comparison across all 14 tasks with RLiable, demonstrating SAVO outperforming the baselines.

[References]

[1] Dulac-Arnold, Gabriel, et al. "Deep reinforcement learning in large discrete action spaces." arXiv preprint arXiv:1512.07679 (2015).