REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes

Further experimental results

Thank you for your valuable feedback on the scope of our evaluations. We agree that expanding the range of environments tested would provide a more comprehensive understanding of REValueD's effectiveness. The current focus on DeepMind Control Suite tasks from proprioceptive inputs was a starting point to establish the method's performance. We acknowledge the importance of testing in more diverse environments as these could offer insights into the versatility and robustness of our approach under varying conditions. As such, we have added results for DecQN and REValueD in a selection of Metaworld tasks (Appendix J). We see that REValueD still maintains superior performance over DecQN in these tasks, generally with smaller variance.

Control suite performance

For the metric used in our results, we agree that using 'Time steps' instead of 'Gradient steps' on the x-axis would align better with conventional standards and enhance readability. We have made the necessary adjustments to the figures to display 'Time steps' (the ratio was 5 env steps per update for all non-quadruped environments, in which case we used 10).

Regarding the inclusion of a continuous actor-critic agent as a baseline, our focus is on discrete action spaces in this study. Particularly in the context of DecQN and REValueD, the focus was driven by the specific challenges and characteristics of high-dimensional discrete action spaces. Since continuous control isn't the primary focus of our work, we believe that adding a continuous actor-critic baseline might not directly contribute to the comparative analysis intended for our research. However, we appreciate the suggestion and will consider it for potential future studies where a comparison with continuous control methods would be more relevant.

Increasing discretisation and sensitivity to hyper-parameters

Thank you for pointing out this discrepancy between the performance in Fig 6 of Seyde et al. and Figs 2,4 of our paper. In our revised paper, we have now plotted figures with environmental interactions on the x-axis which allows a more direct comparison. For dog-walk, after 1M environment interactions the score in Seyde et al. is ~650 and in ours it is ~550. This discrepancy appears to be the result of poor performance on a single seed in which our DecQN was unable to learn past a score of ~50. This can be seen in Fig 2 in which the variance of scores is uncharacteristically high. Removing this seed from our results, we average ~650, more in line with Seyde et al.

For finger-spin, after 1M environment interactions the score in Seyde et al. is ~850 and in ours it is ~750. In this instance, we don’t believe this is the result of an “outlier” seed (variance in scores is low), rather this may be due to slight difference in implementation and/or hyperparameter adjustments as you have suggested.

As for the reason for the differences in changing the hyper-parameters, the simple answer is that when we initially started experimenting with REValueD, this is how our existing code base was implemented. We did test to see whether they effected performance, but there was no significant changes between the implementations.

Computational overhead

Your suggestion to include timing results is indeed pertinent and would offer a more comprehensive understanding of the practical implications of using ensembles, particularly in terms of training time.

In our manuscript, though we cited a paper which claimed 'minimal slowdown' for relatively small ensemble sizes, we agree that providing empirical data to quantify this would be significantly beneficial. To address this, on the same machine we ran 3 seeds each for walker-walk and dog-walk, for DecQN and REValueD. Each seed ran for 500k env interactions (100k updates). We have included the results in Table 3, Appendix C. We observe around a 25% slow down using REValueD, but it is important to consider that REValueD is more sample efficient than DecQN and so less environment interactions are needed to achieve a superior performance. We have also changed the wording from 'minimal' to 'minor'

Comparison with distributional DecQN

You make an important point about distributional critics being an alternative to ensembles in addressing variability due to exploratory sub-actions, and their potential advantage in computational efficiency.

Whilst we briefly mentioned the potential of distributional reinforcement learning in our conclusion, we agree that incorporating a more detailed discussion and experimental comparison would strengthen our paper. We have added a comparison between DecQN with a distributional critic in Appendix L, finding that REValueD maintains its superior performance.

Choice of value decomposition form

In our study, we focused on the DecQN value-decomposition, which utilises the mean operator. This choice was driven by our specific interest in exploring and building upon the DecQN framework. While other forms of value-decomposition, such as the weighted sum, offer interesting alternatives, our study aimed to investigate the implications of the DecQN approach in particular. We acknowledge that other decomposition forms could lead to different insights and outcomes, and agree that exploring these alternatives would be a valuable avenue for future work.

Description of action and sub-action spaces for dog/humanoid

We have added a brief description of the action and sub-action spaces in the humanoid/dog in Appendix C, under environment details. We have elaborated on the unique challenges posed by the humanoid and dog tasks in the DeepMind Control Suite, such as the large number of joints (sub-actions) that need to be controlled and the intricate interdependencies among these joints. This will provide readers with a clearer understanding of the setting and underline the motivation behind our approach, particularly highlighting why these tasks are well-suited for evaluating the performance of REValueD and the implications of our method on managing such complex action spaces.

Design of $|\delta_i|$

Regarding the design of $|\delta_i|$ and its implications, we appreciate the opportunity to clarify this aspect. The concept of $|\delta_i|$ is central to our approach in handling the credit assignment issue. The rationale behind the design is that a large $|\delta_i|$ represents the difference between the current and target utility values for a sub-action. Therefore, a large $|\delta_i|$ might indicate significant impact from other sub-actions or external factors. This is because in complex environments, especially with multiple sub-actions, the effect of an individual sub-action is often intertwined with others. Therefore, when $|\delta_i|$ is large, it indicates that the reward or the state transition is likely not solely determined by the sub-action in question, but rather is influenced by a combination of several sub-actions.

Baselines

Whilst we only compared to two main baselines in the main paper, we have added in the revised version comparisons to BDQ equipped with an ensemble, as well as DecQN equipped with a distributional critic (DecQN-C51), in Appendix K and L, respectively.

Comparison with Tang et al. 2022

Thank you for bringing this piece of research to our attention, which we have now included in our Related Work section. While both studies operate in the realm of Q-function decomposition within factored action spaces, there are two key distinctions with respective to their theoretical analyses.

First and foremost, our analysis focuses on the bias/variance of the error between the true and (function) approximated Q-values, whereas Tang et al. focus on the fundamental properties of the Q-values themselves. The outcomes of our respective analyses reflect how these related, but distinct, attributes change when moving from the standard Q-learning approach to one based on value-decomposition, and hence these analyses can be thought of as complementary rather than contradictory.

Second, there is a minor difference in the value-decomposition itself. We use the decomposition proposed by Seyde et al., which involves taking a mean of utility values to approximate the global Q-values, whereas Tang et al. take the sum of utility values. For completeness, we have added a section in Appendix B that analyses this sum variant under our framework, and we find this time that both the bias and variance are higher. We could of course mirror Tang et al’s. analysis using the mean instead of the sum, but this is beyond the intended scope of our paper which is interested in the error as stated above.

In summary, the apparent discrepancies concerning variance can be attributed to varying methodologies and distinct theoretical frameworks, which when taken together provide complementary perspectives and insights into Q-learning in FMDPs using value-decomposition.

Q-value decomposition

Our analysis assumes that the Q-function can be decomposed for practical approximation, an approach extended to the true Q-function in Equation (4.2). While, as Tang et al. 2022 highlight, such decomposition may not perfectly align with the true Q-function in practice, we posit that it offers a close enough approximation for the purposes of reinforcement learning, where exact representations are often unattainable. This assumption underpins our theoretical framework and the development of REValueD. The strong empirical results, acknowledged by the reviewer, provide further evidence that this decomposition approach is adequate in many practical tasks.

Performance of BDQ

Thank you for pointing out that a more like-for-like comparison of BDQ would be to compare to BDQ with an ensemble. In Appendix K we have added some comparisons with BDQ-ensemeble in the environments where the bin size is 3, and in the variants with larger bin sizes, and find that the performance remains largely unchanged when equipping BDQ with an ensemble.

As for the performance of BDQ in Figures 1 vs. 2, it is important to note that the x-axis scale (also note as per request of reviewer Athy we have changed x-axis scale from updates to environment interactions) changes slightly between Figures 1 and 2 for the dog-walk task. In Figure 1 we cut the figure off after 750k environment interactions, at which point BDQ obtains a score of ~500, whereas BDQ in Figure 2 is achieving approximately this score after 1M env interactions for $n = 10, 30$, so it is taking more environment interactions to achieve the same score when using a higher bin size, though the performance does not diminish as quickly as DecQN when using a higher bin size.

Presentation suggestions

Regarding your suggestion for Table 2, we agree that bolding the best results for each task would significantly enhance readability and quick comprehension of the table. We have incorporated this in the revised version of the manuscript.