Unifying Diverse Decision-Making Scenarios with Learned Discrete Actions

• Scalability：We fully agree that testing in more complex environments would provide a better assessment of the performance of the proposed method. However, we should point out that in our MuJoCo experiments with online RL (Figure 9), we have already included quite complex environments such as Humanoid with observation shape: 376 and action shape: 17. In the hybrid environment experiments with online RL (Figure 4), the action space of HardMove-v0-n10 is also large (1024 discrete action types), and GoBigger features complex structured observations and dynamic environments. The performance in these environments has partially validated the scalability of our method. In the next revision of our paper, we will include baseline results on the D4RL antmaze environment.
• Analysis of the latent action shape $K$:
  • The good performance can be achieved with Hopper when $K=4$. We attribute this to two reasons:
    • First, due to the structure of state-conditioned action, it's as if there are 4 actions available under each state, rather than 4 actions available under all states. This significantly expands the range of optional actions from K to $|S|^K$, where |S| is the number of all possible states.
    • Second, according to the analysis in previous paper "Is bang-bang control all you need? solving continuous control with bernoulli policies.", decent performance can be achieved in these special MuJoCo environments in D4RL just by using bang-bang control. Therefore, the number of effective actions required in this environment is inherently limited.
    • These two reasons explain why a small $K$ is able to obtain satisfactory results.
  • Regarding the direct comparison of manual discretization and direct clustering, we have already carried out these experiments in the original version of our paper.
    • You can find the simple discretization of DQN in Figure 9. As expected, due to the exponential increase in action space dimensions after discretization, the optimization of Q becomes difficult and results in poor performance.
    • In Figure 14, you can find our case study on LunarLander environment in the online RL setup. It's observed that DQN + expert cluster, which directly uses actions from expert trajectories for clustering, then applies DQN, can achieve similar results to GADM+DQN. This indeed shows that reducing the dimensionality of the action space greatly impacts performance.
    • However, we believe that our proposed GADM is more widely applicable, as expert data may not always be available and clustering in complex action spaces can be complicated. GADM can learn discrete actions from scratch, and in the offline RL setting, it can effectively mitigate the pathological latent action space problem (see details in the latest version of our paper, Figure 3) with our proposed latent action mask, which makes a naive DQN with GADM can draw comparable results with several spefically designed offline RL algorithms.

Thanks for your valuable feedback. We will address your comments and questions in the following parts:

• Question about Figure 1: The purpose of Figure 1 is to illustrate the redundancy of action spaces, indicating that similar (s, a) pairs will be mapped to the same latent action. The GADM state-conditioned encoder, which we designed, precisely implements this by inputting (s, a) and outputting the corresponding latent action k (i.e., the encoded features $z^e=e_\phi(s,a)$). For more details, please refer to Section 3.2 of our revised paper.
• Writing: As stated in the "General Response" section, we have made extensive revisions to our paper. Titles of sections with significant updates are highlighted in red in the latest version. We hope these modifications enhance the paper's readability and impact. We are grateful for your insightful feedback and look forward to receiving more constructive comments.
• Code table:
  • As described in Section 3.2, if we directly apply the codetable design of VQ-VAE to GADM and reuse the original exponentially weighted average method for updates, we would obtain poor results. The codetable here needs to serve as the decision-making action space for subsequent RL training. However, the original design of VQ-VAE tends to make most vectors in the codetable very similar, leading to homogeneity in decision semantics. This limits the actions that RL algorithms can select, thereby impacting final performance. A new approach is necessary to ensure the vectors in the codetable remain as diverse as possible. Since RL algorithms only need to use the index of different latent actions, it does not need to generate different styles of images like the original image generation task of VQ-VAE, we can directly fix the value of the codetable so that it does not participate in training, thereby maintaining sufficient diversity.
  • Specific implementation methods are to initialize the codetable with a series of one-hot vectors or bisection point vectors. Suppose the code table contains $K$ candidate latent actions. The one-hot vector means that each latent action also corresponds to a K-dimensional (i.e. N=K) one-hot vector. It is equivalent to each latent action being widely separated, and the distance between two vectors are equal. The bisection point vector is to let $K$ points fall on the $K$ equal points of the line segment connected by $[-1, -1, ..., -1]$ and $[1, 1, ..., 1]$ (N-dimensional vector). Each bisection point corresponds to a vector.
• Latent action confidence predictor:
  • We have thoroughly optimized and restructured the section on the Latent Action Confidence Predictor. We have included an in-depth analysis of the issue concerning the pathological latent action space and have refined Figure 3. We also explain the input and output of the latent action confidence predictor, the training loss, and provide a strict definition of the latent action mask technique. We outline how the latent action mask technique can be applied in concrete discrete RL algorithms in Section 3.3 of our revised paper.
  • Similarities to RND:
    • The main idea of RND is to measure novelty using the prediction error in a specific state. Given the neural network's characteristics in fitting datasets for supervised learning, a larger prediction error at a certain state implies the agent has visited the nearby state space less frequently in the past, suggesting a higher novelty for that state.
    • In contrast, the latent action confidence predictor uniformly samples (s,k) pairs from the replay buffer, where k is obtained by encoding (s,a) using the latest action encoder. We train a classifier-like model, where (s,k) pairs that frequently appear in the replay buffer have a stronger impact on the neural network update, thus yielding a higher confidence score. Conversely, if a pair (s, k') appears less frequently or not at all in the replay buffer, its impact on the neural network update is smaller, resulting in a lower confidence score. Thanks to the unique initialization of the neural network, the initial confidence score is set to $\frac{1}{K}$ (where $K$ is the total number of latent actions), so the output confidence score $C$ for a certain state reflects the distribution of k in the replay buffer. so the output confidence score $C$ for a certain state is proportional to the distribution of k in the replay buffer. While the foundational concept has certain similarities with RND, their application and detailed mechanisms show significant distinctions. Consequently, we have refined and enhanced the relevant descriptions in the latest revision of our paper.

Dear Reviewer Fy3u,

We greatly value your insightful feedback on our paper and appreciate the time you have dedicated to its detailed review. We are eager to address your concerns and clarify any misunderstandings.

(1)

• Indeed, Figure 1 serves as an illustrative example, emphasizing the inherent redundancy found within the raw action space. This redundancy, prevalent in transitions and $Q$-values, can be significantly mitigated through a more compact discretization process, thereby expediting the learning process.
• Although we did not directly illustrate how this redundancy is eliminated in terms of transitions or $Q$-values in the main text, many of our experimental results indirectly substantiate this point.
  • For instance, in the experiments conducted in the online hybrid action space, our proposed method, GADM, markedly outperforms existing algorithms specifically designed for hybrid action spaces. In the Moving environment, for example, once the action type x is set to break, the decision-making process remains unaffected by any value of the action parameter y. This suggests that a substantial number of raw actions are semantically consistent, enabling us to achieve performance superior to algorithms like MPDQN, which are expressly designed for hybrid action spaces, using our learned discretized action space.
  • Additionally, the experiments conducted in the offline D4RL environment corroborate this. Even when setting a relatively small value of $K$, we managed to achieve results comparable to those of offline RL algorithms operating in the raw continuous action space.
• We appreciate your suggestion and will delve further into how GADM minimizes action space redundancy from the perspectives of transitions and $Q$-values in the next revision of our paper.

(2) In response to your second point, suggesting the removal of the part where we compare our intuition to that of RND, we concur with your perspective. Our initial intention was to offer a familiar frame of reference to help readers comprehend our method. However, we acknowledge that this comparison could potentially lead to confusion. We have already eliminated this part in the revised version of our paper.

(3)

• We agree with your suggestion, as well as those from other reviewers, that our model should be compared with more related baseline methods. However, we have some reservations about the view that "I think the action space is not a big deal if you just consider them as continuous number or cut them into bins?"
• Indeed, for continuous action spaces, traditional methods like TD3 and SAC can deliver satisfactory results. However, when dealing with complex hybrid action spaces, which are composed of continuous and discrete components, traditional RL techniques often fail to perform well. Our approach is adept at handling various decision-making problems, including hybrid and continuous ones, in both online and offline scenarios within a unified framework. This could serve as a reference baseline for future research. Specifically, the advantages of learning discrete action spaces are demonstrated through examples in hybrid action spaces.
  • Hybrid action space: For instance, the action spaces of various environments tested in Figure 4. Specifically, the action space of "Moving and Sliding" is (x, y), where the value of the action parameter y depends on the choice of action type x. In addition, the action space of the HardMove-v0-n10 environment includes discrete actions and continuous parameters (x, y) — where x includes $2^{10}$ discrete actions, and y represents a ten-dimensional continuous action. In the GoBigger environment, the action space is defined as [x, y, action_type]. Here, (x, y) signifies the player's control over the ball's acceleration, while action_type is an integer representing the type of action, with [0,1,2] denoting three distinct types of actions. This type of action space poses significant challenges to traditional RL algorithms.
  • In such cases, algorithms specifically designed for hybrid action spaces, such as MPDQN and HPPO, become essential. However, these algorithms are often intricately designed and lack general applicability. As shown in Figure 4, we found that GADM+DQN consistently outperforms baseline methods like MPDQN and HPPO in terms of both performance and stability, providing empirical support for the advantages of learning the discrete action space.

• In addition, if we choose to discretize simply, we would face the issue of exponential growth in the dimensionality of the discretized action space. As shown in Figure 9, in the experiment of the online continuous environment, the performance of DQN (Manually Discretized) gradually deteriorates as the action space expands. Even in some hybrid action space environments where we can manually design a fine-grained discretized action space (e.g., the GoBigger environment shown in Figure 4), our GADM+DQN still surpasses the DQN w/ Manually method to reach a higher performance ceiling.

We thank you again for your thorough and constructive review.

Best Regards.

Authors

Dear Reviewer 9Kqe,

We greatly value your insightful feedback on our paper and appreciate the time you have dedicated to its detailed review. We are eager to address your concerns and clarify any misunderstandings.

• Verbosity and Structure:

  • As indicated in the "General Response," we made comprehensive revisions to our paper. In the latest edition, we have color-highlighted the titles of the sections that underwent significant modifications in red. We anticipate that these changes will improve the paper's readability and influence. Please note that due to the length constraints of the main paper, the visualization analysis of the action space learned in the offline and online settings continues to be included in the appendices. However, we made clear references to it in the experiment section of the manuscript. We thank you for your insightful feedback and look forward to receiving more constructive comments from you.

• Failure Case Analysis:

  • The ablation experiments in both the main text and the appendix partially reflect our analysis of failure cases, e..g.:
    • In the ablation studies for online and offline settings presented in the main body of the paper, we demonstrated that not employing our proposed techniques results in a significant performance drop. Specific methods, such as latent action mask, latent action remapping, and warmup, were extensively discussed in their respective sections.
    • Moreover, in the appendix's experiments on online continuous action space, the performance of DQN with naive discretization was noticeably underwhelming. We have conducted an initial analysis of this. If techniques such as E.A.R. are not implemented, the performance will suffer a notable decrease, as briefly discussed in the appendix.
  • In subsequent revisions of the paper, we plan to incorporate more comprehensive analyses of other failure cases. We appreciate your suggestion.

• Original Figure 3:

  • We appreciate your inspiring suggestions. In response, we have revised Figure 3 of the original paper according to your recommendations and improved the corresponding textual descriptions and captions. Please note that due to space limitations, in the revised version of the paper, we have moved the original Figure 3 to Figure 6 in the appendix.

• "Codetable" vs. "Codebook":

  • We fully agree on the importance of using professional and literature-consistent proprietary terms. We referred back to the original VQVAE paper, where they use the term "embedding table," while some subsequent NLP and CV papers, such as "Zero-Shot Text-to-Image Generation", use the term "codebook". There appears to be no unified standard.
  • Our initial choice of using "codetable" was because it reflects both a lookup table and a type of encoding. We believe that consistent use of a proprietary term in the paper with clear semantics is acceptable. If necessary, we will change it to a more suitable term in the revised version of the paper.

We thank you again for your insightful feedback and look forward to receiving more constructive comments from you.

We highly appreciate your thoughtful feedback and thank you for the thorough review. We look forward to addressing your concerns and clarifying any misunderstandings.

• Insufficient Method Details and Experimental Details: As stated in the "General Response", we have made extensive revisions to our paper. The titles of sections with significant updates are highlighted in red in the latest version. We hope these modifications will enhance the readability of the paper. We thank you for your insightful feedback and look forward to receiving more constructive comments from you.
• Connection to prior work and novelty: We acknowledge that the statement ("other algorithms can be seen as a special case of our framework") may be inappropriate. In original related work, we noted that previous methods such as HyAR only learned the continuous part of hybrid action space. Our method, on the other hand, is capable of handling all types of action spaces and supports both online and offline RL, which led to our initial statement. However, based on reviewer feedback, we recognize that there are indeed many other types of action representation and discretization methods. Our proposed framework cannot unify all previous methods. A more accurate description would be "the first method to unify action representation discretization across all types of action spaces, while simultaneously supporting both online and offline RL". We have corrected this in our revised paper.
• Comparison to action representation learning methods: Our method doesn't claim that action discretization by VQ-VAE scheme is better than other action representation learning methods. However, discrete action space does have many advantages, such as low redundancy, high optimization efficiency, etc. The main contribution of our approach is the first attempt to discretize all types of action spaces and simultaneously address the training stability issues derived in online and offline RL. Therefore, the experimental part is mainly to demonstrate the versatility of different types of action spaces and training settings, and the comparison with other action representation is secondary. We are also testing some action representation learning methods and are preparing to supplement relevant comparison experiments.
• GADM Complexity and scalability:
  • We have introduced a series of components that make up GADM, but it should be emphasized that most modules can be regarded as designs within GADM. The only change to the classic RL training pipeline is to use the mask obtained from the latent action confidence predictor to appropriately modify the RL loss function (such as GADM + DQN mentioned in Section 3.3). Action encoding, decoding, and remapping can be regarded as additional data pre-processing and post-processing modules, thus decoupling from the classic RL code. In contrast, for DQN, if you want to expand to the hybrid action space, the previous approach can only make major changes to the entire optimization process like MPDQN, and if you want to expand to offline RL problems, you need to add new optimization items like CQL terms, but GADM can solve all the above problems through an action representation model and generalize naive DQN to various scenarios.
  • For scalability, we should point out that GADM can be inserted in normal discrete RL algorithms in a plug-in manner, such as passing a new mask variable to control the selection of target $Q$ action in DQN. And we have already validated its performance on a series of RL environments with different challenging action spaces. In our MuJoCo experiments with online RL (Figure 9), we have already included quite complex environments such as Humanoid with obs shape: 376 and action shape: 17. In the hybrid environment experiments with online RL (Figure 4), the action space of HardMove-v0-n10 is also large (1024 discrete action types), and GoBigger features complex structured observations and dynamic environments. The performance in these environments has partially supported the scalability, and we will test more environments in future version.
• Question about state-conditioned design:
  • Without the incorporation of state-conditioning, there would only be $K$ available actions across all states. However, due to the structure of state-conditioned actions, it appears as if there are $K$ available actions under each individual state, rather than $K$ actions shared across all states. This significantly expands the range of optional actions from $K$ to $|S|^K$.
  • We are currently supplementing our experiments without state-conditioning and the preliminary results are consistent with our expectations. When state-conditioning is removed, the decrease in the number of available actions across all states leads to a significant performance decline. We will include a detailed experimental results in our revised paper. We appreciate your constructive suggestion.

• Question about Sampled MuZero:

  • Firstly, we want to clarify that the primary aim of GADM+MuZero is to demonstrate that GADM can extend existing discrete decision-making algorithms to a broader range of applications. For instance, while MuZero is only applicable to discrete action spaces, GADM+MuZero can be applied to continuous action spaces. Due to the issue of pathlogical latent actions—which we have thoroughly defined in our latest paper—there could be stability issues with the algorithm. This problem is more pronounced in offline scenarios where there is no need for interaction with the environment, prompting us to validate the principle on offline D4RL. We paid particular attention to the impact of latent action confidence prediction in GADM on instability in offline scenarios. The results were aligned with our expectations, with GADM+MuZero performing exceptionally in addressing OOD issues in offline RL.
  • Of course, we also plan to supplement our experiments with comparative studies in online settings in subsequent work.
  • Regarding the performance issues with Sampled MuZero, there could be some performance differences since we used open-source implementation (as the original authors did not open-source their code).

• Question about latent action shape and its sensitivity:

  • In the online setting, we actually use a $K$ (the size of the embedding table, i.e., the latent action shape) parameter of 128, not 256; 256 represents the dimension of the embedding vector. To avoid confusion, we have optimized the naming of the variables in our hyperparameter table for clarity.
  • In online settings, the appropriate latent action shape $K$, is often larger than what is set in offline settings.
    • The main reason is that the action distribution in the datasets used for online settings is generally broader than that in offline datasets, thus necessitating a larger number of latent actions for modeling.
    • In online settings, the agent interacts with the environment actively. To find the optimal trajectory, the agent needs to explore sufficiently. Theoretically, the agent will encounter a broader range of the action space. To avoid omission, we need to set $K$ larger to better model all potential actions.
    • In offline settings, the dataset may have been collected from expert or medium policies, so the range of actions is relatively smaller, leading to a smaller $K$ setting.
  • Our latent action mask technique can partially alleviate the pathological problem that arises in the latent action space (a detailed analysis is provided in the revised version of the paper). We expect that the sensitivity to the latent action shape $K$, will not be very high in offline or online settings. However, this needs to be confirmed through our experiments. In the revised version of the paper, we will include the complete experimental results and analysis. Thank you for your valuable suggestions.

Here, we present some supplementary ablation study results focusing on state-conditioned settings and varying K values. A more comprehensive version of these results, encompassing additional environments, will be made available in the next revised version of our paper.

• In the offline D4RL Hopper Medium-Expert scenario, we conducted experimental comparisons of the performances under different K values for both state-conditioned and non-state-conditioned settings. As the experimental results show, in the non-state-conditioned environment, the number of available actions for all states is fixed at K, significantly constraining the expressive power of the model, leading to a significant decline in performance.

• Similarly, we conducted ablation experiments on different latent action shapes (K) under the state-conditioned setting in Hopper Medium-Expert scenario. In a state-conditioned setup, the number of available actions for each state becomes $|S|^K$ (where |S| is the number of all possible states), greatly enhancing the expressive power of the model. It's noteworthy that the model performs well even when K=3. However, when K is reduced to 2, considering the dataset includes both medium and expert policies, the K=2 setting struggles to effectively reconstruct all the core actions under each state, leading to a performance decline. In our tests, K=4 proved to be the optimal choice. It's worth noting that as the value of K continues to increase, although the number of optional actions for each state also increases, it simultaneously raises the difficulty of learning the Q function, resulting in a slight performance drop.

We thank you again for your thorough and constructive review.

2A:

• Reward weighted loss function: The key insight behind this design is that RL does not necessitate the modeling of all possible actions, which significantly reduces the learning burden of GADM. In essence, GADM does not have to accurately reconstruct all action samples, but should focus more on those actions with potential for high returns. Hence, a weighting scheme based on immediate rewards or return-to-go is necessary. However, due to our writing mistakes, the expression of this weight was not rigorous enough in the original paper. We have corrected this problem in the revised version. Specifically, for any RL environment, GADM can directly reuse the reward shaping/normalization scheme commonly used in this environment and then scale it to [0, 1] through simple transformations to serve as weights. As for the case of negative rewards mentioned by the reviewer, the ambiguity arose from our symbolic representation norm(R(s,a)). The original intent was to denote a specific type of normalization method for rewards, such as $(r - r_{min})/(r_{max}-r_{min})$, or other appropriate methodologies.

• Latent action confidence predictor:

  • As stated in the "General Response", we have undertaken a comprehensive optimization and reconstruction of the section pertaining to the latent action confidence predictor, resulting in a more refined representation in Figure 3.
    • We have incorporated a thorough analysis of the pathological latent action space problem, which is caused by redundant and shifted latent actions. This phenomenon frequently results in overestimation and subsequent instability issues affecting both offline and online RL.
    • In addressing these challenges, we propose the development of the latent action confidence predictor. We further elaborate on the input and output specifics of this predictor, its training loss, and provide a stringent definition of the latent action mask technique.
    • For more in-depth details, we kindly direct you to Section 3.2 in our recently revised paper.
    • Moreover, we've outlined how the latent action mask technique can be applied in concrete discrete RL algorithms in Section 3.3 of our revised paper, titled Practical Algorithm. We have included this new section in the updated version of the paper.

• E.A.R.:

  • Extreme Action Regression (E.A.R.) is a technique specifically devised for some environments, such as MuJoCo, which require precise extreme action values as shown in paper "Is bang-bang control all you need? solving continuous control with bernoulli policies.".
  • Acknowledging the feedback from multiple reviewers on the need for more extensive coverage of our core methods and experiments within the main body of the paper, we have made adjustments. Considering that E.A.R. is a technique tailored for specific environments rather than a universally applicable one, it is not our core contribution and we have relocated the detailed introduction of E.A.R. to the appendix under Implementation Details in the revised paper, while maintaining a reference to it in the main paper.

• Warmup: We fully agree with the review's point that "any details that affect reproducibility should be included". Therefore, we have moved the motivation and principle of warmup to Section 3.1 Dual training pipelines for GADM: offline and online RL in the revised paper. Specifically, the GADM for online RL setting’s training pipeline can benefit from an optional warmup phase. During the warmup stage, data is accumulated via a random or expert policy, or from a pre-collected dataset. This data is used to train the action model, providing a solid foundation for the subsequent stage, as shown in Section 4.1.3.

3A:

• Baseline in hybrid action space:

  • In the online hybrid action space environment, we selected MPDQN and HPPO as benchmarks for the following reasons:
    • Firstly, given that our GADM+DQN is also a derivative of DQN, it is fair that we compare it with MPDQN, which stands as the top-performing algorithm that extends off-policy DQN to hybrid action spaces.
    • Moreover, as stated in paper "HyAR", HPPO currently stands out as one of the most effective algorithm specifically designed for hybrid action spaces.
    • While PADDPG is recognized as one of the previous algorithms for handling hybrid action spaces, its performance does not measure up to the two algorithms we have chosen for comparison.

• Error bars in tables: In offline RL, the impact of random seeds on performance is usually not as significant as in online RL. Therefore, many previous papers, such as D4RL, BCQ, CQL, etc., only reported the average return. We adopted this setting as well. However, adding error bars can indeed help readers to better evaluate performance, and we will be including standard deviations in subsequent versions of the paper.

We greatly appreciate your insightful and thorough review. We look forward to addressing your concerns in the following sections.

1A: We will add related works mentioned by the reviewer about action representation and action discretization in the next revised version of our paper. And we will polish the related work part to describe the connections and differences between our proposed method and previous methods more clearly.

1B:

• Prior works can be regarded as a special case of our framework: We acknowledge that this statement may be inappropriate. In our original discussion of related work, we noted that previous methods such as HyAR only learned the continuous representation of hybrid action space, while other methods only extended the discrete action representation to offline RL. In contrast, our method is designed to handle all types of action spaces and is capable of supporting both online and offline RL, which led to our initial statement. However, based on reviewer feedback, we recognize that there are indeed many other types of action representation and discretization methods. Our proposed framework cannot unify all previous methods. A more accurate description would be "the first method to unify action representation discretization across all types of action spaces, while simultaneously supporting both online and offline RL". We have corrected this in our revised paper.
• GADM's potential as a general design of decision-making foundation models: To design effective foundation models, drawing from the rapidly developing field of large language models can offer valuable insights. Establishing a universal training paradigm hinges on unifying input and output types. In the field of natural language processing, the model's output can be unified by predicting contextual words, where the output space is a discretized fixed dictionary. Currently, many works (e.g., BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers) in the field of computer vision is exploring methods to discretize image representations. However, in regard to sequence decision-making problems and reinforcement learning, the format of the output space (i.e., action space) is highly complex. If a unified representation form can be established, and its performance is comparable or superior to classical RL algorithms, then this unity could form an important part of the decision-making foundation model. Our method is an attempt in this direction, and its effectiveness has been initially demonstrated in the environments mentioned in the paper.

1C:

• Details about baselines:
  • We have added a brief introduction to hybrid action space algorithms in the "Online RL Results" section, and provided an analysis as to why their performance in the hybrid action space does not match up to that of GADM+DQN.
  • In the online continuous action space MuJoCo experiments, the baseline algorithm we compared our method with is the classical TD3 method and DQN with naive manually discretized. Due to space constraints, we have only included the corresponding references and provided brief introductions to these methods.
  • In the offline D4RL scenarios, the baseline algorithms we compared our method with include model-free methods such as BCQ, CQL, and ICQ, as well as model-based offline RL methods like DT and TT. For the comparison of our method with BCQ and ICQ, please refer to Figure 6 in the revised paper.
• Reference: As mentioned in the "General Response", we have optimized and rebuilt the experimental part in the revised paper, and added accurate references for all the supplementary experiments in the appendix.

2A:

• Diversity-aware codetable: As described in Section 3.2 of the paper, if we directly use the codetable design of VQ-VAE in GADM and reuse the original embedding loss or exponential weighted average method to update it, we will obtain very poor results. This is because the codetable here needs to be used as the decision-making action space for subsequent RL training. In other words, each vector of the codetable needs to have different decision semantics as much as possible. However, the original design of VQ-VAE will make most of the vectors in codetable become very similar, that is, they may become very homogeneous in decision semantics, which will limit the actions that RL algorithms can select, thereby affecting the final performance. Therefore, a new approach is necessary to ensure the vectors in the codetable are as diverse as possible. Since RL algorithms only need to use the index of different latent actions, it does not need to generate different styles of images like the original image generation task of VQ-VAE, we can directly fix the value of the codetable so that it does not participate in training, thereby maintaining sufficient diversity. Specific implementation methods are to initialize the codetable with a series of one-hotvectors or bisection point vectors.

3B:

• Experiments on standard discrete and continuous online RL environments are missing:
  • We would like to clarify that the appendix of our original paper has already shown the comparison results of GADM+DQN, TD3, and DQN with simple manual discretization in the continuous action space of the MuJoCo environment. Currently, it is located in Figure 9 of the appendix.
  • Regarding experiments in the online discrete action space, we believe that when the dimension of the discrete action space is low, it is not necessary to use action representation learning methods, as the action space is already compact in such cases. For high-dimensional discrete action spaces, it might be necessary to use action representation learning methods, but this will be the focus of our future research, as our current research focuses on hybrid and continuous action spaces.
• Appendix: "We evaluate on 4 environments", but I only see 1 (hopper-v3). Were the results on other tasks not promising?:
  • Similar to the previous question, our baseline experiment was conducted in four continuous action spaces in the Mujoco environment, including hopper-v3, halfcheetah-v3, ant-v3, and humanoid-v3. We apologize if the illustrations were not clear enough and may have caused some misunderstandings.
• Why inconsistent environments between baseline results (only hybrid action space environments) and ablations (HalfCheetah)?:
  • We would like to clarify that our baseline experiments were performed across eight environments. These included four hybrid action space environments and four continuous action space environments within the MuJoCo benchmark. The outcomes from these distinct sets of experiments are depicted separately in Figures 4 and 9, respectively.
  • In the main body of our paper, we presented an ablation study (Figure 5) that was conducted on two specific environments: the continuous action space of HalfCheetah and the hybrid action space of HardMove-v0-n10. These were chosen as representative examples for their respective action space types. We believe this experiment settings provide a comprehensive view of our methodology's performance across different types of action spaces.

3C:

• Standard VQ-VAE v/s diversity-aware codetable:
  • As described in Section 3.2 of the paper, if we directly apply the codetable design of VQ-VAE to GADM and reutilize the original embedding loss or exponential weighted average method for updating, the results will be suboptimal. The reason lies in the nature of the codetable, which needs to serve as the decision space for subsequent RL training. Each vector in the codetable should ideally exhibit maximal decision semantic diversity. However, the initial design of VQ-VAE might lead to a high degree of homogeneity in decision semantics among the vectors in the codetable, thus limiting the action selection of the RL algorithm and impacting its final performance. To ensure the diversity of vectors in the codetable, we propose a new strategy: given that the RL algorithm only needs to use the indices of different latent actions and doesn't need to generate images of varying styles like the original task of VQ-VAE, we can keep the values of the codetable constant, ensuring its diversity during training. The specific method involves initializing the codetable with a series of one-hot vectors or bisector point vectors. We have conducted relevant comparative experiments but due to space constraints, they were not included in the paper. These results will be added to the revised version.
• FiLM and Reward normalization:
  • Our initial paper focused on the effective use of action space discretization methods in various action spaces across both online and offline settings, without including comparative experiments for FiLM and reward normalization. These were not initially included because our choice of highlighted in the "FiLM" paper, namely its ability to perform layer-wise, input-dependent transformations of feature maps. Similarly, we opted for reward normalization and weighted reconstruction loss to allocate more model capacity to the key areas within the overall action space. However, acknowledging the importance of these technical elements for the reproducibility and robustness of our findings, we've decided to incorporate comparative experiments for FiLM and reward normalization in our revised paper. Thank you for your valuable suggestion.

3D:

• Codebook size:
  • As our latent action mask technique can partially mitigate the pathological latent action space problem (detailed analysis is provided in the revised version of the paper), we expect that the experiments on offline d4rl are not very sensitive to the codebook size, i.e., the latent action shape K. However, this needs to be confirmed by our experiments. We will add the complete experimental results and analysis to the revised version of the paper. Thank you for your valuable suggestion.

3A:

• GADM+DQN should be comparison with standard algorithms like TD3 on continuous action space.：
  • We would like to clarify that in the appendix of the original version of our paper, we have conducted a comparative analysis of GADM+DQN and TD3, as well as DQN with naive manual discretization, in the continuous action space of MuJoCo. This can be found in Figure 9 of the appendix. Our apologies if our referencing was not clear enough before, which might have caused you to overlook it. We have now added more explicit descriptions to the experiment section for easier reference.
• MuZero should be compared against on some discrete action task.：
  • Firstly, we want to clarify that the primary aim of GADM+MuZero is to demonstrate that GADM can extend existing discrete decision-making algorithms to a broader range of applications. For instance, while MuZero is only applicable to discrete action spaces, GADM+MuZero can be applied to continuous action spaces. Due to the issue of pathlogical latent actions—which we have thoroughly defined in our latest paper—there could be stability issues with the algorithm. This problem is more pronounced in offline scenarios where there is no need for interaction with the environment, prompting us to validate the principle on offline D4RL. We paid particular attention to the impact of latent action confidence prediction in GADM on instability in offline scenarios. The results were aligned with our expectations, with GADM+MuZero performing exceptionally in addressing OOD issues in offline RL.
  • Regarding experiments in the discrete action space, we believe that when the dimension of the discrete action space is low, it is not necessary to use action representation learning methods, as the action space is already compact in such cases. For high-dimensional discrete action spaces, it might be necessary to use action representation learning methods, but this will be the focus of our future research, as our current research focuses on hybrid and continuous action spaces.
• Why discretization is the right way to go?：
  • Our framework's principal contribution lies in its initial effort to discretize all types of action spaces, while concurrently addressing the training stability issues that occur in both online and offline RL scenarios. We do not claim that the VQ-VAE scheme for action discretization is superior to other representation learning methods. However, we do recognize and highlight some insightful aspects of action space discretization:
    • Discretization need not consider all possible action samples, but rather focuses on the most critical parts needed for the agent's decision-making process.
    • We need to carefully discern the information that genuinely requires compression during the process of discretization. For instance, the content described in Figure 1 of our paper provides a set of examples that can guide our thinking for subsequent module design.
    • It's essential to ensure the diversity and stability of the code table in the process of discretization.
    • Our analysis in the motivation section, bolstered by empirical validation, suggests that discretization yields promising results, both intuitively and experimentally.
  • Despite not being the primary focus of our current research, we understand the importance of comparison with other action representation methods. As such, we are actively exploring alternative action representation learning methods and plan to augment our research with relevant comparative experiments in the forthcoming revised paper.

I appreciate the improvements made in writing, but still there is a lot to be done in terms of

• the experiments that are missing (baselines, ablations, codebook size analysis, etc)
• missing prior work discussion
• clarity in writing. It should be clarified what is claimed as this work's original contribution. And these contributions should be verified experimentally. "the first method to unify action representation discretization across all types of action spaces, while simultaneously supporting both online and offline RL" this by itself is not a contribution without validating why this unified scheme is useful against its alternatives.

I would keep my score as this paper needs most of the experiments that were asked, and the authors agreed to.