Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.

W1: Limited task/domain evaluation: The experiments are confined to a limited benchmark set (Adroit tasks). While this can be a strong testbed, the generality beyond this, especially those tasks or domains heavily demanding for human-like behaviors (e.g., navigation, games, role-playing), is not empirically validated.

Yes, as the reviewer noted, we chose Adroit because it is a strong testbed, which requires continuous control and fine-grained skills. We believe it is well-suited for evaluating human-like manipulation behavior.

However, we also agree with the reviewer that demonstrating the generality of MAQ is important. Therefore, we conduct additional experiments in two Atari environments:

• Venture: a navigation and role-playing style task in which player movements are clearly visible
• Enduro: a third-person racing game with a highly dynamic environment

Next, we train both PPO and MAQ+PPO on these games. For MAQ, the macro-action sequence length is set to 9. To assess human-likeness in the discrete action space, we adopt the method proposed in [1], which calculates the Wasserstein distance between action distributions, where xN represents the aggregated action histogram over N steps. The results below show that MAQ+PPO outperforms PPO in both scores and human-likeness, demonstrating that MAQ can generalize beyond Adroit tasks. We would be happy to include these additional experiments in the revision if the reviewer finds them satisfactory.

* Note: DTW and WD are not normalized; lower is better.

[1] Pearce, Tim, et al. "Imitating human behaviour with diffusion models." arXiv preprint arXiv:2301.10677 (2023).

W2: Negative impact on task success (RLPD): The results for MAQ+RLPD suggest MAQ can sometimes negatively impact task performance, especially when the underlying RL agent is highly performant. This needs a more nuanced discussion or a proposed mitigation.

This is a great question! We were also curious about why MAQ+RLPD performed worse than RLPD, especially in the Hammer environment, where the success rate drops from 1.00 to 0.56. After submitting the paper, we conducted a deeper analysis of this task. We find that in the 25 human demonstration trajectories, humans take an average of 451.4 steps to successfully complete the task. However, in the D4RL environment, the agent is restricted to a maximum of 200 steps per trajectory, after which the episode is automatically considered a failure.

Since MAQ learns from human demonstrations, it tends to favor smooth but slow movements, while non-human-like RL agents tend to complete the task more quickly but less naturally. To validate this hypothesis, we increased the maximum episode length from 200 to 450 steps. As shown in the table below, MAQ+RLPD's success rate improves significantly from 0.56 to 0.72 while maintaining human-likeness scores. We will include this analysis with an explanation in the revision. Thank you for highlighting this!

* Note: DTW and WD are not normalized; lower is better.

W3: Insufficient ablation studies: Lack experimental analysis on the (least) amount of human demonstrations required for MAQ-based RL agents, and the the codebook size K in VQVAE.

Regarding the amount of human demonstrations, we conduct an ablation study using different amounts of training data in Door and Pen environments: 90% (as in the paper), 75%, 50%, and 25%. The results below show that while the success rate decreases as the amount of training data is reduced (as expected), the trajectory similarity scores remain consistent. This demonstrates that MAQ can still capture human-like behavior even with as few as 6 trajectories.

* Note: DTW and WD are not normalized; lower is better.

Regarding the codebook size K, please refer to our Appendix. C: "Impact of Codebook Size on Similarity Score in MAQ", which provides a detailed analysis of how different values of $K$ affect performance.

W4: Lack of strong IL baselines: Comparison only to BC and not to state-of-the-art imitation learning or inverse RL baselines.

We include BC as a baseline to represent a simple and widely used method. As the reviewer noted, BC captures human-likeness to some degree but does not optimize with task success rate. We did not include GAIL for several reasons. First, GAIL relies on high-quality expert demonstrations, while our focus is on imitating human behavior, which can be suboptimal but diverse. Second, GAIL is known to require extensive training time to train in high-dimensional continuous control tasks. For example, training GAIL on the 6-dimensional Walker task takes ~25M steps. In contrast, our method learns macro actions for the 24-dimensional Pen task in only 1M steps. We also attempted to train GAIL for 1M steps on the four D4RL tasks, but GAIL failed to produce any successful trajectories, possibly requiring more training and computational resources.

However, we agree that comparing other imitation learning methods would strengthen our results. To this end, we included IQ-Learn [1], one of the strongest IL algorithms. Since IQ-Learn was not originally tested on D4RL tasks, we applied it (using the hyperparameters recommended in the original paper) to all four tasks. The results below show that IQ-Learn performs poorly in both human-likeness and task success, while MAQ+IQ-Learn improves both performance. We believe this additional experiment offers a more convincing comparison and also demonstrates that MAQ is complementary to IL algorithms. We can include this experiment if the reviewer finds it satisfactory.

[1] Garg, Divyansh, et al. "Iq-learn: Inverse soft-q learning for imitation." Advances in Neural Information

* Note: DTW and WD are not normalized; lower is better.

W5: Limited novelty ...

Our contribution lies in how we integrate these existing components to address the underexplored challenge of achieving human-like behavior in RL. We agree with the reviewer that VQVAE and RHC are not novel in themselves. However, to the best of our knowledge, this work is the first to combine macro actions learned from human demonstrations via VQVAE with RHC to achieve both high task performance and human-likeness across multiple RL algorithms.

While previous human-like RL methods often rely on handcrafted behavioral cost functions, our approach offers a general framework that avoids manual design. We believe this represents a meaningful and sufficiently novel contribution, and a significant step forward for the human-like RL community.

Q1: Human evaluation settings …

There are 19 human evaluators in our human study. We will include this information in the revision. Thank you for pointing it out. We will clarify this in the revision. In addition, we follow the similar method used in [2] reporting 96% confidence intervals for both the Turing Test and the ranking test. Thank you again for raising this point.

[2] Tirinzoni, Andrea, et al. "Zero-shot whole-body humanoid control via behavioral foundation models." arXiv preprint arXiv:2504.11054 (2025).

Q2: Additional computational costs …

Since we only have 25 trajectories per task, the VQVAE can be trained in a very short time. Specifically, training VQVAE takes only 3 to 4 minutes, whereas RL training requires several hours. While MAQ-based RL introduces some additional computational overhead, we conduct a comparison of inference time for each RL algorithm, with and without MAQ, as shown in the table below. Overall, the cost remains acceptable.

It is also worth noting that MAQ can offer faster inference efficiency, as it executes an entire macro action (e.g., 9 consecutive actions in the paper) with a single forward pass, whereas vanilla RL must forward at every timestep. In this case, MAQ-based RL can even achieve faster overall inference time than vanilla RL, especially when longer action sequences are used. We will include these results with a detailed explanation of the computational cost in the revision. Thank you for your suggestion.

We thank the reviewer for all the constructive feedback and the time the reviewer put into helping us improve our submission.

We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.

W1: Figures are poorly described, particularly Figure 1, which lacks subfigure explanations and adequate detail.

Thank you for the suggestion. We will revise the caption to include clearer subfigure explanations and more detailed descriptions in the revision.

W2: If right, The evaluation uses only ~2.5 human demonstration trajectories per task, raising concerns about robustness of human likenliness mesures

Yes, due to the limited number of human demonstration trajectories (25 per task), we adopted a 9:1 train/test to maximize the training data available for learning human behavior. As a result, the test data contains only 3 trajectories. To mitigate concerns about robustness, each algorithm in the paper is trained three times using three different randomly selected train/test splits (while maintaining the 9:1 ratio). This reduces the chance of overfitting to any particular test data and helps ensure the evaluation results are reliable and robust.

In addition, to further assess robustness, we conduct an ablation study using different amounts of training data in Door and Pen environments: 90% (as in the paper), 75%, 50%, and 25%. The results below show that while the success rate decreases as the amount of training data is reduced (as expected), the trajectory similarity scores remain consistent. This demonstrates the robustness of our method.

* Note: DTW and WD are not normalized; lower is better.

W3: There doesn’t seem to be a clear observation that MAQ degrades task performance for the highest scoring RL algorithm, RLPD. Difference is even more obious when looking at the appendix reward curves, Figure 6. Q2: RLPD Degradation: In Table 1 and Appendix Figure 6, RLPD seems to suffer significantly when combined with MAQ. Can the authors clarify this discrepancy between reported success rate and observed reward curves? Can you provide more details on how the success rate is computed vs reward curve normalized score

We were also curious about why MAQ+RLPD performed worse than RLPD, especially in the Hammer environment, where the success rate drops from 1.00 to 0.56. After submitting the paper, we conducted a deeper analysis of this task. We find that in the 25 human demonstration trajectories, humans take an average of 451.4 steps to successfully complete the task. However, in the D4RL environment, the agent is restricted to a maximum of 200 steps per trajectory, after which the episode is automatically considered a failure.

Since MAQ learns from human demonstrations, it tends to favor smooth but slow movements, while non-human-like RL agents tend to complete the task more quickly but less naturally. To validate this hypothesis, we increased the maximum episode length from 200 to 450 steps. As shown in the table below, MAQ+RLPD's success rate improves significantly from 0.56 to 0.72 while maintaining human-likeness scores. We will include this analysis with an explanation in the revision.

* Note: DTW and WD are not normalized; lower is better.

Regarding the success rate and normalized reward, in D4RL, the success rate indicates whether the agent successfully completes the task within the episode limit (e.g., 200 steps). In contrast, the normalized reward is a heuristic score defined by the D4RL environment to reflect how close the agent gets to completing the task. For example, in the Door task, the reward increases as the door gets closer to being fully opened. However, only when the door is completely opened, the task marked as successful. Overall, tasks like those in D4RL are primarily evaluated based on whether the agent actually succeeds since the rewards are often used to guide learning, but do not directly indicate task success. This is why we report and compare success rates in the main experiment.

W4: The motivation for human-like behavior is weakly argued and under-cited, beyond bringing in discussion the Turing Test.

We understand the reviewer's concern. Indeed, the topic of human-like behavior in RL has been relatively underexplored, as most of the prior work has focused primarily on improving task performance rather than human-like behavior. This is why we start with the concept of the Turing Test to emphasize the importance of developing agents that are not only effective but also human-like. We will revise the paper to strengthen the motivation for human-like RL and include a Broader Impact section (see the response to W6). If the reviewer has any recommended citations or related works, we would be grateful for the suggestions.

W5: The number of human evaluators is not reported, and no statistical significance analysis is provided for the Turing Test or ranking results. Q1: Statistical Validity of Human Evaluation: How many human evaluators participated in the Turing Test and ranking tests? Please include this number and report significance tests to support the reliability of these results.

Thank you for pointing this out. In our human study, a total of 19 human evaluators participated in the Turing Test and ranking test. We will clarify this in the revision.

In addition, we follow the similar method used in [1] reporting 96% confidence intervals for both the Turing Test and the ranking test. Thank you again for raising this point.

[1] Tirinzoni, Andrea, et al. "Zero-shot whole-body humanoid control via behavioral foundation models." arXiv preprint arXiv:2504.11054 (2025).

W6: The paper lacks a Broader Impact or Societal Impact discussion, especially relevant when aiming to mimic human behavior. Q4: Motivation and Impact: The paper briefly mentions the importance of human-likeness but does not provide a compelling argument or citations. Could the authors expand on the value and implications of designing human-like agents, including societal risks or limitations? The paper does not adequately discuss the limitations or potential societal impacts of building human-like agents. The Discussion section does not meaningfully reflect on possible negative consequences or ethical concerns (e.g., manipulation, human trust, or deceptive agents). I recommend the authors include a Broader Impacts section that honestly addresses these issues and situates their work in a wider context.

Thank you for the suggestion. We will include a Broader Impact section in the revision to address both the potential benefits and risks of human-like agents. The following are several key points we plan to cover. On the positive side, human-like behavior can offer practical benefits in real-world applications, particularly in scenarios involving human-robot collaboration. For example, a robot that assists with human-like movements can improve safety and trust, reducing the risk of accidents. On the other hand, human-like agents may be misused in deceptive or manipulative ways, such as cheating in competitive games. We will include these aspects in the revision. If reviewers feel there are additional aspects worth addressing, we would be happy to include them.

W7: It remains unclear whether demonstrations used in the human-likeness evaluation are successful ones. This could unfairly bias comparisons. Q3: Demonstration Quality: Were the human demonstrations used in the evaluation (e.g., Turing Test) all successful executions? Were agent demonstrations sampled from successful rollouts? This information is crucial for fair comparison.

Yes, all 25 human demonstrations used in our experiments are successful trajectories, as provided in [2]. We will clarify this in the revision. Thank you for pointing it out.

[2] Rajeswaran, Aravind, et al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations." arXiv preprint arXiv:1709.10087 (2017).

Thank you for your replies and for the time and effort to review our paper and provide thoughtful suggestions. We are glad that our clarifications have addressed your concerns. We will make our best effort to incorporate these improvements into the final version.

We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.

W1: The idea of using macro actions is restrictive for RL agents in contrast to prior work that may be leveraged to achieve similar results. Furthermore, the human-likeliness relies on the assumption of compositionality of macro actions, which may or may not hold in general.

We chose to use macro actions based on the assumption that sequences of actions better reflect human behavior. While prior work has attempted to produce human-like behavior using hand-crafted cost functions or reward shaping at the primitive action level, such approaches often require task-specific domain knowledge, limiting their generality. In contrast, our method relies only on human demonstrations and avoids manual task-specific design, making it more applicable across domains.

To address concerns regarding potential restrictions in executing macro actions, we also conducted an ablation study in which agents still learn macro actions, but at test time only execute the first few steps of each macro action to allow more reactive behavior. Specifically, with a macro-action sequence length of 9, we tested executing only the first $j \in${$1,3,5,7$} actions. The table below shows that both human-likeness and performance remain similar across different $j$. Notably, even when $j=1$, the agent remains human-like while increasing reactivity.

* Note: DTW and WD are not normalized; lower is better.

Regarding the compositionality assumption, we agree that this is an assumption. However, our empirical results suggest that the learned macro actions are reasonably composable in practice. We also agree that a study of macro-action compositionality would be a valuable direction for future work.

W2: VQVAE requires expert demonstrations …

We would like to clarify that our proposed MAQ framework does not require expert or optimal human demonstrations. Instead, MAQ is designed to distill human-like behaviors from any human demonstration and integrate them with RL algorithms to improve task performance.

For example, in D4RL [1][2], two types of datasets are provided: (a) a human demonstration dataset (25 trajectories per task) collected from humans, and (b) an expert demonstration dataset (5000 trajectories per task) generated by a fine-tuned expert RL policy. As shown in the table below, the rewards of the human dataset are substantially lower than those of the expert dataset, indicating that the human demonstrations are far from optimal.

In our experiments, MAQ is trained solely on the human dataset, without using any expert dataset. Although these trajectories are not optimal, our results show that MAQ is still able to achieve both human-likeness and high success rates. This shows that MAQ can effectively capture human behavior even from suboptimal demonstrations.

[1] Rajeswaran, Aravind, et al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations." arXiv preprint arXiv:1709.10087 (2017).

[2] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

W3: … the result is highly domain dependent …

We agree with the reviewer that MAQ requires access to human demonstrations in the target domain, as noted in our Discussion section: "Although MAQ exhibits more human-like behavior, one limitation is its reliance on human demonstrations to distill macro actions. The quality of these demonstrations can affect the effectiveness of MAQ. However, since our primary goal is to pursue human-likeness, obtaining human demonstrations is both necessary and appropriate within the scope of this work."

However, we would like to emphasize that, compared to prior human-like RL approaches that rely on domain knowledge to manually design human-likeness cost functions, our method only requires collecting demonstrations. This greatly reduces the effort and cost of adapting to new domains, making MAQ more practical for real-world applications. While extending MAQ to cross-domain generalization remains an interesting direction for future work, we believe our method already represents a significant step in human-like RL by offering a general, data-driven solution that avoids handcrafted, domain-specific constraints.

Q1: Any reason for why prior works that leverage human demonstrations and can be used to generate "human-like" behaviors are not considered or compared, such as GAIL (see below)?

We include BC as a baseline to represent a simple and widely used method. As the reviewer noted, BC captures human-likeness to some degree but does not optimize with task success rate. We did not include GAIL for several reasons. First, GAIL relies on high-quality expert demonstrations, while our focus is on imitating human behavior, which can be suboptimal but diverse. Second, GAIL is known to require extensive training time to train in high-dimensional continuous control tasks. For example, training GAIL on the 6-dimensional Walker task takes ~25M steps. In contrast, our method learns macro actions for the 24-dimensional Pen task in only 1M steps. We also attempted to train GAIL for 1M steps on the four D4RL tasks, but GAIL failed to produce any successful trajectories, possibly requiring more training and computational resources.

However, we agree that comparing other imitation learning methods would strengthen our results. To this end, we included IQ-Learn [1], one of the strongest IL algorithms. Since IQ-Learn was not originally tested on D4RL tasks, we applied it (using the hyperparameters recommended in the original paper) to all four tasks. The results below show that IQ-Learn performs poorly in both human-likeness and task success, while MAQ+IQ-Learn improves both performance. We believe this additional experiment offers a more convincing comparison and also demonstrates that MAQ is complementary to IL algorithms. We can include this experiment if the reviewer finds it satisfactory.

[1] Garg, Divyansh, et al. "Iq-learn: Inverse soft-q learning for imitation." Advances in Neural Information

* Note: DTW and WD are not normalized; lower is better.

Q2: Three different RL methods are chosen … Why is this efficient use of space given that the learned macro actions are essentially treated as actions (assuming my understanding is correct) so it should generalize to whatever standard RL method?

We fully agree with the reviewer's point! As noted, MAQ is fundamentally a framework that transforms actions into learned macro actions, making it to be easily integrated into various RL algorithms. However, we were concerned that reviewers/readers might expect empirical evidence of this generality. Therefore, we included experiments with three representative RL algorithms, IQL, SAC, and RLPD, to demonstrate that MAQ consistently improves human-likeness while maintaining high task success rates across different RL algorithms. Note that our intention is not to compare these RL algorithms themselves, but to show that each benefits from integrating MAQ, thereby highlighting its flexibility and generality. We welcome any suggestions for improving our experimental design.

Q3: … significant variance in the performance across different tasks.

The performance variance is primarily due to inherent differences in task difficulty. For example, the D4RL paper [2] (Table 2) reports that among the four Adroit tasks, Pen and Door are relatively easier, while Relocate (requiring precise grasping, lifting, and moving of an object) is the most challenging. This result also aligns with our experimental results. In summary, the performance variance is not specific to MAQ, but is based on differences in task difficulty.

Q4: Even though MAQ+RLPD has an overall win rate of 71%, Figure 4 still shows that human demonstration dominates it, no?

It is true that when directly comparing human demonstrations against MAQ+RLPD in the ranking test, human demonstrations have a 62% win rate. However, our original claim is based on the overall win rate across all agent pairs. For example, human demonstrations only achieve an 87% win rate against SAC, whereas MAQ+RLPD achieves a 95% win rate against SAC. This suggests that human evaluators found MAQ+RLPD to be more human-like than human demonstrations when both are compared to SAC. Therefore, our intention is to highlight MAQ+RLPD's overall performance in fooling human evaluators across all comparisons, rather than to claim it outperforms human demonstrations directly. We thank the reviewer for the feedback and will revise the phrasing in the revision to clarify this more precisely.

Q5: How do you set the reward for the macro actions?

The reward of macro action is computed as the sum of each step's reward over the entire action sequence. We will include this information in the revision.

We thank the reviewer for the further clarification and for engaging in the discussion!

My first concern was about the assumption of the availability of human demonstrations AS MACRO ACTIONS, not the availability of human demonstrations in general. …

This is a strong assumption since it requires the identification of macro actions (that are reusable and composable) and then soliciting demonstrations for these macro actions. …

We thank the reviewer for the clarification, and we would like to clarify a possible misreading of our method (we apologize if we misunderstood the reviewer's concern). Our approach does not require identifying macro actions first and then collecting human demonstrations specifically for those predefined macro actions. Instead, we extract fixed-length segments (9 in the paper) from the human demonstrations. Given each state $s$ in a demonstration trajectory, we define a macro action $m$ as the sequence of 9 consecutive primitive actions starting at $s$ (see Figure 1). As a result, every state in the training data is associated with a macro action (a human-like segment). In summary, we first collect human demonstrations, and then macro actions are derived from these trajectories.

Regarding whether these macro actions are reusable and composable, as mentioned in our previous response: since they are extracted from human demonstrations, they inherently contain the behaviors necessary to complete the task.

We also agree with the reviewer that using segmentation methods from option learning [1, 2, 3] to determine variable-length macro actions per state is an interesting direction. However, in the work, our primary focus is not on the segmentation strategy itself, but rather on demonstrating that macro actions derived directly from human demonstrations can allow RL to achieve both a high task success rate and human-likeness. We appreciate this suggestion and will include it in our discussion of future work.

[1] Kipf, Thomas, et al. "CompILE: Compositional imitation learning and execution." ICML 2019.

[2] Chang, Yi-Hsiang, et al. "Reusability and transferability of macro actions for reinforcement learning." ACM TELO 2022.

[3] Konidaris, George, et al. "Constructing skill trees for reinforcement learning agents from demonstration trajectories." NeurIPS 2010.

My concern about the compositionality was due in part to the use of the RL agent. …

Our current method is developed under the standard RL setting where the human demonstrations and the learned RL agents operate within the same MDP. Namely, the environment dynamics and the task specification are identical.

We agree with the reviewer that handling environment perturbations or generalizing across different environments is a very interesting direction. To achieve such generalization, the MAQ framework could be extended to a meta-RL setting [4, 5], where the agent is provided with a distribution over several tasks during training (e.g., tasks involving obstacle avoidance), along with demonstrations from each task. This setup would allow the agent to learn reusable and adaptive macro actions across multiple tasks. In this way, the agent can generalize to similar but unseen tasks during testing (e.g., an environment with an unseen delicate obstacle). We appreciate this suggestion and will include it in our discussion of future work.

[4] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." ICML 2017.

[5] Rakelly, Kate, et al. "Efficient off-policy meta-reinforcement learning via probabilistic context variables." ICML 2019.

We hope the above explanation addresses the reviewer's concerns, and we would be happy to engage in any further discussion to improve our submission.

We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.

W1: The proposed framework would require expert human or optimal demonstration data in order to encode near-human-like behaviors

We would like to clarify that our proposed MAQ framework does not require expert or optimal human demonstrations. Instead, MAQ is designed to distill human-like behaviors from any human demonstration and integrate them with RL algorithms to improve task performance.

For example, in D4RL [1][2], two types of datasets are provided: (a) a human demonstration dataset (25 trajectories per task) collected from humans, and (b) an expert demonstration dataset (5000 trajectories per task) generated by a fine-tuned expert RL policy. As shown in the table below, the rewards of the human dataset are substantially lower than those of the expert dataset (except Pen), indicating that the human demonstrations are far from optimal.

In our experiments, MAQ is trained solely on the human dataset, without using any expert dataset. Although these trajectories are not optimal, our results show that MAQ is still able to achieve both human-likeness and high success rates. This shows that MAQ can effectively capture human behavior even from suboptimal demonstrations.

[1] Rajeswaran, Aravind, et al. "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations." arXiv preprint arXiv:1709.10087 (2017).

[2] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

W2: The paper assumes that the human-demo are optimal trajectories and in that case, it might miss the intent of the task if they are near-suboptimal.

Following our previous response, MAQ does not require optimal trajectories. However, we agree with the reviewer that training on suboptimal demonstrations may affect task performance. We have also noted this in the Discussion section: "The quality of these demonstrations can affect the effectiveness of MAQ."

To investigate the impact of demonstration quality, we conduct an ablation study using different subsets of trajectories with the lowest task rewards. Specifically, we split the training/testing dataset based on the reward of each trajectory and evaluate three scenarios:

• 75% low-reward: train on 75% lowest-reward trajectories and test on the remaining 25% highest-reward ones
• 50% low-reward: train on 50% lowest-reward trajectories and test on the remaining 50%
• 25% low-reward: train on 25% lowest-reward trajectories and test on the remaining 75%

Since reducing the size of the training dataset will affect the performance, we also include an experiment using the same data proportions but with trajectories selected randomly. The results are shown below.

* Note: DTW and WD are not normalized; lower is better.

As expected, training with fewer demonstrations reduces performance in both low-reward and random settings. Moreover, training on low-reward trajectories consistently leads to lower success rates compared to random. This demonstrates that the quality of demonstrations affects task performance. However, DTW and WD scores remain consistent across all scenarios, even when training on the lowest-reward trajectories. This suggests that MAQ can still effectively capture human behavior from suboptimal trajectories. We would be happy to include this additional ablation study in the revision if the reviewer finds it satisfactory.

W3: The paper does not talk about any dynamic environments where the frequency of executing actions might play an important role in having more reactive policies and yet human-like.

To evaluate using MAQ in dynamic environments, we conduct an additional experiment on Enduro, a dynamic Atari environment with rapidly changing scenes, frequent decision-making, and the need for real-time reactions to avoid collisions.

We train both PPO and MAQ+PPO on Enduro. For MAQ, the macro-action sequence length is set to 9. To assess human-likeness in the discrete action space, we adopt the method proposed in [3], which calculates the Wasserstein distance between action distributions, where xN represents the aggregated action histogram over N steps. The results below show that MAQ+PPO outperforms PPO in both scores and human-likeness, demonstrating that MAQ can effectively operate in dynamic environments while preserving human-like behavior.

* Note: DTW and WD are not normalized; lower is better.

[3] Pearce, Tim, et al. "Imitating human behaviour with diffusion models." arXiv preprint arXiv:2301.10677 (2023).

Q1: How are we making sure that we have proper coverage of the state-action pair for human demonstrations for different experiments to have effective/generalized quantization?

To investigate the coverage of state-action pairs from human demonstrations, we analyze whether actions from testing trajectories appear within the VQVAE codebook. Specifically, for each state-macro action pair $(s, m)$ from the testing trajectories, we input the state $s$ into the trained VQVAE decoder to reconstruct all macro actions corresponding to the codebook entries. Then, we check whether the macro action $m$ appears among these decoded outputs.

Since D4RL uses continuous actions, we calculate the L2 distance between ground-truth macro action $m$ and each decoded macro action $\tilde{m}$. If the distance to any codebook entry is below a predefined threshold $t$, we consider $m$ to be covered by the codebook. The table below shows the evaluation results. For comparison, we also applied the same analysis to expert trajectories, which do not exhibit non-human-like behavior. The results demonstrate that our VQVAE trained on human demonstrations achieves high coverage on human trajectories but very low coverage on expert trajectories.

Q2: Any particular way/frequency of updating the codebook (clusters) or having more robust codebooks that don't change drastically with the new information.

We adopt a commonly used technique in VQVAE: codebook reinitialization [4], which helps maintain effective codebook usage and avoids codebook collapse. Specifically, if certain codebook entries remain unused for a period, they are reinitialized with the most frequently used embeddings. This strategy ensures that the codebook remains robust.

[4] Williams, Will, et al. "Hierarchical quantized autoencoders." Advances in Neural Information Processing Systems 33 (2020): 4524-4535.

Q3: Line 123: Are there any ablations for j to have reactivity vs human-likeness metrics comparison?

We conduct an experiment in the Pen environment using MAQ+RLPD with different values of $j$, including 1, 3, 5, and 7. The table below shows that both human-likeness and performance remain similar across different $j$. This is likely because $H$ (action sequence length) is set to 9 during MAQ training, so even when $j=1$, the agent is still able to execute human-like behavior.

We choose $j=H$ in the paper because longer sequences tend to generate smoother, uninterrupted human-style motion. In addition, this setting reduces the replanning frequency, which improves training efficiency.

* Note: DTW and WD are not normalized; lower is better.

Q4: Equation 4: What is this loss term ||m-~m||^2? Is this the L2 norm over the action sequences or log-likelihood?

We use the L2 norm over the action sequences.

Dear AC and Reviewers,

Thank you for your efforts and time in handling and reviewing our papers. We are willing to clarify any further questions or concerns during the remaining author-reviewer period.

Sincerely,

Authors