Reinforcement Learning with Action Chunking

Thank you for your insightful comments and suggestions on our paper. We addressed your main concern about our experiments by evaluating our method, all our baselines (11 baselines, with 3 new baselines) on 25 OGBench tasks (5 tasks for each of the 5 domains). In contrast, FQL evaluates 15 tasks for the offline-to-online setting (1 task for each domain). Our method outperforms all baselines across the board. In addition, we apply action chunking to two of our new baselines (BFN and IFQL) which yield two new variants of our method (Q-LAC-BFN, Q-LAC-IFQL) that also significantly improves the corresponding baselines. In particular, Q-LAC-BFN achieves strong performance across the board on par and sometimes better than Q-LAC. Please see the summary of our new results in our response to Reviewer nCbd.

We believe that there are also several misunderstandings of the reviewer on several key results of our paper that we would like to clarify below:

“What accounts for this difference in performance for FQL compared to the original FQL paper?”

Our results in Figure 4 were an aggregation of both task2 and task4. In contrast, for the offline-to-online RL experiments, the FQL paper only reported performance on the representative task for each domain (e.g., task2 for cube-double). The significantly harder task (e.g., task4) was not evaluated. For the rebuttal, we reran all the experiments on all 5 tasks for each domain, and our reported performance matches the FQL paper on the same task (e.g., task2 on cube-double).

“In Figure 4, in half of the environments, FQL-n performs better than FQL … These results seem to contradict the claimed benefit of action chunking.” “As shown in Figure 4, in about half of the tasks, RLPD achieves performance equal to or better than Q-LAC.”

The results that show FQL-n performs better than FQL on these environments do not contradict the claimed benefit of action chunking. Instead, they show evidence that n-step return does help speed up value learning and Q-LAC uses chunking to enjoy a similar benefit of n-step return backup. In our new results, Q-LAC performs significantly better than FQL-n especially on the hardest domains (cube-triple and cube-quadruple). In terms of RLPD, while it is true that RLPD is a competitive baseline as it can match the online performance on the simpler tasks, it fails on the hardest cube-quadruple domain. In our newest experiments, both FQL and RLPD fail to solve all but one task in cube-quadruple domains. In contrast, Q-LAC solves 4 out of 5 tasks on cube-quadruple.

“Q-LAC performs worse than the RLPD baseline in pure online settings” / “Results don't convincingly demonstrate that Q-LAC's performance is due to action chunking or behavior regularization. It seems more likely that the offline training provides a better starting point for the policy”

On the hardest domain cube-quadruple, Q-LAC exhibits strong performance (solving 4/5 tasks) online with offline pre-trained performance being 0. On the other hand, RLPD methods do not succeed on any of these tasks. This provides strong evidence that the effectiveness of Q-LAC also comes from its ability to learn online effectively rather than just the advantage of being able to pre-train offline as suggested by the reviewer. We would also like to clarify a couple of misunderstandings: (1) None of our methods are evaluated in the pure online setting. All the methods leverage an offline dataset. (2) Figure 2 shows Q-LAC outperforms RLPD, RLPD-AC and RLPD-ACBC after an initial pre-training phase which is opposite to what you seem to suggest in your review. (3) Our paper focuses on the offline-to-online setting which is common to use a pre-training phase [1-3].

[1] "Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning."

[2] "Flow Q-learning."

[3] "Adaptive policy learning for offline-to-online reinforcement learning."

“RLPD consistently outperforms RLPD-AC5. These results seem to contradict the claimed benefit of action chunking.”

RLPD-AC5 is not our method; the figure shows that naively using action chunking (RLPD-AC5) to an online RL algorithm (RLPD) does not work well. Our key insight is that we need to use an expressive behavior constraint (e.g., behavior constraint on an expressive policy) to regularize the RL training (see Section 4.2). RLPD-AC5 (1) uses a Gaussian policy (much less expressive than a flow policy that is used in Q-LAC), and (2) does not use a behavior constraint whereas our method, Q-LAC, is based on FQL that deploys a behavior constraint.

“[action chunking policy] sacrifices reactivity and granularity, and may even be viewed as a degenerate case of [1-step action policies]” / “[1-step action policy] has finer granularity and higher real-time responsiveness”

Our action chunking policy is not a degenerate case of 1-step action policies. Our action chunking policy can not be fully represented by a 1-step action policy because action chunking policy can produce non-Markovian behavior that is not possible to achieve by a 1-step action policy. While 1-step action policy has higher real-time responsiveness, it also exhibits worse temporal coherency in our experiments (see our response to Reviewer kQoc). Temporally correlated/non-Markovian noises have been known for improving online exploration [1, 2], and our method provides a convenient way to generate temporally coherent actions while also being informed by offline data as prior. While it is true that our action chunking policy can only represent a subset of all non-Markovian policies, in practice, we found that using an action chunking policy (Q-LAC) still consistently outperforms 1-step action policy (e.g., FQL).

[1] "Pink noise is all you need: Colored noise exploration in deep reinforcement learning."

[2] "Continuous control with deep reinforcement learning."

Action chunking policies are limited as they cannot be switched and history-conditioned policies are more favorable. “Advantage of action chunking over 1-step action policies and skill-based approaches are not convincing”

While there are limitations to action chunking policies, they do not diminish their strong empirical performances across the board (see the table in our response to Reviewer nCbd). We show in our new experiments that Q-LAC performs better than all methods that use 1-step policies (FQL, RLPD, BFN) and the skill-based approach (SUPE-GT, see the description in our submission). We additionally include two variants of Q-LAC, Q-LAC-BFN and Q-LAC-IFQL, a result of applying action chunking to two newly added offline-to-online RL baselines, BFN and IFQL (see more details in our response to Reviewer nCbd). As we have shown in the result table, action chunking consistently improves the non-chunking counterpart, supporting our claim that introducing action chunking into RL leads to improved performance.

“action chunking policy proposed in this paper is an interesting idea, but it lacks sufficient theoretical support”

We are happy to hear that you think the action chunking policy proposed in this paper is an interesting idea. To provide more theoretical rigor and as recommended by other reviewers, we have included a theorem and proof below that shows the value backup for our chunked Q-function is unbiased. Please refer to our response to Reviewer joyS for more details on the theorem and discussions.

Thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Does action chunking help with offline learning or does it help with online learning?

We do not have direct evidence that action chunking helps with offline RL. From the table, we can see that aggregated over 25 tasks, in terms of offline RL performance, Q-LAC performs almost the same as FQL (38 vs. 37), Q-LAC-BFN performs almost identical to the base algorithm BFN (52 vs. 51), and Q-LAC-IFQL performs similar to the base algorithm IFQL (39 vs. 36). The main focus of our paper is the offline-to-online setting where we evaluate all the methods at the end of the 1M offline + 1M online training steps, and we see that all of our methods improve upon the base algorithms significantly online: Q-LAC vs. FQL (86 vs. 58), Q-LAC-BFN vs. BFN (86 vs. 63), and Q-LAC-IFQL vs. IFQL (59 vs. 47). This provides another evidence to support our claim that Q-LAC explores more effectively online (with higher temporal coherency in actions – please see more details in our response to Reviewer joyS, "How does action chunking policy’s actions deviate from non-chunked policy?"). While it is possible that action chunking can help with offline RL on harder tasks beyond the benchmark that we consider, we leave that to future work.

I wonder if starting from the Q-LAC's offline policy, but following the RLPD online learning process, what would happen?

This would be a really nice comparison that we would have run in our experiments but unfortunately it is not possible because RLPD uses Gaussian policies and our method uses flow policies. It is not a trivial task to convert the flow policies into Gaussian policies while still preserving their effectiveness. Along this line, we additionally compared our methods with the closest offline-to-online RL baseline that we could find in the prior work that shares many similar designs as RLPD, ReBRAC (Gaussian policy, DDPG-style policy extraction). We have found that even with offline pre-training, ReBRAC fails to outperform Q-LAC and only achieves a final aggregated success rate of 50% (vs. Q-LAC/Q-LAC-BFN at 86%).

Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating? If there are specific experiments/comparisons that you would like to see, please let us know!

Thanks for your response and raising the score! We are glad that our previous response was able to address most of your concerns! For the remaining concerns, we address below:

Novelty concern

While using (1) expressive policies and (2) behavior constraints themselves are not novel, the application of them to the action chunking in the context of offline-to-online RL is new and drives the performance of multiple existing offline-to-online RL methods upward significantly (Q-LAC vs. FQL [86 vs. 58], Q-LAC-BFN vs. BFN [86 vs. 63], and Q-LAC-IFQL vs. IFQL [59 vs. 47]). The fact that we have identified these general two design principles (expressive policies and behavior constraints) that make action chunking effective in offline-to-online RL is a valuable insight for the community that has the potential to generalize to other offline-to-online RL methods (different from the three we tested on [FQL, BFN and IFQL]). This aligns with what Reviewer nCbd has noted with a positive stand on the paper, "the fact that action chunking didn't work with RLPD is enough for the current submission to be interesting".

The authors’ attempt to apply chunking to online RL was not successful, leading them to turn to the offline-to-online RL setting.

We believe that the observation that naively applying action chunking to online RL does not work is a valuable insight for the community (as Reviewer nCbd also found this interesting). Our work also shows that action chunking does not directly apply to existing purely online RL methods because they do not use (1) expressive policies and (2) behavior constraints.

More importantly, our work focuses on the offline-to-online RL setting (not just proposing a general RL algorithm). Our main goal is to demonstrate the effectiveness of action chunking for improving existing offline-to-online RL methods. Our choice of specialization the offline-to-online RL is not unique in the literature. There are many existing methods that are also specifically designed for the offline-to-online RL setting [1, 2]. While we focus on the offline-to-online RL problem setting, we believe that the key design principles that we identified would be a valuable insights for future work to develop effective offline-to-online and potentially even purely online RL algorithms by leveraging the principles in our paper. We believe that our work is a valuable step in the direction of tractably learning non-Markovian policies (in the form of action chunking policies) in RL for effective online exploration and improved sample efficiency.

[1] Nakamoto, Mitsuhiko, et al. "Cal-QL: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2023): 62244-62269.

[2] Zhou, Zhiyuan, et al. "Efficient online reinforcement learning fine-tuning need not retain offline data." arXiv preprint arXiv:2412.07762 (2024).

Thank you again for your insightful feedback and detailed review! Hope this response addresses your remaining concerns!

Thanks for your detailed review and insightful comments. We addressed your concern on the lack of analysis and discussion by adding (1) theoretical results that show Q-LAC performs unbiased n-step return backup, (2) empirical analysis that shows Q-LAC’s policy exhibit better temporal coherency compared to the non-chunked baseline, highlighting the effectiveness of action chunking for online exploration, (3) empirical analysis that quantifies the value speedup effects of n-step returns, (4) a more complete ablation study for the effect of action chunk size where we show when a too big of an action chunk size starts to hurt performance, and (5) a detailed analysis of the CPU/GPU memory overhead and computational cost overhead of our action chunking method Q-LAC.

“No formal analysis is provided about unbiased value backup”

To address your concern, we have included a theorem with proof below that shows the value backup for our chunked Q-function is unbiased.

[Theorem: Q-LAC performs unbiased n-step backup]

Let $s_t, a_t, s_{t+1}, a_{t+1}, \cdots, s_{t+n}$ be a trajectory segment generated by following a data collection policy $\pi_\beta(\cdot | s)$ and $r_t, r_{t+1}, \cdots r_{t+n-1}$ be the reward received at each corresponding time step. Let $V^\pi(s_t)$ be the value for a policy $\pi$ starting from state $s_t$. Let $Q^\pi(s_t, a_t, a_{t+1}, \cdots, a_{t+n-1})$ be the Q-value for a policy $\pi$ starting from state $s_t$ and executing the action sequence $a_t, a_{t+1}, \cdots, a_{t+n-1}$.

Under any policy $\pi$ (possibly different from $\pi_\beta$), the $n$-step return estimate $\hat{V}\_{n} := \sum_{t'=t}^{t+n-1}\left[\gamma^{t'-t}r_{t'}\right] + \hat{V}(s_{t+n})$ is unbiased for $Q^\pi(s_t, a_{t}, \cdots, a_{t+n-1})$ as long as $\hat{V}(s_{t+n})$ is unbiased for $V^\pi(s_{t+n})$.

[Proof]

From the definition of $Q^\pi(s_t, a_t, a_{t+1}, \cdots, a_{t+n-1})$, we can write it as an expectation: Q^\\pi(s_t, a_t, a_{t+1}, \\cdots, a_{t+n-1}) = \\mathbb{E}\_{s\_{t'} \sim T(\\cdot | s\_{t'}, a\_{t'})} \\left[\\sum\_{t'=t}^{t+n-1}\\left[\\gamma^{t'-t}r(s\_{t'}, a\_{t'})\\right] + V^\pi(s\_{t+n})\\right],

$= \\mathbb{E}\_{s\_{t+n}, r\_{t+1}, r\_{t+2}, \cdots, r\_{t+n-1}}\\left[\\sum\_{t'=t}^{t+n-1}\\left[\\gamma^{t'-t}r\_{t'}\\right] + V^\pi(s\_{t+n})\\right]$

= \\mathbb{E}\_{s\_{t+n}, r\_{t+1}, r\_{t+2}, \cdots, r\_{t+n-1}}\left[\sum\_{t'=t}^{t+n-1}\\left[\\gamma^{t'-t}r\_{t'}\\right] + \\hat{V}(s\_{t+n})\\right]

$= \mathbb{E}\left[\hat{V}_{n}\right]. \square$

The second line uses the fact that the rewards $r_t, r_{t+1}, \cdots, r_{t+h-1}$ are associated with the trajectory segment $s_t, a_t, \cdots, s_{t+n}$. The third line uses the fact that $\hat{V}(s_{t+n})$ is unbiased for $V^\pi(s_{t+n})$. The fourth line uses the definition of $\hat{V}_{\mathrm{n\text{-}step}}$.

The implication of this theorem is that if we use the n-step return backup for the chunked Q-function $Q(s\_t, a\_t, a\_{t+1}, \cdots, a\_{t+n-1})$ (with a value estimate of $\hat{V}\_{s} \\leftarrow Q(s\_t, a\_t, a\_{t+1}, \\cdots, a\_{t+n-1})$) to convergence it converges to the ground truth $Q$-value under the same policy $Q^\pi(s\_t, a\_t, a\_{t+1}, \\cdots, a\_{t+n-1})$. In contrast, the n-step return backup on the original action space is known to not converge to the correct $Q$-value [1,2].

This might be unsurprising because we are effectively treating the action chunk as the new action space and run RL with this transformed action space. Logically, if there is any RL algorithm that is guaranteed to converge to the optimal 1-step policy in the original action space, running the same algorithm on this transformed action space will also converge to the optimal action chunking policy. While it is true that the optimal action chunking policy cannot represent the optimal 1-step policy in some cases, in our experiments, we have found them to be sufficient (see the new results in our response to Reviewer nCbd).

[1] "Revisiting Peng’s Q($λ$) for Modern Reinforcement Learning."

[2] "Revisiting fundamentals of experience replay."

How does action chunking policy’s actions deviate from non-chunked policy?

To address your question, we analyze the temporal coherency of the actions generated by the Q-LAC policy and the FQL policy in the beginning of online fine-tuning on task cube-triple-task2 (4 seeds). In particular, we look at the distance between the end-effector positions five time steps apart from our Q-LAC policy compared to FQL policy (e.g., $\\|\mathbf{x}^{\\mathrm{eef}}\_t - \\mathbf{x}^{\\mathrm{eef}}\_{t+5}\\|\_2$). A higher distance suggests a higher temporal coherency in the actions taken by Q-LAC and FQL policy. As shown in the table below, Q-LAC exhibits a higher temporal coherency, allowing it to explore in the environment better, and better online sample efficiency (see our new results in our response to Reviewer nCbd).

“The ablation experiment on action chunking size is incomplete.”

We ran a complete ablation study on more tasks with longer chunk sizes. In our original submission, we only ran on cube-triple-task4. Here we show an aggregation of all five cube-triple tasks to give a more complete picture. As shown in the figure, the final performance saturates at around $h=10$. When the action chunk size is too large ($h=50$), the learning fails. This is to be expected as the action prediction for longer chunks of actions is harder, and TD backup n-step return with a large $n$ reduces the stitching capability of Q-learning (e.g., the extreme case is Monte-Carlo regression where the algorithm can no longer do any stitching).

“The article lacks analysis on whether action chunking will bring additional memory or time overhead.”

To address your concern, we analyzed the time and memory that action chunking brings and found that action chunking incurs almost no overhead on CPU & GPU memory and some but manageable overhead on the computational cost especially for online learning where environment steps are still the main bottleneck.

Memory

Action chunking brings little memory overhead on both the CPU memory and GPU memory. For CPU memory, our replay buffer stores the transitions normally and we sample chunks of observations and actions for updating our agent. For GPU memory, action chunks make the input to the policy and the critic larger which make the first layer of our neural network (and the last layer of our neural network for the flow policy) larger. We include the parameter count below:

• Q-LAC: ~5M parameters
• FQL: ~4.9M parameters
• RLPD: ~17.2M parameters

Time

In terms of time, we report the time it requires for each step during the offline phase and the online phase below (in milliseconds). Compared to FQL, Q-LAC is ~40% more expensive offline, and only 10% more expensive in the online phase as the environment step takes most of the time during online training.

“The article mentions that the learning of value is accelerated, but there is no quantified indicator or experiment to measure the extent of this acceleration.”

To address your question, we analyzed the convergence of the average Q-value of the state-action pair in the dataset over the course of offline training for Q-LAC, FQL-n and FQL on one of our hardest tasks, cube-quadruple-100M-task2 (averaged over 4 seeds). To gain insights on the convergence rate of the Q-value, we compute the difference in Q-value from the previous time step and highlight the difference if it is greater than 10. Please refer to our response to kQoc for these results.

Both Q-LAC and FQL-n (FQL with n-step returns) have large Q-value differences in the first 100K to 150K steps and then have small Q-value differences for the rest of the offline training. In contrast, FQL has large Q-value differences for a longer time up to almost half way through the offline training (~400K steps). This suggests that both Q-LAC and FQL-n enjoy a faster convergence of the Q-value compared to FQL, providing evidence that n-step backup does accelerate value learning. It is worth highlighting that while this quantifies the value learning speed, it does not quantify how well the value is learned. In fact, in our experiments, even though FQL-n exhibits a faster value learning, it suffers from an online collapse problem (as shown here), possibly due to the bias in the n-step backup. In contrast, Q-LAC does not suffer from this issue, possibly due to its unbiased n-step backup (see more discussion in Section 4.3 Benefit #1).

Thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Thanks for the insightful feedback and detailed review. We discuss the novelty of our contribution below. Our main contribution is to introduce effective action chunking into RL, and while this modification might seem simple, our experiments show that it is highly impactful. Furthermore, during the rebuttal, we additionally show that the same novel contribution can be applied to multiple different RL algorithms, further supporting our claim that introducing action chunking into RL leads to improved performance. Please see our response to nCbd for the summary table of our new results.

“The core contribution appears largely incremental”

Our main contribution is to introduce effective action chunking into RL, and while this modification might seem simple, our experiments show that it is highly impactful. For example, FQL fails to solve all but one task in cube-triple and cube-quadruple domains. In contrast, Q-LAC solves 4 out of 5 tasks on both domains. In addition, getting action chunking to work in RL is not trivial. Furthermore, during the rebuttal, we additionally show that the same novel contribution can be applied to multiple different RL algorithms (Q-LAC [FQL], Q-LAC-BFN [BFN], Q-LAC-IFQL [IFQL]), further supporting our claim that introducing action chunking into RL leads to improved performance.

“Does action chunking generalize as a performance enhancer when integrated with ​​other​​ offline-to-online RL algorithms?”

Yes it does! For this rebuttal, we include two variants of Q-LAC, Q-LAC-BFN, a result of applying the same principles to a new offline-to-online RL algorithm and Q-LAC-IFQL, a result of applying the same principles to a offline-to-online RL baseline IFQL (as considered in the FQL paper). Action chunking consistently improves the non-chunking counterpart (e.g., Q-LAC vs. FQL, Q-LAC-BFN vs. BFN, Q-LAC-IFQL vs. IFQL), supporting our claim that introducing action chunking into RL leads to improved performance. Please see the summary of our new results in our response to Reviewer nCbd.

“Comprehensive performance metrics (e.g., aggregated scores across all tasks) are absent”

Our summary table (see our response to Reviewer nCbd) now contains the aggregated scores across all tasks.

“scene task results Q-LAC exhibits the highest variance among baselines, potentially undermined its online performance.”

We have fixed this in our newest experiments where Q-LAC can succeed 100% for all scene and puzzle-3x3 tasks online. In general, we have found that the original OGBench scene and puzzle-3x3 domains have high variances because the success rate is misaligned with the reward function (a shaped reward function that provides some reward for each sub-task finished). In some cases, succeeding in a task requires the agent to un-finish some of the sub-tasks. To eliminate any confounding factor from poor reward shaping, in all our new experiments, we use a sparse-reward function (-1/0) for both scene and puzzle-3x3 domains where a reward of -1 is given as long as the overall task has not been completed (regardless of the sub-task completion). With this modification, all the methods exhibit a much smaller variance. In addition, we now test on all five tasks rather than only two tasks for each domain.

“The manuscript lacks substantive discussion or illustration explaining why action chunking is particularly well-suited to non-Markovian settings.”

To address your concern, we analyze the temporal coherency of the actions generated by Q-LAC and FQL. In particular, we look at the distance between the end-effector positions five time steps apart from our Q-LAC policy compared to FQL policy. A higher distance suggests a higher temporal coherency in the actions taken by Q-LAC and FQL policy. Please see our response to joyS ("How does action chunking policy’s actions deviate from non-chunked policy?") for the result table. As shown in the table, Q-LAC exhibits a higher temporal coherency, allowing it to explore in the environment better, and better online sample efficiency

“Formal theoretical analysis is absent. Such analysis would significantly strengthen claims regarding the method's effectiveness and robustness.”

For theoretical rigor, we added a theorem that shows our method performs unbiased n-step return backup. Please refer to our response to Reviewer joyS for more details.

“Why does the action chunking formulation yield limited performance improvements in strongly Markovian or fully Markovian situation?”

We are unsure how to best interpret this question as it is unclear to us which limited performed improvements that you are referring to. Here is our interpretation – “why does action chunking yields limited performance improvement on scene and puzzle-3x3”. In our new experiments (see our response to nCbd), we found that most baselines already perform quite well on both scene and puzzle-3x3 at around 100% success rate for most of the tasks. We believe that the limited performance improvements are due to the fact that these two domains are already saturated by existing algorithms. If we interpret it wrong, could you please clarify the question?

“Equation 8 claims that action chunking avoids bias in value estimation. What is the associated impact on estimation variance, and how does this affect the stability and convergence of the learning process?”

To address your question, we analyzed the convergence of the average Q-value of the state-action pair in the dataset over the course of offline training for Q-LAC, FQL-n and FQL on one of our hardest tasks, cube-quadruple-100M-task2 (averaged over 4 seeds). We compute the difference in Q-value from the previous time step and highlight the difference in red if it is greater than 10.

Both Q-LAC and FQL-n (FQL with n-step returns) have large Q-value differences in the first 100K to 150K steps and then have small Q-value differences for the rest of the offline training. In contrast, FQL has large Q-value differences for a longer time up to almost half way through the offline training (~400K steps). This suggests that both Q-LAC and FQL-n enjoy a faster convergence of the Q-value compared to FQL, providing evidence that n-step backup does accelerate value learning. It is worth highlighting that while this quantifies the value learning speed, it does not quantify how well the value is learned. In fact, in our experiments, even though FQL-n exhibits a faster value learning, it suffers from an online collapse problem (as shown here), possibly due to the bias in the n-step backup. In contrast, Q-LAC does not suffer from this issue, possibly due to its unbiased n-step backup (see more discussion in Section 4.3 Benefit #1).

“How does Q-LAC scale to environments with high-dimensional action spaces? Does effective Bellman updating with chunked high-dimensional inputs necessitate more complex Q-network architectures, and what computational trade-offs arise?”

While it is possible that the chunked action space requires a more complex Q-network, in our experiments, we find that using the same architecture (4 hidden layers with 512 units each) as the non-chunked FQL is sufficient to solve all our tasks. To provide more insights for the scalability of our method, we analyzed the time and memory that action chunking brings and found that action chunking incurs almost no overhead on CPU & GPU memory and some but manageable overhead on the computational cost especially for online learning where environment steps are still the main bottleneck. This suggests that Q-LAC can be easily applied to existing RL algorithms without requiring large overhead on memory and time, making it scalable for more complex problems. Please respond to Reviewer joyS for more details on the memory and time overhead of action chunking.

Thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Thanks for your detailed review and insightful comments. We addressed your main concern on (W3) empirical analysis by conducting a more comprehensive set of experiments (11 baselines and 25 tasks in total) and additional ablation studies requested by the reviewer on FQL-n, (W1-W2) novelty by clarifying the key principles behind our method Q-LAC that differentiate us from prior offline-to-online RL methods (e.g., FQL) and skill-based/macro-action-based methods.

New results! (3 Q-LAC methods + 11 baselines on 25 OGBench tasks)

The newly experiments are summarized below (3 seeds for SUPE-GT and RLPD-AC-BC and 4 seeds for everything else). We report the success rate (%) after 1M offline training steps followed by 1M additional steps of online fine-tuning (offline $\rightarrow$ online). Our methods, Q-LAC and Q-LAC-BFN, outperform other baselines by a large margin across the board.

For each row, we bold each number if its confidence interval overlaps with the best method’s confidence interval. Due to the character limit, we omit the confidence interval in the table and the performance on the individual task, but we are happy to provide them during the discussion period.