Real-Time Execution of Action Chunking Flow Policies

Thank you for your in-depth review! We have attempted to address your concern about computational costs by providing more details, which we will also add to the Appendix in a future revision (in addition to the existing latency breakdown in the Appendix). We have also provided some clarifications about the experiments. Please let us know if you have any other concerns!

In Figure 5, RTC and RTC with hard masking show similar performance when inference delay is zero or high (d=4). Why does soft masking provide no benefit in these cases?

When $d = 0$, hard masking is equivalent to disabling RTC entirely, which performs well because there is no delay. When $d = 4$, hard masking is equivalent to soft masking, because the size of the region where soft masking would normally apply is $H - s - d = 0$.

The paper provides limited discussion of failure modes or scenarios where RTC might struggle. Understanding when and why the method fails would help practitioners decide when to apply it and guide future improvements.

Across our simulated and real tasks, we did not find scenarios where RTC struggles (compared to synchronous baselines), so we would be confident recommending it as a reasonable default strategy.

Can you provide detailed timing breakdowns comparing RTC to standard inference? How does the overhead scale with action dimension, prediction horizon, and number of denoising steps?

The RTC overhead comprises an approximate 50% slowdown of each denoising step. Thus, the overhead is linear in the number of denoising steps, independent of action dimension and prediction horizon. The total overhead depends on the model architecture — in Pi05, with 5 denoising steps, the denoising loop is approximately 20% of the total inference time without RTC, and 30% with RTC. We will add these details to the Appendix, as well as a more detailed table breaking down how much latency each component of the Pi05 architecture contributes (with and without RTC).

What are the fundamental barriers to applying RTC to autoregressive VLAs? Have you explored any modifications that could extend the approach beyond diffusion/flow models?

The fundamental barrier is the lack of inpainting ability. If the actions were represented in standard chronological order, then one could simply “prompt” the model with the first $d$ frozen actions. However, [1] found that this standard action representation does not work well for high-frequency control, and their solution foils a naive prompting strategy. It is possible that alternative inpainting methods for autoregressive models exist, but we have not explored this direction yet.

[1] “FAST: Efficient Action Tokenization for Vision-Language-Action Models.” Pertsch et al.

Thank you for your in-depth review! We have attempted to address your question about inpainting strategies by adding a simulated ablation: it shows that RTC is still valuable with simpler inpainting strategies, but the work we have done to adapt and tune a state-of-the-art method is critical for maximum performance. Additionally, we have clarified the impact of additional computational overhead and sudden perturbations. Please let us know if you have any other concerns!

However, in the limitations section, the authors primarily acknowledge the added computational overhead introduced by their method. I encourage the authors to further examine and report on whether this overhead affects other critical aspects of system performance—such as latency variability, responsiveness to dynamic changes, or robustness under constrained compute conditions.

In Table 2 of the Appendix, we report mean and standard deviations of inference times for each component of the system. We find that even with RTC, latency variance for model inference is very low (<1ms); most of the variance comes from the network component, which is unrelated to RTC. Regarding responsiveness, RTC is necessary to enable responsiveness under delays (see our answer below), so it is difficult to divorce the negative impact of the additional computational overhead from the positive impact of the method itself. Regarding constrained compute conditions, RTC is targeted primarily at offboard inference scenarios, where compute is plentiful, but the network adds inevitable latency. However, if one were to use RTC in onboard inference, we do not see any reason why the backpropagation in RTC would disproportionately affect a smaller GPU (the relative overhead would be the same as on a powerful GPU).

How robust is the proposed method to sudden perturbations?

Unfortunately, we cannot run more real-world experiments, but we can point out that in the simulated benchmark, the actions are perturbed with random Gaussian noise at every step (lines 231-232). Our results (Figure 5) show that RTC is more robust than other methods, and that it strictly benefits from increasingly closed-loop control (Figure 5, bottom left).

A comparison or ablation study exploring different inpainting strategies would greatly enhance the generality and reproducibility of the proposed approach.

We have performed an additional simulated ablation comparing to the inpainting strategy from Diffuser [1], which is simpler and does not include a gradient-based correction. Unfortunately, we cannot attach images to the rebuttal, but the inpainting strategy from Diffuser performs worse than standard RTC across the board; it performs worse than BID at $d = 1$ and $d = 2$, and better than BID at $d = 3$ and $d = 4$; and it performs much better than naive asynchronous and temporal ensembling. Our conclusion is that our full method — with a state-of-the-art inpainting algorithm and soft masking — is important for maximum performance, but that a simpler inpainting strategy can still provide some benefit.

[1] “Planning with Diffusion for Flexible Behavior Synthesis.” Janner et al.

I appreciate the authors’ efforts in addressing the concerns raised in my original review. I have also carefully considered the comments provided by the other reviewers.

With respect to my own feedback, I find that the authors have satisfactorily addressed the key issues in their response. Please ensure that the additional results mentioned are incorporated into the final manuscript.

Regarding the comments from Reviewer jJyz, I agree that the manuscript would benefit from a more comprehensive discussion of the existing literature in control, particularly with respect to explaining why those approaches are not applicable to the present work. Additionally, the connection and relevance of this work to cascade control systems should be clarified. Thus, I suggest the authors to tone down sentences like "this is a machine learning paper in submission at a machine learning conference, not a controls conference."

While the manuscript lacks a thorough discussion of the control literature, I believe the work maintains a clear focus on addressing a specific problem, i.e., executing action chunking using diffusion policies. Overall, I find it to be a solid empirical study with demonstrated performance, and I recommend that the paper be accepted.

Thank you for your response! We will certainly include these additional ablations in the paper, and we appreciate your help in improving this work.

Regarding the field of control, we want to clarify that we in no way disagree that the paper would be improved by a more thorough treatment of existing literature in control, which includes many closely related ideas. We actually had a more detailed discussion of MPC in an initial draft, given its similarities with action chunking. However, we decided to cut it in favor of more thorough discussion of learning-centric related work, which we thought would be more important for the NeurIPS audience. With the additional space afforded by the camera-ready, we can certainly put this back, and clarify why these methods are not directly applicable to our problem setting (essentially, elaborating on lines 96-97: "their reliance on explicit dynamics models and cost functions makes their application difficult in unstructured settings..."). We agree that discussion of cascade control is missing, and that discussing its relationship with RTC -- and action chunking more broadly -- would also be beneficial.

We apologize if our response to reviewer jJyz came across aggressively -- we were specifically responding to their comment that "in a well-understood field of control (rather than diffusion based trajectory generation), more care needs to be taken." We merely wanted to highlight that we did not intend for this to be framed as a "controls paper" per se, and that we fully agree that our work is limited to diffusion-based action generation. We also meant to highlight that the venue (NeurIPS) influenced our writing and framing. We hope that is clear from the manuscript -- especially the introduction, where we deliberately tried to motivate the problem first from a general large-scale deep learning perspective, then from a robot learning perspective. We would welcome any feedback in clarifying the motivation further!

Thank you for your in-depth review! We agree that framing our claims properly is very important, and we appreciate your help in treating this subject carefully. We will attempt to address your concerns by: (1) expanding and clarifying our usage of the term “real-time”, and being more explicit about our assumptions; (2) adding more simulated ablations; and (3) clarifying a number of points that we believe may stem from misunderstandings, regarding which we will improve intelligibility in a future revision. Please let us know if any further issues remain!

Firstly, the paper describes the proposed approach as real-time, but real-time has a specific meaning, which is that there is a guaranteed response time. This paper accelerates execution but there is no discussion of a guarantee on response time, or a response time to what.

This is a great point! We had a discussion among the authors about this exact point, and ultimately decided on “real-time” for the following reason: we can think of each low-level controller timestep as an “event”, and the procurement of the next action as the “response”. Prior synchronous action chunking strategies are not real-time because, every $s$ steps (the execution horizon), the controller pauses and waits for model inference, meaning the event-response time is suddenly and unexpectedly many times higher. In RTC, every response comes immediately (in $< \Delta t$ seconds) because model inference is done asynchronously with controller execution.

We agree that this detail is important, and we will update the paper to explain how RTC satisfies the real-time guarantee, namely that it always returns an action response within the $\Delta t$ deadline. As with any real-time system, it is possible for this guarantee to fail in cases of catastrophic failure, e.g., a temporary network outage that causes $d$ to exceed $H - s$ (this never happened throughout our 480 evaluation episodes). However, we will also clearly state this possibility as a limitation.

The authors describe a minimum delay of somewhere between 76 and 98ms, plus 25ms of transmission time over the LAN, for a minimum of $d \approx 6ms$. However, we see in figure 5 that the performance drops substantially up to $d = 4$, and results are not given beyond that.

We believe that there are a few misunderstandings here, and we will clarify the writing on this point in a future revision. 76ms is the model inference delay for the baselines, and 98ms is the model inference delay for RTC (line 263). $d$ is in units of controller steps, not milliseconds, and is calculated using the equation on line 129: $d = \lfloor(98ms + 25ms) / (20ms)\rfloor = 6$. We will expand that section to clarify the distinction.

Figure 5 describes the simulated experiments, not real-world experiments, meaning that we control $d$ directly independent of model latency. $d = 4$ is the maximum because we train with a horizon of $H = 8$, and the maximum possible $d$ is $\lfloor H / 2 \rfloor$ (mentioned on line 238). This means that our simulated experiments test very high relative latencies — up to half of the prediction horizon — which is why performance drops so much across the board (but less so for RTC).

and inject latencies of 100ms and 200ms, which are large (arguably unrealistic?) latencies given the stated properties of the existing system

These are very realistic latencies. Many VLAs are already much slower than Pi05: “Kim et al. [30], who optimize the 7B OpenVLA model [29] specifically for inference speed, achieve no better than 321ms of latency on an NVIDIA A100 GPU” (line 145). If the scaling properties from vision and language hold in robotics, we should expect the best-performing VLAs to increase in size as time goes on (mentioned in line 37). Additionally, as mentioned throughout the paper, our experiments were conducted in ideal network conditions with powerful workstations; weaker devices and/or slow networks could easily add latency in the hundreds of milliseconds. These justifications for studying injected latency are mentioned on lines 265-266 of the paper.

For instance, "we find that hard masking somewhat underperforms soft masking, particularly when d is smaller." The fact that there are no error bars raises the question of how significant this is

Figure 5, which compares hard and soft masking in the simulated benchmark, has 95% Wilson score intervals shaded in (as stated in the figure caption).

solve rate is not defined. I assume this is the proportion of trials where the task was completed correctly

The Figure 5 caption mentions that each Kinetix environment “involves getting a green object on the left to touch a blue one on the right.” This is what counts as a “success”; we will update the Figure 5 caption to clarify this point and explicitly define the solve rate.

Equation 5 gives the exponential decay of the weights during in-painting -- if the exponential were encouraged to delay faster, or more slowly, how sensitive is the performance of the model?

We have performed an additional simulated ablation that we will add to the paper (unfortunately, the rebuttal response does not allow images). We compared 4 schedules: exponential decay (our default), linear decay, constant ones (no decay), and constant zeros (hard masking; already in the paper). Averaged over environments, exponential performs the best, followed by linear, followed by hard masking, followed by no decay. Exponential and linear both outperform BID by a large margin.

The authors are also not rigorous in the related work in terms of MPC techniques -- the issue of executing a partial control and then generating a new one is a very well-studied problem in control -- there are principled techniques from people like Chris Gerdes who have examined when to to switch to a new control given a partial plan

Could you point us to some more specific references? We are not aware of any particular techniques from the MPC literature that would be applicable out-of-the-box to our setting, but we would be happy to add any relevant references.

What happens in this paper when the flow matching returns quickly? This can lead to thrashing, and is not discussed in this paper.

How often the chunk changes can be controlled independently of how quickly the flow matching returns, and is controlled by the hyperperameter $s_\text{min}$ (see Algorithm 1, line 13, as well as lines 210-212 of the text). If the flow matching returns quickly, it allows for a lower execution horizon, but does not require it (see line 211 of the text). However, RTC can take advantage of a lower execution horizon due to its consistency across chunks (Figure 5, lower left).

If by “thrashing”, you are referring to the phenomenon where the robot switches between different plans too quickly, this problem is discussed throughout the paper and is at the core of the motivation for RTC (lines 45, 154-164, 253-255, Figure 2). Apologies if we are misunderstanding your terminology.

Thank you for your in-depth review! We appreciate your acknowledgement of the timeliness and practicality of our problem setting. Regarding novelty, we would emphasize that we are the first work to study this problem setting (real-time execution with VLAs when the model latency exceeds a single controller timestep), that our baselines have not been applied to this problem setting before, and that we still provide an effective inference-only solution that outperforms these baselines.

We have attempted to answer your questions about soft masking, and we will revise the paper to clarify these points, namely its role as an empirical (but not theoretically-motivated) modification to the existing inpainting algorithm. Please let us know if you have any other concerns!

Innovations like soft masking seems to work well empirically but lacks mathematical justification.

That is a valid point, and we will add it to our limitations section.

Why does soft masking match the first $d$ steps better than hard masking?

In the paper, we mention that “a small $d$ leads to a weak guidance signal and approximation errors can still cause discontinuities” (line 195). What we mean is that the inpainting algorithm is fundamentally a best-effort approximation, and in practice, we found that only using the first $d$ steps as guidance led to a poor match for small $d$. As you pointed out, this is theoretically unsatisfying, but it is an empirical reality.

Also, because soft masking is encouraging closer match, why wouldn't it end up slowing down task progress?

The level of matching is a spectrum: from no matching at all, which is so bad that it is unusable on a real robot (see Figure 2, Figure 5, and the supplemental video), to perfectly matching a very long chunk of future actions, which could slow down task progress by preventing closed-loop correction. The right level of matching is an empirical question, but we found that with existing VLA hyperparameters (an action horizon of ~1 second), our soft masking approach with exponential decay performs well.

Would other types of low level controllers, such as more compliant controllers benefit more or less from this method?

In theory, a more compliant controller could be more amenable to real-time execution, as the impact of sudden discontinuities between chunks is lessened. However, without RTC, the controller would merely provide a smoothing effect similar to that of temporal ensembling, which we have shown performs very poorly both in simulation and in reality.

I appreciate the authors’ efforts in addressing my comments from the original review.

Overall, I find the paper to be a solid empirical study. I encourage the authors to expand their discussion on soft masking, including any alternative design choices they explored during development.