Flow Q-Learning

Thank you for the detailed review and constructive feedback on this work. We especially appreciate the clarification questions about deterministic policies and policy constraints, as well as several helpful suggestions. We also conducted an additional ablation study on ODE solvers. Please find our response below.

• Would a better ODE solver/different marginal probability path improve performance?

Thank you for the question. Following the suggestion, we conducted an additional ablation study to compare different ODE solvers for FQL. We consider three ODE solvers in this experiment: (1) the Euler method (default), (2) the midpoint method, and (3) the Runge-Kutta (RK4) method. The table below compares the final performance of these ODE solvers on $3$ OGBench tasks ($8$ seeds, $\pm$ denotes standard deviations):

The results suggest that the performance of FQL is generally robust to the choice of ODE solver, except that RK4 leads to somewhat unstable performance on $`cube-double`$. While we have not tested different marginal probability paths, we expect similar results, given that the performance of FQL is generally robust to other flow-related hyperparameters (Figure 9). That said, we believe a better ODE solver or marginal path sampler might further improve performance in more complex tasks and datasets.

• About different kinds of policy constraints.

As the reviewer correctly pointed out, Gaussian behavior-constrained actor-critic methods (e.g., ReBRAC) minimize the distance between the RL policy's actions and dataset actions, while FQL minimizes the distance between the RL policy's actions and the BC policy's actions. They both serve the same role of a behavioral regularizer. The main difference is that ReBRAC directly imposes a Gaussian-based constraint, while FQL indirectly imposes a flow-based constraint. We note that FBRAC in our experiment lies precisely between the two: it directly imposes a flow-based behavioral constraint.

We have empirically compared these $3$ policy extraction schemes (ReBRAC, FBRAC, and FQL) in Table 2 of the paper. The result shows that our indirect flow-based behavioral constraint (FQL) indeed leads to better performance than both the direct Gaussian-based behavioral constraint (ReBRAC) and direct flow-based behavioral constraint (FBRAC). Following the suggestion, we will further clarify the differences between these policy constraints in the final version of the paper.

• Is $\mu_\omega$ in L139 a deterministic policy?

As mentioned in the "Notational Warning" paragraph (L126), $\mu_\omega(s, z)$ is a deterministic function. However, this does not mean that we perform distillation into a deterministic policy: even though $\mu_\omega(s, z)$ is a deterministic function, since $z$ is a random variable (sampled from $\mathcal{N}(0, I)$), it serves as a stochastic policy $\pi_\omega(a \mid s)$ when marginalizing out $z$. As a result, the one-step policy clones the flow BC policy for each $\mathbf{z}$ through deterministic distillation, while maximizing the Q function with stochastic actions (Eq. (8)). While we (partly) explained this subtlety in the "Notational Warning" paragraph, we will further clarify this point in the final draft to prevent any potential confusion.

• About the remark box.

The purpose of the remark box is to provide further theoretical insight into the policy constraint of FQL and to discuss its relation to previous offline RL methods (TD3+BC, AWAC, CQL, etc.). We believe this section can be safely skipped if the reader is mainly interested in empirical results and methodology (which is the main reason we formatted this discussion with a separate box).

• Parameter comparisons between policies.

To clarify, we used the same [512, 512, 512, 512]-sized MLPs all networks (including flow policies, one-step policies, and Gaussian policies; Table 5), unless otherwise noted, and they have almost identical numbers of parameters. There are some exceptions (e.g., IDQL), but we used smaller networks only when the smaller ones performed better than the default-sized ones. We fully described the way we chose these hyperparameters in Appendix F.2. We will further clarify this point in the final draft.

• Notational suggestions.

Thanks for the helpful suggestions! We will incorporate them in the camera-ready version.

We would like to thank the reviewer again for raising important questions about FQL. We believe the additional results and clarifications have significantly improved the quality of the paper. Please let us know if you have any additional questions or concerns. If we have addressed your concerns, would you consider raising your rating?

Thank you for the detailed review and constructive feedback on this work. We especially appreciate your pointing out related work in different domains. Please find our response below.

• Inference details of policies.

Thanks for asking this clarification question! In FQL, neither the BC flow policy nor the one-step policy uses CFG (or CG), as there are no separate class labels in our problem setting. More specifically, the one-step policy $\mu_\omega(s, z)$ is simply a standard feedforward neural network (so no iterative sampling procedure is needed), and the BC flow policy $\mu_\theta(s, z)$ is based on the standard Euler method for ODE solving. At evaluation (inference) time, only the one-step policy is used (L160), and no iterative process is involved. We refer to the algorithm box in the main text (Algorithm 1) as well as L679-L688 in Appendix C for the full description of the sampling procedure.

• Related works on image diffusion models.

Thanks for pointing out these relevant works in image diffusion models! We will cite and discuss them in the camera-ready version.

We would like to thank you again for raising clarification questions and suggesting several related works. Please feel free to let us know if you have any additional questions or concerns. If we have addressed your concerns, would you consider raising your rating?

Thank you for the positive feedback about this work! We would be happy to address any additional questions or concerns you may have, so please feel free to let us know. If there are no further concerns or questions, would you consider raising your rating?

Thank you for the highly detailed review and constructive feedback about this work. We especially appreciate your question about the expressivity of the one-step policy, for which we conducted an additional experiment. Please find our response below.

• Is the one-step policy expressive enough?

Thanks for asking this valid question. To empirically address it, we performed a controlled ablation study to compare the expressivity of (1) Gaussian BC, (2) full flow BC, and (3) one-step distilled BC policies. Here, we evaluate their BC performance (not RL performance) to solely focus on their expressivity without being confounded by different policy extraction strategies (in particular, note that there's no clear, straightforward way to train a full flow policy with RL).

Specifically, we trained these three types of policies on $4$ goal-conditioned OGBench tasks, and measured their goal-conditioned BC performance. To ensure a fair comparison, we used the same architecture and common hyperparameters, and ablated only the policy class. The table below shows the mean and standard deviation across 8 seeds at 1M gradient steps (200K steps for $`cube-single`$ due to overfitting).

The results show that one-step distilled policies achieve nearly identical performance to full flow policies on these tasks, and both flow variants generally outperform Gaussian policies.

Overall, we expect that one-step policies should be expressive enough to model complex action distributions in many practical scenarios, especially considering that one-step flow models can generate highly realistic samples [1, 2] even for high-dimensional image generation.

[1] Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2023).
[2] Frans et al., One Step Diffusion via Shortcut Models (2025).

• Potential trade-off by using a one-step policy.

As shown in the table above, one-step policies often have a very similar expressivity to full flow policies (at least on our benchmark tasks). Hence, we expect the "trade-off cost" of using a one-step policy to be generally marginal. That said, this trade-off might come into play if the task requires highly precise control and thus the full expressivity of iterative generative models. In this case, one solution would be to relax FQL to train a few-step distillation policy to strike a balance between precision and the number of recursive backpropagations. We leave this extension of FQL for future work.

• Scalability to higher state-action dimensions and real-world data sources.

Thanks for the question. We would like to first note that we have evaluated FQL on a diverse set of high-dimensional benchmark tasks (e.g., $`visual-*`$ tasks require pixel-based control, $`humanoidmaze`$ requires $21$-DoF whole-body control, and $`adroit`$ tasks require $24$-DoF dexterous manipulation). Moreover, the OGBench tasks used in this work are generally more complex than the D4RL tasks used in prior work, and they feature noisy, non-Markovian, multi-modal action distributions that (partly) resemble real-world datasets. That said, we did not evaluate FQL on real robotics data, as this work focused more on the algorithmic side of the method. We believe applying FQL to real robots with pre-trained VLA BC flow models [3] is a particularly exciting direction for future research. We will mention the lack of real-world experiments as a limitation (and discuss relevant work) in the final version of the paper.

[3] Black et al., π0: A Vision-Language-Action Flow Model for General Robot Control (2024).

• Notational suggestions (e.g., $a^\theta$ instead of $a^\pi$) and minor clarifications.

Thanks for the helpful suggestions! We will incorporate them into the camera-ready version. In the remark box, as the reviewer correctly pointed out, $\pi^\theta$ and $\nu^\theta$ correspond to the same distribution. Our original intention was to distinguish $\nu^\theta$ as a probability measure, but in hindsight, we feel that this distinction is unnecessary, and we will revise the paper to use the same $\pi^\theta$ notation in the remark box.

We would like to thank you again for raising important questions about FQL, and we believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions.