A Differential and Pointwise Control Approach to Reinforcement Learning

We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:

[W1] Lack of a Related Works section

We appreciate this suggestion. In the final revision, we will include a Related Work section before the conclusion, using the additional page provided. We will highlight our contributions, particularly the theoretically derived regret bound. Our bound is notable because it arises from pointwise guarantees on per-step policy error, rather than bounding only cumulative regret.

This approach offers more fine-grained guarantees: instead of bounding just a global sum, we demonstrate that the learned policy remains near-optimal at each timestep, thereby avoiding overfitting to specific cumulative reward paths that may result in inconsistent behaviors.

[W2] Non-standard notations

We will revise the notations and formatting based on the reviewer’s suggestions to improve clarity and consistency. All formatting issues will be addressed in the final revision.

[W3] Lack of investigation into computational requirements/cost

We apologize for not including a more comprehensive analysis of computational requirements. Section 4.2 discusses the time complexity of our method, and Appendix C.2 includes memory storage details. Below, we provide approximate training times (similar across tasks):

While the use of automatic differentiation increases cost during sample collection, the overall training time is comparable to standard methods and lower than ensemble-based approaches like TQC.

[Q1] Why is $\rho_0$ termed an adversarial distribution?

The initial state distribution can adversely impact exploration and the resulting learned policy. We use the term "adversarial" to emphasize that even under challenging initial conditions, so long as some mild regularity holds, we can still establish convergence guarantees and regret bounds, as shown in Section 3.

[Q2] Why $L_1$ loss in Step 6 of the algorithm?

Due to the complexity of the problem settings, outlier samples may occur. We use $L_1$ loss to reduce the sensitivity of the model to these outliers, improving robustness.

[Q3] Incorporate other physical priors and not only the symplectic behavior?

Yes, this is possible. The symplectic structure arises from our formulation via classical optimal control and its connection to Hamiltonian dynamics. Generalizing to other priors, such as those relevant for fluid dynamics or quantum mechanics, is a compelling direction. However, this would require alternative theoretical tools, and we consider it a very promising area for future work.

[Q4] Surface modeling task performance

Grid-based or molecular task is more complex than the surface modeling task. For instance, grid-based problems may involve multiple topological modes, making learning and decision-making more difficult. Molecular tasks involve not only atomic geometries but also their interactions, increasing complexity. In these more challenging settings, DPO has greater opportunity to demonstrate its potential.

[Q5] Handling a more complex forward dynamics that involve a decay on state s for example or dynamics that are not Lipschitz

Our theoretical results rely on regularity assumptions. When these do not hold, theoretical guarantees may break down and performance could deteriorate. However, in practice, we find that algorithmic and computational adaptations can help mitigate such issues. We are also working toward a more general framework that performs reliably under milder assumptions.

We sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work.

We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:

[W1] Evaluation on standard RL tasks

We have additionally evaluated our algorithm on continuous-state, continuous-action versions of three standard RL tasks: Pendulum, Mountain Car, and CartPole.

For Mountain Car, we modify the reward to a continuous version: $R = 100 * \sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\sigma$ denotes the sigmoid function.

For CartPole, we use a continuous reward: $R = \text{upright} \cdot \text{centered} \cdot \text{stable}$, with:

• $\text{upright} = \sigma(-5 \cdot |\theta / \theta_{\text{thresh}}|)$
• $\text{centered} = \sigma(-2 \cdot |x / x_{\text{thresh}}|)$
• $\text{stable} = \sigma(-0.5 \cdot (\dot{x}^2 + \dot{\theta}^2))$

A reward close to 1 indicates that the agent is balanced, centered, and dynamically stable.

Our algorithm demonstrates reasonable performance on these standard tasks. More importantly, its true strength lies in scientific computing domains, where challenges such as optimizing over structured geometric spaces, handling coarse-to-fine grid discretizations, and navigating molecular energy landscapes are more representative. These tasks involve evaluating complex functionals and reflect real-world computational modeling problems.

We are currently extending our method to more complex domains, such as the Black Oil Equations in reservoir engineering, a PDE-based setting similar to our grid-based task, and the molecular problem of virus capsid deformation under therapeutic interactions. These problems are more computationally intensive and domain-specific, and will be addressed in separate works. This paper focuses on the theoretical and algorithmic foundations, which provide a principled basis for these future applications.

[W2] Hyperparameter ablation study

We thank the reviewers for pointing out the lack of an ablation study. We now provide two ablation tables examining sensitivity across key hyperparameters for benchmark algorithms.

In the first table, we vary parameters:

• CrossQ / TQC: number of critics and quantiles
• SAC: entropy coefficient

In the second table, we vary:

• DDPG: action noise type (Ornstein-Uhlenbeck), target update coefficient
• PPO / TRPO: clip coefficient, Generalized Advantage Estimator parameter $\lambda$, and whether to normalize the advantage

For DPO, we use standard hyperparameters including a learning rate of 1e-3 and batch size of 32, without any special tuning. The algorithm performs well under these default settings and should not require extensive hyperparameter selection.

Overall, we observe that hyperparameter variation does not significantly affect relative performance rankings. Hence, we omitted these ablations from the main text due to space constraints. We report results only for the reward-reshaped version here, but the straightforward (S-) versions, omitted due to space, shows similar behavior and robustness across tasks.

[Questions] We have addressed all questions through the responses to the weaknesses.

[Editing/formatting errors] We will make revisions to notations and formattings based on reviewer suggestion to improve clarity.

We sincerely appreciate the reviewer's thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work.

We would like to thank the reviewer for your interest in our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:

[W1] Continuous-time RL related works

We appreciate this excellent suggestion. In the final revision, we will include a dedicated Related Work section (on the additional page) to better contextualize our contributions. In particular, we will highlight how our approach compares to prior formulations in continuous-time RL, including:

Continuous-time RL. Most RL methods are formulated using discrete-time Markov decision processes. However, control theory [Fleming and Soner, 2006] has long offered a natural framework for modeling RL in continuous time. One of the first works to formalize this connection is [Wang et al., 2020], which introduced a continuous-time RL formulation grounded in stochastic differential equations (SDEs), replacing cumulative rewards with time integrals and modeling dynamics via continuous-time Markov processes. Several subsequent works, including ours, build on this control-theoretic perspective.

A line of work [Ainsworth et al., 2020; Yildiz et al., 2021], adopts control-theoretic formulations and proposes continuous-time analogs of policy gradient and actor-critic methods without introducing extensive probabilistic tools. Nonetheless, these approaches require direct access to the reward function and its derivatives pointwise, suffering from the same limitations in scientific computing domains as model-based RL approaches discussed in Section 1.

To extend robust methods like SAC, TRPO, or PPO to continuous time, the Q-function must be redefined so concepts like the advantage function remain meaningful. However, naive extensions of the Q-function collapse to the value function, eliminating action dependence and breaking key components like advantage-based updates. A recent novel line of theoretical work [Jia and Zhou, 2022; Zhao et al., 2023] addresses this by redefining the Q-function as a limiting ratio of expected reward to time, connecting it to the Hamiltonian (see Section 2). [Zhao et al. 2023] further reconstruct key components of TRPO and PPO, offering continuous-time counterparts of these algorithms.

Our work also builds on the control-theoretic formulation (simplified in Eq. 4 with stochastic $f$), but differs in two key aspects. First, we use the continuous-time formulation only as a means to derive the dual of RL: we move to continuous time mainly to construct the dual via PMP, and then discretize the dual. Second, we define the policy over the joint space of state and adjoint variables, treating it as an operator over this extended space. This allows us to capture localized updates more naturally. We conjecture that our "$g$-function" (Section 2.1) aligns with the Hamiltonian-based Q-function in [Jia and Zhou, 2022], and our model corresponds to an iterative procedure refining the continuous-time advantage function within the extended state-adjoint space.

We agree this context was important. To keep the paper accessible, we initially only focused on general RL literature, but we will incorporate relevant continuous-time RL works in the final revision.

[W2] Evaluation on standard RL tasks

We additionally evaluate DPO on continuous-state, continuous-action versions of Pendulum, Mountain Car, and CartPole.

For Mountain Car, we use a continuous reward: $R = 100 * \sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\sigma$ denotes the sigmoid function.

For CartPole, we use the continuous reward $R = \text{upright} \cdot \text{centered} \cdot \text{stable}$, with:

• $\text{upright} = \sigma(-5 \cdot |\theta / \theta_{\text{thresh}}|)$
• $\text{centered} = \sigma(-2 \cdot |x / x_{\text{thresh}}|)$
• $\text{stable} = \sigma(-0.5 \cdot (\dot{x}^2 + \dot{\theta}^2))$

Rewards close to 1 indicate the agent is balanced, centered, and stable.

We report the final mean reward below (best in bold, second-best in italic):

While DPO performs reasonably on standard tasks, its main strengths lie in scientific domains, where policies must optimize over structured spaces, handle coarse-to-fine grid, and navigate complex energy landscapes.

We are extending DPO to more complex applications, such as Black Oil Equations in reservoir engineering, which is similar to our grid-based task, and molecular systems like virus capsid deformation. These settings are more computationally demanding and will be explored in future work. This paper focuses on the theoretical and algorithmic foundation to support such directions.

[W3-part 1] Hyperparameter ablation study

We now provide ablation study varying key hyperparameters:

• CrossQ / TQC: number of critics and quantiles
• SAC: entropy coefficient
• DDPG: noise type (Ornstein-Uhlenbeck), target update
• PPO / TRPO: clip coefficient, GAE $\lambda$, advantage normalization

Hyperparameter variations have limited impact on relative rankings. Due to character limit, we include only one representative table from the reshaped-reward setting. Other ablation tables (including those for S-variants and remaining configurations) show similar trends and are consistent with the findings presented here.

DPO uses default hyperparameters (learning rate 1e-3, batch size 32) and achieves good performance without tuning.

[W3-part 2] Computational/memory cost

Section 4.2 analyzes DPO's complexity; memory details appear in Appendix C.2. Below are approximate training times (similar across tasks):

[Q1] Re-planning assumption

We assume that planners may use learned models instead of a real oracle for re-planning, but this still relies on accurately modeling rewards from arbitrary states. Achieving this requires a well-updated policy, which in turn depends on effective re-planning, creating a circular dependency.

[Q2] 5000 sample steps too small for evaluation

We agree this number may appear low compared to typical RL benchmarks. However, our molecular task is representative of real-world biophysical simulations, where reward evaluations can take minutes per step. Running 50,000 steps would require weeks of compute per trial, making research iterations infeasible. We therefore use 5,000 steps to stress-test sample efficiency under realistic constraints. For all other tasks, we use much larger budgets (e.g., 100,000 steps), ensuring that our overall evaluation remains well-balanced.

[Q3] The forward dynamics for the surface modeling and grid-based modeling tasks

The underlying surface dynamics are inherently complex and nonlinear. For RL modeling, we define control at selected locations (e.g., surface control points), which is a common approach in scientific computing. In surface modeling, for example, one may also choose to modify the magnitude of a surface normal vector, leading to different dynamics and reward characteristics. Choosing one over the other reflects a modeling decision rather than a reduction in task complexity. Our chosen task still preserves the essential challenges of optimization over structured spatial domains.

[Q4] How does the Differential RL formulation apply to partially-observable MDPs?

DPO can be extended to POMDPs using belief states. The dual formulation remains valid, but requires probabilistic estimation of latent dynamics, and this is an exciting direction for our future work.

[Q5] Table 4: Why is the standard deviation close to 0 for some methods on the molecular dynamics task?

The initial state distribution for this task is concentrated in a narrow region, and some methods converge quickly with minimal exploration. This can result in low variance across runs.

We sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work.

We would like to thank the reviewer for the recognition of our work and for the valuable and thoughtful feedback. We address the noted weaknesses and questions below:

[W1] Performance on standard RL tasks and reasoning behind the selection of 3 evaluation tasks

We have additionally evaluated our algorithm on continuous-state, continuous-action versions of three standard RL tasks: Pendulum, Mountain Car, and CartPole.

For Mountain Car, we modify the reward to a continuous version: $R = 100 * \sigma(20 * (position - 0.45)) - 0.1 * action[0]^2$, where $\sigma$ denotes the sigmoid function.

For CartPole, we use a continuous reward: $R = \text{upright} \cdot \text{centered} \cdot \text{stable}$, with:

• $\text{upright} = \sigma(-5 \cdot |\theta / \theta_{\text{thresh}}|)$
• $\text{centered} = \sigma(-2 \cdot |x / x_{\text{thresh}}|)$
• $\text{stable} = \sigma(-0.5 \cdot (\dot{x}^2 + \dot{\theta}^2))$

A reward close to 1 indicates that the agent is balanced, centered, and dynamically stable.

We report the final mean reward below (best in bold, second-best in italic):

Our algorithm demonstrates reasonable performance on these standard tasks. More importantly, its true strength lies in scientific computing domains, where challenges such as optimizing over structured geometric spaces, handling coarse-to-fine grid discretizations, and navigating molecular energy landscapes are more representative. These tasks involve evaluating complex functionals and reflect real-world computational modeling problems.

We are currently extending our method to more complex domains, such as the Black Oil Equations in reservoir engineering, a PDE-based setting similar to our grid-based task, and the molecular problem of virus capsid deformation under therapeutic interactions. These problems are more computationally intensive and domain-specific, and will be addressed in separate works. This paper focuses on the theoretical and algorithmic foundations, which provide a principled basis for these future applications.

[W2] Regret bound compared to previous literature

To evaluate sample efficiency, Regret, the gap between the cumulative reward of the learned policy and the optimal policy over $K$ episodes, is a standard metric. In discrete settings, several works (e.g., [Cai et al., 2020]) have established optimal regret bounds of $O(\sqrt{K})$. However, these bounds depend on the cardinality of the state space, which becomes intractable in continuous state-action settings.

In continuous domains, regret bounds are hard to obtain without strong assumptions. A mild assumption is that of a Lipschitz MDP, under which the minimax regret has a known lower bound of $\Omega(K^{\frac{d+1}{d+2}})$ [Slivkins, 2022], where $d$ is the dimension of the joint state-action space. Within our policy optimization (DPO) framework, we prove a regret bound of similar order: $O(K^{\frac{2d+3}{2d+4}})$ (see Corollary 3.6).

To improve rates, several works impose stronger smoothness assumptions. [Maran et al. 2024] assume reward and transition functions are $\nu$-times differentiable, achieving regret $O(K^{\frac{3d/2+\nu+1}{d+2(\nu+1)}})$, which recovers the optimal $O(\sqrt{K})$ as $\nu \to \infty$. [Vakili and Olkhovskaya 2024] assume kernelized transition and reward functions in a reproducing kernel Hilbert space (RKHS) with Matern kernel of order $m$, yielding regret $O(K^{\frac{d+m+1}{d+2m+1}})$, which also approaches $O(\sqrt{K})$ as $m \to \infty$. With stronger assumption, our work also yields dimension-independent rates (see Corollary 3.6).

Our bound is significant because it arises from pointwise guarantees on the per-step policy error, rather than only bounding the total cumulative regret. For a fixed horizon $H$, we show the expected policy error at each step $j$ across episode segments. Summing over steps yields the global regret in Eq. 21.

Our estimates gives a much more fine-grained guarantees: rather than bounding only a global sum, we show that the learned policy is near-optimal at each timestep, avoiding issues like overfitting specific cumulative reward paths (e.g., reward hacking or physically inconsistent behavior). In this sense, pointwise bounds are stronger than bounding the total regret alone.

[Q1] Hyperparameter selection

For DPO, we use standard hyperparameters including a learning rate of 1e-3 and batch size of 32, without any special tuning. The algorithm performs well under these default settings and should not require extensive hyperparameter selection.

We now provide two ablation tables examining sensitivity across key hyperparameters for benchmark algorithms.

In the first table, we vary parameters:

• CrossQ / TQC: number of critics and quantiles
• SAC: entropy coefficient

In the second table, we vary:

• DDPG: action noise type (Ornstein-Uhlenbeck), target update coefficient
• PPO / TRPO: clip coefficient, Generalized Advantage Estimator parameter $\lambda$, and whether to normalize the advantage

Overall, we observe that varying hyperparameters across all algorithms does not substantially alter the performance ranking or lead to drastic performance shifts. Hence, we did not include these ablations in the original submission. Here we only show results for the reward-reshaped version here, but the straightforward (S-) version, omitted due to space, shows similar behavior and robustness across tasks.

[L1] Limitation on intuitive explanation

We will make revisions to wording and formatting to improve clarity and make the presentation more accessible to readers unfamiliar with control-theoretic concepts.

We sincerely appreciate the reviewer’s thoughtful and constructive feedback. Due to the character limit, we have provided a concise response here, but we will carefully incorporate all suggestions and address the concerns raised in our final revision. Thank you again for your valuable insights and for engaging deeply with our work.

Reference

[Cai et al., 2020] Cai, Q., Yang, Z., Jin, C., Wang, Z.: Provably efficient exploration in policy optimization. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1283–1294. PMLR (13–18 Jul 2020)

[Slivkins, 2022] Slivkins, A.: Introduction to multi-armed bandits (2022)

[Vakili and Olkhovskaya 2024] Vakili, S., Olkhovskaya, J.: Kernelized reinforcement learning with order optimal regret bounds (2024)

[Maran et al. 2024] Maran, D., Metelli, A.M., Papini, M., Restell, M.: No-regret reinforcement learning in smooth MDPs (2024)