Safely Learning Controlled Stochastic Dynamics

Thank you for engaging with the technical details. Your questions raised important clarifications that helped us improve the overall exposition.

(General comment). This work takes a first step toward provably safe learning under unknown stochastic dynamics, a setting largely unexplored compared to prior work assuming known dynamics. Our focus is theoretical and methodological; code will be released for reproducibility. Experiments on more complex problems are part of ongoing work. The Sobolev and RKHS assumptions are standard in learning theory and are commonly used to derive generalization guarantees. While the rates depend on dimensionality, this is inherent to the problem. Sobolev regularity helps reduce this effect and leads to near-optimal rates.

"Does this method suffer from a curse of dimensionality? Would it be feasible to present numerical experiments on systems with higher state dimensions or input parameters?"

While our experiments focus on a 2D setting, the method is theoretically designed to scale to higher-dimensional problems. The curse of dimensionality is fundamental in nonparametric learning, but our approach scales gracefully with smoothness. The convergence rates scale as $\mathcal{O}(N^{-\nu/(m+1+2\nu)})$, where $\nu$ denotes smoothness and $m$ the input dimension. Thus, the curse of dimensionality can be mitigated under sufficient regularity.

The main computational cost arises from Gram matrix inversions, which can be reduced using standard approximations such as Nyström or random feature methods, without sacrificing theoretical guarantees. Appendix B provides full complexity analysis. In particular, the computational cost of our safe method is of the same order of magnitude as that of the (unsafe) standard batch kernel ridge regression.

Extending experiments to higher-dimensional systems is a promising and feasible direction for future work.

"In assumptions A1 and A2, is the requirement for all theta, or there exists a theta? The former is very restrictive but appears at odds with the written text."

We acknowledge that Assumptions (A1) and (A2) may appear strong, but they are minimal in what they require: only one known safe and one known reset point are sufficient to initiate safe exploration. The sets $S_0$ and $R_0$ can be as small as singletons. Without at least one such point, safe learning cannot begin. In many systems, states of the form $S_0 = \{(0, \theta) : \theta \in D\}$ and $R_0 = \{(0, \theta) : \theta \in D\}$ are valid and realistic, as systems are often initialized in known, safe and resettable conditions.

We express the assumptions in general (set-based) form to support more practical and informative choices: our approach leverages non-singleton safe and reset sets to accelerate learning while preserving formal safety guarantees.

"Can you expand more on how smoothness of dynamics relates to drift and diffusion terms?"

While we do not study this explicitly, Sobolev regularity of the drift and diffusion terms is expected to imply Sobolev regularity of the resulting state densities under standard conditions. This follows from classical results in parabolic PDE theory, where under standard assumptions, solutions gain regularity relative to the coefficients, approximately two derivatives in space and one in time. We will add a comment on this in the revised version of the paper. See Bonalli, R., and Rudi, A. (2025). Non-parametric learning of stochastic differential equations with non-asymptotic fast rates of convergence. Foundations of Computational Mathematics, 1–56, for related results.

"It is hard to parse the definition of initial safe resettable set, can you explain the role of t and t'?"

In the definition of $\Gamma_0$, $t$ denotes the planned sampling time and $T$ the reset time. The condition $(\theta, t') \in S_0$ for all $t' \in [0, T]$ ensures that the system is known to be safe at every intermediate time between sampling and reset. Thus, $t$ indicates where the observation is collected, and the condition on $t'$ ensures safety over the whole trajectory. This guarantees that a sample at $(\theta, t)$ lies within a trajectory that is entirely safe and ends in a known resettable state.

As mentioned earlier, $\Gamma_0$ can be as simple as a singleton, e.g., $\Gamma_0 = {(\theta, 0, 0) \mid \theta \in D}$, corresponding to a trajectory of length zero-trivially satisfying the requirement. The general formulation allows the use of larger sets when more prior knowledge is available, enabling faster exploration.

"At the beginning of section 4.2, should it be index i instead of N? Or can you explain the role of these two indices?"

In this section, $N$ refers to the iteration index, i.e., the control $(\theta_N, t_N)$ being evaluated. The superscript in $w_i^N$ indicates that the $i$-th sample trajectory is drawn at iteration $N$ using control $u_{\theta_N}$. So $i$ indexes the sample trajectories for a fixed control, and $N$ identifies which control they correspond to. We agree this could be clarified in the notation and will improve the explanation in the revision.

"Can you further explain the assumption on N in theorem 5.1? It seems unintuitive to upper bound N by assumption. But maybe its actually better understood as a bound on Q, the number of samples per trajectory?"

We agree: the condition is better understood as a lower bound on $Q$ as a function of $N$. It ensures that the number of sample trajectories per control is large enough to guarantee accuracy. The logic is: fix a target accuracy $\eta$, then determine the required $N$, and then determine the required $Q = Q(N)$ to achieve it. We will clarify this in the revised version. More precisely, the assumption requires: $Q \gtrsim N^{(2\nu + n)/(2\nu - n)},$ up to log factors, where $n$ is the state dimension and $\nu$ is the regularity. This reflects the classical trade-off in kernel estimation and helps quantify how regularity mitigates the curse of dimensionality.

Can you compare the size of the learned safe region to that of the known dynamics? If not theoretically, then at least empirically? Why is the definition of safety meaningful? Shouldn't we care about absolute safety integrated over time, rather than just at each instant probabilistically?"

While a theoretical comparison is not provided, Figures 4, 5, and 6 show that the learned safe set expands empirically toward the reachable true region. Looser safety thresholds allow the algorithm to recover a larger portion of the reachable set-i.e., the region accessible without crossing unsafe states or requiring discontinuous jumps.

Regarding the safety definition, we use the quantity

$$s^\infty(\theta, T) = \inf_{t \in [0, T]} \mathbb{P}(g(X(t)) \geq 0),$$

which ensures high-probability safety at each individual time point. While enforcing trajectory-level safety (i.e., $\mathbb{P}(g(X(t)) \geq 0 \ \forall t \in [0, T])$) is indeed a desirable goal, it remains fundamentally harder to verify under unknown dynamics. Our pointwise formulation provides a tractable and meaningful notion of safety, especially in the absence of model knowledge.

To the best of our knowledge, no existing methods for exploration in unknown stochastic systems offer stronger or comparable theoretical guarantees. That said, we would be very interested in identifying and comparing to such approaches and would be glad to include any relevant references.

"Figures too small; label plots directly."

We will revise the figures to improve clarity and annotate plots directly with threshold values.

Thank you for your detailed and practical comments. We address your points below.

(General comment). This work takes a first step toward provably safe learning under unknown stochastic dynamics, a setting largely unexplored compared to prior work assuming known dynamics. Our focus is theoretical and methodological; code will be released for reproducibility. Experiments on more complex problems are part of ongoing work. The Sobolev and RKHS assumptions are standard in learning theory and are commonly used to derive generalization guarantees. While the rates depend on dimensionality, this is inherent to the problem. Sobolev regularity helps reduce this effect and leads to near-optimal rates.

"Some notations and writing could be improved..."

Thank you. We agree and will clarify that $w_i$ indexes the Brownian motion samples and that $M_k$ denotes the number of time steps in trajectory $k$. Allowing $T_k \leq T_{\max}$ enables learning controllers that are safe over shorter time horizons, improving flexibility.

"The paper would benefit from the inclusion of an algorithm table..."

We will include an algorithm table in the revised version. A well-documented Python library has already been developed and will be released as open source. Appendix B provides a detailed breakdown of each algorithmic step and its computational complexity.

"Include comparisons with safe exploration baselines."

We agree this would be valuable. Nevertheless, most existing approaches assume known dynamics and typically lack formal safety guarantees during exploration. We plan to include such comparisons in future work and welcome any relevant references we may have missed.

"Include a practical case study."

Thank you for the suggestion. We agree that practical case studies would help illustrate the relevance of our approach. Our work addresses a challenging and, to our knowledge, previously unaddressed setting: learning general unknown dynamics under safety constraints with formal guarantees. While our focus here is on establishing these theoretical foundations, the method is directly applicable to robotics domains where safety and uncertainty are central. For example, flying humanoid robots like iRonCub3 involve highly nonlinear, safety-critical dynamics, and tasks such as cloth folding require learning deformable dynamics safely (e.g., Amadio, F., Delgado-Guerrero, J. A., Colomé, A., and Torras, C. (2023). Controlled Gaussian process dynamical models with application to robotic cloth manipulation. International Journal of Dynamics and Control, 11(6), 3209–3219). We view these as promising directions for future applications.

"Does the computation of the safety model introduce overhead?"

The overhead is moderate. The per-iteration cost is $\mathcal{O}(i^3 + 3Qi)$ (compared to $\mathcal{O}(i^3 + Qi)$ for standard regression), leading to total complexity $\mathcal{O}(N^4 + QM^2)$. Under sufficient Sobolev regularity, $Q < N^2$ is enough, making the evaluation cost negligible and the overall complexity comparable to batch kernel ridge regression (unsafe).

We appreciate your constructive feedback. We address your points below.

(General comment). This work takes a first step toward provably safe learning under unknown stochastic dynamics, a setting largely unexplored compared to prior work assuming known dynamics. Our focus is theoretical and methodological; code will be released for reproducibility. Experiments on more complex problems are part of ongoing work. The Sobolev and RKHS assumptions are standard in learning theory and are commonly used to derive generalization guarantees. While the rates depend on dimensionality, this is inherent to the problem. Sobolev regularity helps reduce this effect and leads to near-optimal rates.

"Evaluation is restricted to low-dimensional synthetic environment..."

We agree. Our goal is to provide a statistical analysis of the interaction between learning and safety. Theoretical guarantees in our framework explicitly scale with dimension: the convergence rates for the three estimators are of order $\mathcal{O}(N^{-\nu/(m+1+2\nu)})$, provided the number of samples per trajectory satisfies $Q \geq \mathcal{O}(N^{(2\nu + n)/(2\nu - n)})$ (up to logarithmic factors). This implies that, under sufficiently smooth dynamics (i.e., large $\nu$), the curse of dimensionality can be mitigated.

Details on computational complexity are provided in Appendix B. Briefly, the per-iteration cost is $\mathcal{O}(i^3 + 3Qi)$ (versus $\mathcal{O}(i^3 + Qi)$ for standard regression), resulting in total complexity $\mathcal{O}(N^4 + QN^2)$. Under sufficient Sobolev regularity, $Q < N^2$ is enough, making evaluation cost negligible and overall complexity similar to batch kernel ridge regression (unsafe).

"Initial safe/reset sets may be limiting."

We acknowledge that Assumptions (A1) and (A2) may appear strong, but they are minimal in what they require: only one known safe and one known reset point are sufficient to initiate safe exploration. The sets $S_0$ and $R_0$ can be as small as singletons. Without at least one such point, safe learning cannot begin. In many systems, states of the form $S_0 = \{(0, \theta) : \theta \in D\}$ and $R_0 = \{(0, \theta) : \theta \in D\}$ are valid and realistic, as systems are often initialized in known, safe and resettable conditions.

We express the assumptions in general (set-based) form to support more practical and informative choices: our approach leverages non-singleton safe and reset sets to accelerate learning while preserving formal safety guarantees.

"Ablation study on kernel and thresholds."

Thank you for the suggestion. While we did not include an ablation study in this version, we agree it would offer valuable insights. Our framework supports a broad class of kernels, and the safety/reset thresholds directly govern the trade-off between conservativeness and exploration. We plan to investigate these aspects more systematically in future work.

"Sensitivity to mis-specification of the sets in assumptions?"

This is an important point. The algorithm requires at least one correctly specified safe and resettable point to initiate exploration. If this point is misspecified (e.g., not actually safe or resettable), the algorithm may halt immediately, as it cannot proceed without safety guarantees. In this sense, the method is sensitive to gross mis-specification of the initial sets. However, this is standard in safe learning and is typically satisfied in practice by choosing conservative initial conditions, e.g., using $S_0 = \{(0, \theta) : \theta \in D\}$ and $R_0 = \{(0, \theta) : \theta \in D\}$, which are valid in many settings.

"Can this method be applied to higher-dimensional and longer-horizon problems?"

Yes. While our experiments focus on a 2D setting, the method's statistical and computational performance is theoretically characterized in arbitrary dimensions (see above). In particular, convergence rates depend on both the smoothness of the target functions and the dimensionality, allowing scalability under sufficient regularity. The main computational cost lies in kernel regression, which can be mitigated using standard techniques such as Nyström approximations or random features, without sacrificing theoretical guarantees (See, e.g. Rudi, A., Camoriano, R., and Rosasco, L. (2015). Less is more: Nyström computational regularization. Advances in neural information processing systems, 28).

Thank you for your thoughtful review. Below are our responses to the points you raised.

(General comment). This work takes a first step toward provably safe learning under unknown stochastic dynamics, a setting largely unexplored compared to prior work assuming known dynamics. Our focus is theoretical and methodological; code will be released for reproducibility. Experiments on more complex problems are part of ongoing work. The Sobolev and RKHS assumptions are standard in learning theory and are commonly used to derive generalization guarantees. While the rates depend on dimensionality, this is inherent to the problem. Sobolev regularity helps reduce this effect and leads to near-optimal rates.

"The method relies on assumptions as Sobolev regularity, RKHS membership, eventually known safe/reset sets, which might be limiting."

We acknowledge that Assumptions (A1) and (A2) may appear strong, but they are minimal in what they require: only one known safe and one known reset point are sufficient to initiate safe exploration. The sets $S_0$ and $R_0$ can be as small as singletons. Without at least one such point, safe learning cannot begin. In many systems, states of the form $S_0 = \{(0, \theta) : \theta \in D\}$ and $R_0 = \{(0, \theta) : \theta \in D\}$ are valid and realistic, as systems are often initialized in known, safe and resettable conditions.

We express the assumptions in general (set-based) form to support more practical and informative choices: our approach leverages non-singleton safe and reset sets to accelerate learning while preserving formal safety guarantees.

"Validation is limited to a 2D toy example, and scalability is not demonstrated."

We agree that evaluating the method on higher-dimensional systems would be valuable. In this paper, we focused on a low-dimensional setup to clearly illustrate the core theoretical guarantees. Higher-dimensional experiments are planned for future work. Nonetheless, the theoretical results scale with dimension, with convergence rates for the three estimators of order $\mathcal{O}(N^{-\nu/(m+1+2\nu)})$, provided the number of samples per trajectory satisfies $Q \geq \mathcal{O}(N^{(2\nu + n)/(2\nu - n)})$ (up to log factors). This means the curse of dimensionality can be mitigated under sufficient smoothness (i.e., large $\nu$), making the method applicable in higher-dimensional settings.

The main computational cost arises from Gram matrix inversion, which can be reduced using standard approximation techniques such as Nyström methods, without sacrificing theoretical guarantees (see Rudi et al., 2015). Appendix B provides algorithmic details and complexity analysis. In short, the per-iteration cost is $\mathcal{O}(i^3 + 3Qi)$ (vs. $\mathcal{O}(i^3 + Qi)$ for standard regression), leading to overall complexity $\mathcal{O}(N^4 + QN^2)$. Under sufficient regularity, $Q < N^2$ is enough, making evaluation costs negligible and overall complexity comparable to batch kernel ridge regression (unsafe).

"Comparison with other safe learning approaches is limited. Impact of reset unclear."

We agree and thank you for raising this. Our setting differs from most safe RL approaches, which typically assume known dynamics or do not offer formal safety guarantees. In contrast, our method ensures certified safety during exploration under fully unknown stochastic dynamics.

The reset mechanism is optional but beneficial: it prevents the learner from restarting exploration far from the initial safe region, improving sample efficiency in practice. Resets are also common in prior safe learning works, e.g., "Turchetta, M., Berkenkamp, F., and Krause, A. (2016), which rely on resets to known safe states to enable safe expansion. Safe exploration in finite Markov decision processes with Gaussian processes. Advances in neural information processing systems, 29.", which rely on resets to known safe states to enable safe expansion.

Importantly, the impact of resets is reflected in our theoretical analysis. Our convergence rates depend on the Sobolev regularity of both the safety function and the reset mechanism. Thus, even if the safety constraint is highly regular, the reset function can become the bottleneck if it has lower regularity. In that sense, the reset constraint may limit the overall convergence rate.

"The method requires upper bounds on $\|s\|$, $\|r\|$, which are not always available. How sensitive are the safety guarantees to misspecification?" "How are hyperparameters $\beta^s, \beta^r, \lambda, \gamma, R$ tuned?"

We agree this is a key practical limitation. The bounds on $\|s\|$, $\|r\|$, are required for our theoretical guarantees. When not known a priori, they can be conservatively overestimated, which ensures safety but may result in more cautious exploration. In some settings, such as Sobolev RKHS, these norms can be estimated from prior knowledge, for example using bounds on $|p|_{H^\nu}$.

Although a quantitative analysis of how safety degrades under misspecified bounds is still missing, we believe this requirement is a reasonable trade-off for enabling provable safety under fully unknown dynamics.

In our experiments, hyperparameters ($\beta^s, \beta^r, \lambda, \gamma, R$) were selected heuristically based on smoothness assumptions to reduce computational overhead (see Appendix B.3). Developing more adaptive tuning strategies to estimate these quantities more accurately is a promising direction for future work. Following Shalev-Shwartz (2012), one could explore approaches based on the doubling trick over a finite decreasing sequence of candidate values, which may help preserve theoretical guarantees without requiring prior knowledge of the quantities $\|s\|$ and $\|r\|$.

Shalev-Shwartz, S. (2012). Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2), 107–194.

More broadly, this reflects a central challenge in safe learning: guaranteeing safety when both the system and its regularity are unknown. Our framework allows for variable smoothness through flexible kernel assumptions, but understanding safety under uncertainty in regularity remains an open and important problem.

"Suggestion: increase complexity and dimensionality of the experiments; compare to safe RL baselines."

We agree that evaluating on more complex, higher-dimensional environments is an important next step. This work aims to establish a principled foundation for safe learning under unknown dynamics, and scaling to more realistic domains (e.g., robotics) is part of our ongoing research.

Regarding comparisons to safe RL and CBF methods: we acknowledge their relevance and value. However, to our knowledge, most existing approaches either rely on known or structured system dynamics or do not offer provable safety guarantees in the fully unknown setting we consider. We would be grateful for any specific references we may have overlooked and will gladly include relevant baselines in future versions or extended work.

Several notable examples that reflect growing interest in safety-aware learning, while operating under model knowledge, include:

Prajapat, M., Köhler, J., Turchetta, M., Krause, A., and Zeilinger, M. N. (2025). Safe guaranteed exploration for non-linear systems. IEEE Transactions on Automatic Control.

Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30.

Turchetta, M., Berkenkamp, F., and Krause, A. (2019). Safe exploration for interactive machine learning. Advances in Neural Information Processing Systems, 32.

Baumann, D., Marco, A., Turchetta, M., and Trimpe, S. (2021, May). Gosafe: Globally optimal safe robot learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA) (pp. 4452-4458). IEEE.

Sukhija, B., Turchetta, M., Lindner, D., Krause, A., Trimpe, S., and Baumann, D. (2023). Gosafeopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 320, 103922.