ODE-based Smoothing Neural Network for Reinforcement Learning Tasks

> Question 3

Your suggestion is great. We provide an additional test on the backward and forward time. The platform used is an AMD Ryzen Threadripper 3960X 24-Core Processor. The network structures compared are MLP, SmODE, MLP-SN [8] and LipsNet [3]. The latter two algorithms are smoothing neural networks. The power iteration step in MLP-SN is set to 15. The experimental results are shown in Table 4.

Table4 Forward and Backward Propagation Time Test. The experimental comparison algorithms are MLP, SmODE, MLP-SN and LipsNet.

It implies that the computation time of SmODE is high than MLP's. Fortunately, the 1-batch forward time is a small value within 1 ms, which still acceptable in the real-time application. We found that the training time of SmODE is about 2-3 times longer than that of MLP. The bottleneck is the backpropagation through time. It is actually what we are working on recently. We are going to introduce an Adjoint method to eliminate the bottleneck. We would advise to follow our future work.

> Reference

[1] Mysore et al. "How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning", ACM Transactions on Cyber-Physical Systems, 2020.

[2] Chen et al. "Addressing Action Oscillations through Learning Policy Inertia", AAAI, 2021.

[3] Song et al. "LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control", ICML, 2023.

[4] Mysore et al. "Regularizing Action Policies for Smooth Control with Reinforcement Learning", IEEE ICRA, 2021.

[5] Shen et al. "Deep Reinforcement Learning with Robust and Smooth Policy", ICML, 2020.

[6] Yu et al. "TAAC: Temporally Abstract Actor-Critic for Continuous Control", NeurIPS, 2021.

[7] Wang et al. "Smooth Filtering Neural Network for Reinforcement Learning", IEEE TIV, 2024.

[8] Miyato et al. "Spectral normalization for generative adversarial networks," arXiv, 2018.

> Weakness 2

Thank you for your suggestion. Next, we will explain why we choose MLP, LipsNet, and LTC as the baseline network structure and why they can form complete related baselines.

First, let's discuss MLP. In deep reinforcement learning (DRL) control algorithms, the most commonly used policy network structure is the MLP. This structure also represents the core issue we aim to address: the problem of action oscillation that arises when using classic DRL algorithms for control. Thus, using MLP as the baseline network is essential for evaluating the performance and smoothness improvements achieved by alternative network structures.

The second one is LipsNet, which is the SOTA algorithm of smooth neural network. Our paper uses it as the core comparison algorithm. If we can achieve better performance and smoothness than LipsNet in most control tasks, it can strongly prove that our method is effective and has more advantages.

Finally, LTC is one of the classic algorithms of Neural ODE, and it is mentioned in the paper that it has a certain anti-interference ability. Therefore, it is also very important to compare with LTC and achieve better performance and smoothness, which belongs to the same category of Neural ODE. In addition, it can further show that we are the first to solve the two causes of the unsmoothness of DRL control actions (high-frequency noise interference and unconstrained Lipshchitz constant) at the same time. I believe this is very interesting for the RL community.

> Weakness 3

Your suggestion is very good, we have deleted this unscientific statement. From the results of the ablation experiment, we can see that the adaptive constraint on the Lipschitz constant of the neural network plays a great role in the smoothness of the control action and does not affect the performance of the strategy. This is due to the fact that $g(x(t), I(t), t, \theta)$ can adjust the degree of Lipschitz constraint according to the current input and state, as described in Theorem 2. The image in Appendix C is intended to demonstrate the ability to adaptively adjust the degree of Lipschitz constraint based on the speed of dynamic changes in the current state.

> Question 1

Your discovery is very sharp. In the proof of Theorem 2, we did use a specific parameter range, but in fact our proof does not depend on the specific parameter selection. As long as $f(x(t), I(t), t, \theta)>0$ and is bounded, the proof of Theorem 2 is valid.

> Question 2

Your suggestion is excellent. In the Walker2d-v3 task, we replaced the policy network in the TD3 algorithm with SmODE and conducted sensitivity experiments on the hyperparameters $\lambda_1$ (Filtering Hyperparameter) and $\lambda_2$ (Lipschitz Hyperparameter).

Under this setup, we tuned the hyperparameters and found that the optimal values were both 0.01. To explore the sensitivity within the same order of magnitude, we experimented with 0.02; for different orders of magnitude, we experimented with 0.001 and 0.1. When analyzing the sensitivity of one parameter, the other parameter is kept constant at 0.01. Gaussian noisy variance is 0.1. The results are shown in Table 3:

Table 3 Hyperparameter sensitivity analysis. When analyzing the sensitivity of one parameter, the other parameter is kept constant at 0.01. Gaussian noisy variance is 0.1.

The results indicate that $\lambda_2$ has a stronger impact on action smoothness compared to $\lambda_1$. Within the same order of magnitude, the differences in performance and action smoothness are not significant, but across different orders of magnitude, there are substantial differences.

The selection of parameters is divided into two steps:

1. First, set $\lambda_1$ to 0, and select the weight of $\lambda_2$ to 0.01 and 0.001 for experiments to determine which weight can better balance strategy performance and smoothness.
2. Fix $\lambda_2$ unchanged, and change the value of $\lambda_1$ from the same as $\lambda_2$ to 10 times smaller for experiments to determine which weight can better balance strategy performance and smoothness.

Tip: The weight of $\lambda_2$ is greater than or equal to the weight of $\lambda_1$.

We thank you for the careful reading of our paper and constructive comments in detail.

> Weakness 1

It may be that my inadequate quoting has caused you concern. In fact, Mysore et al. (2020) [1] were the first to introduce the use of $\Delta a_t = a_t - a_{t-1}$ to define the degree of action smoothing, which became a key performance metric. The exact formulation is provided in Section 5, "Evaluation," of their paper. Additionally, Chen et al. (2021) [2] employed the AFR as a central measure of action smoothing. A detailed description is available in Section 4.1, "Training and Evaluation," of their paper. However, their work focuses on AFR as a metric for discrete action spaces, while LipsNet (2023) [3] extends it to continuous action spaces. There are also some works [4-6] that directly or indirectly add AFR as a penalty term in the training loss function or reward function to improve the smoothness of the control action, which shows that AFR is a very important index for measuring the smoothness of the action.

After giving numerous citations, we hope you will recognize that this is a generally agreed upon metric, and is one of the most intuitive and effective performance indicators of control smoothness in practical control tasks.

But now that you mention it, we found another test metric named mean weighted frequency (MWF) in [4] [7].

When analyzed in the frequency domain, the unsmoothness of the control policy can be explained by the high-frequency components in the action sequence. Given an action sequence $\{a_{0},a_{1},\cdots a_{T}\}\sim\rho_{\pi}$, the MWF is defined

$\zeta(\pi)=\mathbb{E} _ {\{a_0,a_1,\cdots a_T\}\sim\rho_\pi}\left[\sum _ {i=1}^n\frac{\sum _ {j=1}^kA_{ij}f_{ij}}{n\sum _ {j=1}^kA_{ij}}\right],$

where $A_{ij}$ and $f_{ij}$ is the $j$-th amplitude and frequency component in the frequency spectrum of the $i$-th dimension action sequence.

We selected four robot control tasks, Humanoid-v3, Walker2d-v3, Pusher-v2 and Hopper-v3, and used the MWF indicator for experimental testing. Since the state values of different Mujoco tasks vary greatly, we set two levels of uniform noise for the four tasks, as shown in Table1. For the whole tasks, noise is added to all states.

Table 1 Different levels of uniform noise for different Mujoco tasks. The numbers in the table represent the range of values for uniform noise.

The comparison network structures used in the experiment are MLP, MLP-LowFilter and SmODE, where LowFilter is a low-pass filter that utilizes a three-step history state. The results, which are the averages of five seeds, are shown in Table 2.

Table 2 Experimental results. Average control performance of MLP, MLP-LowFilter, and SmODE for different uniform noise levels, where level 1 is on the left column and level 2 is on the right column. The average mean weighted frequency is indicated in parentheses. Results are expressed as mean ± standard deviation of five independent environmental seeds.

The experimental results demonstrate that AFR and MWF provide consistent measures of action smoothness, with both serving as effective indicators. AFR, being more intuitive and easier to calculate, has been widely used in related works since 2020. Consequently, we selected AFR as the primary indicator for assessing action smoothness.

2.2.2

Our definition of $g(x(t), I(t), t, \theta)$ already guarantees its boundedness. In addition, you can see that $f(x(t),I(t),t,\theta) = \frac{w_{ij}}{C_{m_{i}}} \text{sigmoid}(\cdot)$ is defined in line752, where $w_{ij}\in(0.001,1.0), C_{m_{i}}\in(0.4, 0.6), sigmoid(\cdot)\in(0, 1)$. Therefore, we can get $0 < f(x(t),I(t),t,\theta) < \frac{1.0}{0.4} = 2.5$.

In fact, we can clearly point out that C = 5, which is determined by the upper bound of f: $C = 2 \cdot \max(f(x(t),I(t),t,\theta)) = 2 \cdot 2.5 = 5$. Therefore, $M(x(t), I(t), t, \theta)$ has no direct connection with $C$. I believe that after giving the above detailed explanation, you can believe that our work has a solid theoretical foundation.

> Weakness 3

You are right that the discussion of scalability, computational overhead, and real-time performance is very important.

Regarding scalability, I first introduce the design principles of the number of neurons in the three layers of Smooth ODE Module. You can see Page 19 of the paper, Table 11. The number of neurons in the second and third layers is the same as the action dimension of the control task, while the number of neurons in the first layer is slightly larger than the action dimension. The action dimension of the actual control task is not very large, so the number of neurons in the Smooth ODE Module will not be very high.

To address the issues of training efficiency and inference efficiency, we provide an additional test on the backward and forward time. The platform used is an AMD Ryzen Threadripper 3960X 24-Core Processor. The network structures compared are MLP, SmODE, MLP-SN [2] and LipsNet [3]. The latter two algorithms are smoothing neural networks. The power iteration step in MLP-SN is set to 15. The experimental results are shown in Table 2.

Table2 Forward and Backward Propagation Time Test. The experimental comparison algorithms are MLP, SmODE, MLP-SN and LipsNet.

It implies that the computation time of SmODE is high than MLP's. Fortunately, the 1-batch forward time is a small value within 1 ms, which still acceptable in the real-time application. We found that the training time of SmODE is about 2-3 times longer than that of MLP. The bottleneck is the backpropagation through time. It is actually what we are working on recently. We are going to introduce an Adjoint method to eliminate the bottleneck. We would advise to follow our future work.

> Weakness 4 & Question 3

4.1

Thank you for doing your extensive reading and finding this similar study that was done very close in time. I need to remind you that according to the last QA of ReviewerGuide, this work published on 2024.8.21 (after 2024.7.1) is a concurrent paper and does not need to be compared. However, we found the work of Wang et al. [6] to be quite interesting. After reading it, we will compare and analyze it from theoretical perspective, discuss its advantages and disadvantages.

Theoretically, their work addresses the issue of unsmooth action in neural network control solely from the perspective of filtering, while overlooking the fact that the unconstrained Lipschitz constant of the neural network itself is also one of the core reasons for the unsmooth action. This work can be considered to use the simplest Neural ODE, using a first-order forward Euler expansion. Our Neural ODE uses the bionic modeling approach, has stronger representation capabilities, and uses a semi-implicit Euler discretization with higher solution accuracy.

In terms of experimental settings, they conducted training in a noisy environment by artificially adding noise, which is quite different from the basic RL training environment. In addition, they need to retain the hidden state of 4 steps of history for filtering, which makes it difficult for their algorithm to be seamlessly integrated into other RL algorithms and frameworks. In summary, they gave some additional designs compared to SmODE in order to adapt to Gaussian noise interference.

I would like to raise concerns regarding the publication timeline and technical overlap of submission #7391. While the IEEE paper's publication date (Aug 21, 2024) comes after the submission deadline and falls within the concurrent work window, there are serious concerns about the substantial technical overlap between these papers that warrant further investigation by the program committee.

The fundamental technical framework of both papers shows remarkable similarities. Both papers propose essentially identical solutions to the smoothing RL control actions problem. Although different mathematical notation is used, the underlying principles remain the same. Notably, the ODE-based implementation effectively realizes the same first-order low-pass filtering mechanism as in the IEEE paper.

The methodological approach also demonstrates significant overlap. Both works employ learnable state-based parameters for filtering and focus on dynamic filtering of hidden states. The treatment of high-frequency noise follows similar principles, and both papers ultimately achieve comparable results through analogous technical means.

The authors' rebuttal statement that the published work "addresses the issue solely from the perspective of filtering" appears to understate these similarities. Both papers utilize equivalent mathematical principles, demonstrate similar experimental results, and share core technical approaches. The distinction between ODE-based and filtering-based approaches becomes particularly thin when examining the actual implementation details.

Given these substantial overlaps, I recommend that the program committee investigate this submission for potential academic integrity concerns. A proper investigation would help ensure the integrity of the peer review process and maintain ICLR's high standards.

4.2

4.2.1

Neural CDEs [4] address a fundamentally different problem from ours. While we focus on ensuring smooth control actions in reinforcement learning by designing a neural network with ODEs to reduce oscillations, their work tackles the issue of irregular sampling in time series data, using CDEs to manage time dimension irregularities.

In theory, our paper builds on Neural ODEs, with the key innovation being the use of low-pass filtering and Lipschitz constraints to achieve smoothness. In contrast, their work is based on CDE theory, with the primary innovation being the handling of time dependencies in sequences through control differential equations.

Regarding application, our focus is on control problems in reinforcement learning, such as robot control and autonomous driving, while their work targets general time series modeling tasks, including sensor and medical data.

In summary, this method is not suitable for our experimental comparison. (You may have noticed that in Section 3.4 of their work, they proposed Training via the adjoint method to reduce memory overhead and speed up training. This is exactly the method we want to use in our subsequent work. Thank you for your suggestion.)

4.2.2

We appreciate your suggestion to compare our work with Stable Neural Flows (SNF) [5]. While SNF offers valuable theoretical insights into designing stable Neural ODEs, its focus is on providing stability guarantees through energy functions.

Our work and SNF are complementary: SNF provides a theoretical foundation for stable Neural ODEs, whereas our approach delivers a concrete implementation and experimental validation of smooth control using an ODE-based neural architecture. The absence of implementation code in SNF makes direct experimental comparison infeasible. However, we believe that integrating SNF's theoretical framework with our practical implementation could be a promising direction for future work, combining theoretical stability guarantees with smooth control capabilities. In summary, this method is not suitable for our experimental comparison.

> Question 2

In Weakness3's response, we thoroughly analyzed and compared the computational and reasoning times of SmODE with those of other network structures. We demonstrated that SmODE's 1-batch reasoning time is under 1 ms, which satisfies practical control requirements. Next, we analyze the impact of the number of ODE solver iterations on strategy performance and smoothness.

We experimented with three settings for the Pusher-v2 task, 3, 6 (Standard SmODE), and 9 iterations. We set two levels of uniform noise for the Pusher-v2 task. They are [-0.1, 0.1], [-0.2, 0.2]. The experimental results are shown in Table 3.

Table3 Experimental results. The average action fluctuation rate is indicated in parentheses. Results are expressed as mean ± standard deviation of five independent environmental seeds.

Experimental results show that increasing the number of iterations can indeed improve the accuracy of the solution, thereby improving the performance and smoothness of the strategy, but it also faces the risk of longer inference and training time. Compared with the 3-step iteration to the 6-step iteration, the smoothness improvement from 6 to 9 is relatively small. In addition, compared with MLP, the 6-step iteration has achieved a great improvement in smoothness. In summary, the 6-step iteration solution is a good choice.

> Reference

[1] Lechner et al. "Neural circuit policies enabling auditable autonomy", Nature machine intelligence, 2020.

[2] Miyato et al. "Spectral normalization for generative adversarial networks," arXiv, 2018.

[3] Song et al. "LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control", ICML, 2023.

[4] Kidger et al. "Neural Controlled Differential Equations for Irregular Time Series", NeurIPS, 2020.

[5] Massaroli et al. "Stable Neural Flows", arXiv, 2020.

[6] Wang et al. "Smooth Filtering Neural Network for Reinforcement Learning", IEEE TIV, 2024.

We appreciate the reviewer for the careful reading of our paper and detailed discussions.

> Weakness 1 & Question 1

As you can sense, “Bionic modeling” can be confusing at first glance. We elaborate on the cited article: Neural circuit policies (NCP) enabling auditable autonomy [1]. The researchers in this work reference the structure of the nervous system of Caenorhabditis elegans, a nematode renowned for its near-optimal neural network architecture and coordinated neural information processing mechanisms. NCP mentions that bionic modeling of neurons using imitation Caenorhabditis elegans can enhance their computational capabilities. Therefore, biological concepts such as membrane capacitance are mentioned in our paper as learnable parameters with an initial range of values consistent with those in the NCP.

Since you are very concerned about the impact of the value range of biological modeling parameters on performance, we conducted sensitivity experiments on the four parameters $w_{ij}, C_{m_{i}}, \mu_{ij}, \gamma_{ij}$ in the Pusher-v2 robot control task. With all other parameters held constant, two sets of experiments were conducted by expanding and shrinking the value range of each parameter by a factor of 10, resulting in a total of 8 experiments. The experiment used two different levels of uniform noise, [-0.1, 0.1] and [-0.2, 0.2], and the results are shown in Table 1.

Table1 Experimental results.. The average action fluctuation rate is indicated in parentheses. Results are expressed as mean ± standard deviation of five independent environmental seeds.

The experimental results show that adjusting the value range of biological parameters will affect both the performance and smoothness of the strategy, resulting in a classic seesaw effect. However, the performance and smoothness of all experiments exceeded that of MLP.

Equation 12 introduces several parameters that, while increasing the model's complexity, enhance its biological plausibility and strengthen the connection between Neural ODEs and biological structures. Despite this added complexity, no manual hyperparameter tuning was performed for the biological modeling parameters in any of the experiments presented in this paper. Nevertheless, the experimental results consistently demonstrated high performance and effective smoothing capabilities. Consequently, there is no extensive parameter tuning space that would undermine the reliability of the results. Although the absence of clear empirical rules may seem concerning, it is important to note that none of our experiments required adjustments to the biological modeling parameters. Furthermore, their range remained consistent with that of NCP, which is why we did not explore empirical tuning rules.

> Weakness 2

2.1

You can look at line 242. We define $g(x(t), I(t), t, \theta)=\text{tanh}(h(x(t), I(t), t, \theta))$, where tanh stands for a hyperbolic tangent function, and $h(x(t), I(t), t, \theta)$ stands for a neural network. Thus, $g(x(t), I(t), t, \theta)$ takes values in the range (-1, 1). Therefore there is no problem of lack of boundedness assumption.

2.2

2.2.1

Our definition of $g(x(t), I(t), t, \theta)$ already guarantees its boundedness.

In addition, you may have misunderstood. In the proof, we only examine the sign of $\frac{dx_i}{dt}$ at the special point $x_i(t) = M = \max(0,g(\cdot)_i^{\max})$, instead of requiring $\frac{dx_i}{dt} \leq 0$ for all values ​​of $x_i(t)$. In order to prove that the upper bound $x_i(t) \leq \max(0,g(\cdot)_i^{\max})$ holds, we only need to prove that: when $x_i(t)$ reaches this upper bound, its derivative is non-positive, which ensures that the trajectory will not exceed this upper bound. In fact, the state of the neuron can change freely between the lower and upper bounds, and the derivative can be positive or negative.

The derivation steps for the lower bound are completely symmetrical. You only need to replace $M = \max(0,g(\cdot)_i^{\max})$ with $M = \min(0,g(\cdot)_i^{\min})$. At this time, the derivative is greater than 0, and the proof is complete.

Dear reviewer 7dLr,

It is less than 2 days before the end of the discussion phase. I sincerely hope that you can take some time out of your busy schedule to check some of our responses. Your questions have given us a lot of thought, and we look forward to having in-depth exchanges with you or reaching an agreement on the contribution of this work.

Through careful deliberation and analysis of the distinctions in theoretical frameworks and experimental protocols, we present objective evidence that carries greater scholarly weight than subjective comparative evaluations. Next, we shall demonstrate through the following rigorous arguments that our work is entirely free of academic integrity concerns. (Incidentally, you did not make any valid replies during the three-week discussion period, and on the last day of the discussion phase you revised the visible scope of the review via Revisions, and we did not receive any notifications. Your evident expertise in manipulating OpenReview's notification protocols astounds us.)

1. Although both are first-order low-pass filtering, the work you mentioned is a discrete-time first-order low-pass system in fixed form, while the Neural ODE proposed in this paper is a continuous-time first-order low-pass system in variable form. Therefore, the latter has a higher degree of freedom in neuron structure design and can embed more good theoretical properties (e.g., adaptive Lipschitz constant constraints, adaptive filtering degree adjustment proposed in this paper) according to the demand.

2. The two algorithms are also very different in terms of training approach, as the work you mentioned uses the most basic backpropagation, whereas our work solves continuous-time systems using semi-implicit Eulerian discretization, thus requiring the use of the BPTT method to propagate the gradient.

3. As you mentioned in Strength, our work are the first combination of Neural ODE to address dual issues leading to unsmooth control, and the work you mentioned does not consider the neural network Lipschitz constant unsmooth control problem. This is because they use fixed form discrete first-order low-pass systems that do not have the ability to design adaptive Lipschitz constant constraints.

4. The work you mention uses hundreds or thousands of filter neurons, whereas our work requires only a dozen or so Smooth ODE units.

5. The core mathematical formula for the work you mention is $\tau\frac{\mathrm{d}h(t)}{\mathrm{d}t}+h(t)=x(t),\mathrm{with}h(0)=x(0)$, which after expansion with first-order forward Eulerian expansion is $h_t=(1-\alpha_t)h_{t-1}+\alpha _tx_t,\mathrm{with}h_0=x_0$, and $\alpha_t$ is the filter coefficient. And the core formula of our work is $\frac{\mathrm{d}x(t)}{\mathrm{d}t}=-f\left(x(t),I(t),t,\theta\right)x(t)+f\left(x(t),I(t),t,\theta\right)g(x(t),I(t),t ,\theta)$, where $f\left(x(t),I(t),t,\theta\right)$ is used to adaptively control the degree of filtering and $g(x(t),I(t),t,\theta)$ is used to adaptively control the Lipschitz constant. A semi-implicit Eulerian discretization is used and then solved in continuous time 1s. Our work is based on Neural ODE unfolding, which is fundamentally different as can be seen from the mathematical formulation and the way it is solved.

6. In terms of experimental setup, the work you mentioned trains in a noisy environment by artificially adding Gaussian noise, which is very different from the basic RL training environment. This training environment setting may result in an algorithm that has a significant advantage when dealing with Gaussian noise and may have a degraded performance when dealing with other noises.

7. Their work requires preserving the hidden state of the 4-step history for filtering, which makes it difficult to seamlessly integrate their algorithm into other RL algorithms and frameworks.

8. Finally, our work has been submitted once before 2024.08.21 (the date of publication of the work you mentioned) (the core METHOD is identical), and you can contact us after the review if you need further proof.

In summary, there are no academic integrity issues associated with our work.

> Major issue 4

Your suggestion is valuable; it is essential to test under non-Gaussian noise, as this aligns with the motivation discussed in introduction. Therefore, we decided to experiment with uniform noise as a representative of non-Gaussian noise.

For the MuJoCo environment, we selected four tasks Humanoid-v3, Walker2d-v3, Pusher-v2, and Hopper-v3 for our experiments. Since the state values of different Mujoco tasks vary greatly, we set two levels of uniform noise for the four tasks, as shown in Table1. For the whole tasks, noise is added to all states.

Table 1 Different levels of uniform noise for different Mujoco tasks. The numbers in the table represent the range of values for uniform noise.

Three different policy networks were used in the experiments, the first one is MLP network, the second one is MLP+LowFilter and the third one is SmODE. LowFilter is a low-pass filter that utilizes a three-step history state. The extended Kalman filter is not employed here because it requires state transition and observation equations, as well as noise statistics, which are typically unavailable in general reinforcement learning environments such as MuJoCo. The results, which are the averages of five seeds, are shown in Table 2.

Table 2 Experimental results. Average control performance of MLP, MLP+LowFilter, SmODE for different uniform noise levels, where level 1 is on the left column and level 2 is on the right column. The average action fluctuation rate is indicated in parentheses. Results are expressed as mean ± standard deviation of five independent environmental seeds.

Under different levels of uniform noise, SmODE, functioning as a policy network, achieved the lowest average action fluctuations and the best performance compared to MLP and MLP+LowFilter in the whole Mujoco tasks. Comparing MLP with MLP+LowFilter, we observe that while incorporating a low-pass filter reduces action fluctuations ratio, it may also lead to performance degradation in the Humanoid-v3, Walker2d-v3, and Hopper-v3 tasks.

> Minor issues

Thank you very much for reading the paper so carefully, we have revised all the relevant expressions in the paper, and we believe that these changes can play a very important role in improving the readability and quality of the paper.

> Suggestion

Your suggestion to use $h(t)$ as hidden states is quite good to be consistent with previous work. However, both $h(t)$ and $x(t)$ have been defined in the current paper, and redefining $h(t)$ as hidden states requires more formulae to be modified, which may cause trouble to other reviewers. I hope you can understand that we will not make changes for the time being, and we will do it uniformly after the review is finished, thank you very much.

> Question

We reviewed the work you mentioned [2], which aims to address the challenge of inferring unseen information from raw observations in a partially observable (PO) environment. Their approach proposes a recursive model based on Neural ODEs, integrated with a model-free reinforcement learning framework. Unseen information is inferred by encoding historical observations, actions, and rewards, and its effectiveness is demonstrated across various PO continuous control and meta-reinforcement learning tasks. The core of the model is the GRU-ODE, which combines gated recurrent units with neural ODEs to manage hidden state transitions and potential state transitions.

While both apply Neural ODE to deep reinforcement learning, this work you mention is quite different from the neural network control non-smoothing problem we are trying to solve. Therefore, the methods devised are quite different, but we both agree that Neural ODE is highly robust.

> Reference

[2] Zhao et al. "ODE-based Recurrent Model-free Reinforcement Learning for POMDPs", NeurIPS, 2023.

We appreciate the reviewer for the careful reading of our paper and detailed discussions.

> Major issue 1

Your suggestion is spot on, the discussion about Limitation should really be shown in the main text section. We have revised the paper to include that section at the end of page 10.

> Major issue 2

2.1

Thanks to your suggestion, we have rewritten Equation 12 as

$\frac{\mathrm{d}x _ {i}}{\mathrm{d}t}=\sum _ {j}\left[-\frac{w _ {ij}}{C _ {\mathrm{m} _ {i}}}\sigma _ {i}\left(x _ {j}\right)x _ {i}+\frac{w _ {ij}}{C _ {\mathrm{m} _ {i}}}\sigma _ {i}\left(x _ {j}\right)\cdot\tanh(h\left(x _ {j},\theta\right))\right]+x _ {\mathrm{leak} _ {i}}$

2.2

You can look at line 242. We define $g(x(t), I(t), t, \theta)=\text{tanh}(h(x(t), I(t), t, \theta))$, where tanh stands for a hyperbolic tangent function, and $h(x(t), I(t), t, \theta)$ stands for a neural network. Thus, $g(x(t), I(t), t, \theta)$ takes values in the range (-1, 1).

To describe it further, $g(x(t), I(t), t, \theta)_i$ outputs a scalar $G$ after getting the state input, then $g(x(t), I(t), t, \theta)^{max}_i=|G|, g(x(t), I(t), t, \theta)^{min}_i=-|G|$.

2.3

You're right that this part could be misleading. We redefine $\frac{\mathrm{d}x}{\mathrm{d}t} = L(x)$, where $L(x)=l(x(t),x(t+\Delta t))$. The $l$ in the original equation, although it takes only one parameter in the differential equation, is extended after numerical discretization to accept values at two points in time, which is done to achieve an implicit solution.

> Major issue 3

As you can sense, “Bionic modeling” can be confusing at first glance, but the term actually comes from the work on Neural circuit policies (NCP) enabling auditable autonomy [1]. The researchers in this work reference the structure of the nervous system of Caenorhabditis elegans, a nematode renowned for its near-optimal neural network architecture and coordinated neural information processing mechanisms. NCP mentions that bionic modeling of neurons using imitation Caenorhabditis elegans can enhance their computational capabilities. Therefore, biological concepts such as membrane capacitance are mentioned in our paper as learnable parameters with an initial range of values consistent with those in the NCP.

> Reference

[1] Lechner et al. "Neural circuit policies enabling auditable autonomy", Nature machine intelligence, 2020.

Thank you for comprehensively addressing most of my concerns. I am confident that the final paper is strong and will update my rating from a 6 to an 8. One more note to the authors, the action fluctuation ratio is not a great metric for measuring the amount noise in the output as it is sensitive to the dimensionality of the action space and weighs many small action changes the same as one large action change. I realize that it's an easy metric to compute and that the work you compare against uses it, but I do believe a Fourier transform based metric better captures the notion of action noise reduction. This comment does not detract from existing strength of the work and I do not expect the authors to redo their analysis.

We thank you for the careful reading of our paper and constructive comments in detail.

> Weakness

Your suggestion is valuable; it is essential to test under non-Gaussian noise, as this aligns with the motivation discussed in introduction. Therefore, we decided to experiment with uniform noise as a representative of non-Gaussian noise.

For the MuJoCo environment, we selected four tasks Humanoid-v3, Walker2d-v3, Pusher-v2, and Hopper-v3 for our experiments. Since the state values of different Mujoco tasks vary greatly, we set two levels of uniform noise for the four tasks, as shown in Table1. For the whole tasks, noise is added to all states.

Table 1 Different levels of uniform noise for different Mujoco tasks. The numbers in the table represent the range of values for uniform noise.

Three different policy networks were used in the experiments, the first one is MLP network, the second one is MLP+LowFilter and the third one is SmODE. LowFilter is a low-pass filter that utilizes a three-step history state. The extended Kalman filter is not employed here because it requires state transition and observation equations, as well as noise statistics, which are typically unavailable in general reinforcement learning environments such as MuJoCo. The results, which are the averages of five seeds, are shown in Table 2.

Table 2 Experimental results. Average control performance of MLP, MLP+LowFilter, SmODE for different uniform noise levels, where level 1 is on the left column and level 2 is on the right column. The average action fluctuation rate is indicated in parentheses. Results are expressed as mean ± standard deviation of five independent environmental seeds.

Under different levels of uniform noise, SmODE, functioning as a policy network, achieved the lowest average action fluctuations and the best performance compared to MLP and MLP+LowFilter in the whole Mujoco tasks. Comparing MLP with MLP+LowFilter, we observe that while incorporating a low-pass filter reduces action fluctuations ratio, it may also lead to performance degradation in the Humanoid-v3, Walker2d-v3, and Hopper-v3 tasks.

I apologize if my previous explanation was unclear. The vehicle tracking problem is not a POMDP problem, as we have access to the desired trajectory points over a longer time horizon as part of the observation, along with the vehicle's state information.

There is a brief discussion of using filters in page 6 of CAPS [1]. The discussion emphasizes that neural network-based control behaves fundamentally differently from traditional controllers. Neural network policies are typically not trained with integrated filters, so deploying them with a filter alters the dynamic response expected by the network, potentially leading to anomalous behavior. The paper further suggests that naively introducing filters into the training pipeline, without also incorporating the corresponding state history, can result in catastrophic learning failures. This likely occurs because it disrupts the Markov assumption, which is central to reinforcement learning theory. On the other hand, compensating for this by including the state history to maintain the Markov assumption could significantly increase the representation complexity of the problem, due to the higher-dimensional input states. Our SmODE also achieves very good performance and smoothing results even under conditions that do not require direct input of historical states, suggesting that such compromises are unnecessary.

> Reference

[1] Mysore et al. "Regularizing Action Policies for Smooth Control with Reinforcement Learning," IEEE ICRA, 2021.

thank you for the additional experiments. I updated my score.

However: When you argue that your method works better than related work under non-Gaussian noise -- using uniform noise is a bit obvious. Thats still "nice" noise, i.e. unimodal, "Gaussian-like", compared to non-symmetric (think exponential) or multi-modal distributions.

To really bring the point home an alternative way is to make the benchmark partially-observable (e.g masking one or a small subset of sensors). By that you can achieve complex (and state-dependent) noise.