LipsNet++: Unifying Filter and Controller into a Policy Network

Thank you for your insightful feedback! We are encouraged by the positive aspects of your assessment, i.e., method effectiveness, sound experimental design, and strong benchmarks and figures. Your recognition of the frequency-domain filtering, Section 3.2, and Fig. 12 is especially appreciated.

We now address your comments in detail:

---

> Improved Theoretical Claims

In Line 626, the equality $K=\max_{x'}\left\Vert\nabla f(x')\right\Vert$ follows from Equation (8) in Lemma A.1, rather than a direct application of the mean value theorem. So, the inequality $\leq$ does not apply here. To ensure a more rigorous presentation, we have explicitly included the error term into the theorem in the main text and discussed the limiting case $\rho\to0$. Please refer to our response to reviewer QXnu for the revised theorem.

> Improved Discussion with Additional References

Thank you for pointing out some theoretical filters, such as [1-3]. These works exhibit strong theoretical foundations. In contrast, our approach is more experimental—we don't make any assumptions about system linearity, convexity, or noise distribution. Instead, we introduce a learnable filtering matrix that adaptively learns from data which frequencies are important and which are noise. We have added a detailed discussion in introduction section and don't elaborate here due to space constraints.

> Weakness Concerns

In fact, LipsNet++ differs significantly from LipsNet in structure. First, LipsNet only addresses excessive Lipschitz constant, overlooking the dual causes of action fluctuation. In contrast, our approach introduces the Fourier Filter Layer and Lipschitz Controller Layer, which not only control the Lipschitz constant but also directly address observation noise. Second, while LipsNet has a rigorous Lipschitz constraint, it relies on a strong assumption—only for MLP with piece-wise linear activations (ReLU). Our Jacobian regularization, however, generalizes to any network architectures. Third, LipsNet’s Lipschitz constraint method is too slow for real-time inference, whereas LipsNet++ achieves a 4× faster inference speed, as shown in Appendix H. In summary, the two techniques in our work are both highly novel and impactful, with a structure entirely distinct from LipsNet.

> Comments & Suggestions

The lag when operating at the edge of capability is an interesting topic. Notably, LipsNet++ prevents catastrophic failures when the only option is to use a big acceleration and stop to a halt. The DMControl-Cheetah environment exemplifies this scenario, requiring actions to switch rapidly between -1 and 1 for high-speed movement. The [Link - Figure R1] visualizes the first two action dimensions (rapid switching occurs across all dimensions). As shown in the figures, LipsNet++ preserves the necessary rapid reaction behavior to maintain good control performance (see TAR in Fig. 9) while reducing action fluctuation as much as possible (see AFR in Fig. 9). LipsNet++ can adapt action smoothness dynamically by learning local Lipschitz constants that vary with different obs input.

> Fixed Typos & Unclear Words

Line 160: Typo fixed (even $\to$ even if)

Line 262: Typo fixed (dose $\to$ does)

Line 206: We have clarified this statement by explicitly noting that LipsNet’s Jacobian matrix calculation during inference introduces significant computational overhead. The inference time and their comparisons are already detailed in Appendix H.

Line 557: Fixed by removing 'real'

Appendix typos: Fixed all typos of 'acceleration' and 'obstacle'

> Questions

Q1: We have not tested with adversarial noise, which is a promising direction for future work.

Q2 & Q3: Training is conducted without noise. Our method doesn't require the presence or absence of noise during training. During testing, different levels of noise are added to evaluate performance and action fluctuation under varying noise intensities. Apart from this, the training and test setups remain identical. The ability to achieve filtering even with noise-free training is attributed to Equation (4): the term $\lambda_h\left\Vert H\right\Vert_F$ drives the filter coefficients of frequencies unrelated to control performance to 0, as shown in Fig. 11, 22, and 28. This mechanism is one of our core ideas.

---

Thank you again for your time and valuable feedback! Please let us know if you have any further questions.

References

[1] H. Elad, K. Singh, and C. Zhang. Learning linear dynamical systems via spectral filtering. NeurIPS 2017.

[2] H. Elad, H. Lee, K. Singh, C. Zhang, and Y. Zhang. Spectral filtering for general linear dynamical systems. NeurIPS 2018.

[3] A. Sanjeev, E. Hazan, H. Lee, K. Singh, C. Zhang, and Y. Zhang. Towards provable control for unknown linear dynamical systems. ICLR Workshop 2018.

Thank you for your insightful feedback! We appreciate your recognition of our well-written and convincing results and analyses, reasonable evaluation criteria, nice ideas, the proposed methods that make sense, and the easy-following nature of our paper. The positive aspects of your assessment encourages us.

We now address your comments in detail:

---

> Improved Theoretical Claim

We appreciate your valuable suggestion! We have refined Theorem 3.2 by explicitly incorporating the approximation error, defining the neighborhood area, and describing its asymptotic behavior when $\rho\to 0$, as you suggested. The revised theorem is as follows:

Theorem 3.2 (Lipschitz’s Jacobian Approximation): Let $f:\mathbb{R}^n \to \mathbb{R}^m$ be a continuously differentiable neural network. The Jacobian norm $\left\Vert\nabla_xf\right\Vert$ serves as an approximation of the local Lipschitz constant of $f$ within the neighborhood $\mathcal{B}(x, \rho)$, centered at $x$ with radius $\rho$. The approximation error is given by $\mathop{\max}_{\ \delta\in \mathcal{B}(0,\rho)} \left[ \left(\nabla_x\left\Vert \nabla_x f(x) \right\Vert\right)^\top\delta + o(\delta)\right].$ Moreover, as $\rho \to 0$, the Jacobian norm converges to the exact local Lipschitz constant, i.e. $\left\Vert\nabla_xf\right\Vert \to K(x)$.

> Additional Comparisons with Baselines on DMControl

Beyond the comprehensive baseline comparisons in the double integrator environment, Section 4.2 evaluates LipsNet++ against three baselines (MLP, LipsNet-G, LipsNet-L) across all DMControl environments and Appendix J compares it with one baseline (MLP-SN) on Reacher environment. Extending MLP-SN to all environments is impractical due to excessive hyperparameter tuning (Line 1039).

To further address your concerns, we have added experiments with additional baselines on DMControl environments. Now, LipsNet++ is evaluated against all baselines on DMControl, including MLP, CAPS, L2C2, MLP-SN, LipsNet-G, and LipsNet-L.

TAR comparison: [Link - Table R1], AFR comparison: [Link - Table R2]

As shown in the above tables, LipsNet++ maintains SOTA performance on the non-toy DMControl tasks.

> Improved Discussion with Additional References

• Jacobian regularization: Prior works [1,2] use Jacobian regularization to enhance adversarial robustness for neural networks, with [2] proposing an efficient Jacobian norm approximation method (already cited in Line 214). While common [1-4], it is rarely explored in RL. Studies [5,6] have identified high Lipschitz constant of policy network as a cause of action fluctuation. Our Theorem 3.2 shows the Jacobian norm is an approximation of Lipschitz constant, introducing Jacobian regularization as a practical way to control Lipschitz constant in RL.

• Observation filtering: Methods [7-10] enhance RL robustness in algorithm level, e.g., state-adversarial DRL with SA-MDP modeling for inaccurate observations [7] and robust training framework for safe RL with rigorous theoretical analysis and empirical results [8]. In contrast, our approach improves action smoothness and robustness in network level by designing policy network (Fourier Filter Layer), embedding prior knowledge directly into its structure.

We have added a detailed discussion in introduction section and don't elaborate here due to space constraints.

---

Summary

We hope our revisions adequately address your concerns. Please let us know if you have any further questions.

Thank you for your time and valuable feedback!

References

[1] D. Jakubovitz, et al. Improving dnn robustness to adversarial attacks using jacobian regularization. ECCV 2018.

[2] J. Hoffman, et al. Robust learning with jacobian regularization. arXiv 2019.

[3] W. Yuxiang, et al. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. ICCV 2021.

[4] K. Co, et al. Jacobian regularization for mitigating universal adversarial perturbations. ICANN 2021.

[5] R. Takase, et al. Stability-certified Reinforcement Learning via Spectral Normalization, arXiv 2020.

[6] X. Song, et al. LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control, ICML 2023.

[7] H. Zhang, et al. Robust deep reinforcement learning against adversarial perturbations on state observations. NeurIPS 2020.

[8] Z. Liu, et al. On the robustness of safe reinforcement learning under observational perturbations. ICLR 2023.

[9] M. Xu, et al. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv 2022.

[10] H. Zhang, et al. Robust reinforcement learning on state observations with learned optimal adversary. ICLR 2023.

Thank you for your insightful feedback! We appreciate your recognition of our work as a well-motivated approach that makes a strong contribution and as an excellent specimen of leveraging theoretical insights to solve a crucial issue in RL. Your positive assessment greatly encourages us.

We now address your comments in detail:

---

> Theoretical Improvement

We appreciate your valuable suggestion! We have refined Theorem 3.2 by explicitly incorporating the approximation error and its asymptotic behavior. The revised theorem is as follows:

Theorem 3.2 (Lipschitz’s Jacobian Approximation): Let $f:\mathbb{R}^n \to \mathbb{R}^m$ be a continuously differentiable neural network. The Jacobian norm $\left\Vert\nabla_xf\right\Vert$ serves as an approximation of the local Lipschitz constant of $f$ within the neighborhood $\mathcal{B}(x, \rho)$, centered at $x$ with radius $\rho$. The approximation error is given by $\mathop{\max}_{\ \delta\in \mathcal{B}(0,\rho)} \left[ \left(\nabla_x\left\Vert \nabla_x f(x) \right\Vert\right)^\top\delta + o(\delta)\right].$ Moreover, as $\rho \to 0$, the Jacobian norm converges to the exact local Lipschitz constant, i.e. $\left\Vert\nabla_xf\right\Vert \to K(x)$.

This refinement provides a more precise characterization of the approximation error, as you suggested.

> Question on Lemma A.1

We appreciate your question regarding Lemma A.1. Theorem 5.19 in Rudin's book is an existential result that relates the function’s increment to its derivative at some point, without involving the Lipschitz constant or providing an explicit expression for Lipschitz constant; whereas Lemma A.1 explicitly gives an equivalent expression for the Lipschitz constant. In summary, Theorem 5.19 primarily focuses on the bounding variation of a function, while Lemma A.1 focuses on the local Lipschitz property and provides an approximation method for calculating the Lipschitz constant.

---

Summary

We hope our revisions adequately address your concerns. Please let us know if you have any further questions.

Thank you again for your time and valuable feedback!

Thank you for your insightful feedback! We are encouraged by the positive aspects of your assessment, i.e., well-supported claims, mathematically sound derivations, valuable insight, comprehensive related works, novel integration of filtering and control, clear empirical improvements, and good ablation studies.

We now address your comments in detail:

---

> Examine Impact of Control Types

All DMControl environments in our paper (Cartpole, Reacher, Cheetah, Walker) are torque control. To examine the impact of control types as you suggested, we additionally train on position control (DMControl Fish) and velocity control (Gymnasium Bipedal Walker) environments. The Fish task needs to control 5 actions (tail, tail_twist, fins_flap, finleft_pitch, finright_pitch) under position control type, and the Bipedal Walker needs to control 4 motor speed values under velocity control type. Their TAR and ARF are summarized as follows:

TAR comparison: [Link - Table R3], AFR comparison: [Link - Table R4]

These results show that LipsNet++ achieves good action smoothness across all three control types. Additionally, some studies on robotics control also support the effectiveness of LipsNet-related structures in torque control [1] and position control [2].

In fact, we believe LipsNet++ does not assume the control type to be torque, position, or velocity; it ensures action smoothness in all cases and serves as a general solution. The choice of control type should depend on the task requirements.

> Other Comments Or Suggestions

• Formula in Line 98: Typo fixed.
• Thank you for your suggestion. We have renamed Section 3.2 to “Understanding Key Factors Behind Action Fluctuation” and refined the contribution description accordingly. Additionally, we believe analyzing the root causes of action fluctuation is essential. Equation (1) mathematically identifies two fundamental causes, which play a crucial role in guiding the design of the subsequent network layers (Fourier Filter Layer & Lipschitz Controller Layer). Previous works did not distinguish the two causes and treated them as one; therefore, our analysis actually contributes to the broader community.

> Q1: Scalability to Image-based RL

Yes, LipsNet++ can scale to image observations. The core question here is whether LipsNet++ can handle high-dimensional observation tasks. To demonstrate this, we added two experiments: DMControl Cartpole (pixel observation) and Gymnasium Humanoid (348 observations), with the former has image-based observations and the latter has high-dimensional observations.

Given the high-dimensional input, we adopt the following LipsNet++ architecture:

• DMControl Cartpol: [Convolutional Layer -> Fourier Filter Layer -> Lipschitz Controller Layer]
• Gymnasium Humanoid: [Linear Layer -> Fourier Filter Layer -> Lipschitz Controller Layer]

In the Convolutional Layer and Linear Layer, they perform the feature extraction and dimension reduction.

The TAR and AFR are summarized in [Link - Table R5]. These results demonstrate that LipsNet++ can handle high-dimensional observation tasks and can scale to image-based tasks.

> Q2: Gradient Flow for Matrix $H$

$H$ receives gradients from the policy improvement loss $\mathcal{L}'$, which includes the critic value as described in Equation (4) and the last equation in Section 2.1; but $H$ does not directly receive gradients from the critic itself, as the critic does not have a Fourier Filter Layer.

Integrating a Fourier Filter Layer into the critic is an interesting direction. In this case, the actor and critic could potentially share one $H$ matrix, raising an interesting question of whether $H$ should receive gradients only from the actor or from both. Thank you for your insightful perspective. We will explore this in future work.

---

Summary

We hope our revisions adequately address your concerns. Please let us know if you have any further questions.

Thank you again for your time and valuable feedback!

References

[1] Y. Zhang, et al. Robust Locomotion Policy with Adaptive Lipschitz Constraint for Legged Robots. IEEE RAL, 2024.

[2] G. Christmann, et al. Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies. IEEE IROS, 2024.