FlipNet: Fourier Lipschitz Smooth Policy Network for Reinforcement Learning

Continue here ...

> Minors

All typos have been corrected in the uploaded revision paper.

> Question 2

Yes, $j$ is the imaginary unit. Its definition has been added in the uploaded revision paper.

> Question 3

FlipNet performs well in high-dimensional observation tasks. The newly added Humanoid environment is such a task with 348 observations. And the above results show that FlipNet still maintains good robostness and smoothness.

Additionally, the FFL (Fourier Filter Layer) is an network layer, which can be inserted in arbitrary position beyond the input layer in the paper. This means you can first use a layer for feature dimensionality reduction and then pass the features through the FFL. We added an experiment using Humanoid where the policy network is [Linear(348, 256), ReLU, FFL, Linear(256, 256), ReLU, Linear(256, 17)]. The following results show that the network with FFL placed afterward exhibit comparable robustness and smoothness to those with FFL placed beforehand.

> Question 5

Using complex matrices in the Fourier Filter Layer allows for control over both the amplitude and the phase of frequency components. After the Fourier transform, each element in $X$ is a complex number, i.e. $X_{u,v}=a+bj$. The corresponding frequency amplitude is determined by its magnitude $\sqrt{a^2+b^2}$, and the corresponding frequency phase is determined by its phase $\arctan\left(\frac{b}{a}\right)$.
Similarly, after multiplying by the filter matrix $H$, the filtered frequency amplitude and phase are determined by the magnitude and phase of Hadamard product $X\odot H$, respectively.

(1) When $H$ is a real matrix, the Hadamard product $X\odot H$ cannot change the frequency phases. By adjusting the element values in $H$, only the amplitude of each frequency can be altered (suppressed or enhanced). In this case, the output obtained by the inverse Fourier transform retains a clear physical meaning.

(2) When $H$ is a complex matrix, the Hadamard product $X\odot H$ can change both the frequency amplitudes and the frequency phases. In this case, the sequence obtained by the inverse Fourier transform is not merely a linear combination of frequencies, but rather an arbitrary combination of frequency functions with arbitrary phases. This makes the output more abstract, essentially becoming a type of feature.

Previously, the use of complex matrices in FFT has already been recognized as a feature extractor in NLP field$^{[1]}$.

> Question 6

10 seeds are used in these figure and tables. The number of seeds has been specified in Appendix J now.

> Question 7

Different observation dimensions have different amplitudes. Based on the magnitude of the observations, we selected approximately 10% to 30% of the values as the noise magnitude.

> Question 8

Fig. 10 presents the action curve from a single experiment in a real-world driving task; hence, there is no error bar in it.

Fig. 13 shows the average metrics of four scenarios from Fig. 28, rather than the average across multiple seeds, so there is no error bar in it.

Summary

We hope our response can address your concerns, and be glad to hear any remaining concerns. Given the unusual score from one reviewer, we would be extremely grateful if you could consider increasing your score. Looking forward to your response and appreciating your consideration!

References

[1] J. Lee-Thorp, et al. FNet: Mixing Tokens with Fourier Transforms, ACL 2022.

We thank you for the careful reading and insightful review!

After carefully considering your concerns, we now address each of them below:

> Weakness 1 & Question 1

In RL control tasks, action is computed as $a_t = \pi(o_t)$ and observation is given by $o_t = s_t + \xi_t$, where $s_t$ represents the current state and $\xi_t$ denotes any type of observation noise. The term $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert$ you mentioned is actually $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert = \Vert \frac{\mathrm{d} (s_t + \xi_t)}{\mathrm{d}t} \Vert = \Vert \frac{\mathrm{d} s_t}{\mathrm{d}t} + \frac{\mathrm{d} \xi_t}{\mathrm{d}t} \Vert$. The first part $\frac{\mathrm{d} s_t}{\mathrm{d}t}$, i.e. the change rate of dynamics system, is related to the inherent property of the target dynamic system. Therefore, when high-frequency noise occurs, the second part $\frac{\mathrm{d} \xi_t}{\mathrm{d}t}$ will causes $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert$ to increase significantly. Thank you for pointing out the potential source of misunderstanding. In the revised paper, we have decouple $o_t$ as two terms, i.e. $s_t$ and $\xi_t$, to make it clear. We kindly invite you to check Section 3.1 in the uploaded revision paper.

> Weakness 2

In fact, the noise model for Mini-Vehicle Driving have already been illustrated in Appendix M of our original paper. We kindly invite you to refer to the first paragraph in Appendix M, where both the noise amplitude and uniform distribution were clearly described. Additionally, we would like to emphasize that FlipNet does not require specific noise models. This is because the Fourier Filter Layer learns from the data whether each frequency should be suppressed or enhanced. The learned optimal filter matrix depends on the training data rather than predefined noise models.

> Weakness 3 & Question 4

(1) For hyperparameters: We have already done sensitivity analysis for $\lambda_k$, $\lambda_h$, and $N$ in Appendix F and G of our original paper. We kindly invite you to refer to Fig. 17, Fig. 18, and Fig. 19. The results show that all the hyperparameters have very low sensitivity, making FlipNet convenient for tuning and easy to use.

(2) For backward time: We greatly thank you for poing out the difference between backward time and training wall-clock time. 10x backward time does not mean the training time is 10x slower. Backward propagation is just one step in RL. RL involves additional time-consuming steps beyond backward, such as sampling and evaluation. According to your suggestion, we list the training wall-clock times for 1M iterations of TD3 on DMControl envs below.

On average, the training time of FlipNet is 1.6 times that of MLP, which more accurately illustrates the impact on training time. The result is updated in Appendix H. We kindly invite you to check it in the uploaded revision paper.

In conclusion, the hyperparameters have low sensitivity and the training time is acceptable.

> Weakness 4

In addition to DMC, we also evaluate on the real-world Mini-Vehicle Driving task, which is a more valuable benchmark. We excluded DMC's Dog, Quadruped, Hopper, and Ant because TD3 performs poorly in these challenging environments, which fall outside the scope of our paper. Similarly, the baseline method, i.e. LipsNet, excluded these 4 environments in their paper for likely the same reason.

In order to address your concern, we added 3 new environments in Gymnasium: Hopper, Ant, and Humanoid (Quadruped and Ant are the same environment in Gymnasium).

Total average return:

• |Env|Noise $\sigma$ | MLP | LipsNet | FlipNet | |---|---|---|---|---| |Hopper| 0.05 | 3304 ± 4 | 3461 ± 3 | 3460 ± 8 | || 0.1 | 3127 ± 388 | 3302 ± 21 | 3459 ± 15 | |Ant| 0.05 | 3610 ± 44 | 3791 ± 38 | 3804 ± 44 | || 0.1 | 1903 ± 206 | 2712 ± 301 | 3390 ± 449 | |Humanoid| 0.05 | 5021 ± 46 | 5267 ± 15 | 5503 ± 21 | || 0.1 | 4984 ± 39 | 5069 ± 22 | 5505 ± 13 |

Action fluctuation ratio:

• |Env|Noise $\sigma$ | MLP | LipsNet | FlipNet | |---|---|---|---|---| |Hopper| 0.05 | 1.00 ± 0.02 | 0.95 ± 0.01 | 0.90 ± 0.01 | || 0.1 | 1.22 ± 0.05 | 1.13 ± 0.03 | 1.01 ± 0.02 | |Ant| 0.05 | 1.30 ± 0.01 | 1.31 ± 0.01 | 1.30 ± 0.01 | || 0.1 | 1.56 ± 0.04 | 1.42 ± 0.02 | 1.35 ± 0.01 | |Humanoid| 0.05 | 1.00 ± 0.02 | 0.79 ± 0.02 | 0.67 ± 0.02 | || 0.1 | 1.07 ± 0.01 | 0.90 ± 0.02 | 0.72 ± 0.02 |

In conclusion, FlipNet has significantly better robostness and smoothness in high noise situations.

Continue in the next reponse ...

We thank you for the careful reading and insightful review!

After carefully considering your concerns, we now address each of them in turn:

> Weakness 1 & Question 1

Sorry that you might be misled by our paper writing. In fact, MLP-SN and LipsNet-G have already been compared in our original paper using the double integrator environment. We kindly invite you to refer to Fig. 8, Tab. 3, and Tab. 4, which contain detailed comparison data. Also, the comparison results have already been discussed in Line 378 ~ Line 384.

As for the DMControl environments, we have already compared MLP, LipsNet, and FlipNet. Also, the MLP-SN is compared on DMControl's reacher environment, whose result can be found in Tab. 13 in our original paper. What we want to emphasize is that compare MLP-SN on all DMControl environments is not realistic. Because this would necessitate the manual tuning of spectral norm hyperparameters for each layer in MLP-SN, which have an unwieldy number of potential hyperparameter combinations, as said in Appendix J.

> Weakness 2

Thank you for pointing out. We have added figure number and caption in Section 3.4, and kindly invite you to check it in the uploaded revision paper.

> Question 2

In RL control tasks, action is computed as $a_t = \pi(o_t)$ and observation is given by $o_t = s_t + \xi_t$, where $s_t$ represents the current state and $\xi_t$ denotes any type of observation noise. The term $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert$ you mentioned is actually $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert = \Vert \frac{\mathrm{d} (s_t + \xi_t)}{\mathrm{d}t} \Vert = \Vert \frac{\mathrm{d} s_t}{\mathrm{d}t} + \frac{\mathrm{d} \xi_t}{\mathrm{d}t} \Vert$. The first part $\frac{\mathrm{d} s_t}{\mathrm{d}t}$, i.e. the change rate of dynamics system, is related to the inherent property of the target dynamic system. Therefore, when high-frequency noise occurs, the second part $\frac{\mathrm{d} \xi_t}{\mathrm{d}t}$ will causes $\Vert \frac{\mathrm{d}o_t}{\mathrm{d}t} \Vert$ to increase significantly. Thank you for pointing out the potential source of misunderstanding. In the revised paper, we have decouple $o_t$ as two terms, i.e. $s_t$ and $\xi_t$, to make it clear. We kindly invite you to check Section 3.1 in the uploaded revision paper.

Summary

We hope our response can address your concerns, and be glad to hear any remaining concerns. Given the unusual score from one reviewer, we would be extremely grateful if you could consider increasing your score. Looking forward to your response and appreciating your consideration!

Continue here ...

> Question 1

Yes, we have considered it. However, although the Butterworth filter has lower computational overhead, we aim to avoid introducing a predefined response curve shape, which could significantly degrade control performance. In contrast, our Fourier Filter Layer learns from the data whether each frequency should be suppressed or enhanced, offering higher flexibility without sacrificing performance. If the data happens to be suitable, our Fourier Filter Layer can learn a Butterworth filter. Additionally, its forward time is within 0.2 ms, meeting the requirements for real-time processing.

> Question 2

FlipNet performs well in rapid reaction environments. The DMControl-Cheetah is such a environment, where the actions need to change rapidly between -1 and 1 to make cheetah move fast. Some figures and data are provided in LINK to illustrate its effectiveness.

> Question 3

Using complex matrices in the Fourier Filter Layer allows for control over both the amplitude and the phase of frequency components. After the Fourier transform, each element in $X$ is a complex number, i.e. $X_{u,v}=a+bj$. The corresponding frequency amplitude is determined by its magnitude $\sqrt{a^2+b^2}$, and the corresponding frequency phase is determined by its phase $\arctan\left(\frac{b}{a}\right)$.
Similarly, after multiplying by the filter matrix $H$, the filtered frequency amplitude and phase are determined by the magnitude and phase of Hadamard product $X\odot H$, respectively.

(1) When $H$ is a real matrix, the Hadamard product $X\odot H$ cannot change the frequency phases. By adjusting the element values in $H$, only the amplitude of each frequency can be altered (suppressed or enhanced). In this case, the output obtained by the inverse Fourier transform retains a clear physical meaning.

(2) When $H$ is a complex matrix, the Hadamard product $X\odot H$ can change both the frequency amplitudes and the frequency phases. In this case, the sequence obtained by the inverse Fourier transform is not merely a linear combination of frequencies, but rather an arbitrary combination of frequency functions with arbitrary phases. This makes the output more abstract, essentially becoming a type of feature.

Previously, the use of complex matrices in FFT has already been recognized as a feature extractor in NLP field$^{[3]}$.

Summary

We hope our response can address your concerns, and be glad to hear any remaining concerns. Given the unusual score from one reviewer, we would be extremely grateful if you could consider increasing your score. Looking forward to your response and appreciating your consideration!

References

[1] G. Shani, et al. A Survey of Point-based POMDP Solvers, AAMAS, 27, 1-51, 2013.

[2] D. Hafner, et al. Mastering Atari with Discrete World Models, ICLR 2021.

[3] J. Lee-Thorp, et al. FNet: Mixing Tokens with Fourier Transforms, ACL 2022.

We thank you for the careful reading and insightful review!

After carefully considering your concerns, we now address each of them in turn:

> Weakness 1

Using historical observations is a very common practice for the following three reasons:

(1). You can easily find ReplayBuffer classes for storing historical observations in various RL training tools. For instance, the num_steps parameter in TensorFlow Agents (link), the stack_num parameter in Tianshou (link), and the sequence parameter in RLlib (link) are all for historical observations usage.

(2). In POMDP$^{[1]}$ or RL algorithms like Dreamer$^{[2]}$, they all use historical observations as a very common setting.

(3). In real-world control tasks, using historical observations is essential—for example, in autonomous driving, this has become a widely accepted consensus.

> Weakness 2

Thank you for your feedback. Now, the code is accessible in our anoymous GitHub (link).

> Weakness 3

(1). For the forward time: As shown in Tab. 6, FlipNet's 1-batchsize forward time is 0.16ms, and that of MLP is 0.10ms. Although 0.16ms appears to be 2x slower than 0.10ms, the absolute value of 0.16ms is very small and fully meets the requirements for real-time inference.

(2). For the backward time: 10x backward time does not mean the training time is 10x slower. Backward propagation is just one step in RL. RL involves additional time-consuming steps beyond backward, such as sampling and evaluation. To more accurately illustrate the impact on training time, we list the training wall-clock times for 1M iterations of TD3 on DMControl envs below.

On average, the training time of FlipNet is 1.6 times that of MLP, which more accurately illustrates the impact on training time. The result is updated in Appendix H. We kindly invite you to check it in the uploaded revision paper.

> Weakness 4

Besides not being network-level methods, we did not evaluate CAPS and L2C2 also due to their high sensitivity to penalty coefficients. Tuning these hyper-parameters to balance their performance and smoothness for all tasks in our paper is not practical. However, in order to address your concern, we have added comparative experiments with CAPS and L2C2 by carefully tuning in the double integrator environment.

Total average return comparison:

• |Noise $\sigma$ | MLP | CAPS | L2C2 | MLP-SN | LipsNet-G | LipsNet-L | FlipNet | |---|---|---|---|---|---|---|---| | 0.01 | -51.0±0.1 | -52.1±0.1 | -55.0±0.1 | -62.0±0.1 | -53.2±0.1 | -55.3±0.1 | -56.5±0.1 | | 0.05 | -53.5±0.2 | -53.0±0.2 | -55.4±0.4 | -62.3±0.4 | -54.3±0.3 | -55.6±0.4 | -56.8±0.2 | | 0.1 | -59.5±0.6 | -56.9±0.6 | -57.5±0.5 | -62.8±0.7 | -54.2±0.7 | -56.0±0.6 | -57.1±0.6 | | 0.2 | -78.4±1.8 | -67.9±1.0 | -63.4±1.5 | -65.9±1.7 | -59.8±1.1 | -58.7±1.4 | -57.9±0.6 | | 0.3 | -103.2±3.7 | -85.9±3.0 | -82.3±4.4 | -71.8±2.3 | -74.3±2.1 | -65.3±1.6 | -59.9±1.9 |

Action fluctuation ratio comparison:

• |Noise $\sigma$ | MLP | CAPS | L2C2 | MLP-SN | LipsNet-G | LipsNet-L | FlipNet | |---|---|---|---|---|---|---|---| | 0.01 | 0.02±0.01 | 0.01±0.01 | 0.01±0.01 | 0.01±0.01 | 0.01±0.01 | 0.01±0.01 | 0.00±0.01 | | 0.05 | 0.11±0.01 | 0.07±0.01 | 0.05±0.01 | 0.03±0.01 | 0.04±0.01 | 0.03±0.01 | 0.01±0.01 | | 0.1 | 0.19±0.01 | 0.15±0.01 | 0.10±0.01 | 0.06±0.01 | 0.07±0.01 | 0.06±0.01 | 0.02±0.01 | | 0.2 | 0.34±0.02 | 0.26±0.01 | 0.20±0.01 | 0.13±0.01 | 0.17±0.01 | 0.12±0.01 | 0.03±0.01 | | 0.3 | 0.48±0.02 | 0.37±0.02 | 0.33±0.02 | 0.20±0.01 | 0.28±0.01 | 0.20±0.01 | 0.05±0.01 |

The results show that CAPS and L2C2 cannot balance performance and smoothness well, and FlipNet is still the SOTA method. We kindly invite you to check Tab. 3, Tab. 4, and Fig. 8 in the uploaded revision paper, where the new results have been added.

Continue in the next reponse ...

Thank you for your feedback.

> Weakness 1

We respect your workloads on real robots, where using history observations is widely recognized as essential. While it's possible to list RL libraries that do or do not support historical observations, this is beyond the scope of our paper; our focus is on facilitating smoother control for your real robot scenarios.

Furthermore, the support of history observation can be achieved with minimal effort—using a single line of code to call a wrapper gymnasium.wrappers.FrameStackObservation (link), applicable to any RL library. This approach also preserves the Markov property, as the augmented state $[s_t, s_{t-1}, ..., s_{t-N}]$ forms a new MDP. Given your workloads with real robots, we trust you may familiar with this straightforward solution.

We don’t focus on which RL library you use or how you implement the wrapper; replacing the MLP with FlipNet in tasks with historical observations achieves smoother control—this is our straightforward contribution.

> Weakness 3 & 4

Thank you for suggesting a Limitation Section. We have added Appendix N for this purpose.

> New concern

1. The reference link focuses on the global Lipschitz constant, while our Theorem 3.1 emphasizes the local Lipschitz constant. Neural networks often exhibit varying behaviors across regions, whose global Lipschitz constant dominated by steep-gradient areas. This makes it less representative and meaningful for most input points, especially in complex machine learning tasks. Our theorem fills this gap by offering a localized analysis.
2. Unlike the reference link, which provides an upper bound equaling the Lipschitz constant only in convex domains, our Theorem 3.1 imposes no such assumptions and instead establishes that the Jacobian norm serves as a first-order approximation of the local Lipschitz constant.
3. Most importantly, prior works like MLP-SN and LipsNet directly optimize the Lipschitz constant—a challenging approach. In Theorem 3.1, the relationship between local Lipschitz constant and Jacobian norm shows that regularizing Jacobian norm is a simpler way to smooth policy networks, offering valuable insight for the community.

We sincerely appreciate your active feedback and specific suggestions.

It is clear that the authors are not taking my negative suggestions seriously, the limitations section they added is buried as the last section of the appendix, and does not represent the limitations in any meaningful manner. Instead, the section is now about future work more than it is about limitations, and the limitations themselves are heavily downplayed. No, the FrameStackObservation wrapper doesn't solve the problem. Adding history to the observational space means adding it to the inputs of the Value functions that these RL algorithms use, which is not what this method intends to do, it also wastes memory as the observations are duplicated for each state to keep it an MDP. Even if I accept the wrapper argument, this allows for having historical observations during training, not during inference where the constraints of implementations apply. It's clear the authors will only give arguments against my concerns rather than incorporate them improving the paper. Even though they have addressed some of my other concerns, my main issue was with limitations representation, as this is still far from improved in the paper, I will lower my score.