Perlin Noise for Exploration in Reinforcement Learning

Thank you for your thoughtful feedback and for highlighting areas where additional clarification could improve our work. We’ve provided responses below, and please don’t hesitate to ask if more questions arise.

Weaknesses:

“Clarity is my biggest concern. Many math symbols, terms and performance criterion are undefined. Such as t and i in line 211 or the Mean squared Jerk are not properly defined.”

i is the action dimension index, t is the timestep index. These definitions should not have been omitted, we will add them. We thank the reviewer for this important notification. We will also formalize Mean Squared Jerk in the appendix. Jerk is the derivative of acceleration. We provide the mean of the squares of the calculated jerks over a series of rollouts during deterministic evaluation in the environments at regular intervals during training.

“The results produced by the authors are also different from existing work. Pink noise [X] shows that they are outperforming white noise in many tasks and the results are evaluated in a statistical way. But in this work, pink noise seems to be significantly worse than white noise.”

The referenced work tested Pink noise (Colored Noise with $\beta = 1$) in an off-policy setting, benchmarked using SAC. As pointed out in 3.3, one of the problems with Colored Noise is the inability to actually stay on-policy. We expect this to be less of an issue with algorithms designed for off-policy usage. We refer to [1] as a previous work investigating Colored Noise in an on-policy setting. Here Pink ($\beta = 1$) was also found to perform rather badly, the authors instead recommend Colored Noise with $\beta = \frac{1}{2}$, which we refer to as 'HalfBeta' in our work.

“The authors claim that Perlin noise can be used for hard-exploration tasks. But in the selected tasks, there aren't many tasks involving sparse rewards.”

We focused on environments with task-level exploration, such as those provided by MujocoMaze and our custom harder mazes. Some of the tested MujocoMaze environments use sparse rewards, we refer to the MujocoMaze documentation regarding the exact reward definitions: https://github.com/kngwyu/mujoco-maze. We generally found simply converting existing benchmark environments into sparse-reward environments to be ill-suited to test exploration performance as these were build with current algoritms in mind, leading to solvability issues across the board when just sparsifying the rewards. We further believe our experimental evaluation to be quite broad already.

“Smoothness would be the key benefits of using Perlin Noise in robotics. Thus, to fairly compare the benefits, it would be more interesting to compare white/pink noise with a simple PID as the filter.”

While it is true that a PID-based approach can sometimes be applied, as shown in the "Walking in the Park" paper [4], where a policy network predicts motor positions and a PID controller generates the actions to achieve them, this setup relies on access to specific internal states of the environment and their response to actions. In RL, such assumptions are not part of the problem definition, making PID control incompatible with standard RL environments.

Our work aims to remain fully compatible with RL environments, which provide only observations and rewards, without requiring modifications or access to internal environment states. Our method is designed as a drop-in replacement for exploration strategies, ensuring broad applicability without additional demands on the environment.

We would be happy to discuss specific test cases where you believe a PID-based comparison could provide valuable insights.

“Computational cost is not mentioned. How much extra compute cost would Perlin noise bring?”

From a formal perspective, the Complexity is O(n) in the number of action dimensions. (Or more precisely $O(a \cdot n \cdot 2^n)$, where a is the action dimensionality and n is the dimensionality of the spanned Perlin noise, which is always 2 in our case.) From a practical perspective, the provided reference implementation is using a perlin implementation written in vanilla python (no numpy). The training time required per generation is still almost identical between perlin and the other noises. It would be possible to increase the performance significantly by using an efficient implementation.

We thank the reviewer for this important feedback, we have extended the document with a description of the complexity of our sampling mechanism (Section 3.1, line 230)

“No ablation study has been carried out to show the robustness of Perlin noise.”

We kindly refer the reviewer to the Questions section of our response as we believe that their question was answered there.

[Continued in second post due to character limit]

I first would like to thank the authors' efforts on providing more information, which helped me a lot for understanding the paper. I thus, have increased my score. But I do believe that this work still needs more polish, which could not be simply addressed during the rebuttal phase as they might change the scope of the paper. Here are some of my suggestions:

• Currently, the paper title is too ambitous given the results we have seen. Another reviewer has also pointed out that as a general exploration algorithm, applying to both off-policy and on-policy algorithms would be necessary.
• In the appendix, in some of the environments, white noise could still outperform Perlin noise. A proper explaination of these failure cases would also be great for the practationer.
• The paper writing should be improved.
  • Visualization: Legends/x,y-axis label/ticks font size are too small. Unclear definition of the figure, such as the blue dot in fig. 1.
  • The normalization function and its calculation are unclear to me when reading first through it.
• For on-policy algorithms, such as PPO, the number of parallel enviroments would also be one essential parameter to investigate. Especially, in the current version, only on-policy is tested and all benchmarks are mostly robotics tasks which actually require smoothness. Then it is straightforward to me to see how perlin noise performs when number of envs in parallel increases since in robotics, parallel training is a common approach.

Thank you for your valuable feedback and for taking the time to engage so thoughtfully with our work. We’ve provided responses to your concerns below and hope these address the questions you raised. Please feel free to reach out if additional clarification is needed.

Questions:

“Regarding Figure 5, I understand that the sampled action trajectories are drawn from a fixed underlying distribution, and it is evident that Perlin noise generates much smoother trajectories. However, these smooth trajectories (e.g., over 100 actions) are based on a fixed underlying distribution. In real RL learning or inference, the stochastic policy distribution is conditioned on the current state, meaning the policy distribution changes at each step. In this scenario, does the sequence smoothness still hold? In other words, would generating Perlin noise for a changing policy trajectory still result in smooth action sequences?”

We refer to our 'Response to Reviewer Feedback' for an explanation of what smoothness we are capable of providing. Our Perlin-based disturbances have no impact on the policy mean generated by the policy NN. PPO (according to the original paper and reference implementation) does not contextually parameterize the variance; it is modeled as a simple parameter. Therefore, the variance tends to slowly decrease across the training generations. We guarantee that the applied disturbances are smooth. Whether this also results in actually smooth actions is further dependent on the policy parameters. If the NN learns to parameterize high-frequency actions, the resulting trajectories will also be jerky. We provide measured mean squared jerk of the actions as a measure of action smoothness in the appendix.

We generally observe that the smoothness in action disturbances also leads to higher smoothness of sampled actions during training as can be seen in Appendix B.

“Does the use of smooth action trajectories imply that action changes are relatively slow? Could this affect the speed of learning or inference? For instance, in scenarios that require rapid action adjustments (such as emergency braking in autonomous driving), would this smoothness impact the effectiveness of the policy?”

Our idea was in parts inspired by recent progress with movement primitive-based trajectory-centric RL. These techniques often achieve higher performance, but at the cost of being unable to generate high-frequency actions (and not being directly applicable to normal step-based RL envs). Remaining in the step-based paradigm, but carrying over the smooth trajectory disturbance behavior from these (as we do with Perlin-based exploration), we see that we can increase the performance of such traditional step-based algorithms as PPO. We also carry over a bias for smooth actions. However, contrary to MP RL, we are not unable to parameterize high-frequency actions; we merely disfavor them during training. This will still harm the ability (extend required time) to converge for certain tasks that require such high-frequency actions (we see this as the reason for the sub-par performance indicated in Figure 12(e): DMC Hopper Hop), but contrary to MP RL, we do remain able to learn high-frequency motions if required eventually. Exploration methods tend to be disabled when deploying policies (switching to deterministic policy evaluation), so we have no impact on the performance of a deployed system, apart from maybe having helped the agent to learn a better policy.

“Besides generating smooth action sequences, does Perlin noise effectively increase the diversity of the sampled actions? In model-free RL, efficient exploration, i.e., collecting more diverse samples, is critical for improving sample efficiency. How does the introduction of Perlin noise influence the breadth of exploration?”

We believe it is highly depending on the definition of 'diversity'. As can be seen in Figure 1, we achieve a higher coverage of the state space. We would regard this as an indication of higher diversity of trajectories. But this requires looking at the diversity of whole trajectories. For a per-action treatment, one could also try to formalize the 'diversity' of an action via its entropy. In this case, Perlin-based exploration will actually decrease the 'diversity', as we favor consistent actions to follow one another, therefore decreasing the entropy for each individual step. Perlin restricts the diversity of any individual action in favor of higher diversity of complete interactions with the environment by being able to escape from local optima and expanding the scope of states reached.

[Continued in second post due to character limit]

Thank you for your careful reading of our work and for your insightful feedback. We appreciate the opportunity to clarify and expand upon our contributions. Should further questions arise, we would be more than happy to provide additional clarification.

Weaknesses:

“Perhaps my main concern with this paper is that, at a high level, it simply proposes and investigates a different noise function for PPO. While simple solutions especially if they solve major problems are not necessarily bad, the novelty and technical contribution here is quite limited.”

Yes, it’s simple—like PPO’s advantage over TRPO, simplicity here is intentional and valuable. Perlin noise adds novel structured, smooth exploration that current noise types lack, ensuring fluid motions and overcoming local optima with minimal changes. While many complex methods can boost performance, they often aren’t adopted due to implementation and tuning difficulty. In contrast, we see our approaches simplicity and drop-in usability as an advantage, making it feasible for practical use and further research.

“Since the claim is that Perlin noise provides more smooth, structured exploration independent of the algorithm or task, it seems necessary to have at least one more algorithm beyond PPO to validate this claim.”

We agree with the reviewer that further research across algorithms is valuable. In this work, we focused on on-policy methods, as they tend to suffer more from converging on local optima and imitating jerky actions seen during training. PPO was chosen as the de facto standard for on-policy, but future work could explore Perlin noise’s impact on other on- and off-policy methods. But we see this as outside the scope of this paper and this rebuttal phase.

“Figure 5 needs some context on what the task is, what kind of policy was used to collect the trajectory (is it a random policy? Trained policy? Hand-engineered one?), etc.”

Other reviewers asked similar questions. We refer to our 'Response to Reviewer Feedback' for an answer and have updated the paper for more clarity (changes also described in the 'Response to Reviewer Feedback').

“On lines 59-60 the authors contrast against intrinsic-motivation and trajectory-level noise methods by saying ‘However, these methods require additional treatment of the underlying optimization method by changing the objectives or introducing higher-level abstractions of the actions.’ Why is this a problem?”

We regard not requiring more extensive adjustments as a benefit, as it allows using our technique as a drop-in replacement. Often, such adjustments also result in new additional problems, as the inability to generate high-frequency actions we observe with trajectory-level RL.

Thank you again for your constructive feedback and for helping us refine our paper. We hope these responses clarify our approach and motivation.