Action Sequence Planner: An Alternative For Offline Reinforcement Learning

Thank you for your valuable feedback. We appreciate the time and effort you have taken to evaluate our work. 1.We acknowledge your concerns about the clarity of our notation and presentation. To improve the readability and comprehension of our method, we have removed the mathematical formulas and notations in Section 3.2. Instead, we have included a more detailed explanation of the method in the appendix, along with its implementation code. We believe this change provides a more intuitive and accessible way for readers to understand our approach. 2.To address your concerns about the empirical results, we have added an additional experiment that reports the performance of our method when the window length k is set to 1. As shown in the updated results, the performance significantly degrades with very small window lengths, underscoring the suitability of our method for planning over longer sequences. Due to time constraints, we were only able to include the experimental results for k=1，however, we believe this is sufficient to demonstrate that this hyperparameter is indeed sensitive in our method. We also acknowledge the importance of a more comprehensive hyperparameter sensitivity analysis in offline RL algorithms, which we plan to explore in future work.

1.We understand that the lack of a full theoretical derivation of the policy gradient theorem under our proposed objective may raise concerns about the validity of the approach. Due to time constraints, we were unable to provide a comprehensive proof in this version of the paper. However, we have included more detailed implementation code, which we believe will give readers a clearer understanding of our approach. We plan to open-source our code in the near future, and we are confident that making it publicly available will allow others to experiment with our method and, through additional empirical studies, demonstrate its effectiveness.

2.We acknowledge your concerns regarding the clarity of our presentation. To improve the readability and comprehension of our method, we have made significant revisions in the paper. Specifically, we have removed some of the more complex mathematical formulas and notations in Section 3.2 to enhance clarity. In their place, we have included a more detailed and intuitive explanation of our approach in the appendix, along with its implementation code. We believe this change provides a more accessible way for readers to grasp our method, and we are confident that the code provided will help further elucidate the key ideas behind our approach.

3.We acknowledge that our current experiments are limited, particularly in terms of the number of environments tested. Due to time constraints, we have focused on a subset of tasks, specifically from the D4RL benchmark, but we plan to expand our evaluation in future work.

1.Thank you for your insightful question regarding the potential relationship between trajectory length and the value of k. As discussed in the manuscript, there is no strong or necessary dependency between these two variables. In practice, we observe that setting k to the length of an episode (epi_len) generally yields satisfactory results in most scenarios.However, when the environment exhibits high randomness, we recommend reducing the value of k. This adjustment allows the model to perform more frequent re-planning, which can help to mitigate the effects of such randomness. The trade-off between the trajectory length and the value of k is essentially a balance between the model’s ability to plan at a global scale versus its ability to adapt to changes and re-plan dynamically.

2.Regarding the impact of smaller vs. larger values of k, we have conducted additional experiments in the revised version of the manuscript, which include the case where k=1. These results clearly show a significant performance degradation when k is set to smaller values.In our approach, larger values of k are more advantageous, as they allow the model to plan over a larger window, facilitating more coherent global planning. When k is larger, the model is able to capture long-range dependencies and plan for extended action sequences, which results in a more stable and effective policy. On the other hand, smaller values of k limit the model’s ability to plan over long trajectories, thus reducing its overall performance. We have updated the manuscript to include these additional experiments and results, which should provide further clarity on this point.

3.Position embeddings are particularly useful when k is not equal to the trajectory length (i.e., when we are performing multi-step planning). In such cases, the model needs to plan actions in segments rather than over the entire trajectory at once. Position embeddings help the model identify the current step or position within the trajectory, which is crucial for effective multi-step planning. Intuitively, by knowing its current position in the action sequence, the model can more accurately plan future actions and make adjustments as needed. This mechanism is similar to how positional information is used in sequence models like Transformers, where the model benefits from knowing the relative position of tokens in a sequence. We believe this is a natural and effective way for the model to handle intermediate planning steps, especially when dealing with long action sequences.

1.Thank you for your comments and concerns regarding the cross-entropy loss computation and the use of Gaussian outputs in our model. We understand that the explanation of how our loss components are computed may have been unclear, and we appreciate the opportunity to clarify this.Thank you for your comments and concerns regarding the cross-entropy loss computation and the use of Gaussian outputs in our model. We understand that the explanation of how our loss components are computed may have been unclear, and we appreciate the opportunity to clarify this.To address your first point, while our neural network outputs both the mean and standard deviation of a Gaussian distribution, only the mean is used to compute the cross-entropy (CE) loss. The mean represents the central tendency of the predicted action distribution, which is treated as the action in our loss computation. The standard deviation, together with the mean, is used when calculating the Kullback-Leibler (KL) divergence to regularize the policy. We have now revised the manuscript to clarify this distinction and removed the more complex mathematical symbols and formulations to make the presentation clearer. The detailed, intuitive explanation and the specific implementation code are now included in the appendix, which we believe will further aid in understanding our approach.Regarding the issue of multidimensional actions, the neural network outputs a sequence of action vectors, where each action vector has dimensions of 2×action_dim×k, with k representing the number of discrete action bins. After generating epi_len/k action sequences, we concatenate these sequences to form a full action trajectory. Once the trajectory is formed, we perform normalization to ensure that the actions are appropriately scaled for the training process. We have added additional details and code examples in the appendix to address this specific concern.

2.As you pointed out, both cross-entropy and KL minimization are closely related, and their mathematical equivalence in some contexts can lead to confusion. We would like to clarify that cross-entropy loss and KL divergence are used in different stages of our training process and are not intended to be combined into a single objective.

Cross-Entropy Loss: The CE loss is used during the training process to directly match the predicted action (mean of the Gaussian) to the target action, providing a stable gradient for the network. This loss is computed using only the mean output of the Gaussian distribution. KL Divergence: The KL divergence is used as a regularization term to ensure that the learned policy does not deviate too much from a prior distribution, and both the mean and standard deviation are involved in its calculation. Thus, these two losses serve different purposes: CE loss helps align the predicted actions with the target distribution, while KL divergence regularizes the network to maintain a reasonable exploration-exploitation balance. We have revised the paper to clearly differentiate these two objectives and explain their distinct roles.

1.Thank you for your insightful comment regarding the use of discounted return as a solution for inaccurate value estimation. We acknowledge the concern that directly using discounted returns, without a more sophisticated value function, may result in higher variance and thus affect training stability. However, our approach, which replaces the value/advantage function with discounted returns, serves as an initial, simplified step toward addressing the challenges posed by inaccurate value estimation and error accumulation. 2.We appreciate your concern regarding the use of fully connected (FC) layers to directly generate action sequences. While it is true that sequence models like RNNs and Transformers are generally preferred for sequence modeling due to their ability to capture long-range dependencies, our choice of FC layers was motivated by the goal of providing a simple and computationally efficient baseline for our method. In our experiments, this approach yielded stable results with relatively low time cost during training, which is particularly important for verifying the effectiveness of our proposed loss function. We also recognize that sequence models such as Transformers and RNNs might perform better in capturing temporal dependencies in more complex scenarios. We plan to extend our method in future work by incorporating sequence models like RNNs or Transformers, and will validate these alternatives in subsequent experiments. For now, our focus has been on establishing a straightforward approach that provides a solid foundation for further exploration. 3.We agree that if a constant is added to all actions in a trajectory, it could lead to an unchanged distribution, which may pose issues during training. To mitigate this, we apply a normalization step to the action distribution, ensuring that the trajectory distribution is adjusted appropriately, and that the learned action sequence remains meaningful. 4.We acknowledge your concerns about the clarity of our notation and presentation. To improve the readability and comprehension of our method, we have removed the mathematical formulas and notations in Section 3.2. Instead, we have included a more detailed explanation of the method in the appendix, along with its implementation code. We believe this change provides a more intuitive and accessible way for readers to understand our approach. 5.Regarding the performance of ASP-MLE compared to BC, the lower performance of ASP-MLE in our experiments is indeed due to the instability introduced by using the maximum likelihood estimation loss on entire action sequences. As illustrated in the gradient magnitude plots, this instability results in erratic gradients and can lead to incorrect updates during training. In contrast, our method with cross-entropy loss provides smoother gradient updates, leading to more stable training and better performance.