Offline Behavior Distillation

Q1: Related work on offline data generation of RL should be provided and compared. The proposed Av-PBC method is only compared with random policy, are there other related works? Is this paper the first work of offline data generation in RL?

A1: Thanks for your comments. To the best of our knowledge, our paper is the first to achieve behavioral data distillation from offline RL data. We introduced the novel OBD algorithm Av-PBC and demonstrated its effectiveness by comparing it with two baseline methods, DBC and PBC. Our method shows significant improvements in both performance and speed across multiple datasets.

The most closely related work to ours is (online) behavioral distillation [1]. We have detailed the differences between [1] and our work in the Related Works section (Line 103-114). Our focus is on the offline scenario, where the RL environment is not accessible. This distinction is crucial as it highlights the unique challenges and contributions of our approach; please refer to Line 103-114 for more details.

[1] Andrei Lupu, Chris Lu, Jarek Luca Liesen, Robert Tjarko Lange, and Jakob Nicolaus Foerster. Behaviour distillation. In The Twelfth International Conference on Learning Representations, 2024.

Q2: It is not clear what is the difference between AvPBC and Av-PBC in Section 1.

A2: Thanks and addressed. "AvPBC" is "Av-PBC" in Section 1.

Q3: The network architecture is not clear. Since data generation is required. A generating network should be used in addition to a policy network. More details about the data generation model should be provided.

A3: Thanks for your suggestion. We would like to clarify that our OBD method does not involve the use of generative models. Instead, we optimize the distilled data through backpropagation, applying an element-wise update as outlined by the $`GradDescent`$ procedure in Algorithm 1.

In the first step of $`GradDescent`$, we calculate the meta-gradient $\nabla_\mathcal{D_**syn**}$ via Backpropagation Through Time (BPTT) as described in Equation 3. We then update $\mathcal{D_**syn**}$ by $\mathcal{D_**syn**} \rightarrow \mathcal{D_**syn**} - \alpha \nabla_\mathcal{D_**syn**}$. We will provide additional clarification and detail about this process in our revised paper.

Q4: What is the number of offline datasets as the number of offline data samples influences performance?

A4: Thank you for your comments. We list the sample size of each offline dataset in Table 3 of the PDF file, which can also be found in Table 4 of the Appendix. These datasets range from 0.3 million to 2 million samples. Our Av-PBC significantly outperforms other approaches across these varying dataset sizes, demonstrating its robustness and effectiveness.

Q5: The time and performance of Av-PBC and OffRL should be compared under varying numbers of generated data.

A5: Thanks for your suggestion. We explored the effect of different sizes of synthetic datasets on OBD performance during the response period. The results, presented in Table 2 of the PDF file, indicate that as the size of the synthetic data increases, OBD performance improves. This enhancement is attributed to the larger synthetic datasets conveying more comprehensive information about the RL environment and decision-making processes.

Q1: Corollary 1 and Corollary 2 compare the policy for dataset distillation to the policy of offline RL, but we are more concerned about the performance of any policy trained on the distilled dataset comparing to the optimal policy. It would be better to propose a new metric to evaluate the performance of OBD for RL.

A1: Thanks for your comments. As there is no assumption w.r.t. policies, we would like to clarify that in Corollaries 1 and 2, the policy $\pi^\ast$ can represent either a decent policy learned from an offline RL algorithm or the optimal policy. Since the environment and optimal policy are not accessible in the offline setting, we use advanced offline RL algorithms to derive an "estimated expert policy" $\pi^\ast$ from the offline RL data for the loss computation of OBD. Corollaries 1 and 2 theoretically show that this approach effectively assesses the performance of policies trained on the distilled dataset in relation to the practical benchmark policy $\pi^\ast$, given the constraints of the offline RL scenario.

Q2: It is also a challenge to decide the size of the synthetic dataset to strike a balance between training efficiency and performance.

A2: Thanks for your comments. We have tested various sizes of synthetic datasets, and the results are presented in Table 1 of the PDF. This table demonstrates that OBD performance improves with larger synthetic datasets, as they provide more comprehensive RL environment and decision information during the OBD process.

However, a larger synthetic dataset also presents challenges: (1) it increases the number of parameters in the bi-level optimization process, raising computational costs; and (2) it can reduce efficiency when training policies on the synthetic data. Determining the optimal size of the synthetic dataset depends on the specific RL environment and the coverage and quality of the pre-collected data. We plan to conduct a more detailed analysis on this topic in future work.

Q3: Scaling issue: if the synthetic dataset is large, the dataset distillation has a high computational cost.

A3: Thank you for bringing this up. While there is a trade-off between performance and efficiency related to the synthetic data size in OBD (see A2 above), OBD is primarily intended to distill a compact synthetic behavioral dataset for efficient policy training. As the synthetic dataset grows larger, the benefits of OBD diminish. Consequently, we focus on optimizing OBD performance within a limited synthetic data budget. We will incorporate this discussion into our revised manuscript.

Q1: The dependency on an offline RL algorithm makes Algorithm 1 fails to claim that OBD could improve training efficiency when compared to direct application of the offline RL with the whole dataset.

A1: Thanks for your comments. Though Algorithm 1 incorporates a complete offline RL process, once the distillation is complete, the distilled data can be reused in various scenarios. This allows us to train policy networks much more efficiently on distilled data compared to using the large original dataset. Improving the efficiency of OBD is a key focus for future work, particularly given the challenges of bi-level optimization, as discussed in Limitation. Notably, Algorithm 1 significantly enhances distillation speed compared to naive OBD methods such as DBC and PBC.

Q2: The result has limited discussion on the impact of the quality of the initial dataset on the effectiveness of the Av-PBC objective, limiting its applicability in practice.

A2: Thanks. Please refer to A2 in the global response.

Q2.1: The quality of the dataset $D\_\text{off}$ seems to control the effectiveness of the proposed approach in Table 1. The approach only performs better than BC (whole) on the lowest-quality dataset (M-R). Can the authors discuss this finding in more detail?

A2.1: Thanks for your observation. The quality of offline RL data $D\_\text{off}$ directly affects the expert policy $\pi^\ast$ learned through offline RL algorithm, thereby impacting the final performance of Av-PBC by the loss in Eq. 7. On the other hand, the policy $\pi$ trained on distilled data typically performs below of $\pi^\ast$ due to optimization errors. When original data is of lower quality, such as when sampled from sub-optimal or random policy (like M-R), the offline RL algorithm (Offline RL, whole) can learn a much better policy $\pi^\ast$ than BC (whole) because of its specially designed Bellman backup. This significant performance gap between offRL (whole) and BC (whole) explains why our OBD algorithm performs better than BC (whole) with low-quality $D\_\text{off}$.

Q2.2: The "interesting phenomenon for Av-PBC" suggests that offline datasets with better state coverage lead to better OBD performance. Therefore, the Medium-Replay (M-R) dataset should have higher state-action coverage than the M dataset, but Av-PBC with the M dataset performs better.

A2.2: Thanks for your comments. We respectfully argue that we only claim that state coverage is crucial for OBD performance, and data quality is also another essential factor, as discussed in A2.1. Although M-R datasets have better state coverage, M datasets have superior data quality, which contributes to its better performance in the Halfcheetah environment. Our additional experiments further explore the impact of state coverage on OBD, revealing that greater state coverage enhances robustness against data noise, which benefits OBD performance. Please refer to A2 in the global response for more details.

Q2.3: Why does the Av-PBC based ensemble not improve OBD performance with the M-R and M datasets of the Hopper environment?

A2.3: Thanks for pointing this out. The reason the Av-PBC based ensemble does not improve OBD performance with the M-R and M datasets in the Hopper environment is that Hopper is simpler compared to Halfcheetah and Walker2D. Specifically, Hopper has a state dimension of 11 and an action dimension of 3, while Halfcheetah and Walker2D have state dimensions of 17 and action dimensions of 6. In more complex or high-dimensional environments, there is greater policy or model uncertainty, and policy ensembles are more effective at reducing this uncertainty and improving performance. In contrast, the simpler Hopper environment shows fewer improvements with the ensemble approach.

Q3: What does the capital $D$ in Eq. 1 refer to?

A3: Thanks. The capital $D$ refers to distilled data $D\_\text{syn}$.

Q4: Missing clear definition of the surrogate loss $H$ makes it difficult to understand that minimizing $H$ is equivalent to maximizing the expected return $J$ in Eq. 1 and Eq. 2.

A4: Thanks for your suggestion. We use the surrogate loss $H$ to approximate the expected return $J$. When $H$ is an appropriate proxy, minimizing $H$ effectively corresponds to maximizing $J$. We will clarify this in the revised paper.

Q5: In Algorithm 1, "$D_{syn}$" should be "$B$" in "Compute $H(\pi ,D_{syn})$"

A5: Thanks and addressed.

Q6: In Algorithm 1, could the authors clarify the semantics of applying $`GradDescent()`$ to update ${D}_\text{syn}$?

A6: Thanks. In the first step of $`GradDescent`$, we compute the meta-gradient $\nabla_{D_**syn**}$ via BPTT as shown in Eq. 3. Then we update $D_\text{syn}$ by $D_\text{syn} \rightarrow D_\text{syn} - \alpha \nabla_{D_\text{syn}}$.

Q7: It will make the paper more convincing if some examples of such critical states discovered by Av-PBC could be added to the paper.

A7: Thanks for your suggestion. Please refer to A1 in the global response.

Q8: Generalization to Other Environments for proposed Av-PBC

A8: Thanks. We have tested our approach in three widely-used environments of Halfcheetah, Hopper, and Walker2D with various data qualities. Our experiments show that our Av-PBC consistently and remarkably improves the OBD performance compared to other methods in these nine datasets. We will generalize our Av-PBC to more environments in future work to further validate its effectiveness.

Q9: A discussion on the impact of dataset quality on Av-PBC's performance and practical guidance on applying Av-PBC across different scenarios would enhance its usability and robustness.

A9: Thanks. We provide additional insights on the impact of data quality and state coverage; please refer to A2 in the global response.

Q1: There are no major weaknesses, but ablation studies and further corollaries would strengthen the paper.

A1: thanks for your acknowledgement and suggestion. We have conducted additional empirical studies during the response period to strengthen our paper:

1. We explored the effect of different sizes of synthetic datasets on OBD performance. The results, presented in Table 2 of the PDF file, indicate that as the size of the synthetic data increases, OBD performance improves. This enhancement is attributed to the larger synthetic datasets conveying more comprehensive information about the RL environment and decision-making processes.
2. We examined the role of state coverage in offline RL datasets $\mathcal{D}_\text{off}$ in enhancing OBD. Our empirical findings demonstrate that greater state coverage increases robustness against data noise, which is beneficial for OBD. For a detailed explanation, please refer to A2 in the global response.
3. We provided illustrative examples of distilled Halfcheetah behavioral data in Figure 1 of the PDF file. The top row displays the distilled states, while the bottom row shows the subsequent states after executing the corresponding distilled actions in the environment. The figure reveals that (1) the distilled states emphasize "critical states" or "imbalanced states" (for the cheetah) more than "balanced states"; and (2) the subsequent states after taking distilled actions are closer to "balanced states" than the initial distilled states. These examples offer insights into the explainability of reinforcement learning processes.

Q2: I'd like the authors to add their thoughts on how the proposed methods can serve as building blocks towards resolving the limitations mentioned in the paper.

A2: Thanks for your suggestion. In our paper, we highlighted two primary limitations.

1. The first limitation involves synthesizing distilled data for reinforcement learning. Our proposed Av-PBC partially addresses this issue by distilling high-informative behavioral data for imitation learning, one of the offline RL methods. This approach helps in effectively capturing essential information of the behavioral data, making it a valuable step towards resolving this limitation.
2. The second major limitation concerns the high computational resources required for Offline Behavioral Distillation (OBD) due to the bi-level optimization process. Av-PBC offers a significant improvement in this area by enhancing the distillation speed. Specifically, it reduces the required time to one-quarter of what is needed by the naive OBD methods like DBC and PBC. This efficiency gain makes Av-PBC a practical and resource-efficient option. We will provide a more detailed discussion in the revised version of the paper.