Federated Ensemble-Directed Offline Reinforcement Learning

We thank the reviewer for their valuable feedback. We are delighted to know that the reviewer finds our work original, our problem significant, and our paper well written. Below, we address the reviewer's concerns and hope they will consider increasing their score.

1. Some technical details need to be explained. For example, the ensemble learning and its role.

Response: The idea of ensemble learning in our algorithm is to use the data distributed among different clients to learn a federated policy collectively. In Section 4.1, we have mentioned that `ensemble heterogeneity' is one of the key challenges in federated offline RL. In our work, we proposed an ensemble approach to overcome this challenge. Specifically, our approach uses the performance of the local policies as a proxy to weigh each client's contribution to the federated policy. This ensures that policies from clients with higher performance data have a greater influence on the federated policy. We have explained our approach in Section 5.1, including the mathematical equation that translates our idea to an actual algorithmic step. In addition to the experimental results given in Section 6 that show the superior performance of our FEDORA algorithm, we have also included additional ablation experimental results in Appendix C.1 which show the significance of our ensemble method w.r.t. other ingredients of FEDORA. Please let us know if there are any specific aspects that need to be elaborated further.

2. The novelty of this paper needs further clarification, and what is the main difference between this proposed method and existing studies? It seems that there is only a simple combination of two technologies.

Response: We respectfully disagree with the reviewer's comment that our proposed algorithm is ``is only a simple combination of two technologies''. We emphasize that a simple combination of federated learning and offline RL is insufficient, as we have explained in our paper (see Fig. 1 and the explanation there). Significant algorithmic innovations are necessary to overcome some unique challenges of federated offline RL, as we have explained in Section 4.1. Our contributions include four key innovations: $(i)$ Ensemble-Directed Learning over Client Policies (Section 5.1), $(ii)$ Federated Optimism for Critic Training (Section 5.2), $(iii)$ Proximal Policy Update for Heterogeneous Data (Section 5.3), and $(iv)$ Decaying the Influence of Local Data (Section 5.4). We have given detailed experimental evidence on the superior performance of this method, see Section 6 and Appendix. Moreover, we have done ablation experiments that show the importance of each of these proposed innovations.

3. Numerically, the authors could consider comparing their method with more baselines. There are some studies on federated learning for offline RL

Response: We have demonstrated the superior performance of our FEDORA algorithm against four different baseline algorithms through simulation experiments, see Section 6.1 and Appendix C. We have also evaluated the performance of FEDORA in the real-world using TurtleBot, a two-wheeled differential drive mobile robot, see Section 6.1, and compared it with the same baseline algorithms. We have also included a video of this real-world demonstration. We sincerely believe that these experiments and real-world demo clearly show the superior performance of FEDORA against the standard baselines. We would appreciate further guidance on specific baselines the reviewer would like us to include in our comparisons.

4. What is the technical drawback of the proposed method? E.g., the effectiveness of the agent weight by ensemble approach

Response: We address our limitations in Appendix E. We make the assumption that all clients have the same MDP model (transition kernel and reward model), and any statistical variances between the offline datasets are due to differences in the behavior policies used to collect the data. In future works, we aim to broaden this to cover scenarios where clients have different transition and reward models.

Reading the second remark about the effectiveness, please note that we have already included detailed ablation experiments to analyze the effectiveness of different components of our algorithm, see Appendix C1 and C2.

5. Does this proposed method work for other RL algorithms?

Response Indeed! The FEDORA framework that we propose is general and can work with any actor-critic-based offline RL algorithms. We have mentioned this in our paper, please see lines 126-128.

We thank the reviewer for their comments and are happy to note that they find our work novel, our experiments extensive, and our paper well-written. Below, we address their concerns and hope that they consider increasing their score.

1. No theoretical guarantees have been given for the algorithm though it does build upon foundational work.

Response: Thank you for your comment. Providing a theoretical guarantee for our FEDORA algorithm is indeed a challenging problem that requires technical analysis of multiple complicated components, including offline policy evaluation, pessimistic estimation, ensemble-style quality-based federation, dealing with heterogeneous data, and analyzing these components jointly to derive the final performance bound.

We, however, emphasize that, to the best of our knowledge, ours is the first paper on federated offline deep RL with an algorithm that performs really well in a variety of settings. Our design is analytically driven, identifies each issue of learning from the ensemble of policies, and builds up the algorithm methodically one step at a time corresponding to each analytical insight. We sincerely believe that our rigorous empiricism-driven approach is valuable based on its algorithmic contributions.

2. I believe that the authors should explicitly discuss limitations/ opportunities for future work in the paper.

Response: We discuss the limitations and future directions of our research in Appendix E. For the final submission, we will move this discussion to the main part of the paper to make it more accessible and prominent.

3. It is important for the algorithm pseudocode to be included in the main material as is the norm in such papers...Another minor issue is that the figures are placed very far away from where they are referred to in text.

Response: We pushed the pseudocode to the appendix due to space constraints. However, for the final version, we will move the pseudocode back to the main text, as we will have an extra page for the final submission. Additionally, we will adjust the placement of figures to be closer to the relevant text.

4. How far do the authors perceive that this model can be pushed? i.e. the assumption that all clients have the same MDP is restrictive but understandable for a first set of experiments

Response: We plan to extend FEDORA to a meta federated learning setting, wherein we can learn with clients having different transitions and reward functions. This extension is discussed in Appendix E, and we aim to explore this direction in future work.

5. Have any experiments been run using D4RL-random datasets? It would be interesting to see whether this collapses learning. With regards to FEDORA outperforming centralized training I think a deeper discussion on this would be useful.

Response: Yes, we have run experiments using the D4RL random dataset and compared it with centralized training (See Figure 3 in Section 6.1). We also conduct experiments with clients having different datasets (including random datasets) in Appendix C.6.

6. What is the main reason for this? Heterogeneous data though previous work has successfully mixed datasets

Response: The use of heterogeneous datasets in centralized offline RL is a significant challenge. One reason for the drop in performance when pooling data from behavior policies with different expertise levels is that it can exacerbate the distributional shift between the learned policy and the individual datasets, leading to poor performance [1].

References

[1] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:11501–11516, 2021.484

We thank the reviewer for their positive endorsement of our work. We are happy to know that the reviewer finds our work novel, our experiments extensive and our paper well written. Below we address the concerns of the reviewer.

1. "Collect wisdom" can be replaced by more rigorous exposition. Same goes with "ambitious targets".

Response: We thank the reviewer for their suggestion, we will incorporate this in the final version of the paper.

2. The number of communication rounds needed for FEDORA to converge is still quite high.

Response: The number of communication rounds that FEDORA takes depends on factors such as the complexity of the problem and the number of local epochs performed during each round of federation. We believe that the number of communication rounds can be reduced by increasing the number of local epochs performed in each round of federation.

3. Given how well the algorithm does, some sort of theoretical analysis could further strengthen the work.

Response: Thank you, we are indeed working on this problem. Providing a theoretical guarantee for our FEDORA algorithm is a challenging problem that requires technical analysis of multiple complicated parts corresponding to offline policy evaluation, pessimistic estimation, ensemble-style quality-based federation, dealing with heterogeneous data, and analyzing these components jointly to get the final performance bound.