Federated Maximum Likelihood Inverse Reinforcement Learning with Convergence Guarantee

Thank you very much for your valuable comments and feedback. Below, we address the identified weaknesses and questions.

Weakness

• Presentation: Thank you for highlighting the presentation issues. We provide the following clarifications to address your concerns: In “Apprenticeship Learning via Inverse Reinforcement Learning,” the authors assume that the reward function is a linear combination of features and recover the weights of these features to define the reward function. This reward function aids in the apprenticeship learning process of finding an optimal policy. In our work, f in Eq. (3) refers to the objective function in the optimization problem of federated learning. Thank you very much for pointing out the typos, we have corrected them and updated the paper.

Questions

In our algorithm, the expert trajectories are sampled from the local trajectory databases. As a result, the number of trajectories contributing to model training is the same for each local client in every iteration. For simplicity, we chose to average across clients rather than apply weights based on the number of trajectories owned by each client. However, we acknowledge that this may not fully account for variance differences between clients. Moving forward, we plan to extend our approach by incorporating weighted averaging to address these differences and improve robustness.

We sincerely appreciate your comments and hope these clarifications address your concerns.

Thank you for your valuable feedback. Below, we address the identified weaknesses and questions.

Weakness 1: The unique challenges addressed in this paper include the design of a federated learning algorithm tailored for maximum likelihood inverse reinforcement learning and the theoretical proof of its convergence guarantee. These contributions are central to our work and distinguish it from prior approaches. In our setup, the inner RL problem is formulated as soft Q-learning for Maximum Likelihood Inverse Reinforcement Learning. To ensure compatibility with this specific RL form, we decided to perform aggregation only on the Q-function rather than both the Q-function and the policy. This design choice simplifies the aggregation process and facilitates convergence.The most challenging part of the convergence proof lies in addressing the influence of aggregation. Our proof focuses on the tabular case, where the state and action spaces are finite. Prior ML-IRL work only considered a single agent problem and didn’t analyze it in a federated setting as how global aggregation impacts local models. On the other hand, traditional federated learning convergence analyses cannot be directly applied here because the lower-level problem is a soft Q-learning RL problem and we do not have a gradient-based update rule for the tabular case. Therefore, our construction of convergence is based on soft Q-functions and the entropy-regularized Boltzmann policy updates, which prevents us from utilizing existing proof techniques in the federated learning area. Facing above challenges, we carefully conduct analysis to bound the errors caused by the aggregation of Q-values and reward parameters to evaluate the optimality of the inner-loop RL problem with respect to the Q-values and the outer-loop maximum likelihood estimation.

Weakness 2: Thank you for highlighting concerns regarding the literature review. While there are related topics discussed in [1], [2], and [3], their focuses are notably different: [1] addresses inverse constrained reinforcement learning, [2] explores inverse optimal control, and [3] investigates two-player zero-sum games. Each of these works employs fundamentally different methodologies and solutions compared to our approach. We believe that specific problems, such as federated inverse reinforcement learning, require targeted analysis that directly addresses the unique challenges of the domain.

Weakness 3: In fields like cybersecurity and healthcare, demonstration data are often confidential and distributed across different locations. The data cannot be transmitted and collected for centralized IRL methods. This creates a significant challenge in data sharing due to privacy concerns and data scarcity. Our federated learning approach is designed to address these issues by enabling collaboration across distributed datasets without violating privacy constraints.

Weakness 4: Thank you for raising this important question. While [4], [5], and [6] present general federated bilevel learning algorithms, the specific challenges of inverse reinforcement learning are distinct. Unlike general optimization problems, the low-level RL problem involves interaction with an environment and is characterized by sequential decision-making processes. These unique features necessitate targeted analysis, as different RL models require tailored solutions that consider the dynamics of their specific settings.

Questions: For your question, please refer to our clarification under Weakness 1, as it directly addresses the uniqueness of our contributions in algorithm design and theoretical guarantees.

We sincerely appreciate your comments and hope these clarifications address your concerns.

Thank you very much for your detailed feedback. Below, we address each identified weakness.

Weakness 1: We acknowledge that our algorithm builds on the maximum likelihood IRL framework and employs FedAvg for the global updates. However, the problem we are solving is a bi-level optimization. due to the reinforcement learning (RL) nature of the inner loop, federated learning requires careful adaptation to ensure compatibility with the RL algorithm. We believe that our contributions lie in the design of this tailored algorithm and the accompanying convergence analysis, which address these challenges in a meaningful way.

Weakness 2: Thank you for suggesting additional papers on the topic. We are familiar with R1: Federated Inverse Reinforcement Learning for Smart ICUs with Differential Privacy, as it came up during our literature review. While R1 focuses primarily on application-specific aspects, it lacks a general problem formulation and does not provide theoretical guarantees. On the other hand, R2, R3, and R4 are imitation learning papers, which differ fundamentally from inverse reinforcement learning. R5 focuses on inverse constrained reinforcement learning, which adopts a different IRL formulation, and its solution could not be generalized to other IRL models such as ML-IRL, which we focused on. To the best of our knowledge, we are the first to study this specific problem for ML-IRL and provide theoretical convergence analysis.

Weakness 3: For large-scale discrete state and action spaces, the Q-function and reward function can be represented by neural networks, and their parameters can be communicated across clients. In our approach, both the global policy and local policies are derived from the soft-Q function. As a result, there is no need to communicate policies directly; instead, we compute the global policy based on the aggregated Q-function, ensuring efficient and scalable communication.

Weakness 4: Thank you for your observations regarding the experimental setup. In our experiments, we aim to demonstrate that federated learning provides a more effective solution compared to a purely local approach that is constrained by privacy concerns and limited to partial data. To reflect real-world dilemmas, we allow the federated framework to access diverse datasets across clients, whereas the centralized baseline is restricted to a single dataset. This setup is intended to highlight the advantages of federated learning in leveraging distributed data under privacy constraints.

Weakness 5: Thank you for pointing out these issues. We address them as follows:

1. In Equation (5), $\theta^i_{m,k+1}$ should indeed be $\theta^i_{m,k}$.
2. The time-varying property of $\alpha$ is mentioned in the proof to illustrate its role in determining the convergence rate. 3 & 4. In Theorem 1 Equation (18), $\bar{\pi}^i_{m,T}$ and $\bar{\theta}^i_{m,T}$ should be $\bar\pi_{m,T}$ and $\bar\theta_{m,T}$, respectively.
3. The typos in Steps 1-3 on Page 8 are corrected to make it consistent.

Please check the updated supplement material for the updated full paper.

We sincerely appreciate your comments and hope these clarifications address your concerns.

Thank you very much for your thoughtful comments.

Weakness 1: Thank you for your comment. For the proof, we focus on the tabular case where both the state and action spaces are finite, allowing Q-values to be aggregated. When extending to Q-Networks, both direct aggregation of the network parameters $\phi$ and aggregation of gradients are standard practices. For instance, the original FedAvg algorithm [1] aggregates neural network parameters, and existing work in federated reinforcement learning [2] also aggregates the parameters of both Q-Networks and policy networks.

[1] McMahan, Brendan, et al. “Communication-efficient learning of deep networks from decentralized data.” Artificial Intelligence and Statistics. PMLR, 2017.

[2] Jin, Hao, et al. “Federated reinforcement learning with environment heterogeneity.” International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

Weakness 2: Since the reward function is represented by a neural network, for similar reasons as weakness 1, we believe parameter aggregation is also valid for the reward function. Both the Q-function and the reward function, represented as neural networks, have sufficient capacity to generalize to functions of arbitrary complexity.

Weakness 3: Thank you for looking into the details of our proof. The most challenging part of the convergence proof lies in addressing the influence of aggregation. Our proof focuses on the tabular case, similar to many existing work [3, 4] in this space aiming at optimality proofs. Prior ML-IRL work only considered a single agent problem and didn’t analyze it in a federated setting as how global aggregation impacts local models. On the other hand, traditional federated learning convergence analyses cannot be directly applied here, because the lower-level problem is a soft Q-learning RL problem and we do not have a gradient-based update rule for the tabular case. Therefore, our construction of convergence is based on soft Q-functions and the entropy-regularized Boltzmann policy updates, which prevents us from utilizing existing proof techniques in the federated learning area. Facing above challenges, we carefully conduct analysis to bound the errors caused by the aggregation of Q-values and reward parameters to evaluate the optimality of the inner-loop RL problem with respect to the Q-values and the outer-loop maximum likelihood estimation.

[3] Mei, Jincheng, et al. "On the global convergence rates of softmax policy gradient methods." International conference on machine learning. PMLR, 2020.

[4] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

Weakness 4: In the experiment, we compared the centralized agent with medium-level performance to mixed expert demonstrations of varying performance levels in both federated and centralized settings. We found that demonstrations with lower performance can cause the recovered policy using mixed expert demonstrations to underperform compared to using only medium-level performance demonstrations. This phenomenon could be accounted for by the intuition that suboptimal demonstrations can adversely affect policy recovery.

Question 1: In Algorithm 1, lines 16-20, the index being looped refers to the agent index.

Question 2: Standard federated learning cannot be applied directly to inverse reinforcement learning (IRL). This has also been shown in several recent works that have developed decentralized IRL solutions [5, 6]. The inner RL problem typically has a unique form that is distinct from standard optimization problems. For instance, in our setup, the inner RL problem is formulated as soft Q-learning for Maximum Likelihood Inverse Reinforcement Learning. To ensure compatibility with this specific RL form, we decided to perform aggregation only on the Q-function rather than both the Q-function and the policy. This design choice simplifies the aggregation process and facilitates convergence.

[5] Liu, Shicheng, and Minghui Zhu. "Distributed inverse constrained reinforcement learning for multi-agent systems." Advances in Neural Information Processing Systems 35 (2022): 33444-33456.

[6] Reddy, Tummalapalli Sudhamsh, et al. "Inverse reinforcement learning for decentralized non-cooperative multiagent systems." 2012 ieee international conference on systems, man, and cybernetics (smc). IEEE, 2012.

Question 3: The large deviations observed in our results may stem from differences between the environments. Due to the time and resource-intensive nature of the experiments, we were unable to run multiple trials. However, we plan to conduct further experiments and fine-tune the parameters to mitigate the effects of varying task settings. We appreciate your understanding and will incorporate this in future work.

We sincerely appreciate your comments and hope these clarifications address your concerns.