Remote Reinforcement Learning with Communication Constraints

1. This paper introduces the novel problem of remote RL, provides canonical solutions, and demonstrates how they can be improved. To the best of our knowledge, there are no existing methods that specifically address this setting, making our work a significant contribution to this unexplored area.

2. The specifics of the channel simulation algorithm have been reproduced in Appendix B. The decoding consists of generating $n^*$-th sample from $Q$. We will ensure these details are clarified in the pseudocode for improved accessibility.

3. Remote sampling allows to send samples from P where the cost is $D_{KL}(P,Q)$ where $Q$ is available to both encoder and decoder. Minimizing the distance between P and Q minimizes the communication required.

4. Remote sampling allows efficient communication of samples from $P$, with a cost proportional to the KL divergence $D_{\text{KL}}(P \| Q)$, where $Q$ is known to both the encoder and decoder. By minimizing the divergence between $P$ and $Q$, the communication cost is reduced. In our setting $P = \pi_C(\cdot|s_t)$ and $Q=\pi_A(\cdot|S_t)$.

5. The ASC benchmark simulates a standard RL scenario where all information is aggregated at the controller side, effectively representing a classic RL setup.

We thank the reviewer for their insightful feedback. We are actively exploring comparisons across all proposed environments using quantized and compressed reward signals to provide a more comprehensive evaluation.

In general, the rewards are real numbers they might require significant communication rate. Additionally, in multi-agent or complex environment settings, transmitting rewards alone would require each agent to learn independently, which is often suboptimal. This approach would also necessitate an additional feedback or aggregation mechanism to coordinate learning across agents, further increasing communication demands and making such system more complex.

Response to Weaknesses:

1. Access to information does not necessarily mean bandwidth usage. For example, a robot might know its position via GPS, while the controller has access to stationary video data or sensors. Furthermore, bandwidth constraints are often asymmetric; a base station connected to the power grid can communicate more efficiently compared to a battery-powered sensor or satellite. In this work, we analyzed the case where both the controller and the agent have access to full observations. However, modifying the information available to either the controller or the agent is an interesting direction for future research.

2. We agree with the reviewer that supervised learning can impose computational demands. In our experiments, we used the same architecture for simplicity. However, exploring smaller network architectures or alternative methods for action probability estimation is a natural next step in this line of research.

3. In general, in an MDP formulation, rewards as real numbers. In practice, they are represented as floating-point numbers. Communicating stochastic actions in GRASP via channel simulation is possible since the action is not chosen directly by the controller but it's a random outcome of the channel simulation algorithm.

Response to Questions:

1. Both the controller and the actor learn the actor's policy based on the same set of actions and states. This process is deterministic, ensuring that the learned policies remain consistent and identical.

2. The policies $\pi_A$ and $\pi_C$ represent the actor's and controller's policies, respectively. In Algorithm 1 and 2, $P = \pi_C(\cdot|s_t), Q=\pi_A(\cdot|S_t)$.

3. Channel simulation encoding is a randomized function that maps from pair of distribution $P,Q$ to a sting of bits. Channel simulation decoding is a randomized function that maps $Q$ and string of bits into a sample $x$, which looks as if it was sampled directly from $P$.

4. Channel simulation encoding is a randomized function that maps a pair of distributions $(P, Q)$ to a string of bits. Channel simulation decoding is a randomized function that maps the reference distribution $Q$ and the string of bits back into a sample $x$, which follows distribution $P$.

5. We trained a single policy conditioned on the agent's identity and observations, allowing GRASP to adapt to MARL settings. We acknowledge that there are numerous possible ways to apply GRASP in MARL scenarios. GRASP is a general tool for solving remote RL problems which can be paired with any RL/MARL algorithm.

Response to Weaknesses:

1. GRASP is designed to address the unique challenges of remote RL with constrained communication, combining techniques such as (channel simulation and behavioral cloning) in a novel manner. While we acknowledge that it does not introduce entirely novel components, its contribution lies in the integration of these techniques to solve a practical and underexplored problem.

2. Theoretically, ASC and GRASP sample from the same policy distribution, so the diversity of actions should be identical. However, noise introduced during channel simulation slightly affects the samples but does not noticeably harm or benefit training, as demonstrated in our experiments.

3. The benchmark that transmits rewards directly to the actor would yield identical results to ASC since, in both cases, rewards, actions, and states are aggregated in one location—at the controller for ASC and at the actor for reward-sending. This equivalence is why results for this benchmark are not explicitly included.

Response to Questions:

1. Importance sampling introduces minimal variance. It is only used to transmit samples from the policy distribution, ensuring consistency with the original action samples, and has negligible impact on training performance.

2. No, importance sampling is a tool to transmit samples efficiently at very low bit rates. Its purpose is to enable remote RL with limited communication without altering or enhancing the underlying training process.

3. As stated in Weakness 3, the performance of the reward-sending benchmark is identical to ASC, as both approaches aggregate information in a single location

1-2. Centralized training with decentralized execution (CTDE) is a method of training MARL systems which addresses the non-stationarity of MA-POMDPs. Unlike GRASP, CTDE does not solve the remote RL problem. However CTDE can be used in GRASP as it centralizes knowledge at the controller.

3. We agree with the reviewer, we are unaware of any guarantees for deep RL, and this remains an open challenge.

4. We introduce the remote RL problem and the GRASP algorithm, which is compatible with any RL problem. Our experiments focused on PPO, ContinuousPPO, DDPG, and DQN as baselines. A comparison with imitation learning algorithms such as VIPER or DAGGER is an interesting direction for future work, however GRASP is not an imitation learning algorithm.

5. The MARL settings CooperativePong, PistonBall and Spread are presented in Table 1 and Figure 3.