Multi-Agent Domain Calibration with a Handful of Offline Data

Q1: The objective in Sec. 4.1.

Eq. 4 is obtained through reasonable approximations.

The explanations for each derivation step are as follows:

• Eq. 2 is derived from Eq. 1 based on the definitions of trajectory distribution and KL divergence, transforming the objective into the expectation under the parameter distribution and trajectory distribution of the source domain.
• Eq. 3 further decomposes Eq. 2. Since $\log \frac{q_{\phi}(\xi)}{p(\xi)}$ is independent of the trajectory distribution $d_{\pi,\mathcal{M}_{\xi}}$, the expectation can be decomposed into two terms.
• Eq. 4 approximates Eq. 3. As the parameter distribution and trajectory distribution used to calculate the expectation are difficult to compute, we use Monte Carlo sampling on the source domain to approximate the expected results. To enhance sampling efficiency, we store the samples in the replay buffer, converting the trajectory-based objectives into transition-based objectives, following the off-policy RL paradigm.

We will add the detailed explanations in the revised manuscript.

Q2: Some terms are not defined in Sec. 4.2.

The calibration critic and the calibration policy (actor) together constitute a calibration agent. The calibration critic is responsible for evaluating the accuracy of the parameters output by the calibration actor. The absolute calibration error represents the absolute difference between the parameters output by the calibration actor and the target parameters.

Q3: The latter part of Sec. 4.2 is not understandable.

"The identity $i$ of each parameter" represents a specific identification symbol for each parameter. Specifically, in the experiments, these identities refer to a set of one-hot encodings used to distinguish different parameters. Besides, some key contents were missed: the identity can determine $s'$ given $s, a, \xi$ by keeping the other parameters $\xi^{-i}$ fixed, similar to the approach in COMA [1]. By only changing $\xi_i$, the effect of this parameter on the dynamics can be reflected, thereby facilitating parameter grouping . We apologize for the confusion and will add explanations in the revised manuscript for better understanding.

Q4: Why do we need to match action distribution?

We need to match the action distribution to make the dynamics of the source domain closer to that of the target domain.

Eq. 1 is used to match the trajectory distribution (state distribution and action distribution). The underlying motivation is to minimize the transition dynamics gap. If only the state distribution is matched, multiple combined solutions of the policy and the transition function can exist. Therefore, to obtain the target transition function, we need to match the action distribution as well. To verify our idea, we conducted relevant experiments: for Eq. 1, we no longer matched the action distribution, denoted as Madoc_st. The experimental results, shown below, indicate performance degradation in all three scenarios.

Q5: Why do we need the multi-agent method?

We use the multi-agent method for better optimization efficiency.

The search space of domain parameters grows exponentially with the number of parameters, hindering the optimization of both single-agent methods and previous multi-agent methods. Fortunately, with the advent of linear value factorization algorithms like VDN [2], researchers have improved scalability by decomposing the joint action space theoretically and practically [3], resulting in a much lower action space for each agent. Consequently, Madoc has higher optimization efficiency and can calibrate parameters more accurately. Certainly, the multi-agent method is not applicable to all RL environments. When there is only one agent in the environment with a low action dimension, a single-agent RL method might be more straightforward, providing satisfactory policy optimization without the added complexity.

Q6: How about methods like "imitation learning from observation"?

Madoc is different from and superior to these methods.

Imitation from observation [4] methods (denoted as IfO) can be formalized as learning a discriminator $D(s, s')$ and a policy $\pi(a,\xi|s)$ simultaneously, while Madoc involves training a discriminator $D(s, a, s')$, a calibration policy $q(\xi)$ and a running policy $\pi(a|s)$. There are mainly two differences:

• Madoc decouples tuning the domain parameters and the policy, making it more accurate and efficient in reducing the dynamic gap.
• IfO requires matching the state distribution of the expert trajectory, which imposes additional quality requirements on the dataset [5]. However, Madoc only requires datasets of arbitrary quality to match the transition dynamics.

We also tested the performance of IfO. The first table below reports the mean return during the test stage.

The second table below compares the mean absolute calibration error.

Reference:

[1] Counterfactual Multi-Agent Policy Gradients. AAAI, 2018.

[2] Value-Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward. AAMAS, 2018.

[3] Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization. NeurIPS, 2021.

[4] Recent Advances in Imitation Learning from Observation. IJCAI, 2019.

[5] Robust Learning from Observation with Model Misspecification. AAMAS, 2022.

Q1: Weak empirical results.

Our method Madoc has achieved a trade-off between high mean and low variance in return performance.

• On the one hand, Madoc requires online interaction with the source domain to search for the domain parameters that best match the offline dataset. Consequently, the random seed significantly influences exploration and exploitation, leading to larger variance for Madoc. In contrast, pure offline algorithms like CQL and MOREC learn on a fixed dataset in a conservative manner and do not need to explore. Therefore, they are less influenced by the random seed and have smaller variance.
• On the other hand, algorithms with lower variance, namely H2O, DR+BC, CQL, and MOREC, obtain conservative policies by penalizing the value functions on OOD actions or directly constraining the policies against the behavior policies. As a result, their mean performances are also limited by the dataset. Our method has achieved a trade-off between high mean and low variance, attaining optimal performance compared to baselines in most scenarios.

Regarding the large variance problem of Madoc, we have made some improvements. Once domain calibration is completed, we no longer use pure SAC to train the policy on the source domain from scratch. Instead, we combine SAC with BC to impose appropriate constraints on the learned policy, referred to as Madoc+online_bc. The results are shown in the table below. We can observe that the mean performance of Madoc+online_bc decreased slightly but became more stable, confirming our approach.

The concern you mentioned is of crucial significance, and we will incorporate its discussion in the experimental section of the revised manuscript.

Q2: It's unclear if Madoc is indeed reducing the dynamics gap between the source and target domains.

Our method Madoc indeed reduces the dynamics gap between the source and target domains. Some experimental results are as follows:

As shown in the table, we used "mean absolute calibration error" (i.e., the absolute difference between the physics parameters of the source and target domains) to measure the dynamics gaps of different algorithms. It can be observed that Madoc minimized the mean absolute calibration errors in all three tasks. The complete results are presented in Appendix E.1. We sincerely apologize for the confusion caused by the lack of appendix guidance in the manuscript and will add this guidance in the revised version.

Q3: Could the authors provide more details about Fig. 2?

Fig. 2 demonstrates that, compared to the single-agent method, the multi-agent method can more effectively evaluate the accuracy of each physics parameter output by the calibration policy. The calibration agent consists of a calibration critic that evaluates the accuracy of the physics parameters and a calibration actor (policy) that adjusts the output physics parameters. The plotted metrics are the values output by the calibration critic for a specific physics parameter (i.e., the gravity coefficient) under different parameter conditions. When the parameters output by the calibration actor are closer to the target parameters (indicating a smaller absolute calibration error), the evaluation value output by a "good" critic should be higher. The results in Fig. 2 show that our multi-agent method can train such a good critic, whereas the single-agent method fails to correctly evaluate the accuracy of the physics parameters due to the lack of reasonable reward allocation and the large search space. We have corrected the legend of Fig. 2 in the left half of the PDF of the "global" response and will provide more detailed explanations about it in the revised manuscript.

Q1: Method Complexity and Comparisons with Offline RL.

Madoc adds more modules compared to CQL, but the model complexity and GPU memory cost remain acceptable.

We conducted experiments on the hfctah-med-rep task of the D4RL benchmark to evaluate the model complexity of different methods. The results are shown in the table below, with all networks in the form of MLP. Our method achieves a balance between model complexity and performance.

Q2: Surprising Results in Table 2.

The performance of MOREC is not as good as in the original paper because we used a smaller amount of offline data. MOREC is sensitive to the quantity of offline data as it initially needs to train a generalizable dynamics reward function from offline data and accordingly learn dynamics models for generating transitions. When the quantity of offline data is insufficient, both the dynamics reward function and the dynamics models tend to underfit. As depicted in Fig. 4(a), when using the same large dataset as MOREC, the performance gap is small. However, as the amount of offline data decreases, the gap becomes increasingly larger. We will provide an explanation of it in the revised manuscript.

Q3: Smallest Target Task Dataset Size.

Madoc utilized a sufficiently small amount of offline data.

We present the smallest target task dataset size of different methods below:

SSR [1] proposes a data-efficient pipeline for offline RL with limited data and still used 2e5 training samples. This is because the continuous control tasks possess larger state and action spaces compared to discrete RL environments. Thus, the smallest target task dataset of Madoc, which contains only 5e4 samples, can be regarded as using a handful of offline data.

Q4: What is the computation cost of training your method compared to other Offline RL methods?

Madoc incurs higher computational costs compared to CQL, but this is acceptable given its performance improvement.

We conducted experiments on the hfctah-med-rep task to evaluate the computational costs of different methods. The results are shown below:

The table indicates that Madoc involves three training stages and consumes more computational cost compared to CQL. However, given the significant performance improvement, this additional computational cost is justifiable. We will also include this discussion in the revised appendix.

Q5: How good is the reward model?

The reward model can accurately reflect the dynamics gap under different parameters.

To verify the role of the reward model, we designed an experiment to test whether it can act as a scoring function to evaluate the truthfulness of transitions. As shown in the right half of the PDF of the "global" response, we stored the checkpoint of the trained reward model and used it to test the reward results (the reward range should be [-4, 4]) it outputs under different dynamics. In this simple test, we only changed two parameters: the gravity coefficient and body_mass_1 (the target values are -9.81 and 6.36, respectively). We sampled 256 transitions with the behavior policy under each parameter condition to calculate the mean reward. The results, shown in the heat map, indicate that when the domain parameters are closer to the target parameters, the output reward is higher. This suggests that it can accurately reflect the dynamics gap under different parameters.

Q6: Comparison with independent learning methods.

Independent SAC agents perform worse than Madoc agents in domain calibration.

Independent SAC (ISAC) treats each agent as an independent individual, optimizing each policy with shared rewards without considering the joint policy. We conducted relevant experiments and the results are shown below:

The results indicate that ISAC performs worse than Madoc in domain calibration. We believe the main reason is that all physics parameters are interrelated and cooperative. The independent learning method ignores the policy changes of other calibration agents, leading to non-stationary problems [2]. IPPO [3] performs well on some SMAC tasks because these tasks do not have high requirements for cooperativeness.

Reference:

[1] Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data. NeurIPS, 2022.

[2] Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning. arXiv, 2019.

[3] Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? arXiv, 2020.