CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

Question 4: Training Setup: Can you provide more details about the training setup, such as the GPU types, number of hours required for training, and hardware resources used? This would help improve reproducibility and enable more accurate comparisons in future work.

Please refer to general response Question 2 and we have added the training details in the updated Appendix A.3.1.

[1] Hang Qiu, Pohan Huang, Namo Asavisanu, Xiaochen Liu, Konstantinos Psounis, and Ramesh Govindan. Autocast: Scalable infrastructure-less cooperative perception for distributed collaborative driving. arXiv preprint arXiv:2112.14947, 2021.

[2] Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17252–17262, 2022.

Thanks for your detailed review and valuable feedback. We address your concerns below.

Weakness 1: Modalities in Experimentation: It’s unclear which modalities are used for the experiments. While some sections mention depth, RGB, and LIDAR, the modalities should be clearly stated in Section 5.1 (Data Collection). Clarifying the exact modalities would provide better insight into the data used.

Thank you for your valuable feedback. In the experiments of collaborative decision-making in CAV, for each vehicle, RGB and LiDAR are used during training, while only RGB is used during testing. We have added exact modalities and clearly stated more experimental settings in the updated paper, please see general response Question 2 and Section 5.1 Data Collection.

Weakness 2: Complexity of Driving Scenarios: How realistic were the driving scenarios used in the experiments? Did you incorporate domain randomization, and how many pedestrians or vehicles were simulated? Additionally, are there plans to test the approach in real-world settings?

Thank you for your insightful feedback. The driving scenarios are indeed realistic which are built on AutoCast (Qiu et al., 2021, Cui et al., 2022) using CARLA, with a background traffic 30. They involve complex interactions such as overtaking, lane changing, and red-light violation, situations that are inherently accident-prone. And the number of connected vehicles is also varying, during the data collection, at each timestamp, the ego vehicle has a maximum of three collaborative vehicles provided their distance is within a threshold of 150 meter. We have also added new experiments with real-world data for aerial-ground vehicles collaborative semantic segmentation. Please see the general response Question 1 and the updated Section 5.2 and Appendix A.2.

Weakness 3: Limited Analysis of Failure Cases: The paper focuses on scenarios where CAML performs well, but lacks an exploration of its weaknesses or failure cases. To strengthen the paper, include an analysis of these cases and discuss possible improvements or limitations of the approach.

Thanks for your valuable comment. We have added a failure case analysis and limitations of the research in the updated paper, please see Section 6 Conclusions and Limitations.

Weakness 4: Insufficient Training Details: More details about the training setup would improve reproducibility. It would be useful to specify the GPU type, training time, and hardware resources used for training. Providing these details ensures transparency and supports future replication.

Thanks for pointing this out. Please see training details in general response Question 2 and we have added these in the updated paper.

Typos: L377: “CAML ." -> “CAML.” There is a space between “L” and “.”

Thank you, this typo has been fixed.

Question 1: Clarification on Modalities: Can you clarify the exact modalities used in the experiments? Section 5.1 (Data Collection) does not specify this clearly, and it is inferred that depth, RGB, and LIDAR were utilized from other sections. Are these the only modalities involved, or are there others?

We have added the clarification on modalities in the updated paper, please see general response Question 2.

Question 2: Complexity of Driving Scenarios: How realistic were the driving scenarios used in the experiments? Did you incorporate domain randomization, and how many pedestrians or vehicles were simulated? Additionally, are there plans to test the approach in real-world settings?

The driving scenarios used in our experiments are indeed realistic, as they involve complex interactions such as overtaking, lane changing, and red-light violation, situations that are inherently accident-prone. These scenarios were designed to represent typical real-world conditions using AutoCast (Qiu et al., 2021, Cui et al., 2022) with a background traffic 30. Yes, we plan to test the approach in real-world settings in the future. And we have already added new experiments with real-world data of aerial-ground vehicles for collaborative semantic segmentation, please see the general response Question 1 and paper Section 5.2 and Appendix A.2.

Question 3: Failure Case Analysis: What are the specific failure cases where CAML underperforms? Including more analysis or examples of these cases would provide a clearer understanding of the framework’s limitations and potential areas for improvement.

We have added failure case analysis and limitations, please see general response Question 3 and Section 6 Conclusions and Limitations in the updated paper.

Dear Reviewer,

As the rebuttal is going to end soon, could you please kindly let us know whether our response and the revised paper with additional detailed experiments and clarification have influenced your evaluation?

Thank you for taking the time to review our work, your insights have been incredibly helpful to our research. And we would be deeply grateful if you could update the rating since we addressed your concern.

Sincerely,
The Authors

Question 2: What about heterogeneous scenarios where agents possess different modalities, such as in an RGBD context, including single agents with either RGB or depth data?

Thank you for raising this point. CAML can handle heterogeneous scenarios where agents possess different modalities. As we described and explained in Section 3 of the paper, each agent can process a different number of modalities during training, different agents can also have different main modalities and auxiliary modalities.

Thanks for your review and we really appreciate your comments. We address your concerns below.

Weakness 1: Lack of ablation studies on a wider range of modalities. For instance, what happens when training involves both LiDAR and RGB modalities, while testing uses only RGB or LiDAR? Would the performance still improve when the modality gap is significant?

We appreciate the reviewer's insightful comment. We would like to point out that we already have the case of training with both LiDAR and RGB modalities, while testing using only RGB, in the experiments of collaborative decision-making in CAV, for both CAML and AML ( Please see Section 5.1 Baseline Comparison in the paper).

We have conducted more experiments where we have both LiDAR and RGB for both training and testing, which we refer to as the teacher models for both single-agent and multi-agent settings, as shown below. Then we compare the performance of ADR (accident detection rate) and EIR (expert imitation rate) with AML and CAML. The performances of both CAML and AML decrease compared to respective teacher models because the missing LiDAR modality during testing, but the performance still improves by up to 58.3% when comparing CAML to AML. This addresses the concern in modality gaps, while the performance still improves when comparing AML vs. CAML.

We also validate with new experiments and real-world data, with more modalities such as RGBD and ablation studies, the experiments of the collaborative semantic segmentation for aerial-ground vehicles. Please see the general response Question 1 for more details, we also updated in the paper Section 5.2 and Appendix A.2.

Weakness 2: Lack of validation on real-world benchmarks. Validating real-world data is crucial, especially for applications in connected autonomous vehicles. Numerous real-world benchmarks exist, such as DAIR-V2X, V2V4Real, and TUMTraf, making real-world validation essential.

We appreciate the reviewer’s feedback highlighting the importance of validating our approach on real-world data. To address this, we have conducted new experiments involving real-world data from aerial-ground vehicles for collaborative semantic segmentation. Please see the general response Question 1 for more details, we also updated in the paper Section 5.2 and Appendix A.2 for more results and ablation studies.

Weakness 3: Lack of quantitative analysis regarding uncertainty reduction and data coverage. Additionally, providing qualitative visualization examples would enhance the presentation.

We appreciate the reviewer’s insights regarding the need for quantitative analysis of uncertainty reduction and data coverage, as well as qualitative visualization. We would like to point out that our experimental results do provide quantitative evidence of these aspects, as demonstrated by the improved prediction accuracy, which reflects reduced uncertainty and enhanced data coverage facilitated by multi-agent collaboration. Additionally, we added qualitative visualization examples in the paper to enhance the presentation, please see Appendix A.1 Data Coverage, where we show in a red-light violation scenario in connected autonomous driving, the ego vehicle’s view is obstructed, rendering the occluded vehicle invisible. However, collaborative vehicles are able to detect the occluded vehicle, providing critical complementary information. This additional data helps the ego vehicle overcome its occluded view, enabling it to make more informed decisions and avoid potential collisions with the occluded vehicle.

Question 1: If different modality embeddings are combined through concatenation, can the system only accommodate a fixed modality?

Thank you for your question. We would like to emphasize that CAML is a general framework and it can accommodate various modalities. Concatenation does not restrict the framework to a fixed modality, any modality embeddings can be concatenated as long as the embeddings are compatible in terms of their dimensionality after extraction from the respective encoders. And different embeddings can also be combined in other ways like cross-attention, concatenation is one of them.

Dear Reviewer,

Apart from the response, we have also updated the paper based on your constructive feedback. Today is the last day to submit a revised PDF.

We hope that our responses and revisions have addressed your concerns. We really appreciate if you could consider adjusting your score accordingly.

We deeply appreciate your time and effort in reviewing our work and look forward to any further feedback you may have.

Best regards,
The Authors

Dear Reviewer,

The reason we chose the CoPed dataset for real-world collaborative perception is that it covers different scenarios, the aerial-ground multi-robot collaboration scenarios, which is most suited for collaborative, multi-agent decision-making when using CAML. We also have experiments of autonomous ground vehicles with different sensors, another common practical scenario, so we use different scenarios to validate our approach. We will validate on benchmarks like DAIR-V2X, V2V4Real, and TUMTraf in future work.

Since we have addressed most of your concerns with detailed experiments and explanations, could you please consider updating the score accordingly? We are deeply grateful for that.

Question 2: What is CAML's time and space complexity concerning the number of connected agents? From your description, I presume it is quadratic. Would this trade-off in extra time and space complexity be worth it?

In CAML, the agents' embeddings are shared based on whether the system operates in a centralized or decentralized manner. If the system is a centralized, all collaborative agents share their data with one designated ego agent for centralized processing. Each of the $N-1$ collaborative agents performs its local computation independently, contributing $O(T_c)$ time and $O(S_c)$ space each. Thus, the total computation time and space for all collaborative agents are $O(T_c(N-1))$ and $O(S_c(N-1))$, respectively. For simplicity, assuming each communication from one collaborative agent to the ego agent consumes $O(D)$ time and $O(M)$ space, the total communication time and space complexity for gathering information at the ego agent are $O(D(N-1))$ and $O(M(N-1))$, respectively. Then the ego agent aggregates the received data, running a model, having a time and space complexity $O(T_e)$ and $O(S_e)$, respectively. So the total time and space complexity are $O(T_c(N-1)+D(N-1)+T_e)$ and $O(S_c(N-1)+M(N-1)+S_e)$, respectively.

If the system is decentralized, each agent performs its local computation and shares information with other agents. For simplicity, let the computation time for each agent be $O(T)$, and assume that communication from one agent to another agent requires $O(D)$ time and $O(M)$ space. For $N$ agents, the total computation time is $O(NT)$. In the worst case, each agent share data with all other agents, this can result in $O(N^2D)$ for pairwise sharing. So the total time complexity is $O(NT+N^2D)$. For space complexity, the storage requirement for all agents is $O(NS)$, where $S$ is the storage per agent. Communication between agents adds an additional complexity of $O(N^2M)$. So the total space complexity is $O(NS+N^2M)$. In the typical case, if each agent communicates with only other $k$ agents ($k \ll N$) rather than all $N-1$ agents. The total time and space complexity becomes $O(NT+NkD)$ and $O(NS+NkM)$, respectively. We have updated these in the paper, please see Appendix A.3.2.

The trade-off in extra time and space complexity is worth it. By having multi-agent collaboration, we can capture a more comprehensive understanding of the environment and have improved robustness and more accurate prediction. These are especially important for real-world scenarios where performance is critical like autonomous driving and collaborative perception, otherwise may cause accident.

Question3: Can you compare applying AML on training the entire fusion module to using it only for the encoder and decoder on single vehicles (pre- or post-fusion)?

Thanks for your valuable suggestion. We have conducted new experiments involving real-world data from aerial-ground vehicles for collaborative semantic segmentation. And we added pre-fusion version of CAML as ablation studies and compared with the proposed CAML. Please see the general response Question 1 for more details, we also updated in the paper Section 5.2 and Appendix A.2.

[1] Judy Hoffman, Saurabh Gupta, and Trevor Darrell. Learning with side information through modality hallucination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 826–834, 2016.

[2] Lan Wang, Chenqiang Gao, Luyu Yang, Yue Zhao, Wangmeng Zuo, and Deyu Meng. Pm-gans: Discriminative representation learning for action recognition using partial-modalities. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 384–401, 2018.

[3] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 103–118, 2018.

[4] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Learning with privileged information via adversarial discriminative modality distillation. IEEE transactions on pattern analysis and machine intelligence, 42(10):2581–2593, 2019.

[5] Nathan Piasco, Desire Sidibe, Valerie Gouet-Brunet, and Cedric Demonceaux. Improving image description with auxiliary modality for visual localization in challenging conditions. International Journal of Computer Vision, 129(1):185–202, 2021.

Thanks for your review and we address your concerns below.

Weakness 1: This paper's main contribution is an extension of an existing maturely developed machine learning technique, AML, to the context of cooperative perception, which limits its scope of novelty. Despite experiments conducted by the authors to investigate the benefits of CAML compared to single-agent AML, they only investigate one of the plausible paradigms for applying the AML - ego-only paradigm. Since we are looking into a cooperative perception task, will it be more sensical to use AML for training the encoder for single vehicles in parallel before fusion (pre-fusion) and compare this application with the proposed CAML, which is essentially applying a joint-version of AML to train the fusion module (on-fusion)?

We appreciate your feedback and the suggestion regarding different paradigms for applying AML.

First, we would like to emphasize that our proposed Collaborative Auxiliary Modality Learning (CAML) is not merely an incremental extension of Auxiliary Modality Learning (AML). CAML specifically addresses the unique challenges of multi-agent systems, including heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making with uncertainty reduction. CAML has broader applications of connected autonomous driving, collaborative search and rescue, as well as intelligent transportation. To the best of our knowledge, this work is the first to propose a flexible and principled framework capable of tackling a broad range of collaborative tasks in multi-agent systems where limited modalities are available for inference, especially in real-world scenarios.

Second, to evaluate different paradigms for applying AML, we have further conducted new experiments involving real-world data from aerial-ground vehicles for collaborative semantic segmentation. We added a variant of CAML called Pre-fusion CAML as ablation studies where we apply AML on single-agent to fuse different modalities first. The mIoU of the Pre-fusion CAML is similar to that of CAML. Although the fusion order is different, both versions benefit from robust feature aggregation and multi-agent collaboration, which ultimately results in better segmentation performance. Please see the general response Question 1 for more details, we also updated in the paper Section 5.2 and Appendix A.2.

Weakness 2: The authors have cited several related works to demonstrate the benefits of applying AML to a perception system. However, it would be much better if the authors could present a comparative study of CAML, AML, and other cooperative perception methods on traditional cooperative perception tasks, such as objective detection, to showcase the need for applying AML in cooperative perception.

We appreciate the reviewer's suggestion. We have conducted new experiments involving real-world data from aerial-ground vehicles for collaborative semantic segmentation. The real-world dataset does not just cover the traditional cooperative perception tasks, but also cover the aerial-ground scenarios, which is more inspiring to see the benefits of CAML. Please see the general response Question 1 for more details, we also updated in the paper Section 5.2 and Appendix A.2.

Question 1: Can you justify the need to use AML for cooperative perception in general? If it was beneficial, why had no one before this paper explored using it for more traditional tasks such as cooperative object detection?

The use of AML is particularly valuable in scenarios with constrained data where limited modalities are available for inference. Especially in real world, we cannot always guarantee that we have the same data modalities in both training and testing, some data modalities may be missing during testing compared to training. AML addresses this issue by improving the system's robustness to missing or incomplete modalities during inference. Second, AML promotes computation and resource efficiency during testing, which is very important for deployment in real world. Also as pointed out by Reviewer fWcH, the significance of CAML's principles can be applied to other multi-agent, multi-modal learning scenarios.

There are actually some previous work of exploring AML or similar ideas to perception tasks (Shen et al., 2023; Hoffman et al., 2016; Wang et al., 2018; Garcia et al., 2018, 2019; Piasco et al., 2021), as we described in the Related Work. AML is still a relatively new research area. Shen et al. (2023) formalized various related research problems as the AML framework. But these earlier works have primarily focused on single-agent settings, which is not collaborative. In this paper, we took a novel effort to integrate multi-agent systems into AML, which can help deal with more complex real-world scenarios.

Dear Reviewer,

Thanks for your valuable comment. We have specified definitions of all symbols in the updated paper Appendix A.3.2 and also removed the assertion about robustness on Line 774. If you have further questions or comments, please do not hesitate to let us know, as tomorrow is the deadline for uploading a revised PDF.

If you feel that our response and revisions adequately address your concerns, we would be really grateful if you could consider adjusting your score accordingly.

We greatly appreciate your time and effort in reviewing our work and look forward to any further feedback.

Best,
Authors

Thanks for your review and valuable feedback. We address the concerns below.

Weakness 1: Many experimental settings are not described clearly. For instance, is there a correlation between the number of agents and the number of modalities? The number of agents appears not to be explicitly stated in the experiment. Additionally, I am curious about how the experiment is devised when the number of modalities exceeds 3, and what the outcome would be.

We appreciate your attention to the clarity of the experimental settings. To clarify, the number of agents does not correlate with the number of modalities. The modalities are independently handled by each agent locally with respective encoders (e.g., RGB, LiDAR). We have clarified this in the updated paper Section 3 and we also explicitly stated the number of agents in the experiments in Section 5.1 Data Collection.

As for handling more than three modalities, we believe that the CAML framework can still work, such as text, audio, vision and other sensor inputs, without fundamental limitations. We plan to explore the integration of more diverse modalities as a direction for future work.

Weakness 2: Accurate numbers should preferably be marked on the histograms in the experiment for convenient viewing. For example, in Figure 4(a), it is difficult to tell the highs and lows. Meanwhile, I'm also curious about why while STGN uses both RGB and depth data during testing, CAML relies solely on RGB, yet achieves comparable, or even better performance.

Thank you for your valuable feedback. Accurate numbers have been marked on the histograms in the updated Figure 3, 4 and 5. Regarding your question that CAML achieves comparable performance with STGN, we believe there are two main reasons. First, CAML effectively leverages LiDAR data as an auxiliary modality during training to boost performance through knowledge distillation. Second, in STGN, depth is used to predict object 3D bounding boxes with YOLO, and the predicted 3D position is then used for downstream tasks. But the YOLO model is only pretrained, not finetuned, so it may generate less accurate 3D positions.

Weakness 3: There should be a comparative analysis of training complexity. Multiple agents inevitably increase the parameters of the network. Is the source of the performance improvement due to the increase in network parameters?

Thank you for the insightful comment. We have added the comparative training complexity analysis of AML and CAML in the updated paper, please see Appendix A.3.1. It is true that incorporating multiple agents introduces additional parameters to the system, the primary source of performance improvement in CAML is not solely due to the increase in network complexity. Instead, the performance gains come primarily from the collaborative nature of the multi-agent collaboration, where agents share and aggregate information across modalities, thus improving the overall robustness and accuracy of predictions.

Weakness 4: Discussions on limitations are necessary as there is no description of the limitations of the proposed method.

Thank you for the valuable comment. We have added discussions on limitations in the paper, please see general response Question 3 and Section 6 Conclusions and Limitations.

Ablation Studies. In the ablation studies, we explore another variant of CAML called Pre-fusion CAML, applied to the experiment of aerial-ground vehicles collaborative semantic segmentation. However, it is important to note that this variant can be applied to other domains and experiments as well. In this variant, each vehicle first locally extract feature maps of size $7\times7$ for both RGB and depth modalities. Instead of separately fusing the RGB and depth features between the vehicles, we first fuse the feature maps of RGB and depth within each single vehicle using cross-attention. Then we share and merge the fused RGBD features between vehicles via concatenation. We also apply $1\times1$ convolution to reduce the feature maps to the original channel dimensions. The multi-agent, multi-modal feature aggregations then pass through the decoder. Finally, we obtain the output map by upsampling to match the input image size. The mIoU of the Pre-fusion CAML is similar to that of CAML, achieving 59.16% and 65.78% for indoor and outdoor environments, respectively. Although the fusion order is different, both versions benefit from robust feature aggregation and multi-agent collaboration, which ultimately results in better segmentation performance. For more details, please refer to Appendix A.2.2.

Question 2 (97B9, fWcH): More experimental and training settings, exact modalities used in the experiments.

In the experiments of collaborative decision-making for CAV, during data collection at each timestamp, the ego vehicle has a maximum of three collaborative vehicles, provided their distance is within a threshold of 150 meters. For each vehicle, both RGB and LiDAR data are used during training, while only RGB data is used during testing in CAML. We have updated this in Section 5.1 Data Collection.

In the experiments of collaborative semantic segmentation with aerial-ground vehicles, we use the dataset CoPeD (Zhou et al., 2024), with one aerial vehicle and one ground vehicle, in two different real-world scenarios of the indoor NYUARPL and the outdoor HOUSEA. Additionally, we introduce noise to the RGBD data collected by the ground vehicle. For both aerial and ground vehicles, RGB and depth data are used during training, while only RGB data is used during testing in CAML.

For training, we employ a batch size of 32 and the Adam optimizer with an initial learning rate of $1e{-3}$, and a Cosine Annealing Scheduler to adjust the learning rate over time. The model is trained on an Nvidia RTX 3090 GPU with AMD Ryzen 9 5900 CPU and 32 GB RAM for 200 epochs. We have updated these details in Appendix A.3.1. For the comparative training complexity of AML and CAML in both experiments, we list in the following Table 2 and 3.

Table 1: Training complexity of AML and CAML in collaborative decision-making for connected autonomous driving.

Table 2: Training complexity of AML and CAML in collaborative semantic segmentation for aerial-ground vehicles.

Question 3 (97B9, fWcH): Limitations and failure case analysis.

Even though the advancements of CAML, there are some limitations and failure cases. One failure case is that if the modalities are misaligned, the model may struggle to perform effective fusion, leading to incorrect predictions. For instance, if modalities such as RGB and depth are captured at different time intervals or from non-overlapping fields of view, combining them effectively can be difficult. The auxiliary modalities or views from collaborative agents may become noise, useless or even degrading performance. Another limitation is the increasing system complexity. As the number of agents increases, the complexity of the system grows. The fusion of multi-agent and multi-modal data introduces challenges related to coordination overhead, which may lead to delays in the collaborative learning process. We have added this analysis in Section 6 Conclusions and Limitations.

[1] Yang Zhou, Long Quang, Carlos Nieto-Granda, and Giuseppe Loianno. Coped-advancing multi- robot collaborative perception: A comprehensive dataset in real-world environments. IEEE Robotics and Automation Letters, 2024.

[2] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.