CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

Thanks for your comments and feedback. We are glad that you found our paper technically sound with effective performance improvements. We address your concerns below and will incorporate all feedback into the final version. We hope you could consider raising the score.

Contributions of CAML

Please note that CAML is not simple combination of AML and cooperative system. Multi-agent settings introduces substantial complexity, requiring careful consideration of new challenges. CAML specifically addresses the unique challenges of multi-agent systems, including heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making. CAML has broader applications of connected autonomous driving, collaborative search and rescue, as well as intelligent transportation. To the best of our knowledge, this work is the first to propose a flexible and principled framework capable of tackling a broad range of collaborative tasks in multi-agent systems where limited modalities are available for inference. Unlike prior work that either focuses on multi-agent collaboration but without addressing missing modality at test time, or explores AML ideas solely in single-agent settings, CAML unifies these directions for the first time.

The agents in the framework are not restricted to the RGB modality. CAML is a general framework and can work with any modality. In the experiments of CAV and aerial-ground robot collaboration, we use RGB as the available modality during inference because it is the most commonly used and widely available in robotics and autonomous driving.

Additionally, we conduct new experiments where LiDAR is retained while RGB is removed during testing, please see the results in the following table, with mean and std over four repeated runs. The performances of retaining RGB or LiDAR are similar but retaining LiDAR shows slightly better performance as LiDAR provides more precise spatial localization which is better for accident detection.

Performance comparison to late cooperation

We conduct additional experiments using a late cooperation strategy for the CAV tasks. For each vehicle and each modality (RGB, LiDAR), we compute its decision output (action logits). Specifically, we use the same encoders as in CAML, ResNet-18 for RGB and PointTransformer for LiDAR, to extract features independently from each modality. These features are then passed through MLP prediction heads to generate action logits. To perform late cooperation, we first fuse the logits of the same modality from connected vehicles to the ego vehicle by averaging. Next, we fuse the averaged logits across modalities by taking their mean. The resulting fused multi-modal, multi-agent logits are used to determine the ego vehicle’s control actions for accident detection. We then apply the same knowledge distillation procedure as CAML, distilling from the RGB-LiDAR teacher model to a student model that only uses RGB at test time. The results are presented in the following table. As shown, CAML outperforms the late cooperation approach, as it enables the model to learn richer and more complementary representations across agents and modalities. In contrast, late cooperation only fuses output decisions, which may lead to information loss.

Difference between other RGB-based intermediate cooperation methods

CAML is an intermediate cooperation approach. It has following advantages compared to existing RGB-based intermediate cooperation methods.

1. Robustness under modality reduction: Unlike typical RGB-based cooperation methods that assume consistent modality availability during both training and inference, CAML is explicitly designed to handle modality reduction at test time. It enables robust inference even when high-cost or unreliable modalities (e.g., LiDAR) are unavailable during deployment.

2. Flexibility to agent and modality heterogeneity: CAML is a general framework that supports a varying number of agents and heterogeneous modality availability per agent. This makes it particularly suited for resource-constrained environments where sensors may fail, communication may be intermittent, or hardware capabilities differ across agents.

3. Cross-agent, cross-modal knowledge distillation: CAML goes beyond traditional distillation by performing cross-agent, cross-modal knowledge transfer. It enhances the RGB representation by leveraging auxiliary modalities and collaborative knowledge shared among agents, leading to improved performance even in RGB-only inference settings.

Training complexity of pre-fusion CAML

In terms of training complexity, Pre-fusion CAML is very similar to that of CAML, since both share a similar overall network architecture and use same modality-specific encoders. We illustrate in the table below with the collaborative semantic segmentation task.

Both CAML and its variant Pre-fusion CAML have their advantages, the preferable choice depends on the specific task requirements. If cross-agent modality alignment and consistency in feature representation are critical, CAML is more suitable. If preserving agent-specific contextual understanding and reducing inter-agent communication are priorities, Pre-fusion CAML may be preferred.

Thanks for your comments and feedback. We are pleased that you found our collaborative multi-agent design effective in enabling complementary information exchange and enhancing system robustness. We are also encouraged that you recognized the practical value of CAML for deployment, as validated on real-world data. Below, We address your concerns and will incorporate all feedback into the final version. We hope you'll consider raising the score.

Additional experiments of retaining LiDAR and removing RGB

We conduct additional experiments by retaining LiDAR and removing RGB during testing for the experiments of collaborative decision-making for CAV. We present the results in the following table, with mean and std over four repeated runs. The performances of retaining RGB or LiDAR are similar but retaining LiDAR shows slightly better performance as LiDAR provides more precise spatial localization which is better for accident detection.

Modalities

Yes, the proposed framework can generalize to settings with more modalities, and it is capable of handling arbitrary missing modalities at inference time. Please note that we did not evaluate only two modalities, in the experiments of CAV, we evaluated RGB and LiDAR, in the experiments of aerial-ground robot collaboration, we also evaluated RGB and Depth.

In addition, we conduct additional experiments, using three modalities for the CAV task, where each agent receives RGB, LiDAR, and state information (position and velocity) during training. At test time, only the RGB modality is available for each vehicle. As shown below, incorporating state information during training improves the system performance.

Handling information-missing situations

Yes, CAML can handle such information-missing situations. It is not only a multi-modal system but also a multi-agent framework. From the multi-modal perspective, CAML addresses missing modalities at the modality level. From the multi-agent collaboration perspective, CAML explicitly handles information-level missingness, such as when the ego vehicle has a limited field of view, by leveraging complementary information from surrounding vehicles through V2V communication. This enables the system to compensate for partial observations by integrating diverse views across agents, thereby enhancing perception and decision-making capabilities.

Discussion on related work

Thanks for the suggestion. CAML differs from the two papers in following:

1. CAML leverages multi-agent collaboration and knowledge distillation to address both missing modality and partial agent coverage, which is not explored by the two referenced single-agent works.

2. The other two works focus on improving single-agent robustness to missing sensor inputs, either through transformer fusion strategies or prompting, without multi-agent collaboration or cross-agent information transfer.

We will be sure to discuss these related works in the camera-ready version.

Thanks for your comments and feedback. We are glad that you found our approach preserving collaborative insights, flexible for modality/agent, with strong empirical results. Below we answer your questions and will incorporate all feedback to the final version. We hope you'll consider raising the score.

Key innovation

Please note that our proposed CAML is not merely an extension of AML. Multi-agent settings introduces substantial complexity, requiring careful consideration of new challenges. CAML specifically addresses the unique challenges of multi-agent systems, including heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making. CAML has broader applications of connected autonomous driving, collaborative search and rescue, as well as intelligent transportation. To the best of our knowledge, this work is the first to propose a flexible and principled framework capable of tackling a broad range of collaborative tasks in multi-agent systems where limited modalities are available for inference. Unlike prior work that either focuses on multi-agent collaboration but without addressing missing modality at test time, or explores AML ideas solely in single-agent settings, CAML unifies these directions for the first time.

Knowledge distillation

The knowledge is distilled from the teacher to the student by having the student mimic the teacher’s behavior. For example, in the CAV experiments, we first train a teacher model offline using a cross-entropy loss, where each vehicle has access to both RGB and LiDAR data. Then, we train a student model that receives only RGB input, learning to replicate the behavior of the teacher model. For each data sample, the student receives the same RGB image as the teacher. The training loss for the student model combines two terms: (1) a distillation loss using KL divergence between the student and teacher outputs, which encourages the student to closely match the teacher’s output, and (2) a task loss using cross-entropy between the student’s prediction and the ground-truth label. We provided these details in Appendix A.5 in paper.

Performance difference from AML to CAML

Please note that the performance improvement of CAML over AML is not marginal, but substantial. In collaborative decision-making for CAV, CAML achieves up to a 58.1% improvement in accident detection compared to AML. Additionally, in real-world aerial-ground robot experiments for collaborative semantic segmentation, CAML achieves up to a 10.6% improvement in mIoU over AML. These results highlight the significant advantages of CAML compared to AML.

Working in existing collaborative perception method

Yes, CAML can work with existing collaborative perception methods. It is a general framework that is not limited to specific modalities and can flexibly incorporate different types of sensory inputs. CAML is particularly well-suited for resource-constrained environments where certain modalities may be missing during inference. By enabling effective reduced-modality inference, CAML helps reduce computational cost and improve system robustness during inference, otherwise the system may not work due to missing modality.

We would like to clarify that our technical innovations are not limited by transferring knowledge distillation from single-agent to multi-agent settings.

First, we also introduce technical innovations including: 1) System design, a collaborative teacher-student architecture tailored to the multi-agent setting. 2) Cross-agent, cross-modal knowledge transfer that extends beyond traditional KD, enabling better handling of partial observability and supporting team-level coordination. 3) Efficient runtime design: as shown in Table 1 in paper, our model achieves much lower communication bandwidth and inference latency than prior collaborative methods.

Second, our use of KD is motivated by practical deployment constraints of multi-agent systems, where certain modalities may be unavailable at test time due to missing data or resource limitations. To address this, we leverage auxiliary modalities during training and distill their information into a student model for robust and efficient inference. This goes well beyond simply applying KD from the single-agent context.

Third, multi-agent systems pose unique challenges such as heterogeneity in sensing modalities, dynamic team compositions, and team-level decision-making. Transitioning from single-agent to multi-agent learning is fundamentally non-trivial, multi-agent system learning or Agentic AI are established and complex research areas in their own right.

The usefulness of knowledge distillation in CAML is verified through extensive and in-depth experiments. We distill knowledge from a powerful teacher model into a student model, so that the student model learns to handle missing modalities at test time and operate efficiently in resource-constrained environments. This is demonstrated across diverse scenarios, including collaborative decision-making in CAV and aerial-ground collaborative segmentation, using multiple modalities such as RGB, LiDAR, and Depth, please see Fig. 3–5 and Tables 1–3 in paper. We further validate this by conducting additional experiments incorporating state information as a third modality, showing that distilling state information during training improves the system performance, as presented in the following table.

Thanks for your comments and feedback. We’re glad you found our paper well-written, well-motivated, and the architecture sound with clear discussion of both communication setups and complexity analysis. Below we address your concerns and will incorporate all feedback in the final version. We hope you'll consider raising the score.

Experimental evaluation

We conduct additional experiments as requested, using three modalities for the CAV task, where each agent receives RGB, LiDAR, and state information (position and velocity) during training. At test time, only the RGB modality is available for each vehicle. We show the results in the following table, with mean and std over four repeated runs. As we can see, having state information as another modality improves the system performance. This setup further demonstrates CAML’s ability to leverage richer multi-modal data during training while maintaining effective performance under reduced-modality conditions at inference.

Please note that our evaluation does not only rely on simulation data and scenarios using the AUTOCASTSIM benchmark, which is used for the experiments of the connected autonomous vehicles (CAV). In addition, we evaluated on aerial-ground robot collaboration with real-world dataset CoPeD for collaborative semantic segmentation. This verifies the effectiveness of our approach in real-world environments.

Thank you for the suggestions to evaluate on tasks such as collaborative tracking and to explore datasets like RCooper. We agree that these directions are valuable and complementary to our work, but incorporating such additional experiments within the very limited rebuttal period is not feasible. Moreover, our collaborative decision-making experiments for CAV actually already implicitly involve aspects of collaborative tracking, as vehicles must first be localized and tracked before making decisions. We appreciate the suggestions and will be sure to include discussions of these potential extensions in the future work section of the final version.

Modalities

We conduct additional experiments including state information (position and velocity) as another modality for the CAV tasks, as shown in the above table. Moreover, our work does not only consider RGB and LiDAR modalities. In the aerial-ground robot experiments, we incorporate Depth as a different modality. These evaluations further demonstrate CAML’s flexibility in handling diverse sensory inputs across different tasks and environments.

Yes, it is possible to handle modalities dynamically at test time or when modalities become available intermittently. CAML does not learn modality preferences implicitly, it can utilize all modalities available at time time. In the student model of CAML, the modalities for each agent can be configured individually. The agents do not all need to have the same modalities. Although the current student model training does not dynamically change the modalities over time. As noted in the future work section, we can handle dynamic changes without changing the architecture. Our planned solution is to incorporate a modality dropout strategy. During training of knowledge distillation, we can randomly drop out or mask some modality for each agent in the student model. This allows the student model to learn robustness to dynamic modality availability and adapt effectively over time.

Discussion in limitation section

In extreme scenarios where all available modalities are uninformative (e.g., due to adverse weather), system performance may degrade. CAML improves robustness by leveraging auxiliary modalities and multi-agent collaboration during training. Additionally, our system can support uncertainty estimation to access modality effectiveness. CAML can be extended to incorporate uncertainty estimation mechanisms to further improve system robustness when available modalities are noisy. This would allow the system to detect and better handle such cases. We will include these discussions into the limitations section of the final version.