Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies

[Q1] What if the generated 3DGS results are not ideal?

[A1] Our 3D Gaussian estimation module has been pretrained and finetuned on large-scale datasets that encompass a wide range of variations, including diverse viewpoints, lighting conditions, and occlusions. To further assess its robustness, we conducted additional experiments that introduce challenging lighting variations—including random changes in light color, direction, and intensity—as well as random distractor objects (10 task-irrelevant objects randomly placed beside the manipulated objects on the table), following RoboTwin 2.0.. Despite these perturbations, our method maintained SOTA performance, achieving the highest task success rate:

This demonstrates that our Gaussian estimation is robust and reliable even in complex environments.

[Q2] How much do the 3DGS results matter?

[A2] The 3D Gaussian representations serve two critical roles in our framework. First, they act as a shared global scene representation that facilitates coordination among multiple manipulators in collaborative settings. Second, they serve as a self-supervised structural prior that helps each local agent infer fine-grained spatial cues from 2D observations. In both respects, the quality of the 3DGS representation significantly contributes to the effectiveness of our manipulation pipeline.

[Q3] Does it mean there might be a better "latent" representation of the shared 3D scene representation?

[A3] Yes! We appreciate the reviewer for raising this insightful point. This line of thought aligns well with our ongoing research. The current work represents our initial attempt to incorporate 3D Gaussian representations into manipulation tasks, where we explicitly use Gaussians to represent low-level geometry and appearance. However, we believe there is considerable potential in developing a more expressive and compact latent representation.

In particular, we are currently exploring a Gaussian tokenization framework that integrates high-level semantic cues and latent action information into the 3D representation. This would enable the representation to not only reflect the spatial structure of the scene but also better support reasoning and action generation. We envision such a latent representation as a more efficient interface for downstream tasks, including integration with VLMs or VLAs.

We sincerely thank the reviewer for the thoughtful and insightful questions.

[Q1] Per-step latency and computational expense

[A1] We evaluated the inference latency per action chunk during simulation on an NVIDIA RTX 5090 GPU. The results are shown in the table below:

The inference latency of GauDP can be decomposed into two parts:

• Gaussian Estimation: This component relies on a Transformer to compute self-attention and cross-attention. Thanks to extensive prior work on CUDA optimizations for Transformer computations, this part achieves high inference efficiency.
• Action Generation: This part is identical to that of DP, except for a minor additional CNN module. Consequently, its latency is the same as DP, with the main latency arising from the iterative denoising process.

Therefore, our GauDP method introduces only a marginal increase in inference latency compared to DP and DP3.

While DP3 achieves the highest FPS in simulation, it's important to note that the point clouds used during DP3 simulation have been preprocessed to remove background and table clutter. This is feasible in simulation due to access to privileged semantics and optimized rendering. However, in real-world scenarios, acquiring high-quality point clouds and performing such preprocessing (e.g., filtering irrelevant points and applying farthest point sampling) is both time-consuming and computationally expensive. Therefore, simulation efficiency of point cloud-based policies may not translate directly to real-world deployments. In contrast, GauDP leverages 2D observations and reconstructs 3D Gaussians efficiently.

[Q2] Robustness to real-world visual disturbances and execution errors

[A2] Our Gaussian estimation module has been fine-tuned on large-scale datasets that already encompass diverse conditions, such as varying viewpoints and illumination. Additionally, we enhance the accuracy and robustness of Gaussian estimation by incorporating local depth constraints from multiple viewpoints. This design endows our method with inherent robustness against visual disturbances.

To further evaluate robustness, we conducted experiments under novel settings that introduce challenging lighting variations—including random changes in light color, direction, and intensity—as well as random distractor objects (10 task-irrelevant objects randomly placed beside the manipulated objects on the table), following RoboTwin 2.0. In these settings, our method still achieved the highest task success rate:

Robustness to execution errors

• Gaussian Estimation: Regarding robustness to execution errors, our method reconstructs the 3D scene from two or more 2D observations. Since execution noise and actuation errors have limited impact on the 2D observations, they do not significantly affect the quality of the estimated Gaussians.
• Action Generation: The core contribution of our work is the proposal of a novel framework that leverages 3D Gaussian representations to coordinate 2D image inputs. Our policy learning technique remains consistent with that of DP and DP3, so the robustness of action generation to execution errors shares the same inherent limitations as diffusion policies. However, an interesting observation from our experiments is that policies based on GauDP tend to attempt re-execution of the current subtask more readily when encountering mid-task failures.

[Q3] Significant performance improvement on Lift Barrier task

[A3] Thank you for this insightful question. We believe that Lift Barrier and Grab Roller are particularly suitable for evaluating multi-agent coordination, as success requires tightly synchronized manipulation from both arms. Any premature or delayed movement by one agent typically results in failure.

The Barrier object is made of thin white metal wires and appears as subtle lines in 2D observations, making it difficult to perceive using standard 2D encoders. In contrast, it is much more distinguishable in 3D point cloud representations, especially processed by FPS (Farthest Point Sampling), which explains the higher success rate of DP3 compared to DP.

Our method bridges this gap by reconstructing 3D Gaussians from multiple 2D views in a self-supervised manner. This allows us to recover the spatial structure of challenging objects like the Barrier without requiring explicit 3D supervision or point cloud input. The improved success rate of GauDP on Lift Barrier demonstrates the effectiveness of our learned 3D representations for precise and coordinated manipulation.

[W1] missing tasks of RoboFactory in the main paper

[A1] Thank you for your kind recognition and valuable suggestion. We agree that including more diverse tasks is important for comprehensive evaluation. However, the tasks "pass shoe (2-agent)" and "long pipeline delivery (4-agent)" are relatively weakly cooperative in nature. These tasks can be decomposed into sequential pick-and-place subtasks that each agent can complete independently with minimal coordination. Since our framework is designed to emphasize and promote tight and synergistic collaboration among agents, we focused on tasks that require strong inter-agent dependency and coordinated decision-making.

[W2] "Stack cubes" tasks have a low or 0 success rate

[A2] We appreciate your insightful comment. Indeed, the “stack cubes” tasks are challenging despite appearing simple at first glance. There are two major contributing factors to their low success rates:

• Accumulated manipulation difficulty: Stacking cubes requires precise pick-and-place actions for each individual cube. Even a small deviation in position can cause failure in stacking. Given that several such precise actions must be executed in sequence, the overall success rate decreases approximately exponentially with the number of cubes.
• High task complexity due to spatial and sequential constraints: Each cube has a unique color and must be stacked in a specific order. Agents must select the appropriate cube based on color and determine which agent is closest and most suitable to pick it up. Since cube positions are randomly initialized, the spatial layout and action sequence vary significantly across demonstrations. This introduces substantial variability and demands that the policy generalize across many permutations—much more than in other tasks.

To further investigate, we fine-tuned Pi0 (a strong VLA-based method) on the “stack cube three” task. The success rate remained extremely low (~1%), suggesting that the difficulty arises not from our specific method, but from the fundamental complexity of the task.

In summary, the “stack cubes” tasks uniquely combine the need for high-precision manipulation, spatial reasoning, and decision-making under dynamic layouts. These factors make them the most difficult among our experiments, and we hope our analysis helps clarify the observed performance gap.

Dear Reviewer,

Please comment on the rebuttal ASAP. Thanks.

AC.

We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate the suggestions regarding reproducibility, fairness of comparison, and computational analysis. All of our hyperparameters were aligned with the configurations used in prior work. We will include additional hyperparameter details in the main paper and release all related code and configuration files to ensure full reproducibility. We have carefully addressed each point as follows:

[A1] Hyperparameter Details for Reproducibility

We agree with the reviewer that providing complete hyperparameter settings is essential for reproducibility. We will include these details in the revised version of the main paper. Specifically, our proposed GauDP and all baseline methods adopt identical hyperparameters following prior diffusion policy works and existing benchmarks:

• Action prediction horizon: 8
• Number of observation steps: 3
• Number of action execution steps: 6
• Denoising schedule: Our experiments follow the optimal configurations of DP and DP3 to ensure fair comparison. Specifically, both GauDP and DP adopt DDPM with 100 denoising steps for both training and inference, while DP3 uses DDIM, also with 100 steps.

[A2] Training/Inference Time, Parameter Count

We have conducted a thorough analysis of training and inference runtimes, as well as model size.

Training time depends on the trajectory length of each task. Take the Lift Barrier task as an example, the number of elapsed steps is approximately 100:

• Training time (100 epochs on A100 GPU):

  • DP: 4.8 GPU hours
  • DP3: 2.5 GPU hours
  • GauDP: 6.5 GPU hours

  GauDP has comparable training time to DP, thanks to two factors:

  • The Gaussian estimation module is frozen during the whole policy training phase;
  • We implemented extensive optimizations on data I/O and training efficiency for all these methods. (The relevant codes will be released after the review process.)

• Inference speed of action chunk (Simulation evaluation on NVIDIA 5090 GPU):

  • DP: 1.49 FPS
  • DP3: 1.57 FPS
  • GauDP: 1.28 FPS

• Parameter counts (depend on the number of agents):

Table 1

In the table above, GauDP-full refers to the complete model including the Gaussian estimation module, while GauDP-policy only includes the policy learning component. To ensure a fair comparison in terms of parameter count, we introduce LargeDP, which scales up the original DP model to match the parameter size of GauDP-full. The corresponding performance is reported in Table 2. It is also important to note that the majority of parameters in GauDP are frozen during policy learning, whereas all parameters in LargeDP are learnable. This comparison highlights that GauDP introduces minimal overhead within the policy module, yet achieves a significant improvement in success rate, demonstrating the effectiveness of our framework in enhancing multi-arm coordination.

[A3] Fairness of Comparison

We appreciate the reviewer’s concern regarding fair comparison. To clarify, our policy architecture is strictly aligned with DP: we use the same observation encoder (Resnet18) and the same 1D conditional diffusion module as DP. The only additions in GauDP are:

• A fine-tuned NoPoSplat module with local depth constraints for estimating the Gaussian representation;
• Two 3-layer CNNs: one for channel adjustment and one for pre-fusion of the Gaussian image.

This design choice allows us to isolate and evaluate the benefits of our proposed Gaussian-Image Synergy framework under controlled conditions. By keeping the core policy module unchanged, we ensure that observed improvements result from the proposed representation rather than from architectural scaling. We will further clarify the model architecture details in the supplementary material to avoid any potential confusion.

[A4] Model Size Normalization

We further scaled up the baseline DP model into LargeDP to match or even exceed the parameter count of GauDP, as shown in Table 1. GauDP still achieves the highest success rate, as shown in Table 2, reinforcing that the performance gain stems from our proposed framework rather than increased model capacity. LargeDP still lacks an effective representation for global perception and fine-grained understanding of 3D structure and spatial relationships. This further supports the effectiveness of our method in facilitating multi-agent cooperation.

Table 2

[A5] Revisions to Main Text and Supplementary Materials

We will include additional hyperparameter details in the main paper and provide background information on diffusion policy in the preliminaries. Additionally, we will shorten the ablation study section in the main paper and provide more detailed descriptions of the model architecture in the supplementary materials accordingly in the revised version.