How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation

We sincerely appreciate the reviewer’s thoughtful and valuable feedback.

C1: Contribution of performance improvement

As in Table 8 of our supplementary material, using only texture-prior and no-prior queries (row 3) performs worse than the combination of all three query types (last row). This indicates that the performance improvements primarily stem from our novel fusion strategy—the combination of 3 types of queries in the PQG module—rather than solely relying on pre-trained strong image models, which only influence texture-prior query generation.

C2: Discussions with other top-performing LiDAR-only methods

Thank you for suggesting these two references. We will include them in the final version and provide the following discussion:

LidarMultiNet: IAL outperforms LidarMultiNet on all key panoptic segmentation metrics despite using only a single frame and standard annotations, whereas LidarMultiNet leverages extra temporal frames, detection annotations, and multi-task learning. While LidarMultiNet slightly surpasses IAL in mIoU, this metric primarily reflects 3D semantic segmentation and is not the main evaluation metric for 3D panoptic segmentation. Moreover, some of LidarMultiNet’s effective components, such as GCP and two-stage, do not conflict with our multimodal method and could potentially be integrated into IAL to further enhance multi-modal 3D panoptic segmentation performance.

PUPS: PUPS exibits inconsistent performance across datasets. Specifically, it significantly outperforms the baseline method Panoptic-PHNet on SemanticKITTI (Table 1 & 2 of PUPS) but achieves only comparable performance to Panoptic-PHNet on nuScenes validation set (Table 3 of PUPS). In contrast, IAL consistently achieves superior result over Panoptic-PHNet on both datasets. Since PUPS has not released its code for reproducibility, we hypothesize that its inconsistent performance trend may stem from overfitting to the simpler SemanticKITTI. Compared to SemanticKITTI, nuScenes presents greater challenges due to its larger scale, more diverse domains. IAL attains the highest performance on nuScenes, further demonstrating its robustness and practical effectiveness.

C3: Inference speed and model size comparison

Good catch! We provide a comparison between across all methods in the table below.

To isolate the impact of external image models (SAM, GroundingDINO), we exclude 2D preprocessing step and report the efficiency of the remaining pipeline in 3rd row (denoted by *). Notably, the inference speed of our core pipeline is faster than LCPS, showing the efficiency of our design.

We replace the heavy 2D preprocessing step (GroundingDINO and SAM) with a lightweight alternative, Mask R-CNN (ResNet50 backbone), as reported in 4th row. This variant strikes a balance between speed and performance, running at 2.7 FPS (1.6× faster than LCPS) while maintaining 81.7 PQ. These results indicate that while external 2D mask generation models introduce additional computational costs, our query generation design remains flexible, accommodating different 2D mask proposal models to balance high-quality performance and fast inference speed. More importantly, our core framework remains highly efficient and robust to different 2D preprocessing choices. Also, further optimizations such as model pruning or quantization could further accelerate inference.

C4: Robustness when a modality is degraded

Great suggestion! We evaluate the performance of IAL by comparing the LiDAR-only branch with the full model on the nighttime split (602 scans) of the nuScenes val set (6,019 scans), where image quality is significantly degraded. The results, (63.2 vs 70.5 PQ), show that despite the low image quality, our IAL (full model) still achieves a +7.3% PQ gain over the LiDAR-only model. This improvement can be attributed to two factors: 1. Modality-synchronized augmentation (PieAug), which exposes the model to more diverse samples, including nighttime scenarios, by mixing synchronized LiDAR and image data. It allows the model to generalize better to rare conditions like night scenes. 2. The combination of 3 types of queries in PQG, where no-prior queries complement the texture-prior and geometric-prior queries, helping the model to effectively identify potential instances. Additionally, pre-trained Grounding-DINO and SAM models contribute to enhancing the robustness of 2D mask generation in night conditions, as they have been trained on vast amounts of data, improving generalization to such challenging scenarios.

We sincerely thank the reviewer for the positive feedback and constructive suggestions.

C1: Suggestion for adding additional references.

Thank you for pointing out these two papers. We will add them in the final version and provide the following discussions:

1. UniSeg (ICCV’23) proposes a unified multi-modal fusion strategy for LiDAR segmentation, focusing on fusing voxel-view and range-view LiDAR features with image features. However, UniSeg primarily targets semantic segmentation and does not explore how to utilize multimodal information for instance segmentation, which focuses on thing classes. In contrast, our design of prior-based query generation is specifically tailored for 3D instance segmentation. Furthermore, while UniSeg employs Cartesian voxelization, our method uses cylindrical voxelization, which has been proven more effective for panoptic segmentation in prior works like PolarNet (CVPR'20) and Cylinder3D (CVPR'21). We also incorporate Geometric-guided Token Fusion (including scale-aware positional encoding) to better accommodate this cylindrical representation.

2. MM-FSS (ICLR’25) focuses on few-shot 3D semantic segmentation in indoor scenarios by leveraging both explicit textual modalities and implicit 2D modalities for cross-dataset adaptation. This differs significantly from our work, which focuses on fully-supervised multi-modal 3D panoptic segmentation in outdoor scenarios. Additionally, the fused modalities in MM-FSS (text, image, and point cloud) differ from our approach, which uses multi-view images and LiDAR point clouds.

C2: Could the authors provide the efficiency analysis such as comparing the computational cost and the parameter count with previous methods?

Thank you for your suggestion! We provide a comparison of inference speed (FPS), parameter count, and performance (PQ, the primary evaluation metric) between our method and LCPS, the main baseline, as shown in the table below.

To better understand the computational cost and model size of our framework’s major components, we report a lightweight variant of our model (denoted by * in the 3rd row), which excludes the 2D preprocessing step (Grounding-DINO and SAM). This analysis reveals that the main computational cost and parameter count stem from the sophisticated 2D preprocessing step. However, our core components - including GTF, geometric-prior query generation, 2D and 3D encoder, and transformer decoder - are highly efficient, achieving 4.0 FPS with only 81.8M of parameters.

To further evaluate efficiency, we replace Grounding-DINO and SAM with Mask R-CNN (ResNet50 backbone), a lightweight 2D mask generation alternative, as reported in the 4th row. This variant achieves a well-balanced trade-off between speed and performance, with the full pipeline running at 2.7 FPS (1.6× faster than LCPS) while maintaining 81.7 PQ (+1.9% over LCPS). These results demonstrate that while external proposal models contribute to computational cost, our core method remains highly efficient and adaptable to different 2D preprocessing choices.

We sincerely thank the reviewer for their constructive feedback and positive comments on our contributions.

C1: Fig. 3 is too far from the Sec. 3.1

Thank you for the suggestion. We will update the placement of Fig. 3 in the final version to ensure that PieAug is introduced and illustrated in a more cohesive manner.

C2: The impact statement on page 9 should be in the supplementary material

Thank you for the suggestion. However, we followed the official guidelines, which recommend placing the impact statement in a separate section at the end of the paper, co-located with the Acknowledgements, before the References.

C3: Add qualitative comparison among ablation studies

Great suggestion! Unfortunately, due to the constraints of the rebuttal policy, we are unable to include images or provide links to them on the OpenReview platform. However, we will add these qualitative comparisons in the final version of the paper.

C4: Moving Tab. 8 to the Main Paper

Thank you for your suggestion. We agree that Table 8 is crucial for showing how different prior queries impact the results. We will move it to the main paper in the final version.

C5: Will the mask prior strongly impact texture queries' effectiveness? E.g., replacing Grounding SAM with Mask DINO or Mask R-CNN?

Good catch! We have replaced the mask generation module (Grounding-DINO and SAM) with alternatives, including Mask R-CNN (as you suggested) and its follow-up, the HTC model [1], and report the results in the table below. As shown, the performance of our model remains robust to the choice of mask priors from different mask proposal methods, with only a minimal performance drop when using Mask R-CNN or HTC. This indicates that our texture-prior query generation design is flexible, accommodating different 2D mask proposal models, and can be integrated with advanced off-the-shelf methods (in the future) for further performance improvements.

[1] Chen Kai, et.al. Hybrid task cascade for instance segmentation. CVPR'19.

Thank you for your valuable feedback on the voxel representation, the model efficiency, and the design of query initiation. Regarding your questions, we provide the following detailed explanation.

Q1: Justification of the adoption of cylindrical representation in the LiDAR branch.

While Cartesian voxelization ensures uniform scales, it inherently struggles with the density imbalance in LiDAR point clouds. Due to radial sparsity—where points are denser near the sensor and sparser at a distance—Cartesian voxels fail to accommodate this variation, potentially causing uneven feature distribution and information loss, particularly in distant regions.

The advantage of cylindrical over Cartesian voxel representations is well-documented in prior work, e.g., Cylinder3D (in CVPR’21, Sec. 3.2) and PolarNet (in CVPR’20, Sec 3.3), showing superior performance through adaptive partitioning of sparse distant points. Additionally, adopting cylindrical representation aligns with high-performing baselines like P3Former (IJCV’25) and LCPS (ICCV’23), ensuring consistency and fair comparison.

C2: Computation cost comparison

We compare inference speed (FPS), model size (Params), and performance (PQ, the primary evaluation metric) between our method and LCPS, the main baseline. Additionally, we report a lightweight variant of our model, denoted by *, which excludes the 2D preprocessing step (Grounding-DINO and SAM) to highlight the efficiency of our framework’s major components. Since the 2D preprocessing can be replaced with any off-the-shelf model, this variant demonstrates the trade-off between speed and performance. All latency measurements are conducted on a single NVIDIA A40 GPU with batch size 1. For fair comparison, we measure LCPS latency using its official codebase on our hardware.

As shown in the table above, with Grounding-DINO and SAM, IAL achieves slightly lower FPS than LCPS (0.9 vs. 1.7) but significantly improves PQ (82.3 vs. 79.8), making it ideal for accuracy-critical applications. Removing the 2D preprocessing step (row 2) makes our model 2.4× faster than LCPS. The choice of 2D preprocessing is flexible, balancing accuracy and efficiency. For instance, replacing Grounding-DINO and SAM with Mask R-CNN (ResNet50 backbone, IAL-MaskRCNN, row 4) yields 2.7 FPS (1.6× faster than LCPS) while maintaining a strong PQ of 81.7 (+1.9% over LCPS).

C3.1: Novelty of prior-based query generation

We respectfully disagree with the comment. Our prior-based query generation (PQG) is both novel and well-motivated, as acknowledged by reviewer #DcuJ. PQG fundamentally differs from TransFusion and FSF in both design and query generation mechanism.

From a design perspective, unlike TransFusion and FSF, which solely rely on prior-based queries, our no-prior query solution specifically targets objects missed by geometric (3D) and texture (2D) priors, a critical gap for cross-modal fusion that has been unaddressed in prior works.

In terms of query generation, PQG introduces a more advanced approach that enhances prior utilization. While TransFusion fuses image features with BEV LiDAR features for heatmap prediction (which doesn't fully leverage the rich texture features and may harm each modality during fusion), our PQG module generates 2D and 3D prior-based queries independently. This allows us to fully leverage the rich texture information without information loss and perform late fusion, better complementing the two modalities. As for FSF, they do not explicitly use queries; instead, they extract potential instances from both 2D and 3D branches and merge them via self-attention for final prediction. Our 2D query generation differs in that we use clustering to group points within a frustum and then apply Farthest Point Sampling (FPS) to select potential candidates, effectively mitigating outliers (e.g., background points). In contrast, FSF assumes all points within a mask frustum belong to the same semantic object, which can cause semantic ambiguity.

C3.2: Performance explanation when adding "no-prior queries"

We would like to clarify that we keep the total number of queries the same (as stated in L616-617) across all settings in Table 8. This ensures that the observed differences in performance are not influenced by the total number of queries, but instead reflect the contribution of different query combinations. Thus, the performance improvement when adding no-prior queries cannot be attributed to an increased number of queries, but rather to the complementary benefits of no-prior queries in enhancing the model’s ability to handle missed objects.