IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals

W1: Inefficient pipeline: Two-stage process for SSC and PSC.

We appreciate your insightful feedback. To address your concerns, we conduct additional ablation experiments, shown in Tab. 1. We apologize for the limited formatting due to Markdown.

Single-stage methods (p, q) struggle to balance SSC and PSC, with single-head configuration (p) performing the worst due to its inability to separate semantic and instance-level learning. Single-stage, dual-head methods (a, b) also fall short, as the lack of stage-wise optimization hinders instance registration. Two-stage methods highlight the advantages of stage-wise training: SSC-focused approaches (e, f, k) excel in SSC but underperform in PSC due to limited adaptation, while PSC-prioritized methods (j, n) improve instance registration but compromise semantic consistency or balance. IPFormer, method (r), decouples SSC and PSC optimization, achieving strong instance registration and balanced performance across both tasks, with a moderate SSC trade-off.

We also improve IPFormer’s efficiency by integrating non-learnable operations, such as panoptic aggregation, noted as efficient by Reviewer [rTop]. Additionally, we use a single intermediate grid resolution, unlike methods like OccFormer and PaSCo, which rely on multi-scale voxel features.

Table 1: Additional ablation experiments. Note that $\lambda_{\text{depth}}$ was mistakenly reported as $1$ in our manuscript, but is actually $0.0001$. We apologize.

W2: Ground truth artifacts in SemanticKITTI and the need for evaluation on cleaner datasets.

We thank you for noting that SemanticKITTI contains trace artifacts from aggregated LiDAR scans. To address this, we cross-validate IPFormer on the SSCBench-KITTI-360 dataset, a clean dataset without trace artifacts, similar to Occ3D and nuCraft. As shown in Tab. 1 of the rebuttal to Reviewer [rTop], IPFormer consistently outperforms its closest competitor, CGFormer+DBScan, across all PSC and SSC metrics, demonstrating that instance delineation in our method is not influenced by ground-truth artifacts.

W3 & Q3: Comparison with SparseOcc for image-only PSC on the Occ3D-nuScenes dataset.

We appreciate your effort in highlighting the camera-based occupancy method SparseOcc. However, we argue that a comparison with IPFormer is not equitable. SparseOcc uses (i) $6$ surround-view cameras ($6$ multi-view images) and (ii) up to $96$ images per forward pass (8 or 16 frames, each with 6 images) to reconstruct a panoptic grid. Additionally, SparseOcc evaluates performance using non-standard metrics, differing from classical panoptic metrics established in the field.

Regarding the Occ3D-nuScenes dataset, we kindly refer to Tab. 1 of the rebuttal to Reviewer [rTop], where we cross-validate IPFormer on the SSCBench-KITTI-360 dataset, a clean dataset without trace artifacts. These results on zero-shot generalization demonstrate that IPFormer consistently outperforms its closest competitor, CGFormer+DBScan, across all PSC and SSC metrics.

Q1: What is the main contribution to reduce the runtime? No additional DBSCAN computation?

We thank you for raising these important questions. The main contribution of our work is not the removal of DBSCAN to reduce runtime, as this advantage is a byproduct of addressing PSC in an end-to-end fashion. Instead, our primary contribution lies in the introduction of context-adaptive instance proposals, which extend state-of-the-art instance queries that are static at test time.

As shown in Tab. 2 of our manuscript, this approach improves Thing-related metrics by an average of $18.65$%. This highlights the effectiveness of context-adaptive proposals and marks a key advancement in 3D vision research.

Q2: Consideration of invisible voxel features and their impact on efficiency and instance-voxel alignment.

We appreciate these insightful questions and provide additional ablation results in Tab. 2.

Including invisible voxels demonstrates superior performance across all PSC metrics. For SSC metrics, the configuration without invisible voxels achieves slightly better IoU, while our method attains marginally higher mIoU. Importantly, GPU utilization remains unchanged, as IPFormer operates on a dense voxel grid.

Table 2: Ablation experiments on the influence of invisible voxels, on the SemanticKITTI dataset.

Closing Remark.

The Reviewer's effort and constructive suggestions are sincerely appreciated. We hope our efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification.

Thank you for your comment. We are glad that some of the concerns could be addressed, and are pleased to provide clarification on the remaining concerns.

Novelty of our method compared to SparseOcc and PaSCo:

Unlike our method, the core methodological contributions of SparseOcc and PaSCo (apart from PaSCo being LiDAR-based), primarily address computational efficiency by leveraging sparsity mechanisms. PaSCo employs a multi-scale sparse generative decoder to reconstruct scene geometry and semantics while relying on static queries (called “query features” in the underlying Mask2Former decoder architecture) for instance prediction. Similarly, SparseOcc introduces a sparse voxel decoder and sparse instance queries to predict semantic and instance occupancy. However, a key limitation of both methods is their reliance on static queries, which are fixed during inference and cannot adapt to the observed scene. IPFormer directly addresses this limitation by introducing context-adaptive instance proposals that dynamically adapt to the scene at both training and test time. This dynamic adaptation allows IPFormer to better capture scene-specific details and improve instance-level reasoning. Therefore, the motivation of our work is different, as we aim to overcome the rigidity of static queries. Our core methodological contribution lies in introducing context-adaptive instance proposals, which alleviate the limitations posed by static queries in both PaSCo and SparseOcc, marking a significant advancement in 3D vision research.

To this end, Reviewer [8FSR] does not question the novelty of our approach but actually highlights it as a strength. Specifically, Reviewer [8FSR] states that

DETR-style queries, which are learned during training and fixed at test time, have limited representational capacity and ignore input image's context which may hurt generalization. Therefore the proposed dynamically adaptive instance proposals which are generated from the input image context seems like a sound improvement over DETR-style instance queries. This improvement is backed by the ablation in Table 2 which shows consistent improvement in almost all metrics when switching from learned but fixed instance queries to their proposed context adaptive instance proposals.”

This feedback, along with Reviewer [rTOP]’s acknowledgment of “addressing limitations of static transformer queries”, underscores the novelty and methodological contribution of our work.

Sparse voxel representations:

As discussed above, our primary goal is to address the limitations of static instance queries by introducing context-adaptive instance proposals for vision-based PSC. The use of dense voxels is a deliberate design choice to ensure effective reasoning about voxel-instance relationships, which is successfully demonstrated by the ablation experiment presented in Tab. 2 of the rebuttal. Incorporating sparsity mechanisms, while beneficial in other contexts such as SparseOcc’s handling of up to 96 temporal frames, would diverge from our primary research objective of evaluating the effectiveness of context-adaptive instance proposals. By isolating our investigations to this specific contribution, we ensure scientific rigor, allowing us to clearly demonstrate the advantages of our proposed solution. This focused approach ensures that our contributions are both targeted and impactful, as evidenced by our extensive ablation studies, the state-of-the-art performance on panoptic metrics for in-domain data, superior performance on out-of-domain data via zero-shot generalization, and the runtime reduction of over 14× compared to baseline methods, underscoring both the impact and efficiency of our solution.

We hope that the remaining concerns are clarified, and we will incorporate these clarifications into the revised manuscript, as your feedback is valuable to improve the quality of our work. Please let us know if any concerns remain, as we would be pleased to provide further clarification.

We are sincerely thankful for your valuable feedback and thoughtful suggestions, as well as for taking the time and effort to thoroughly understand the detailed technical intricacies of our work – we deeply appreciate it.

W1: Clarification on instance proposal generation and its adaptation from CGFormer.

We thank you for pointing out the need for additional clarity regarding the technical details of our instance proposal generation. We acknowledge that the formulation in our manuscript could have been more explicit and are pleased to provide further elaboration.

CGFormer’s query generator performs two key steps: (1) it generates a dense voxel grid of context-dependent queries, and (2) enhances their representational capacity by adding learnable embeddings. This understanding is derived from CGFormer’s official implementation on GitHub, as the details are not fully described in the paper. In our work, we extend these ideas to a novel domain by introducing the first context-adaptive instance proposal mechanism for PSC. Specifically, we enhance our proposal generator by integrating learnable embeddings to improve its representational capacity. This approach represents a significant advancement in 3D vision research, as no prior work has explored context-adaptive instance proposals for PSC. Technically, our proposal generator $g$ (Eq. 5 in the manuscript) can be more precisely expressed as:

$

\mathbf{I}_P = g(\text{DSA}(\tilde{\mathbf{V}}_\text{vis}^\mathbf{x},\tilde{\mathbf{V}}_\text{vis},\mathbf{x})) = \text{DSA}(\tilde{\mathbf{V}}_\text{vis}^\mathbf{x},\tilde{\mathbf{V}}_\text{vis},\mathbf{x}) + \mathbf{W}_\mathbf{I}

$

To validate the effectiveness of this approach, we conduct an additional ablation experiment to analyze the performance of context-adaptive instance proposals with and without learnable embeddings. These results, presented in Tab. 1, demonstrate that the inclusion of learnable embeddings consistently improves SSC and PSC metrics, further supporting the value of this design choice.

Table 1: Ablation on IPFormer’s instance proposals wiht and without the addition of learnable embeddings, on the SemanticKITTI dataset.

W2: Performance trade-offs between IPFormer and CGFormer across PSC and SSC metrics.

We appreciate the insightful comments on the performance comparison between IPFormer and CGFormer+DBScan. To address this concern, we want to emphasize the following points:

CGFormer itself does not outperform IPFormer. Its performance gains stem from the addition of DBScan clustering, which enables instance registration within the SSC prediction. For end-to-end solutions like IPFormer, the difference in SSC and PSC performance is primarily influenced by their distinct objectives. This observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors of PaSCo note that SSC and PSC metrics are not directly correlated, a conclusion further supported and validated by our findings. Additionally, since PaSCo is LiDAR-based, our method provides the insight that this phenomenon appears to extend across distinct data modalities.

A more nuanced interpretation of the performance comparison is that CGFormer dedicates its capacity to optimizing SSC, while DBScan’s exhaustive deterministic clustering facilitates instance registration. In contrast, IPFormer addresses PSC in an end-to-end fashion, by dedicating its capacity to both the SSC and PSC objectives. While this leads to performance results that appear less favorable for SSC, IPFormer surpasses CGFormer+DBScan in primary panoptic metrics PQ$^\dagger$ and PQ-All, performs on par in individual metrics, and achieves a significant runtime reduction of over 14$\times$.

W3: Clarification of contributions and distinctions from CGFormer and PaSCo in the methodology.

We are thankful for the careful reading of our manuscript and acknowledge the opportunity to clarify the distinctions between our contributions and prior works.

Regarding the voxel visibility and voxel proposals in Section 3.3, we agree that voxel visibility could be cited more precisely, as it is a well-established process in existing works on voxel-based scene understanding, including CGFormer, OccFormer, Symphonies, and VoxFormer. In contrast, voxel proposals are explicitly stated in line 191 of our manuscript to align with the works referenced therein.

For Section 3.4, we note in line 197 that the 3D Local and Global Encoder is based on CGFormer. This is the only component of the IPFormer encoder based on prior works, which is why no additional references are mentioned in this section. In the context of Section 3.5, we indicate in line 224 that the panoptic aggregation is inspired by PaSCo. However, our implementation extends this approach to operate on dense voxels without incorporating uncertainty calculations, which represents a key difference from PaSCo.

We are pleased to provide these clarifications and will incorporate the details into the revised version of our manuscript to improve clarity and attribution.

Q1: Clarification of the definition and role of $S$ in relation to $\mathbf{F}_{3D}$ and point coordinates.

We appreaciate your effort to point out the confusion regarding the definition of $S$ in line 151 and are pleased to clarify this aspect of our methodology.

We confirm that the point coordinates $(x, y, z)$ are determined by the pixel coordinates $(u, v)$ and the depth bin $d$, independent of the feature $\mathbf{F}_{\text{3D}}$. The correct notation is therefore given by $S =${$(u,v,d) \mid \mathcal{Q}((u,v,d)) = (x,y,z)$}.

We will revise the manuscript to ensure that the distinction between spatial quantization and feature aggregation is more explicit.

S1: Suggestion to update Fig. 2 to clarify depth estimation from RGB input.

Thanks for this suggestion. We will update Fig. 2 in the revised manuscript to include an arrow from the RGB image to the depth map, clarifying that the depth is estimated, and ensuring to properly convey that our method is RGB-based, not RGBD-based.

S2: Improvement of the description of 3D deformable cross-attention and clarification of reference points, sampled points, and the definition of $\mathbf{\Pi}$.

We thank you for pointing out this potential ambiguity and are glad to clarify.

In deformable cross-attention, "reference points" are predefined query locations in the target space ($\mathbf{V}\_\text{vis}$), and "sampled points" are specific locations in the source space ($\mathbf{F}_{\text{3D}}$) around the reference points, selected based on learned offsets to aggregate relevant information.

Furthermore, in our manuscript, $\Pi$ consistently refers to the projection function, mapping 3D query positions $\mathbf{x}$ to their corresponding 2D reference points. This projected reference point $\Pi(\mathbf{x})$ is a 2D location on the feature map $F_{\text{3D}}$, which corresponds to the $(u, v)$ coordinates in the image plane. The offsets $\Delta \mathbf{p}$ are continuous displacement vectors calculated relative to the projected reference point $\Pi(\mathbf{x})$, enabling the model to sample features flexibly from locations in the 3D feature map using trilinear interpolation, which is denoted as $\psi(\cdot)$ in Eq. 5 of the manuscript. We will revise the manuscript to clarify this and ensure consistency in the explanation of $\Pi$.

S3: In line 265, I think it was meant to say “Stuff-instances” not “Thing-instances”

Thank you for the careful reading. We agree that the term should indeed be "Stuff-instances" rather than "Thing-instances". We will correct this in the revised manuscript and appreciate the Reviewer’s attention to detail.

Closing Statement.

We deeply appreciate the Reviewer's thorough analysis and exceptional attention to technical detail, as well as the constructive suggestions, which we will thoughtfully incorporate into the revised version of our manuscript. In light of the new evidence regarding (1) comprehensive clarifications on our contributions, technical details, and performance trade-offs, (2) extensive additional ablation experiments (see Tab. 1 in the rebuttal to Reviewer [Kgxr]), and (3) additional results on zero-shot generalization (see Tab. 1 in the rebuttal to Reviewer [rTop]), we are confident that our work represents a significant advancement in 3D vision research.

We hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification.

Thank you for your response. We are pleased that most of the concerns have been addressed and appreciate the opportunity to clarify the remaining questions regarding performance.

Performance against CGFormer:

Before addressing the concerns, it is important to highlight that DBScan clustering is the currently established method for generating ground truth (GT) for PSC from GT SSC. Given the absence of established baselines for vision-based PSC, we adopt the approach proposed in LiDAR-based PaSCo, in which vision-based SSC is combined with DBScan to create reliable proxies. However, it is crucial to note that these proxies hold no practical relevance, due to their significant computational inefficiency.

We confirm that both CGFormer+DBScan and IPFormer achieve comparable state-of-the-art performance on PSC metrics, with IPFormer exhibiting suboptimal results on SSC metrics. However, labeling CGFormer as being trained for a "different task" is not entirely accurate. PSC is inherently a generalization of SSC, where SSC tasks (occupancy + semantics) form a strict subset of PSC tasks (occupancy + semantics + instance registration). In baseline methods like CGFormer+DBScan, the PSC objective is achieved by first optimizing for the SSC task, followed by instance registration through DBScan clustering. Crucially, the PSC performance of CGFormer+DBScan is directly dependent on the quality of its SSC predictions, as DBScan itself is a heuristic-based clustering algorithm with no independent learning capacity. This dependency highlights that CGFormer+DBScan’s performance on PSC cannot be attributed solely to the clustering step but is fundamentally tied to the strength of its SSC outputs.

To further contextualize this dependency, we emphasize the importance of zero-shot generalization. As presented in Tab. 1 of the rebuttal to Reviewer [rTop], our method significantly outperforms CGFormer+DBScan in both PSC and SSC metrics on out-of-domain data, with the exception of a marginal difference of 0.01% in $PQ$-Thing and $RQ$-Thing. Importantly, metrics initially reported as weaker on in-domain data, such as $SQ$-Thing and $SQ$-Stuff, demonstrate substantial improvements with our method when evaluated on out-of-domain data. Specifically, $SQ$-Thing improves from 20.06 for CGFormer+DBScan to 22.76 for our method, while $SQ$-Stuff improves from 16.19 to 25.89. This trend is also reflected in SSC metrics, where our approach outperforms CGFormer+DBScan on out-of-domain data, despite being outperformed on in-domain data. Moreover, analyzing the relative generalization gap, as also presented in Tab. 1, further underscores these findings. Across all PSC and SSC metrics, our method exhibits superior relative generalization, with the most substantial improvements observed in $SQ$-Thing, $SQ$-Stuff, $IoU$, and $mIoU$ metrics.

These results highlight a critical limitation of the baselines, such as CGFormer+DBScan: their reliance on SSC predictions and heuristic clustering limit their ability to generalize to in-the-wild data. In contrast, our method not only improves efficiency by reducing inference time by over 14$\times$, but also achieves superior zero-shot generalization, surpassing CGFormer+DBScan in both absolute and relative PSC and SSC metrics on out-of-domain data. By providing a unified, end-to-end solution, our approach addresses the inherent limitations of heuristic-based PSC methods, delivering both competitive in-domain performance and superior generalization in real-world scenarios.

Tradeoff between PSC and SSC:

We acknowledge that our earlier phrasing may have unintentionally caused confusion regarding the findings in PaSCo, and we apologize for any misunderstandings. To clarify, while CGFormer+DBSCAN and our method exhibit comparable performance on PSC metrics, there is a notable discrepancy in SSC performance, with our method performing suboptimal. This outcome is unexpected, as similar PSC performance might intuitively suggest comparable SSC performance. However, this discrepancy generally aligns with the conclusion in PaSCo, which states that PSC and SSC metrics are not directly correlated. Furthermore, our zero-shot generalization results (Tab. 1 in the rebuttal to Reviewer [rTop]) qualitatively align with the findings in PaSCo. Specifically, while our method demonstrates significant improvements in PSC performance compared to CGFormer+DBSCAN, the gains in SSC performance remain marginal. These findings reinforce the notion that SC and SSC objectives are not inherently linked and further support the conclusions drawn in PaSCo.

We will incorporate these clarifications into the revised manuscript, as your feedback is valuable to improve the quality of our work. Please let us know if you have any additional questions; we would be pleased to provide further clarification. Thank you for your time and effort.

Thank you for pointing out the additional table with further results. These findings on generalization ability certainly do help highlight the strengths of the proposed approach over CGFormer+DBSCAN. Please do include these results in the main paper, as without them, the statement that "PSC performance of CGFormer+DBSCAN is directly dependent on the quality of SSC predictions" works against the proposed method rather than for it (my major concern was exactly that a method trained for SSC and post-processed could perform as well as a method explicitly trained for PSC). As this has helped address my major concern, I have chosen to raise my score to borderline accept.

With that being said, the stated relationship between PSC and SSC by the authors is still contradictory. An argument is still trying to be made that PSC and SSC performance are not directly correlated, yet it has been stated that "PSC is inherently a generalization of SSC" and the proposed PSC method is first trained with SSC objectives. Moreover, the results themself are mixed and inconclusive as to what is the exact relationship between PSC and SSC performance. I suggest dropping this conclusion/finding from the paper as it seems forced just as an attempt to align with a previous work, and also just leads to more confusion in regards to task and motivation (e.g., why are they not correlated in some manner if PSC is a generalization of SSC? why is the proposed method trained with SSC objectives if SSC isn't relevant to PSC performance?).

We sincerely appreciate your insightful feedback and constructive suggestions. We address the concerns thoroughly.

W1: Struggles with rare classes.

We thank you for your detailed observation. As shown in Tab. 6 of our manuscript, this issue affects all methods and arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4.53% of all voxels.

We kindly direct you to Sec. W2 of the rebuttal to Reviewer [z9JQ], where we elaborate on countermeasures.

W2 & Q4: High memory usage limits accessibility & scalability to higher voxel grid resolutions.

We value your concern regarding the high memory usage and its implications on increased voxel grid resolution. We agree that this can be a limitation.

System tools report $52.8$ GB memory usage due to CUDA overhead and preallocation, while detailed profiling shows the model itself uses $35$ GB. Unlike IPFormer, baselines like OccFormer do not perform PSC directly and require post-hoc clustering, adding extra computation. Thus, comparing GPU memory across SSC and PSC methods is not an entirely equitable assessment.

Noteably, reliance on advanced hardware such as A100 GPUs has become increasingly common in recent 3D vision research. For instance, FB-OCC (CVPR 2023) [1] and VGGT (CVPR 2025) [2] were trained on 32 and 64 A100s, respectively. In comparison, IPFormer operates on a significantly lower computational budget, positioning it on the more efficient end of the spectrum while still achieving competitive performance.

W3: Dependency on 2D models, error propagation to 3D, and generalization beyond KITTI.

We appreciate raising this concern. MobileStereoNet3D undergoes robust training to enhance generalizability. The model is pre-trained on the large synthetic SceneFlow dataset [3] (approx. 40,000 samples). Subsequently, the model is fine-tuned on two real-world datasets: DrivingStereo [4] (approx. 181,000 samples) and KITTI Scene Flow 2015 (KITTI2015) [5] (~400 samples). Note that SemanticKITTI is distinct from KITTI2015.

Additionally, the official MobileStereoNet GitHub repository highlights strong zero-shot cross-dataset generalization capabilities. Moreover, the significant difference in the number of samples among the training datasets argues against overfitting.

Finally, to further mitigate potential 2D-to-3D error propagation, we incorporate a depth refinement network, which enhances the quality of depth predictions before they are used for 3D scene completion.

Q1: Robustness to varying lighting, weather, heavy occlusion, and zero-shot generalization.

We appreciate the Reviewer’s insightful questions regarding the performance for certain scenarios and its zero-shot generalization ability.

1. Zero-shot Generalization: We cross-validate IPFormer on the SSCBench-KITTI-360 dataset [6], which is distinct from the SemanticKITTI dataset it has been trained on. Results in Tab. 1 show that IPFormer demonstrates superior zero-shot generalization capability. Especially in comparison to its closest competitor, CGFormer+DBScan, which it outperforms across all PSC and SSC metrics.

2. Weather/Lighting Variations: While KITTI-360 is not intentionally diverse, its larger sample count (12,764 images) introduces more scene variations. As per the NeurIPS Program Chairs’ rebuttal guidelines, we cannot submit a global rebuttal or an additional PDF with figures to visually support our findings. However, this quantitative results on zero-shot generalization demonstrate robustness under varying conditions.

3. Visibility Sampling: The visualizations in the manuscript and the supplementary video provide compelling evidence of IPFormer’s ability to handle heavily occluded objects. For instance, starting at 4:30 min. in the video, distant occluded cars with minimal visual cues are precisely recognized and segmented, especially when compared to the baselines.

Table 1: Zero-shot generalization for CGFormer+DBScan and IPFormer, by training on SemanticKITTI and cross-validating on SSCBench-KITTI-360.

Q2: Rationale for dual-head training and its impact on cross-task synergy.

To adequately address these questions, we are pleased to provide an extensive set of additional ablation experiments, which are presented in Tab. 1 in the rebuttal to Reviewer [Kgxr]. Because the ablation results are relevant to other Reviewers as well, we kindly direct you to this rebuttal, where we discuss the ablation setup in detail.

Across the design space, methods that emphasize SSC (e, f, k) achieve strong semantic scores but degrade PSC performance, especially PQ-Thing. Conversely, approaches prioritizing PSC (j, n, q) boost PQ-Thing or PQ-Stuff at the cost of SSC quality or overall balance. Joint or single-head variants (a, b, p, q) further struggle with instance registration or overall consistency. In contrast, our dual-head, two-stage method (r) effectively decouples objectives, yielding the best PQ-All and strong performance across PQ-Thing and PQ-Stuff, with only a minor SSC trade-off.

The interplay between PSC and SSC tasks primarily driven by their inherent objectives rather than being significantly influenced by stage-wise training, though slight deviations exist. Notably, this observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with the findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors note that SSC and PSC metrics are not directly correlated, a conclusion further reinforced by the results of our experiments. Moreover, since PaSCo is LiDAR-based, our method offers the insight that this phenomenon appears to extend across distinct data modalities.

Q3: Progress on PanoSSC’s dataset and evaluation on its relaxed IoU threshold.

Despite our efforts to gain access to this dataset, it remains unavailable. Consequently, we have not been able to train IPFormer on this post-processed dataset or evaluate it under PanoSSC’s relaxed $20$% IoU matching threshold. We remain committed to presenting representative evaluation results should access to the dataset become possible in the future.

However, in Tab. 2, we provide evaluation results of IPFormer under PanoSSC’s relaxed $20$% IoU threshold. Expectedly, RQ metrics increase substantially, since more instances are recognized, while these are segmented with less fidelity. Thus, SQ metrics decrease.

Table 2: Ablation on IPFormer’s PSC and SSC performance under PanoSSC’s relaxed $20$% IoU matching threshold on the SemanticKITTI dataset.

Closing Statement.

We sincerely appreciate your effort and thoughtful suggestions, which we will carefully incorporate into the revised version of our manuscript. In light of the new evidence regarding (1) comprehensive clarifications on the robustness to heavy occlusion, performance on rare classes, and dependency on 2D models, (2) extensive additional ablation experiments on cross-task-synergy, stage-wise training, and joint optimization, as well as (3) additional results on zero-shot generalization, we are confident that our contributions represent a key advancement in 3D vision research.

We hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating. Should there be any remaining questions, we would be happy to provide further clarification.

[1] Li et al., FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation, CVPR 2023 3D Occupancy Prediction Challenge

[2] Wang et al., VGGT: Visual Geometry Grounded Transformer, CVPR 2025

[3] Mayer et al., SceneFlow: A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation, CVPR 2016

[4] Yang et al., DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios, CVPR2019

[5] Menze et al., Object Scene Flow for Autonomous Vehicles, CVPR 2015

[6] Li et al., SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views, ICLR 2024

Thank you for the rebuttal. It has satisfactorily addressed most of my concerns, and I am leaning toward a positive evaluation of the paper. The authors are encouraged to release the code of this paper as it involves many steps to reimplement the proposed method.

We sincerely thank thank you for the insightful comments and constructive suggestions. We provide detailed responses to all the questions raised and aim to address the concerns adequately.

W1-1: Clarify differences between PQ and PQ$^{\dagger}$.

We appreciate the effort in pointing this out.

The standard PQ metric requires a predicted segment to match a ground-truth segment with an $**IoU** >0.5$. However, this strict threshold can be overly conservative for Stuff classes, which typically lack well-defined boundaries. The metric PQ$^\dagger$ relaxes the matching criterion specifically for Stuff classes.

Formally, PQ$^\dagger$ retains the same formulation as PQ:

$

\text{PQ}^\dagger = \frac{\sum_{(p,g) \in \text{TP}^\dagger} \text{IoU}(p,g)}{|\text{TP}^\dagger| + \frac{1}{2}|\text{FP}^\dagger| + \frac{1}{2}|\text{FN}^\dagger|} ,

$

but relaxes the matching condition used to define true positives ($\text{TP}^\dagger$), and thus false positives ($\text{FP}^\dagger$) and false negatives ($\text{FN}^\dagger$). Specifically,

• for Thing-classes, predicted and ground-truth segment pairs $(p,g)$ are matched if $\text{IoU}(p,g) > 0.5$, identical to the original PQ definition;
• for Stuff-classes, matches are accepted if $\text{IoU}(p,g) > 0$, thereby allowing any overlapping prediction to contribute to the metric.

This relaxation acknowledges the inherent ambiguity in delineating stuff regions and reduces penalties for minor misalignments.

W1-2: Unclear GPU memory and training time of IPFormer.

We apologize for any misunderstanding and are pleased to provide a summary of Tab. 7 and lines 521–522 of our manuscript:

IPFormer requires $52.80$ GB of GPU memory during training and approximately $3.5$ days to train each of its two stages. Note that we further elaborate on this topic in Sec. W2 of the rebuttal to Reviewer [rTop.]

W2 & Q2-2: More ablation experiments regarding SSC performance and hyperparameters.

We are thankful for this suggestion and are glad to provide an extensive set of additional ablation experiments in Tab. 1 of the rebuttal to Reviewer [Kgxr]. We kindly direct you to this rebuttal, as further ablation experiments are also of interest to other Reviewers.

Since our primary objective is to address the PSC task, we mainly focus on analyzing the influence of Stage 2 hyperparameters in the context of both the SSC and PSC objectives. To this end, we primarily train the SSC task in Stage 1 using established hyperparameters for the cross-entropy, Semantic-SCAL, and Geometric-SCAL cost functions. All weights $\lambda$ associated with these functions are set to $1$, consistent with state-of-the-art and established SSC works, such as CGFormer (NeurIPS 2024 Spotlight), OccFormer (ICCV 2023), and MonoScene (CVPR 2022). Nevertheless, we investigate the effect of removing the Depth loss, as its impact has not been extensively studied. The corresponding weight $\lambda_{\text{depth}}$ was mistakenly reported as $1$ in our manuscript, but it is actually $0.0001$. We apologize for this and will revise it.

Furthermore, we analyze the effect of varying hyperparameters of the Sigmoid Focal Loss [2], specifically designed to down-weight easier examples and focus the training process on harder-to-classify examples:

$

\text{FL}(p) = -\alpha_t (1 - p_t)^{\gamma} \log(p_t) ,

$

where $p_t$ is predicted probability for the true class after applying the sigmoid function, $\alpha_t$ is the class-balancing weight, and $\gamma$ controls the down-weighting of well-classified examples.

Reducing the weights of second-stage SSC losses improves SSC performance but leads to a significant drop in PSC metrics, particularly PQ-Thing. Conversely, increasing the Sigmoid Focal Loss weight enhances PSC performance, especially PQ-Thing, but negatively impacts PQ-Stuff and overall SSC metrics. Methods that balance these weights achieve more consistent PSC results but may sacrifice SSC performance.

Finally, our proposed IPFormer configuration, method (r), achieves a balanced PSC performance by obtaining the best score for PQ-All and the second-best results for both PQ-Thing and PQ-Stuff, despite a reduction in SSC performance. This observation, where SSC performance may be suboptimal despite balanced PSC performance, aligns with the findings reported in PaSCo (CVPR 2024), the method that introduced the PSC task. The authors of PaSCo note that SSC and PSC metrics are not directly correlated, and the findings of our ablation experiments further support and validate this result. Moreover, since PaSCo is LiDAR-based, our method offers the insight that this phenomenon appears to extend across distinct data modalities.

We hope this analysis addresses your concerns by demonstrating that our method effectively balances SSC and PSC objectives. The observed trade-offs do not undermine the method's effectiveness or superiority but rather highlight its flexibility in addressing the distinct challenges. If further clarification is needed, we are happy to provide additional details.

W3-1: Quantitative PSC performance on the SemanticKITTI test set.

Thanks for the opportunity to clarify on this topic. Since the SemanticKITTI test set is hidden to ensure fair evaluation, prevent overfitting, and maintain leaderboard integrity, instance labels cannot be derived for panoptic scene completion (lines 502-503 in the manuscript). Therefore, it is not possible for us or other researchers to evaluate PSC performance on the test set.

W3-2: Lack of description on why Thing-classes showcase suboptimal performance.

We appreciate the detailed observation regarding the suboptimal performance on Thing classes. As shown in Tab. 6 of our manuscript, this issue affects all methods and arises from the significant class imbalance in the SemanticKITTI dataset, where Thing classes make up only 4.53% of all voxels. To address this, we employ the Sigmoid Focal Loss (see Sec. W2), which down-weights easier examples and focuses on harder-to-classify ones, particularly rare Thing classes. We kindly direct the Reviewer to Tab. 1 of the rebuttal to Reviewer [Kgxr], where extensive additional ablation results demonstrate its effectiveness. Our approach balances performance between Thing and Stuff classes, achieving state-of-the-art results by prioritizing equitable performance across both categories rather than solely optimizing for Thing classes.

We hope this explanation clarifies the concern and encourage asking for further clarification if needed.

Q1: Enlarge text in figures.

Thank you for the suggestion. We will enlarge the text in the figures to improve readability in the revised version of our manuscript.

Q2-1: Insufficient experiment details.

We appreciate the Reviewer's concern regarding the experimental details. We provide experimental details in the main manuscript and the technical appendix. This includes

• Datasets: SemanticKITTI and PaSCo ground-truth, voxel resolution, splits, and DBScan clustering parameters.
• Metrics: PQ, SQ, RQ definitions and formulations, as well as category-specific evaluations.
• Model and subnets: Overall architecture and structure of individual sub-networks.
• Training setup: Hyperparameters, batch size, number of epochs, optimizer, learning rate schedule, and runtime analysis.
• Hardware requirements: For GPU and CPU

Moreover, we are pleased to provide the following details:

• Software Framework: PyTorch with an fp32 backend.
• Baseline Comparisons: We utilize pre-trained checkpoints from the official publicly available implementations.

We will add these details to the revised manuscript. If there are specific details that the Reviewer feels are unclear, we kindly ask for clarification so that we can address them.

Q2-3: More visualizations should be included for a comprehensive comparison.

We appreciate the Reviewer's suggestion to include more visual comparisons for a comprehensive evaluation. Unfortunately, as per the rebuttal guidelines outlined in the email from the NeurIPS Program Chairs (dated July 27), it is not possible to provide a global rebuttal, including a single-page PDF containing additional figures. We sincerely apologize for this limitation.

However, we kindly direct you to the video provided in the supplementary material of the initial submission, which contains extensive visualizations (starting from 3:52 min.), including more than $75$ camera frames and their corresponding predictions (incl. the full panoptic scene, and instances-only) for all baselines and IPFormer, as well as the ground-truths. Furthermore, starting from 5:36 min., we provide an additional $20$ camera frames and corresponding predictions of IPFormer.

We value your suggestion and will gladly incorporate additional visual comparisons in the revised version of our manuscript.

Closing Statement.

We sincerely appreciate your effort and constructive suggestions, which we will thoughtfully incorporate into the revised version of our manuscript. In light of the new evidence regarding (1) comprehensive clarifications on experimental details, evaluation metrics, and performance evaluations, (2) extensive additional ablation experiments on two-stage training, the dual-head architecture, and sensitivity to hyperparameters, as well as (3) additional results on zero-shot generalization (see Tab. 1 in the rebuttal to Reviewer [rTop]), we are confident that our contributions represent a key advancement in 3D vision research.

We hope these efforts are acknowledged by the Reviewer and encourage an upgrade of the rating, as you have mentioned in the review. Should there be any remaining questions, we would be happy to provide further clarification.

[1] Porzi et al., Seamless Scene Segmentation, CVPR 2019

[2] Lin et al., Focal Loss for Dense Object Detection, ICCV 2017