SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment

We thank reviewer Cqtb for recognizing our problem setting as well as the original and elegant contribution of our unified query design. Our detailed response to the feedback is as follows.

[W1,Q1] Stronger motivation for the Unified Query Decoder. Direct ablation study against a baseline that runs a state-of-the-art 2D segmentation model like Mask2Former and lifts the results to 3D.

[Motivation]

Thanks for raising this concern. Our Unified Query decoder is built upon the contributions of 2D segmentation giants such as Mask2Former and SAM. However, different from previous 2D methods that can only handle single-image inputs or specific segmentation tasks, our Unified Query Decoder enables our model to accommodate multi-view inputs while ensuring multi-view mask consistency, as well as multi-task understanding including instance, semantic, panoptic, and text-referred segmentation.

We note that these special designs essentially serve the target of this work: building a pixel-aligned representation-based simultaneous understanding and 3D reconstruction network, where the core is how to leverage pixel-aligned representation to combine multi-view masks and 3D Gaussians to achieve 3D-level understanding within a unified model.

Taking this into consideration, the motivation of Unified Query Decoder is twofold: (1) Leverage unified queries shared across views to ensure multi-view mask consistency, which in turn ensures 3D-level segmentation accuracy. (2) Leverage unified queries shared across tasks to enable multi-task segmentation, which extends our approach to versatile 3D understanding applications. We promise we will include the above discussions in our revision.

[Ablation]

Thanks for the constructive suggestion. We implement two models: (1) the proposed method trained with only Gaussian Decoder (Reconstruction Only), (2) Mask2Former trained with multi-view masks (Segmentation Only). Both models are re-trained under the same settings as ours. Following the suggestion, we directly lift 2D masks from Mask2Former to the 3D Gaussians. We refer to this approach as "Mask2Former-GS". The comparison results are shown in the table below. We can see that both efficiency and performance of Mask2Former-GS degrade compared to the original SIU3R model.

As for efficiency, the degradation of inference speed is caused by the extra computation from the additional image encoder of Mask2Former, highlighting the efficiency advantage of our unified model design.

As for performance, we attribute the degradation of both reconstruction quality and segmentation accuracy to the lack of encoder feature sharing between Gaussian and mask decoders, indicating an interesting potential correlation between the reconstruction and segmentation tasks, which we leave to future work for further exploration.

These observations further demonstrate our "idea of using unified queries to handle diverse 3D understanding tasks within a single feed-forward pass is also an original and elegant contribution" as mentioned by the reviewer.

[W2,Q2] The smooth heuristic of continuity loss is likely to be ineffective for objects with complex or curved geometries. Controlled study where the continuity loss is applied to a vanilla 3D Gaussian Splatting framework using ground-truth semantic masks?

[Effectiveness]

Thanks for raising this concern. In our experiments, we only apply the mask-guided continuity loss to 3x3 neighbored pixels. We find such small-range regularization rarely causes over-smoothness in reconstructed 3D Gaussians, even for objects with complex or curved geometries. We have also prepared some qualitative results for demonstration, and found the visualized depths are decent for objects with complex or curved geometries. However, due to the updated rebuttal policy, we cannot provide the images here. We promise we will include the results in our paper and code repository in revision.

[Controlled Study]

Thanks for the constructive suggestion. We optimize vanilla 3D Gaussian splatting fields with and without mask-guided depth continuity loss for each scene in our validation set, and compare their depth and render quality in the table below. We can see that, with such intra-instance depth continuity constraint, the vanilla 3DGS also exhibits better depth and render quality, further demonstrate the effectiveness of our mask-guided depth continuity loss for per-scene optimization methods, validating the universality of our insight that "understanding helps reconstruction".

[W3] Algorithm 1 does not explain how the model reaches a consensus on a semantic label for a 3D Gaussian when the input views provide conflicting information. Definition of the segmentation loss L_mask.

[Algorithm 1]

Algorithm 1 primarily illustrates, at a pseudocode level, how to utilize multi-view 3D Gaussians to lift multi-view mask logits from 2D to 3D and achieve 3D-level multi-view aggregation based on rasterization. All operations correspond to PyTorch code and can be reproduced using the code in our supplementary materials.

To explain how this process reaches a consensus on mask label for a 3D Gaussian when the input views provide conflicting information, we will include the following description in our revision.

Specifically, before the "multi-view mask aggregation step" in Algorithm 1, different input views may exhibit conflicting information when decoding multi-view masks, which causes masks of the same object having different IDs across views. To address this, we reuse the predicted geometry-related 3D Gaussian parameters (i.e., mean, opacity, scale, rotation) to lift multi-view mask information (class-wise query probability maps) to the 3D space. Through the alpha-blending mechanism of rasterization, pixels from different views but corresponding to the same object are explicitly forced to perform weighted averaging at the 3D level, eliminating implicit 2D-level semantic inconsistencies before aggregation. Leveraging the aggregated mask information for mask decoding, we can further obtain multi-view consistent masks for 3D-level segmentation.

[L_mask]

The definition of the segmentation loss L_mask is as follows: $L_{mask}=\lambda_{bce} L_{bce} + \lambda_{dice} L_{dice} + \lambda_{ce} L_{ce}$. Specifically, $L_{bce}$ denotes binary cross-entropy loss that measures pixel-wise differences between the predicted foreground probability and ground truth. $L_{dice}$ denotes dice loss that measures the overlap ratio between the predicted foreground and ground truth. $L_{cls}$ denotes multi-class cross-entropy loss that measures differences between the predicted category probability and ground truth for each mask. In practice, we follow Mask2Former and set $\lambda_{bce},\lambda_{dice},\lambda_{cls}=5.0,5.0,2.0$. We will include this in our revision.

[W4] More controlled experiments on the R$\to$U and U$\to$R modules. They are not well-isolated from the rest of the proposed architecture.

Thanks for the constructive suggestion. We acknowledge that our R$\to$U and U$\to$R modules can indeed be used as plugins and integrated into other network architectures. Specifically, the R$\to$U module is a training-free module that can work as long as multi-view 3D Gaussians and mask logits are provided. The U$\to$R module is a model-free mask-guided loss function that can work as long as predicted / ground-truth multi-view masks are provided.

To verify their effectivenesses under different network architecture, we implement a variant model without other components of our method (i.e., Unified Query Decoder), and directly combine this Gaussian-only network with Mask2Former ("Mask2Former-GS") as mentioned in [W1,Q1]. We conduct ablation studies of R$\to$U and U$\to$R modules for this variant and show the results in the table below. We can see that both of the two modules show similar improvements in reconstruction and understanding just as they do in the original architecture.

[Q3] Could you please provide more details on the implementation of Algorithm 1? Please refer to [W3] for details.

[Q4] Could you please explicitly define the L_mask loss function? Please refer to [W3] for details.

We thank reviewer qLPk for recognizing our novel alignment-free framework, in-depth exploration of mutual benefits, and good experimental results. Our detailed response to the feedback is as follows.

For your information, in addition to addressing the concerns mentioned in this review, we have provided more experimental results (e.g., extend SIU3R to support more input views as suggested by reviewer oN8Y, oS3w) that may be of your interest.

[W1] Would it be possible to train a generalized network that, similar to DUSt3R, possesses strong generalization abilities across different scenarios?
Due to the limited time of rebuttal phase, we are unable to collect and process a large amount of datasets comparable in scale to DUSt3R (9 datasets) with both multi-view images and mask annotations. Therefore, we process two representative indoor multi-view datasets with different sensor characteristics (i.e., ScanNet++, Replica) as additional training / evaluation data. Specifically, we conducted two experiments, i.e. zero-shot generalization on ScanNet++ & Replica, and train a general model by treating ScanNet++ & Replica as additional training data. The detailed results are shown in table below.

It can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.

To further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.

Besides, we collect some multi-view images in real-world scenarios using hand-held cellphones to further validate our generalization capability. We observe that our zero-shot performance in real-world scenes is surprisingly good in both reconstruction and segmentation, which is qualitatively on par with that on ScanNet dataset. Due to the updated rebuttal policy this year, we cannot provide the qualitative results here. We promise we will include the results in our paper and code repository in revision.

[W2] Compare with other feed-forward methods like DUSt3R, MAST3R, and VGGT.
Thanks for the constructive suggestion.

Since DUSt3R, MAST3R, and VGGT are only designed to reconstruct scenes as 3D points rather than 3D Gaussians / masks, we note that they cannot be evaluated for the tasks of novel view synthesis and multi-task understanding. The comparison results are shown in the table below.

As for reconstruction performance, although DUSt3R, MAST3R and VGGT used much more training data (DUSt3R: 9 datasets, MASt3R: 14 datasets, VGGT: 17 datasets) and ground-truth depths for explicit geometry supervision, our method is only slightly inferior to them in terms of depth accuracy. We think such performance gap is reasonable because our method learns reconstruction solely through the supervision of novel view synthesis (NVS) task and was trained only on a single dataset. Besides, it is worth noting that our method exhibits the best depth accuracy among all the methods that support the novel view synthesis task (NVS), which focuses more on the rendering quality rather than merely on the geometry quality.

As for computational efficiency, our method outperforms DUSt3R, MAST3R and VGGT by a clear margin. We attribute this to the adoption of complex point post-processing for DUSt3R and MASt3R, and the significantly larger model size for VGGT (ours: 650M, VGGT: 1.1B).

We will include the results in our revision.

[Q1] How does SIU3R perform on other diverse datasets and real-world scenarios to demonstrate its true generalization capabilities? Please refer to [W1] for details.

[Q2] How does SIU3R compare with DUSt3R, MAST3R, and VGGT in terms of reconstruction quality and computational efficiency? Please refer to [W2] for details.

We thank reviewer oS3w for recognizing our lightweight and intuitive designs, as well as our significance in training efficiency compared to LSM. Our detailed response to the feedback is as follows.

[W1] Performance on datasets with ​​different sensor characteristics remains unverified
To fully verify our performance on other datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.

It can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.

To further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.

[W2, Q3] How the architecture would scale computationally and effectively with more views. Modifications for efficient multi-view scaling?
Thanks for raising this concern.

Inspired by concurrent works on feed-forward multi-view reconstruction (e.g., VGGT), we make minor modifications to our model and make it support more input views. Specifically, compared to the model with two views, where cross-attention is only performed between the tokens of the two views, for multiple views, we perform cross-attention between the tokens of each view and the concatenated tokens of all other views. This is followed by our Gaussian decoder and Unified Query Decoder to predict multi-view 3D Gaussians and masks, respectively. We re-trained 4 variant models with different number of input views (i.e., 2, 4, 6, 8 views). The quantitative results are shown in the table below. We can see that, as No. Views increases, our model achieves better performance in not only reconstruction but also 3D-aware segmentation on novel views. The main reason is that, with more view information available to reduce geometry uncertainty, the quality of 3D reconstruction improves, which, thanks to the design of our R$\to$U module, in turn enhances the segmentation performance.

We have also prepared some qualitative results of handling 8 views and larger scene scales. However, due to the updated rebuttal policy, we cannot provide here. We promise we will include the results in our paper and code repository in revision.

As for efficient multi-view scaling, we believe this remains an open question within the field. However, from an engineering perspective, we can utilize FlashAttention and efficient optimizer tools (e.g., DeepSpeed ZeRO-3) to enable efficient training.

We also note that, at the time of our initial submission, most prior feed-forward reconstruction methods (e.g., pixelSplat, MVSplat, NoPoSplat, LSM) primarily utilized two-view images as input. We adopted the same setting as these prior works to maintain consistency. Additionally, we acknowledge that recent feed-forward multi-view reconstruction methods (e.g., VGGT) can be regarded as our concurrent works.

[W3] (1) Multi-task training without both R$\to$U and U$\to$R, and (2) independent training for each task.
Thanks for the constructive suggestion.

(1) We implement a variant model w/o R$\to$U and U$\to$R, and show the comparison results in the table below. We can see that adding either module alone outperforms this variant model, demonstrating the effectiveness of the our R$\to$U and U$\to$R module designs.

We will include the results in our revision.

(2) We implement two variant models, one with only photometric and depth continuity losses ("Recon. Only"), and another with only mask and text-matching losses ("Segm. Only"). For the "Recon. Only" variant, we ablate our U$\to$R module (i.e., mask-guided depth continuity loss) on it to validate its effectiveness on independent reconstruction task, where we use ground-truth masks as the depth guidance due to the absence of predicted masks. Note that we cannot add our R$\to$U module to "Segm. Only" variant without the reconstructed 3D Gaussians, thus we only implement "Segm. Only w/o R$\to$U" in the experiments. The results are shown in the table below.

We can see that, for reconstruction-only task, our U$\to$R module still significantly improves depth accuracy and novel view rendering quality, demonstrating its effectiveness for independent reconstruction task. We can also see that, without reconstructed 3D Gaussians to help lifting multi-view masks to 3D, we cannot achieve 3D-aware segmentation on novel views. Moreover, without our R$\to$U module that can leverage 3D Gaussians to improve multi-view mask consistency, the segmentation performance would also degrade significantly. These outcomes support the insight of this paper to unify understanding and reconstruction tasks into a single framework. We will include the results in our revision.

[W4] Narrow down the claim and research scope.
Thanks for the constructive suggestion. Indeed, this paper primarily focuses on multi-task 3D segmentation, which falls under the narrow scope of fine-grained 3D understanding tasks and does not include broader tasks such as 3D captioning and 3D question answering.
To address this, we will narrow down our claim by replacing all occurrences of "3D understanding" with "multi-task 3D segmentation" in our revision.

Note that our initial claim on 3D understanding is inherited from previous works (e.g., OpenGaussian[1], FMGS[2]) that also focus on reconstruction-based segmentation, which is inappropriate given the scope of 3D understanding is broader than before.

[1] OpenGaussian: Towards Point-Level 3D Gaussian based Open Vocabulary Understanding, NeurIPS'24

[2] FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding, IJCV'25

[Q1] What if more than two instances of the same category appear in the scene?
Yes. In our experiments, we observe that, if the text refers to a specific instance (e.g., "Chair nearest to the door"), no matter how many other instances of the same category appear in the scene, the segmentation result corresponding to the query with the highest matching similarity can effectively locate the instance. If the text refers to no specific instance but a category (e.g., "Chair"), the queries with topK matching similarities would find every single instance of this category, which is functionally similar to instance segmentation. We have prepared some qualitative results for verification. However, due to the updated rebuttal policy, we cannot provide here. We promise we will include them in paper and code repository in revision.

[Q2] Primary factors for training efficiency compared to LSM.
The main reason for our training efficiency compared to LSM is that we do not require high-dimensional feature rasterization to align 3D Gaussian features with 2D features. We find that such feature-alignment paradigm requires much more training time to achieve convergence than our alignment-free method as shown in the table below.

We thank reviewer oN8Y for recognizing our strong motivation, novelty, multi-task unification and evaluation. Our detailed response to the feedback is as follows.

[W1] Limited to two views.
Thanks for raising this concern.

Building upon the insights of concurrent works on feed-forward multi-view reconstruction (e.g., VGGT), we make minor modifications to our model to support more input views for simultaneous understanding and 3D reconstruction. Specifically, compared to the model with two views, where cross-attention is only performed between the tokens of the two views, for multiple views, we perform cross-attention between the tokens of each view and the concatenated tokens of all other views. This is then followed by our Gaussian decoder and Unified Query Decoder, which predict multi-view 3D Gaussians and masks, respectively. We re-trained four variant models with different number of input views (i.e., 2, 4, 6, 8 views). The quantitative results, shown in the table below, demonstrate that as the number of views increases, our model exhibits improved performance in not only reconstruction but also 3D-aware segmentation on novel views. We think this improvement is primarily due to the availability of additional view information, which reduces geometry uncertainty, and enhances the quality of 3D reconstruction. This, in turn, boosts 3D scene understanding performance, thanks to our mutual-benefit mechanism (i.e., R$\rightarrow$U). We will include the quantitative results in our revision.

Besides, we have also prepared some qualitative results to demonstrate our capability of handling 8 views and larger scene scales. However, due to the updated rebuttal policy, we cannot provide the results here. We promise we will include the results in our paper and code repository in revision.

It is also worth noting that most prior feed-forward reconstruction and understanding methods (e.g., pixelSplat, MVSplat, NoPoSplat, LSM) primarily utilized two-view images as input at the time of this paper's submission. To maintain consistency, we adopted the same setting as these previous works in our initial submission. Additionally, we acknowledge that recent feed-forward multi-view reconstruction methods (e.g., VGGT) can be regarded as our concurrent works.

[W2] Single-dataset training/evaluation.
Thanks for raising this insightful concern.

To fully verify our performance on more datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.

It can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.

To further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.

[W3] How does the model handle the case that the number of instances in a scene is much larger than the number of queries?
As desribed in Table I of our appendix, we set number of queries to 100, which denotes that the maximum number of potential objects in a scene is up to 100. Empirically, we observe that this setting is good enough to handle most cases in ScanNet, where the average number of valid objects is less than 10 for two-view inputs.

However, as the scene scales increase, the number of objects actually present in the scene may exceed 100. In consideration of this, we increase the number of queries from 100 to 200 to accommodate scenes with more objects, and present the comparison results in the table below. Note that we conduct this experiment just for verifying that our method can accommodate more objects by increasing the number of queries. Since the average number of objects (< 10) in scenes is far fewer than 200, the redundancy in queries would incur significant query-to-mask matching ambiguity during training, which leads to slight performance degradation for segmentation tasks.

[W4] Finetune other baselines on ScanNet for fair comparison.
As described at P7-L218, "all baseline methods are evaluated on ScanNet dataset under the same protocols as ours for fair comparison". To be more specific, all of the baseline methods were re-trained on the same data as ours from scratch following the training protocols of their official implementations. We think such an experimental setup can ensure that our comparison is fair.

For clarity, we will highlight this in implementation details in our revision.

[Q1] Which weights were or weren't updated in "Early Aggregate w/ train"? Why reconstruction quality of "Early Aggregate w/ train" was worse than "Early Aggregate w/o train"?
(1) All the weights were learnable for "Early Aggregate w/ train" variant during training. Specifically, for this variant, we train SIU3R model from scratch and perform feature rasterization to aggregate multi-view feature information before mask decoding. By doing this, the model could learn to adjust Gaussian parameters for better multi-view aggregation.

(2) As described above, all the parameters of "Early Aggregate w/ train" variant could be updated after training, which inevitably causes its performance inconsistency with "Early Aggregate w/o train". As for the minor performance degradation, the main reason is that the predicted Gaussian parameters are used for not only reconstruction purpose but also feature aggregation purpose during training, which can lead to suboptimal reconstruction quality. This also highlights the motivation of our training-free R$\rightarrow$U module.

We will include the above details and discussions in our revision.

[Q2] What is the black part in the middle of the semantic map in Fig5(b)?
Since the segmentation masks were originally annotated on 3D meshes rather than 2D images for ScanNet dataset, the rendered ground-truth 2D masks may occasionally have some unannotated pixels (e.g., the black part in Fig5) due to the errors of 3D mesh reconstruction.

[Q3] How did you choose the loss weights?
We set the loss weights this way because it keeps the values of each loss at approximately the same magnitude. Empirically, we find that this configuration is optimal in most cases.

[Q4] Can you provide some tests on in-the-wild image pairs?
We collected some in-the-wild images using hand-held cellphones to further evaluate our generalization capability. Remarkably, our model achieves surprisingly good zero-shot performance in real-world scenes for both reconstruction and segmentation, qualitatively matching its performance on ScanNet dataset. However, image/video results are not available during the rebuttal phase due to the updated rebuttal policy. We promise we will include the results in our paper and code repository in revision.

Hi Reviewer oN8Y,

As mentioned, you plan to improve the score, but it is currently still at borderline reject. Do you intend to increase the rating or keep the current one?

In addition, it would be helpful to specify how to revise the motivation, backbone design, and experimental evaluation to make the submission stronger.

Bests,
AC

We thank reviewer bcN2 for acknowledging our contribution in component designs (i.e., unified queries, R$\rightarrow$U, U$\rightarrow$R modules) and performance gains (i.e., surpass strong baselines across all tasks). Our detailed response to the feedback is as follows.

[W1] Generalization to other datasets besides ScanNet.
Thanks for raising this insightful concern.

To fully verify our performance on other datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.

It can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.

To further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.

[W2] Quantitative evaluation for language grounding.
We note that we have included such quantitative evaluation in Table 1 (measured by mIoU_t) of our initial manuscript. The definition of mIoU_t can be found at P7-L241. For clarity, we will highlight in the caption that mIoU_t is the metric for language in our revision. Here we attach the results in the table below, where we can see that our SIU3R method outperforms strong baselines such as LSM and LSeg in both context-view (2D-only) and novel-view (3D-aware) language grounding tasks. Besides, the ablation study further demonstrates the effectiveness of our R$\rightarrow$U and U$\rightarrow$R modules in language grounding tasks.

[Q1] Baseline implementation details.
As described at P7-L218, "all baseline methods are evaluated on ScanNet dataset under the same protocols as ours for fair comparison".

To be more specific, for pixelSplat, MVSplat, NoPoSplat, Mask2Former, and LSM, we re-train their models following the training protocols of their official implementations using the processed ScanNet dataset same as ours. For reconstruction-only methods (i.e., pixelSplat, MVSplat, NoPoSplat), we only use their original rendering losses for supervision. For understanding-only method (i.e., Mask2Former), we only use its mask losses for supervision. For LSeg, since it already possesses general vision-language understanding capabilities, we directly adopt its pre-trained weights for evaluation following previous feature-alignment-based 3D understanding methods such as LSM and LangSplat [1]. In our evaluation, all baseline methods are evaluated on the same validation set as ours. For clarity, we will add these details in our revision.

[1] LangSplat: 3D Language Gaussian Splatting, CVPR 2024

[Q2] Ablations on initialization and pre-training for fair comparison with LSM.
Thanks for the constructive suggestion. We agree that different initialization and pre-training strategies may incur different performance outcomes. By treating initialization and pre-training as two varying factors and combining them in different ways, we obtained the following four variants of SIU3R to fully investigate the effects: (1) SIU3R w/ DUSt3R initialization and w/o COCO pre-training, (2) SIU3R w/ MASt3R initialization and w/o COCO pre-training, (3) SIU3R w/ DUSt3R initialization and w/ COCO pre-training, (4) SIU3R w/ MASt3R initialization and w/ COCO pre-training (original version).

The comparison is shown in the table below. We can see that the metrics for reconstruction exhibit similar results in terms of novel view synthesis, indicating that the impact of different initializations (via either Mast3R or Dust3R) on performance is not particularly significant for NVS. Furthermore, SIU3R performs consistently better than LSM in terms of NVS for either initialization.

Since neither MASt3R nor DUSt3R has been pre-trained on any understanding tasks, directly using their weights trained solely on the reconstruction task leads to a significant decline in segmentation performance. Despite this, SIU3R still performs competitively as LSM even not pretrained on COCO.

This aligns with the knowledge in the field of vision-based understanding. For 2D detection/segmentation methods (e.g., DETR [2], Mask2Former), it is common practice to use models pre-trained on COCO as initialization for tasks in vertical domains. Similarly, for 3D-level methods like LSM, their segmentation capabilities stem from LSeg, which is also pre-trained on datasets like ADE-20K [3], comparable in scale to COCO. By considering this, we humbly think the evaluations against LSM is fair in terms of COCO pre-training.

It also demonstrates the importance of pre-training on large-scale 2D perception datasets for improving downstream 3D understanding tasks, which could potentially provide good insight for future works.

[2] End-to-End Object Detection with Transformers, ECCV 2020

[3] Scene Parsing through ADE20K Dataset, CVPR 2017

We sincerely appreciate the reviewer's feedback.

As shown in the table below, we compared the statistics between different datasets used for pre-training and fine-tuning. We can see that, COCO has a larger number of images (i.e., No. Images = 123k) but relatively low segment density (i.e., No. Segments / Image = 7.3). The ADE20K dataset, has fewer images (i.e., No. Images = 22k) but a much higher segment density (i.e., No. Segments / Image = 19.6). Besides, the ScanNet dataset used for fine-tuning has 1513 scenes and 3.4 segments per image in average.

As suggested, we replaced the pre-training dataset from COCO to ADE20K and performed fine-tuning on ScanNet, with the results shown in the table below. We can observe that pre-training on ADE20K achieves comparable performance on both reconstruction and understanding tasks compared to pre-training on COCO, and performs better than LSM which is also based on ADE20K pre-trained model. This indicates that, in terms of pre-training, the impact of ADE20K and COCO on downstream tasks is not significantly different. This also implies that our performance advantage over LSM is primarily attributed to the design of our method and the alignment-free paradigm.