SegMASt3R: Geometry Grounded Segment Matching

Limited ablation:

The method uses a learnable dustbin to handle unmatched segments. It would strengthen the paper to include an ablation study evaluating its contribution.

The table below shows the effect of removing the dustbin parameter from the SegMASt3R pipeline. The performance sharply degrades across all metrics without the dustbin parameter.

Ablation Study on Learnable Dustbin

MobileSAM Mask Evaluation

According to Supplementary Table 2, SegMASt3R is evaluated only using ground-truth masks, which are often unavailable during inference. It would be beneficial to also report results using masks from MobileSAM.

During inference, we can rely on segment generators like MobileSAM. In Reviewer jith's rebuttal, we show results on evaluation using noisy masks as obtained from FastSAM in the "Performance under noisy masks" section. Additionally, in our robotic navigation task (L253-290) we use masks generated from FastSAM on the HM3D dataset (Table 4). This clearly shows that our method is not reliant on ground truth masks. We will mention this more explicitly in the paper.

Were Pre-trained Checkpoints Used

It is unclear whether the baseline methods are re-trained or evaluated using their publicly released checkpoints. This is especially critical in wide-baseline settings, where fine-tuning or data augmentation strategies may significantly affect performance.

Yes, we use pre-trained checkpoints. This is indeed a missing detail on our part and we will add the pre-trained checkpoints used for evaluation in the paper. For DINOv2, we use the DINOv2 ViT-G pre-trained checkpoint. For CroCo, we use the ViT-L pre-trained checkpoint provided in their repo. For other models (SAM2, MASt3R, etc.), we use the official checkpoints available in their respective repos.

Typos

In Line 111, the channel dimension of V1, V2 should be 768, instead of 24.

Thank you for pointing this out. We will fix the typo.

Novelty

it is a straightforward adaptation of MASt3R with an added head comprising an MLP and Sinkhorn-based matching.

While some individual components used in our method are known within the community, we demonstrate that even these simple elements when integrated algorithmically can achieve outstanding results across a range of vision tasks, including wide-baseline segment matching, robotic navigation and instance segmentation. Crucially, we integrate these components in a novel and non-trivial configuration, which is key to the performance gains observed. Our approach also serves as an effective exploration of segment-based representations, which, despite the simplicity of our methodology, is shown to enhance performance in multiple robotic-vision settings showcasing the usefulness of segments as a useful intermediate-level scene representation. To our knowledge, this still remains under-explored. Furthermore, the introduction of a segment matching task and wide-baseline evaluation adds an additional layer of novelty, pushing the boundaries of current approaches in this domain.

3D Instance Segmentation

Training requires ground truth 3D instance segmentation, which is not always easy to obtain:

Our method relies more on 2D instance masks than 3D. The benefit with 3D instance masks is they are more boundary aware. 2D instance masks can be easily generated to both train the network as well as use the network to evaluate on novel scenes like shown in the quantitative MapFree results of jith's rebuttal. Our method, even when trained only on indoor datasets generalizes better than compared to baselines on the outdoor MapFree.

Noisy Masks

An additional limitation not covered by the paper is how robust the method is to noisy segmentation masks.

In reviewer jith’s rebuttal we show results for the MapFree dataset in the Generalization / Outdoor Dataset section which covers both outdoors and noisy masks as the masks generated are via SAM2. Additionally, we show results on noisy masks as generated by FastSAM in jith's in the Performance under noisy masks section. The table shows that the performance patterns are still similar to that observed in Table 1 with GT masks.

Dependence of performance on the number of segments:

the paper mentions using 100 segments per image to be 'sufficient to cover the entire image while keeping computation affordable, but this leaves space for a discussion on how performance would change adding more segments.

Our method is not reliant on a particular number of segments. The M (set to 100) parameter in L117 is an upper bound on the segments in the scene. It allows for more efficient batch processing. For the missing masks, we just pad them with 0s. We will add this important detail to the paper (in Line 117). Additionally, during inference time, the number of masks can be set arbitrarily regardless of the set M during training. We presume beyond a certain point having really small segments would indeed degrade performance, but most segmenters (like SAM2, FastSAM) yield less than a 100 segments. Additionally, in the main paper, and in the rebuttal, we have explored several results with different segmentation algorithms. FastSAM is used in the robotic navigation experiment (Table 4 on the HM3D dataset), SAM2 AMG (Automatic Mask Generator) results are shown for the Performance under noisy masks section for reviewer jith. The various segmenters used also naturally cover a varying number of segments, showing that indeed our methods are robust to this quantity.

We thank Reviewer vtn9 for taking the time to review our paper and responding to our rebuttal. We’re glad we were able to address their concerns.

Typos: We thank the reviewer for pointing out the typos.

Cross-view Encoding Transformer

Line 99: “Cross-view encoding Transformer”. Encoding transformer or decoder?

Thank you for pointing this out we will make the required change. It is indeed the decoder.

Description of Dustbin

$\alpha$ Definition. Description of $\alpha$ in L152 is confusing:

The learnable logit $\alpha$ is a threshold below which any score from the score matrix will be classified as not-matched [34]. It is useful in situations when there are no common masks between an image pair, as it then maps such masks to the dustbin. We will clarify this in the paper.

Resolution and Averaging of Features

Lines 199–200: “A segment descriptor is the masked average of its feature map at a suitable resolution” - what is the suitable resolution? “Average of its feature map” - how is the average conducted?

This is a valid point since we haven't explicitly documented the formulation in our paper. For DINOv2 specifically, we work with patch level features at 1/14th the image resolution $(H/14, W/14, 1536)$. To assign segment level features, we choose an intermediate resolution (1/3rd image resolution, decided empirically), and upsample the patch-level features to $(H/3, W/3, D)$ via bilinear interpolation. Binary segemnt masks are downsampled to this same resolution $(H/3, W/3, M)$ via nearest-neighbour interpolation. Finally to obtain our mask level features, we perform a mask-average pooling operation, implemented as a mat-mul: $(M, H/3 \times W/3) \times (H/3 \times W/3, D) = (M, D)$ to obtain a single D-dimensional vector per segment.

Local Feature Matching for Evaluation

Table 1: Local Feature Matching (LFM). Are the results from local visual descriptor matching or from segment matching?

In this paper we believe we are the first to introduce wide-baseline segment matching. To develop a suitable baseline of methods, amongst others we chose to add LFM based baselines as they have been repurposed as segment matchers (for example in [15,31]) and there is a straightforward methodology to turn them into segment matchers. We will rectify this in the baselines subsection to clarify our motivation behind using them in the main paper.

DUSt3R evaluation and fine-tuning MASt3R

Line 287: The ablation with CroCo encoder cannot demonstrate the effectiveness of the “explicit formulation of image matching”. To demonstrate it, it is better to use the DUSt3R backbone as it is not trained for the matching task. Have you conducted ablations on fine-tuning the cross-attention decoder in MASt3R jointly with the segment-feature head? Could this further enhance segment matching?

In Table 5 in the main paper and the Impact of the Feature Encoder (L281-L290) section, we were interested in performing an experiment to understand whether knowledge of the image pair and the subsequent cross-attention that features from both these images undergo is indeed a consequential part of these backbones for segment matching. As mentioned in the section, even though CroCo was not trained for the matching task and it was trained on much-less data than DINOv2, it still performs better than DINOv2 confirming our intuition. We explicitly did not choose DUSt3R because it is trained on a matching task and we imagined it would perform similar to MASt3R. We add the DUSt3R backbone comparison here for completion. As can be clearly seen SegMASt3R is much better than DUSt3R with the same training scheme as SegMASt3R on ScanNet++. Additionally, we observe that fine-tuning the MASt3R decoder layers does not lead to further improvements in performance. Here we report by fine-tuning a total of 4 decoder layers (2 in each decoder).

Generalization / Outdoor Dataset

The evaluation on the outdoor MapFree dataset is limited to qualitative results, without quantitative metrics or comparisons.

Here's a quantitative comparison on the outdoor MapFree (MF) dataset using SAM2 masks. SegMASt3R (MF) is the model which we have now trained on the MapFree dataset (in response to Reviewer jith, mKs5 comments). SegMASt3R (SPP) is the same model as that used in the original manuscript. SegMASt3R (SPP, Dustbin MF) differs from the previous only in the choice of dustbin parameter $\alpha$ which we borrowed from the SegMASt3R (MF) model to demonstrate that the indoor-to-outdoor generalization gap can be further reduced easily (e.g., this single parameter could simply be grid-searched on a small calibration set as an alternative to full training on MapFree).

Ground Truth: Most outdoor real-world datasets, including MapFree, don’t have ground truth instance masks, which is why we hadn't evaluated it quantitatively. The above evaluation uses a pseudo ground truth, obtained from SAM2 video propagation on the image sequence of a given scene. This works well in general because of the narrow baseline in adjacent video frames. We manually verified this ground truth on randomly selected scenes. All compared methods use the same pseudo ground truth.

Dataset/Training: The official train split of the MapFree dataset comprises 460 unique scenes with 0.5 million images in total. To train SegMASt3R, we uniformly sampled 50 scenes to obtain 31K image pairs. Since the MapFree test split ground truth is not available publicly, we report results on their official val split using 7.8k image pairs from 13 scenes sampled uniformly from 65 total val scenes. Similar to our ScanNet++ evaluation, we use pose binning to present results across narrow and wide baselines.

Performance under noisy masks

How does SegMASt3R perform when using imperfect or noisy segment masks?

Our method is robust to noisy masks, e.g., those obtained from SAM2/FastSAM. This is evident from the use of SAM2 masks in the above results on the MapFree dataset, and FastSAM masks in the navigation results on the HM3D dataset (Table 4). The following table presents results on the ScanNet dataset using FastSAM, which shows that the performance patterns are still similar to that observed in Table 1 with GT masks:

Novelty

The core components are all adapted from prior work

Please refer to our response to Reviewer mKs5, titled Novelty.

Thanks very much, jith, for your response.

Novelty

In addition to our response in the “Novelty” section in the rebuttal, we emphasize the novelty of our paper in terms of the new knowledge/insights that have never been reported in the prior art. We have established, for the first time, that “feature distinctiveness at the pixel level does not necessarily translate to the instance level” (L225). We show that a highly accurate local feature matching method (MASt3R) falls short at segment/instance matching when used off-the-shelf (Table 1). Furthermore, our ablation studies (Table 5 and 2nd table in mks5’s rebuttal) show a surprising trend inversion in how different leading backbones behave: DINOv2 (a 2D foundation model) is better than MASt3R (a geometric foundation model) for off-the-shelf segment matching (SegMatch block in Table 1), but (Seg)MASt3R is better than DINOv2 when trained for the task of segment matching (Table 5), which reinforces our claim that “geometric context is essential” (L286) for the segment matching task. Finally, we would like to re-emphasize the simplicity of our proposed pipeline (which in itself never existed before) as a rather positive aspect of this work that is able to deliver remarkable results for the underexplored segment matching task as well as the downstream tasks of mapping and navigation.

Backbone difference

Additionally, we clarify that our segment-feature head is separate from MASt3R's local feature matching head (Line 113-115) and that we only initialize with local feature head’s weights. Since we keep the decoder frozen (Line 108), we retain MASt3R’s original local feature matching head, enabling simultaneous matching capability at both point and segment level through a single forward pass. Further, we have demonstrated that a separate segment-matching head is necessary since the MASt3R’s original local features are not amenable to segment-level aggregation/description, as observed in Table 1 (MASt3R in the SegMatch block).

Noisy Masks on ScanNet++:

In the Table below, we now present results on ScanNet++ using FastSAM masks for both the reference and the query image, which can be directly compared with Table 1. It can be observed that SegMASt3R achieves superior results across the board by a large margin, whereas even on the narrow baseline setting our framework is on par with the best of the methods being compared.

(continued in next comment to adjust for character limit)

Thanks for your response, jith.

Updated Evaluation with Noisy Segments

the numbers are halved … Does it indicate that the proposed method is sensitive to noisy masks?

No, it is not the presence of noisy masks per se that affects performance, but their misalignment with GT masks during evaluation. In other words, the lower numbers are due to an evaluation artefact than due to the matching performance of the proposed method. Our method consistently outperforms all baselines regardless of the quality of masks produced by the upstream Automatic Mask Generator (AMG - FastSAM here) used in the evaluation. Below, we explain our main evaluation strategy in more detail and also provide results through an additional mode of evaluation that decouples this misalignment from the models' matching performance.

In our previous evaluation setup, the lower numbers compared to Table 1 arise because they explicitly incorporate both “noisy segmentation” and “matching performance conditioned on those noisy segments”. AMG predicted (noisy) masks often do not perfectly align with GT masks (i.e., IoU < 1). To evaluate matching conditioned on these predicted segments, we employ an IoU threshold ($\theta$) that discards predicted matches corresponding to segments with IoU < $\theta$. In our primary evaluation, we used $\theta$ = 0.5, which reflects a moderate overlap. However, to perform a comparison inline with Table 1 we evaluate against the full set of ground-truth correspondences, including those corresponding to predicted segments that were discarded due to not meeting the IoU threshold ($\theta$). This amounts to applying a penalty based on the misalignment between predicted and ground-truth segments in respective reference and query images.

To isolate the effect of matching performance from this misalignment (i.e., the penalty in our previous evaluation setup), we now report results where low-IoU segments are excluded from evaluation after predicting matches - this ensures that the matcher still sees noisy AMG inputs. Specifically, now during evaluation, we filter the ground-truth correspondences to only include entries for which at least one pair of predicted segments was retained after IoU thresholding. In the table below, titled “Evaluation Without Penalty”, it can be observed that our method maintains a clear lead in matching segments despite variations in segmentation noise. For comparison, we also report results for the “Evaluation With Penalty” scenario, which extends the results we shared in our previous response.

Evaluation Without Penalty

Evaluation With Penalty

DinoV2 and MASt3R Feature Resolution

Q1. Why downsample the masks for the DINOv2 and MASt3R baselines, rather than interpolating their features to match the mask resolution?

We only downsample the masks for DINOv2, not for MASt3R (we will rectify this in L202). For DINOv2, the following table compares different feature/mask resolutions. It can be observed that performance does not change significantly. We had originally tested this on a subset and chose a lower resolution ([112, 171] - one-third of the original resolution) for efficiency.

Number of Masks, Non-GT Masks For Training, Running Time for Mask Generation At Inference

Q2. The masks extraction is a crucial step in creating high quality segments. ScanNet++ and Replica dataset provide masks, however, in cases where masks are not provided annotating 100 masks (line 123) for every input image seems unpractical. When ground truth masks are not available, how do the authors propose to obtain it? What is the expected running time for this step?

We respond to this in three parts: a) the choice of M=100, b) the need for ground truth (GT) masks during training, and c) expected runtime of non-GT masks during inference.

Number of Masks

a) Number of Masks / M=100 masks: This is simply an upper bound for convenience in batch processing. Most training/test pairs do not have 100 masks, regardless of using GT or SAM/FastSAM. The actual number of masks per image is between 20 to 30 when training using GT. For the missing masks, we just pad them with 0s. We will add this important detail to the paper (in Line 117). Additionally, during inference time, the number of masks can be set arbitrarily regardless of the set M during training.

Non-GT Masks For Training

b) Need for GT masks during training: It is indeed possible to train SegMASt3R with weak supervision based on mask annotations obtained from SAM2 video propagator. To show this, we have now trained a model on the MapFree dataset, which does not have GT instance masks but has image sequences per scene. We use this with SAM2's video propagator (which works well due to narrow baseline between adjacent frames) to generate segment correspondences for different image pairs within the image sequence. This pseudo ground truth is used for both training and evaluation. For results on MapFree which involves noisy (non-GT masks) please refer to the Generalization / Outdoor Datasets section of reviewer jith's.

Running Time for Mask Generation At Inference

c) Non-GT masks during inference: When GT masks are unavailable, we can use SAM /FastSAM to obtain them. FastSAM works in real-time, so mask extraction should not be a bottleneck either in our training or inference and have real-time applicability.

Gaussian Grouping

Q3. Have the authors considered comparing to "Gaussian Grouping: Segment and Edit Anything in 3D Scenes"?

We thank the reviewer for pointing out Gaussian Grouping. We will cite the paper as it is related to this research, but can’t do a fair comparison, as our method is based on only pairwise image based segment matching. Gaussian Grouping on the other hand uses Gaussian splatting which requires multiple overlapping views of a scene to generate accurate segmentations.

Statistical Error Bars

Q4: Statistical Error Bars: Question 7 checklist: I don’t think it is correct – there is no statistical error bars in any of the results (at least not mentioned) also in 3.4.1 the time is not with standard deviation.

This was an oversight on our part. We meant to write no statistical error bars.

Generalization to outdoor datasets

All quantitative results are reported exclusively on indoor datasets.

Please refer to the Generalization / Outdoor Dataset section in reviewer jith's rebuttal.

comparison with DINOv2 and MAStr3 methods

This is covered above in Q1.

Novelty

The proposed approach mainly combines existing components without introducing substantial methodological advances.

While some individual components used in our method are known within the community, we demonstrate that even these simple elements when integrated algorithmically can achieve outstanding results across a range of vision tasks, including wide-baseline segment matching, robotic navigation and instance segmentation. Crucially, we integrate these components in a novel and non-trivial configuration, which is key to the performance gains observed. Our approach also serves as an effective exploration of segment-based representations, which, despite the simplicity of our methodology, is shown to enhance performance in multiple robotic-vision settings showcasing the usefulness of segments as a useful intermediate-level scene representation. To our knowledge, this still remains under-explored. Furthermore, the introduction of a segment matching task and wide-baseline evaluation adds an additional layer of novelty, pushing the boundaries of current approaches in this domain.

Abstract

Inaccuracy in the abstract

We thank the reviewer for recognizing that we are introducing a new task (wide-baseline segment matching) and its evaluation methodology. To develop reasonable baselines, we chose to use SAM2 video propagator which is not inherently trained for this task, because it is the industry and academia standard for segment matching across multiple frames and is also trained on a large amount of data (more than MASt3R). We chose LFM methods because they have been repurposed as segment matchers (for example in [15,31]) and there is a straightforward methodology to turn them into segment matchers. It was our oversight that we only mentioned the baselines but did not explain their motivations, and will add these motivations to the main paper.

Learnable Logit Interpretation and Learnability

The necessity for the learnable logit in the learnable dustbin is unclear.

The logit is basically a threshold below which any score from the score matrix will be classified as not-matched. It is useful in situations when there are masks not common to both images and allows for such masks to be mapped to the dustbin. While we could also have let it have been a tuneable hyper-parameter, we chose to make it learnable to allow for a more natural discovery of its appropriate value. We will clarify this in the paper.

Limitations

The paper overall limitation is that it requires a 3D ground truth annotation, which are quite rare. Furthermore, as a direct result of that, they cannot generalize well to other scenes and datasets. Those issues were addressed by the authors in the Sup. Material.

We have now presented results on the MapFree dataset where we use SAM2 video propagator instead of ground truth annotations for training/evaluation. While this strategy may not apply for all types of datasets, it shows that our method does not strictly depend on ground truth annotations.

We note that our earlier response provided details addressing the reviewer’s concerns on novelty and the requirement of 3D annotation. If Reviewer mKs5 has any remaining specific points, we are happy to clarify further.