Thank you for your detailed review and thoughtful comments. We’re grateful that you endorse our approach of modeling instance merging as a tracking task and acknowledge our method’s performance and efficiency.

Q1: Preliminary of online 3D segmentation.

A1: We study streaming 3D semantic segmentation where each time step provides an RGB image with its aligned LiDAR point cloud. We consider two regimes.

Offline 3D segmentation. The model receives the entire spatio-temporal sequence before inference and predicts segmentations for all frames in one pass. It can exploit global temporal context and is not bound by real-time constraints on latency or memory.

Online 3D segmentation. The model processes a stream. At each step it sees only the current image and point cloud and may carry a compact state from previous steps. It must output labels for the current frame immediately. The procedure is causal and must meet strict per frame latency and memory budgets.

Q2: Importance of merging inter‑frame instances.

A2: Thanks for the suggestion, we clarified the importance below.

• Merging inter-frame instances in LTM. Enforcing consistent object IDs across frames is essential to reconstruct a complete scene segmentation. Without ID continuity, per‑frame predictions cannot be merged into a coherent multi‑frame output.
• Merging inter-frame instance features in STM. Aggregating feature representations over recent frames enriches each object’s descriptor. Historical context improves segmentation accuracy by reinforcing correct object boundaries and reducing noise.

Q3: Section order.

A3: Thank you for pointing this out. We agree that aligning the section order with the pipeline shown in Fig.2 (i.e., the inference flow) will improve readability.

• Planned revision. In the revised manuscript, we will re‑sequence Sections 3.2–3.4 to mirror the left‑to‑right flow of Figure 2 and add explicit step labels in both the subsection titles and the figure to ensure one‑to‑one correspondence. We will also add forward references to Figure 2 at the start of each subsection.
• Rationale for the current organization. Our present ordering follows how the system was actually constructed: we first introduce the tracking backbone (LTM), then present STM, Learning‑Based Mask Integration, and Instance Consistency Mask Supervision as targeted extensions built on that tracking framework. To make this design logic transparent, we will add a short “Motivation & Roadmap” paragraph at the beginning of Sec. 3 explaining this build‑up while making clear that the exposition now follows the inference pipeline depicted in Fig. 2.

Q4: Motivation of each module.

A4: Thanks to the reviewer for asking this.

• LTM as core tracking motivation. We reformulate online segmentation as a tracking problem, using long‑term ID associations to link object proposals over time.
• STM for temporal enhancement. Tracking is inherently a temporal modeling task. STM injects recent frame features into the current frame to improve both tracking stability and segmentation quality.
• LMI and ICMS for enhanced segmentation. In online 3D segmentation, the matching process is highly dependent on each frame’s segmentation outputs, so redundant or erroneous segments can trigger a cascade of errors. To address this, we integrate Learning‑based Mask Integration (LMI) and refine single‑frame masks with Instance Consistency Mask Supervision (ICMS), which reduces over‑segmentation and delivers cleaner, less noisy 3D segmentation results to the tracker.

Q5: Difference from ESAM and main contributions.

A5: Our approach departs from ESAM in how it learns identity over time, how it manages memory, and how it fuses and consolidates segmentations. The central idea is explicit identity supervision with stable lifecycle control and learnable fusion. This design yields stronger temporal reasoning while keeping inference nearly unchanged.

Shared Components. We share only the backbone network and the decoder head with ESAM. Importantly, ESAM's backbone comes from the prior Online3D [1] work rather than being their original design.

Core idea. We recast online 3D segmentation as continuous instance tracking, where each VFM‑derived mask becomes a track query. This tracking view enables stronger temporal reasoning, robust memory management, and principled handling of fragmented masks.

Key differences and contributions.

1. Different training supervision: direct tracking (ours) versus indirect feature alignment (ESAM). We apply an explicit one-to-one matching mechanism/loss. For each object instance, this loss supervises the network to produce precise correspondences across frames. In contrast, ESAM's set-based supervision collapses each instance's features into a single centroid vector. It then encourages centroids of the same object across frames to align while repelling all others. (see Xyr3-Q4 for more discussion)
2. Different lifecycle management. Once ESAM instantiates an instance query, even if it represents a false detection, it stays in memory indefinitely. This lack of an expiration mechanism lets spurious detections persist and spread matching errors to later frames. Our method incorporates a simple yet effective lifecycle policy. Any tracked object that goes unmatched for more than T frames gets automatically removed from long-term memory. This approach stops stale or erroneous entries from accumulating, which cuts down on cascading false matches and boosts overall tracking stability in continuous real-time settings.
3. Learnable fusion for inter-frame object matching. During inter-frame object matching, ESAM relies on manually designed fusion rules to integrate appearance features with geometric information. In contrast, our approach uses a learnable fusion mechanism that enables the model to adaptively weight each source of information.
4. Novel instance-Level Temporal Information. Unlike the original ESAM, our Short-Term Memory module refines each instance's embedding through distance-aware cross-frame attention. This process injects fine-grained temporal context while suppressing background noise.
5. Novel Learnable Mask Integration Module. We introduce a learnable LMI to address within-frame fragmentation during inference. It estimates affinities among fragments, clusters and merges high-affinity groups, and then re-pools features to restore coherent instances. LMI operates only at inference with minimal overhead. During training, we leave fragments unmerged to serve as implicit augmentation.
6. Different Training Strategy. We propose Instance Consistency Mask Supervision alongside a dual-branch decoder that separates two objectives. In the selection branch, self-attention remains enabled, and a one-to-one loss preserves the optimal fragment. In the consistency branch, self-attention stays disabled, and one-to-many supervision stabilizes fragmented queries. Both branches activate only during training, so inference costs stay the same. As a result, the model gains robustness to fragmentation while keeping precise assignments.

Why this matters. Together, LTM and STM reduce drift and false matches by combining long‑horizon memory with fine‑grained temporal refinement. LMI fixes within‑frame fragmentation when it actually hurts outputs, and ICMS improves mask quality without runtime penalty. Ablations in the paper show that each component provides a meaningful, non‑redundant gain and that the improvements accumulate, leading to a substantial overall lift on online 3D segmentation.

We will integrate these clarifications and reorganize the manuscript as needed.

[1] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024

Q6: Potential to be applied to edge devices and improve efficiency.

A6: We appreciate this insightful question.

1. Potential for Edge‑Device Deployment.

   Our approach relies exclusively on widely supported operators such as attention layers and convolutions. Its end‑to‑end inference latency remains close to that of ESAM, indicating strong potential for real‑time execution on resource‑constrained edge devices.

2. Opportunities for Further Efficiency Gains.

   • Model‑architecture simplification

     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.

     FASTSAM	Backbone	Decoder	LTM	STM	LMI	Total
     18ms	64ms	5ms	5ms	3ms	4ms	99ms

   • Precision quantization

     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.

Building on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.

Thank you for your thoughtful comments and for the time you have invested in our work. We have posted our response and hope the above clarifications and the additional experiments sufficiently addressed your concerns. If any questions remain unresolved, please let us know. We would be grateful to clarify further and are happy to discuss specific points.

Thank you for your detailed review and thoughtful comments. We’re very grateful to see your recognition of our writing quality, the significance of our research, and our performance.

Q1: Runtime comparison and memory usage with ESAM or point cloud reconstruction-based methods.

A1: We apologize for any confusion caused.

1. In Table 1 of the manuscript, we have already provided a comparison of end-to-end inference runtime.

2. To offer further clarity, the table below presents a more detailed analysis of efficiency and resource requirements. We infer that the "point cloud reconstruction-based methods" mentioned by the reviewer refer to offline paradigms, such as SAI3D in our Tab.1 of the paper. Given that these are not online streaming processes, we computed their average FPS across an entire sequence for fair comparison. As illustrated in the table, our method incurs only a negligible computational cost compared to the baseline ESAM while achieving substantial performance improvements, thereby demonstrating both the efficiency and effectiveness of our approach.

   Method	AP↑	AP₅₀↑	AP₂₅↑	FPS↑	Memory	Training Time(GPU hours)
   SAI3D	28.2	47.2	67.9	0.24	8052M	training free
   ESAM-E	43.4	65.4	80.9	10.6	2932M	61h
   Ours	46.2	67.9	81.7	10.1	3016M	66h

Q2: Discuss the architectural complexity and deployment for the proposed "LTM + STM".

A2: We appreciate the reviewer's concern regarding the potential overhead introduced by our dual-memory system. To address this, we highlight the following key aspects:

1. Maintained Complexity with Negligible Overhead: Our dual-memory system seamlessly integrates long-term memory (LTM) and short-term memory (STM) into the original end-to-end network, resulting in only negligible increases in training time and inference latency relative to the baseline ESAM. This design fully preserves real-time performance, as evidenced by the detailed comparisons in the table provided in our response to Q1.

2. Ease of Deployment: Both LTM and STM rely solely on simple linear projections and scaled dot-product attention mechanisms, introducing no additional dependencies. Furthermore, since the memory allocation of LTM is statically bounded at initialization, it eliminates the need for dynamic memory allocation, thereby facilitating straightforward deployment.

3. Opportunities for Further Efficiency Gains.

   • Model‑architecture simplification

     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.

     FASTSAM	Backbone	Decoder	LTM	STM	LMI	Total
     18ms	64ms	5ms	5ms	3ms	4ms	99ms

   • Precision quantization

     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.

Building on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.

Q3: Threshold of Learning-based Mask Integration (LMI).

A3: In practice, we utilize a fixed threshold across all test scenarios, which consistently delivers robust results. To minimize false positives, we deliberately set this threshold at a conservatively high level. Consequently, minor perturbations around this value exert only negligible impacts on overall performance. To empirically substantiate these assertions, we conducted evaluations of our method using thresholds ranging from 0.5 to 0.9. As shown in the table below, the results exhibit virtually identical AP scores across this range, thereby confirming the robustness of our LMI strategy to threshold variations.

Q4: Differences between the proposed long-term memory mechanism and ESAM’s query merging.

A4: We highlight two essential differences between our method and query merging in ESAM, one during training and the other at inference.

1. Training supervision：Direct tracking versus indirect feature alignment. Per-object precise supervision. Our network learns with an explicit one-to-one matching loss. Each object instance is supervised to establish accurate correspondences across frames. Set-based coarse supervision in ESAM. Query merging compresses every instance into a single centroid vector. The loss then pulls centroids of the same object together while pushing different objects apart.

   The feature alignment through centroids in query merging of ESAM can be problematic since: a) Viewpoint and occlusion changes. Consecutive LiDAR sweeps expose different surfaces of the same object. Forcing identical centroids penalizes legitimate variations. b) Loss of spatial structure. Averaging points erases intra-instance geometry, so the network cannot tell stable regions from naturally varying ones. c) Over-regularization. Aggressive centroid overlap reduces sensitivity to fine details, which harms accuracy in cluttered scenes.

2. Inference behaviour: Lifecycle management. In ESAM, once a query is created, it remains in memory forever, even if it was a false positive. Without any expiry, these spurious entries keep matching in later frames and propagate errors. We introduce a simple but effective policy: a track that stays unmatched for more than T frames is removed from memory. By discarding stale or erroneous objects, we curb cascading false matches and deliver steadier online tracking.

Beyond the gaps above, ESAM relies on hand-crafted rules to fuse appearance and set-level features. Our model instead adopts a learnable fusion module, allowing the network itself to decide the contribution of each cue.

Q5: Is the threshold in Mask integration (LMI) robust to fragment granularity and could it learn this threshold adaptively?

A5: We thank the reviewer for this insightful question.

1. Robustness to Fragment Granularity.

   The learned mask integration (LMI) threshold is not highly sensitive to fragment granularity. We use a single fixed threshold across datasets and categories without per‑scenario tuning, and observe consistent performance. This robustness comes from the fusion rule that combines geometric proximity with feature‑similarity scores, which reduces reliance on any one threshold. We further validate our method by sweeping the threshold and reporting per-class metrics over four representative categories, ordered from fine to coarse namely picture, chair, table, bed. Picture captures small planar instances that often break into many tiny fragments, chair contains multiple thin parts and yields medium to high fragmentation, table has a large contiguous surface with fewer thin elements, and bed is a bulky object that produces coarse fragments. As shown in the table below, our method maintains strong results across different thresholds and fragment granularities, and it consistently outperforms the variant without LMI. These findings indicate that LMI is insensitive to granularity variations.

   Threshold	picture↑	chair↑	table↑	bed↑
   without LMI	43.9	63.2	52.1	45.9
   0.6	44.2	63.9	52.0	48.5
   0.8	44.4	64.1	52.5	47.5
   0.9	43.9	64.0	52.2	47.3
   0.7 (current)	44.4	63.6	52.9	48.4

2. Adaptive Threshold Learning.

   We appreciate this insightful suggestion and plan to explore two complementary strategies in the future work:

   • Statistical Separation: Fit the confidence scores of matched versus unmatched fragments to two Gaussian distributions, then choose the threshold that maximizes their separation (e.g., via the Bayesian decision boundary).

   • Network‑Predicted Thresholds: Extend the model so that, in addition to mask outputs, it predicts a per‑object threshold at inference time. We would train these thresholds end‑to‑end by evaluating merge quality, allowing each instance to adapt its own threshold based on scene context.

These future directions promise to make our mask‑fusion process even more flexible, while our current results already demonstrate strong robustness under fixed‑threshold operation. Thank the reviewers for this insightful idea!

Thank you for your detailed review and thoughtful comments. We’re delighted by your recognition of our reconceptualization of online 3D segmentation as an instance‑tracking problem and our spatial consistency learning, as well as your positive evaluation of our experimental results.

Q1: Balancing model complexity with computational efficiency for deployment and detailed running time.

A1: We apologize for any confusion caused.

1. In Tab. 1 of the manuscript, we have already provided a comparison of end-to-end inference runtime.

2. To offer further clarity, the table below presents a more detailed analysis of efficiency and resource requirements. We infer that the "point cloud reconstruction-based methods" mentioned by the reviewer refer to offline paradigms, such as SAI3D in our Tab.1 of the paper. Given that these are not online streaming processes, we computed their average FPS across an entire sequence for fair comparison. As illustrated in the table, our method incurs only a negligible computational cost compared to the baseline ESAM while achieving substantial performance improvements, thereby demonstrating both the efficiency and effectiveness of our approach.

   Method	AP	AP₅₀	AP₂₅	FPS↑	Memory	Training Time(GPU hours)
   SAI3D	28.2	47.2	67.9	0.24	8052M	training free
   ESAM-E	43.4	65.4	80.9	10.6	2932M	61h
   Ours	46.2	67.9	81.7	10.1	3016M	66h

3. Opportunities for Further Efficiency Gains.

   • Model‑architecture simplification

     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.

     FASTSAM	Backbone	Decoder	LTM	STM	LMI	Total
     18ms	64ms	5ms	5ms	3ms	4ms	99ms

   • Precision quantization

     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.

Building on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.

Q2: Effectiveness on other types of datasets.

Q2: We evaluated our model on Matterport3D [1] dataset. Experimental results show that our approach consistently outperforms ESAM. These findings attest to the strong generalization ability of our method.

[1] Matterport3D: Learning from RGB-D Data in Indoor Environments

Thanks for the authors' detailed response, which has addressed most of my concerns. I appreciate the authors' efforts in rebuttal once again, and will (for now) keep my initial score.

Thank you for your detailed review and thoughtful comments. We greatly appreciate your positive evaluation of our task objectives, performance, generalizability, and efficiency.

Q1: Improvements of individual components.

A1: Thank you for the focus on component-level contributions.

• Summary. Tab. 4 shows that our complete system raises the baseline from 41.6 AP to 46.2 AP, an absolute gain of 4.6 points. This represents roughly an 11% relative increase.

• Per‑module contributions.

  • Architectural enhancements.

    1. LTM: +2.5 AP (41.6 → 44.1), with AP₅₀ +2.9 and AP₂₅ +2.0.

    2. STM: +1.3 AP (41.6 → 42.9), with AP₅₀ +0.9 and AP₂₅ +1.3.

    3. LTM +STM: +3.2 AP (41.6 → 44.8), with AP₅₀ +3.8 and AP₂₅ +2.3.

       These numbers indicate that both memories address distinct failure modes and that their combination yields a clear net improvement on a strong baseline.

  • Instance Consistency Mask Supervision (ICMS).

    Adding ICMS on top of LTM+STM yields +0.8 AP (44.8 → 45.6) versus LTM+STM. As a supervision‑only change, ICMS does not add inference‑time computation, so this accuracy gain comes effectively at no runtime cost.

  • Learning‑Based Mask Integration (LMI).

    Adding LMI on top of LTM+STM yields +0.7 AP (44.8 → 45.5) versus LTM+STM. This module is model‑agnostic by design, and here it produces a reproducible boost without altering the backbone architecture. In fact, when integrated into ESAM, it delivers an additional +0.9 AP improvement.

• Cumulative and complementary gains of ICMS and LMI. With the baseline of 44.8 AP, adding ICMS alone raises the score by 0.8 to 45.6 AP, while adding LMI alone raises it by 0.7 to 45.5 AP. When both modules are enabled, the score climbs by 1.4 to 46.2 AP, essentially the sum of their individual gains. The almost perfect additivity shows that ICMS and LMI complement each other, where each tackles a different source of error so neither one diminishes the impact of the other. Their benefits therefore accumulate, yielding a clear and meaningful improvement over the baseline.

Q2: Improvements of temporal reasoning instead of model complexity.

A2: We thank the reviewer for raising this point, it motivated us to include the ID-switch analysis to disentangle temporal reasoning from added capacity.

1. Regarding model complexity, our approach adds only a modest number of extra parameters. As the table below illustrates, inference latency and GPU memory usage stay almost identical to the baseline. These findings confirm that our model size has not increased substantially.

   Method	AP↑	AP₅₀↑	AP₂₅↑	FPS↑	Memory	Training Time(GPU hours)
   ESAM-E	43.4	65.4	80.9	10.6	2932M	61h
   Ours	46.2	67.9	81.7	10.1	3016M	66h

2. We agree that labels like long‑term and short‑term emphasize temporal reasoning, yet accuracy gains alone do not prove stronger temporal understanding. To separate temporal effects from added capacity, we evaluate tracking consistency with ID‑switch. ID‑switch is the CLEAR MOT [1] metric that counts how often a ground‑truth trajectory matched to tracker ID A at time t becomes matched to a different tracker ID at time t+1 under optimal one‑to‑one matching, so fewer ID‑switches imply more stable cross‑frame identity. In our results, ID‑switch drops markedly, which is direct evidence that the gains come from enhanced temporal modeling rather than auxiliary complexity.

   Method	ID-Switch↓
   ESAM	1899
   Ours	1707

[1] Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics

Q3: Details of Fig. 1.

A3: We apologize for any confusion caused by Fig. 1. We will improve this illustration in the revision.

1. Temporal Learning. Figure 1 presents a comparison of ESAM and our method. In ESAM (top), the single‑frame query $Q_t$ is sent directly to the decoder and produces per‑frame predictions without any cross‑frame memory. In our pipeline (bottom row), each frame first passes through the Short‑Term Memory (STM) block before decoding, which brings in information from the previous frame.
2. Tracking and Matching. Our method also employs explicit one-to-one assignments for robust tracking. Red dotted links indicate one‑to‑one assignments to ground truth, and the small GT panel on the right collects these per‑frame pairings that provide supervision. The Long‑Term Memory (LTM) panel depicts a memory bank that persists across $t$, $t+1$, and $t+2$. Objects that are matched in the current frame are updated, as marked by the dotted “Updating” annotation, while unmatched objects are carried forward. Consistent colors denote the same object across time. Together, STM refines current queries using immediate temporal context and LTM preserves identity through time, providing explicit temporal modeling that is absent in ESAM.

Q4: Impact of each module and ablation for cross-dataset setting.

A4: Thank you for this insightful question.

1. Impact of each module. For the performance gains contributed by each module, please refer to our response to Q1.

2. Ablation for cross-dataset setting. We have already reported an overall ScanNet200→SceneNN transfer in the manuscript . The sustained performance arises from tracking that models feature‑level correspondence and identity consistency, which are transferable across datasets.

   To further demonstrate each module’s generality, we added a cross‑dataset evaluation in which models are trained on ScanNet200 and evaluated on SceneNN without any fine‑tuning. We keep the backbone, training schedule, and inference protocol unchanged to ensure a fair, apples‑to‑apples comparison. Across this ScanNet200→SceneNN transfer, each module continues to yield consistent improvements over the corresponding ablated variant. We will include the full table (with the same metrics as Tab. 3) in the revision. These results support that the gains are not dataset‑specific and that the proposed components improve robustness under distribution shift.

   Setting	LTM	STM	LMI	ICMS	AP↑	AP₅₀↑	AP₂₅↑
   ①	–	–	–	–	24.3	44.8	63.9
   ②	✓	–	–	–	27.0	49.2	68.4
   ③	–	✓	–	–	25.6	46.7	66.9
   ④	✓	✓	–	–	27.8	51.1	70.2
   ⑤	✓	✓	–	✓	28.4	52.1	70.8
   ⑥	✓	✓	✓	–	28.8	52.4	71.0
   ⑦	✓	✓	✓	✓	29.7	53.6	71.9