Understanding while Exploring: Semantics-driven Active Mapping

We sincerely thank the reviewer for the feedback. We’re grateful for your recognition of the novelty and value of combining active exploration with semantic and geometric uncertainty, and we appreciate your acknowledgment of our strong performance across comprehensive experiments on the Replica and MP3D datasets.

Q4.1 Dependency on semantic predictions

Yes, relying on pseudo labels from a pre-trained OneFormer model can introduce errors, particularly in complex or occluded scenes. However, this reflects a more realistic setting, as ground-truth semantic masks are never available in real-world applications. ActiveSGM is explicitly designed to handle such noisy inputs. First, our use of a soft semantic representation allows the model to retain uncertainty rather than commit to potentially incorrect labels. Second, the semantic exploration criterion actively guides the agent to revisit uncertain or ambiguous regions, effectively reducing semantic noise through repeated observations. As shown in our experiments, this approach enables ActiveSGM to refine the initial pseudo labels over time, resulting in robust semantic reconstruction despite the inherent limitations of pre-trained segmentation models.

Dear Reviewer,

Thank you again for your detailed feedback and for taking the time to review our submission. We hope our rebuttal has addressed your concerns clearly. If you have any additional questions or thoughts, we would greatly appreciate the opportunity to clarify further during the discussion period.

Please let us know if there is any aspect of the paper or our response that you’d like us to elaborate on.

Best regards,

Authors

We sincerely thank the reviewer for the thoughtful and encouraging review. We truly appreciate your recognition of the novelty and clarity of our method, the thoroughness of our literature review and experiments, and the clear explanation and design of our tables and figures. We are also grateful that you found the paper enjoyable to read and acknowledged its potential impact and relevance to the research community. We respond to the suggestions and concerns as following:

Q3.1 Semantic vetor and Entropy 

Each Gaussian stores only a cropped semantic vector for memory efficiency, but entropy is computed using the full vector during rendering (see supplement Line 20, Sparse Rendering). In practice, low-probability categories contribute negligibly to the entropy since $plog(p)$ approaches zero, so the approximation remains accurate.

Q3.2 No real-world hardware experiments 

Deploying and evaluating based on physical robots is challenging due to the lack of well-annotated data, especially to get the grounth-truth semantic label for evaluation. For this reason, prior methods are evaluated only in simulation, making photorealistic environments the most practical and fair testbed for benchmarking. We will consider to extend our work into real-world application in the future.

Q3.3 lambda values 

Yes, the λ values in Equation 5 were selected through empirical tuning, but the method is not highly sensitive to these choices. As long as the two semantic loss terms are balanced in terms of numerical magnitude, the pipeline remains stable and performs well.

Q3.4 Support Open-Vocabulary Model

We consider this an exciting direction for future work. Incorporating open-vocabulary semantic features would require reliable uncertainty quantification to ensure robust integration into our framework.

We sincerely thank the reviewer for recognizing the strengths of our work. We appreciate the positive remarks on the clarity and simplicity of the method, as well as the acknowledgment of its strong empirical performance. We're especially grateful for the recognition that the approach shows potential for impact and adoption by the research community. Here, we respond to the suggestions and concerns raised by the reviewer.

Q2.1 [Quality] "Super category" for semantic representation

Thanks for the suggestion. We will explore integrating "super categories" for other classes in future work.

Q2.2 [Quality] Motivation of the exploration criterion

Our semantic exploration criterion is designed to favor observations that

(1) cover previously unseen regions (areas without existing Gaussians),

(2) reduce semantic uncertainty (entropy of the predicted class distribution), and

(3) low travel cost to encourage efficient exploration.

These components are combined to form a practical and lightweight criterion that balances coverage, semantic gain, and efficiency. A similar heuristic is used in Ref [71], Eq. 9.

Q2.3 [Quality] Next-best-view vs. entire trajectory planning

We use step-wise next-best-view planning with a shorter horizon (0.5 m) to enable more efficient and adaptive exploration. Long-horizon trajectory planning, like in He et al. (mentioned by Reviewer fEfZ), requires dense sampling and costly post-optimization. In practice, planning is cheaper than moving the robot. Our method integrates new observations on-the-fly and avoids continual learning issues in NeRF.

Q2.4 [Quality] Real-World Applicability

We account for real-world constraints by maintaining an occupancy grid with a 0.2 m collision buffer, restricting the agent to move only within free voxels. These ensure that planned viewpoints are both reachable and obstacle-free.

Q2.5 [Quality] Simulation vs. Real World

Real-world evaluation is challenging due to the lack of well-annotated RGB-D-semantic datasets and the difficulty of deploying all baselines on physical robots. For this reason, prior semantic mapping methods are also evaluated only in simulation, making photorealistic environments the most practical and fair testbed for benchmarking.

Q2.6 [Quality] Comparisons with Competing Methods

We thank the reviewer for highlighting the concern regarding comparing our method with a range of techniques addressing slightly different problems. We agree that such comparisons need to be handled carefully, and we clarify our intentions and methodology as follows:

Semantic-SLAM vs. Semantic Mapping: To the best of our knowledge, most NeRF/3DGS-based semantic mapping methods—such as NIDS-SLAM, DNS-SLAM, SNI-SLAM, and SGS-SLAM are formulated as semantic SLAM systems that jointly solve for localization and semantic mapping. In contrast, our method assumes known camera poses, focuses solely on semantic mapping. Therefore, we agree that direct performance comparisons may not be entirely fair, as SLAM inherently involves the harder task of localization. Furthermore, many SLAM methods use ground truth semantic labels, while our method uses noisy semantic predictions from a pre-trained segmentation model, adding further difficulty to our task. To account for this, we include the yellow band results in Table 1 solely for reference and completeness. We explicitly do not claim superiority over SLAM systems in this context.

Fair Comparison via Controlled Setup (Table 1 – SGS-SLAM variant):

To enable a fairer comparison with semantic SLAM methods, we select SGS-SLAM as the strongest representative baseline and reconfigure it to function as a semantic mapping-only system: (1) Disable its tracking module and use ground truth poses. (2) Replace ground truth semantics with semantic predictions. This allows us to directly evaluate the semantic representation quality, which is a key contribution of our method. We then disable our own active exploration module to ensure a passive, mapping-only setting, enabling a fair apples-to-apples comparison between semantic representations.

Active vs. Passive Mapping (Table 1 – Blue): By comparing ActiveSGM (with exploration enabled) against its passive variant and against the above baselines, we highlight our method achieves better segmentation with fewer observations/steps, demonstrating the effectiveness of our active exploration strategy guided by semantic and geometric uncertainty.

Active 3D Mapping and View Synthesis (Tables S.6, S.7): While some baselines focus on active mapping tasks without semantics (e.g., for 3D reconstruction or novel view synthesis), we ensure fair comparisons by using posed RGB-D input across methods. Our semantics are predicted from RGB data, not an extra input. This allows us to show that even with semantic reasoning added, our system remains competitive in standard active mapping tasks.

Active Semantic Mapping Baselines (Table 1 – Red): We also include comparisons to recent active semantic mapping baselines, which are more aligned with our task definition. These further validate the advantage of our sparse semantic representation and active policy.

We hope this clarifies the rationale behind our evaluation setup and demonstrates that we took significant care to ensure fair and interpretable comparisons across different problem settings. We will make these clarifications more explicit in the final version of the paper.

Q2.7 [Quality] Evaluation of Semantic Representation

Please also see supplement Table S.4, in which we also train ActiveSGM in a passive maner using the same frames of SGS-SLAM. The result shows that our sparse semantic representation maintains more accurate semantic understanding than RGB-encoding representation.

Q2.8 [Clarity] SLAM or Mapping

We will clarify the related work. Here we mainly focus on the mapping moudle of previous methods, specificlly for those using 3DGS or NeRF as mapping backbones.

Q2.9 [Clarity] SplaTAM Misstatement

Apologies for the misstatement — we use the mapping backbone of SplaTAM, not the full SLAM system. We will correct this in the revised version.

Q2.10 [Clarity] Missing related work discussion (SSMI and TARE)

These were missed due to oversight and will be included in the final version. They are used as baselines in Ref [92].

Q2.11 [Clarity] Keyframe Definitions

Please see L220-230. Local keyframes are the those have the highest overlap within all observations, as designed in SplaTAM. The global keyframes are described in L226-230, and are designed for maintaining global semantic information.

Q2.12 [Clarity] Path Planner

Due to the limited pages, we described the path planner in the supplement, L 51-58. The complete path planner is: we use the exploration criterion to decide the destination, and use the local path planner to decide the path from the current position to the destination.

Q2.13 [Clarity] Questions about 3DGS

We follow the standard training scheme for the Gaussian components as in SplaTAM (Ref [51]) and 3DGS (Ref [25]); please see Supplement L146–L150 (all training configurations are available online). Equation 1 follows the standard volumetric rendering process using alpha blending. $f_i(p)$ in Eq. 1 denotes the probability that a ray stops at sampling position $p$ (see also Eq. 3 in Ref [22]). In practice, we apply an early stopping criterion when the accumulated transmittance drops below 0.01, effectively ignoring negligible Gaussian contributions.

Q2.14 [Clarity] RGB-D Requirement

Our method requires RGB-D input, as we focus on semantic mapping with accurate geometry. If only RGB images are available, an additional depth estimation module would be needed to generate the necessary 3D structure.

Q2.15 [Clarity] Entropy-Based Filtered Mask

Apologies for the confusion. The entropy filter at L207 is applied on the OneFormer prediction, not on the rendered semantic mask.

Q2.16 [Clarity] Geometric Exploration Criterion

The unobserved regions do not contain any Gaussians, so the silhouette mask value in those pixels is 0 by definition.

Q2.17 [Clarity] F-1 / Top-1 / Top-3 Accuracy

Other baseline methods report only mIoU. To provide a more comprehensive evaluation, we select the strongest baseline (SGS-SLAM) and additionally compute F-1, mAP, Top-1, and Top-3 accuracy, as shown in Table S.4 of the supplement.

Q2.18 [Clarity] Yellow Results in Table 1

Yes, the yellow-band results in Table 1 were copied from the original publications. We include them for reference because they are among the few methods that use 3DGS or NeRF as a semantic representation.

Q2.19 [Clarity] SLAM vs. Mapping Results

For the experiments in Table 1 (blue) and Tables S.4–S.7, all baselines are run with RGB-D inputs. To ensure fairness, we disable the tracking modules in SLAM baselines and use ground-truth camera poses.

Q2.20 [Clarity] Results from Zhang et al.

Due to space constraints in the main paper, we report averaged scores from Table 2 of Zhang et al. Full per-class results are provided in Table S.5 of the supplement.

Q2.21 [Clarity] Purpose of Figure 3

The purpose of Figure 3 is to illustrate the color-coding ambiguity in RGB-encoded semantics used by SGS-SLAM. In contrast, ActiveSGM maintains a global semantic distribution, avoiding such ambiguity. To eliminate pose-related confounds, we provide ground-truth poses to SGS-SLAM in this comparison.

Q2.22 [Originality] Significance of ActiveSGM, v.s. ActiveGAMER

Please refer to our comparison between ActiveGAMER and ActiveSGM (Q1.2), where we discuss the significance of our contributions in terms of accuracy, efficiency, and semantic fidelity.

Q2.23 Other minor issues

Thanks for pointing out the issues. We promise to fix them in the final version.

Detailed comparison with ActiveGAMER

Experiement design and comparison

Intentional revisiting of uncertain semantic regions

Practical challenges

We thank the reviewer for the thoughtful review and encouraging feedback. We appreciate your recognition of our practical setup using predicted semantics, the potential of combining semantic understanding with 3DGS, and the clarity of our pipeline and exploration heuristics. We're glad the evaluation effectively highlighted our method’s capabilities. Here, we respond to the concerns and suggestionsraised by the reviewer.

Q1.1 Experiments section lacks clarity on what drives performance

We will substantially rewrite the experimental results section to better delineate concepts we inherited, such as benchmarks, metrics and baselines, and the contributions of our work.

Q1.2 Advantage over ActiveGAMER is not clearly justified

While ActiveGAMER focuses on actively acquiring geometric and photometric information for 3D reconstruction, ActiveSGM is specifically designed for semantic-aware active mapping. There are three key differences that contribute to ActiveSGM’s improved performance:

1. Semantic-Aware Exploration: ActiveSGM maintains a semantic map with sparse representation during exploration, enabling the system to reason explicitly about semantic coverage and uncertainty—something ActiveGAMER does not address.

2. Joint Semantic and Geometric Criteria: Our exploration strategy leverages both geometric completeness and semantic uncertainty to select informative viewpoints. This leads to more targeted exploration, particularly in semantically rich or ambiguous regions.

3. No Post-Refinement Required: Unlike ActiveGAMER, which relies on a post-exploration photometric refinement stage, ActiveSGM’s semantic feedback loop encourages natural revisiting of uncertain areas, making the mapping process more efficient and integrated.

Together, these design choices allow ActiveSGM to produce higher-quality reconstructions with stronger semantic fidelity, even without relying on additional refinement stages.

Q1.3 Component-wise differences to prior work are underexplained

To address this, we highlight that our three core experiments are designed to isolate and evaluate specific aspects of our semantic mapping pipeline. The bullets below refer to Table 1.

1. Replica-SLAM (yellow): We compare against NeRF/3DGS-based methods using dense RGB-encoded semantics and ground-truth labels, while our method uses only predicted semantics and still shows robustness by correcting noise over time.
2. Replica-NVS (blue): We compare fairly against SGS-SLAM by disabling its tracking module and providing GT poses, isolating mapping performance and showing that our sparse semantic representation is both more memory-efficient and more accurate than dense alternatives.
3. MP3D (red): We evaluate generalization in large-scale scenes and show that ActiveSGM, with full 3D path planning and no reliance on GT semantics, outperforms both geometry-only and semantic mapping baselines under the same RGB-D input.

These comparisons collectively demonstrate the contribution of each design choice in our pipeline. We also refer the reviewer to the response Q2.6 to Reviewer 7urF for a more comprehensive comparison.

Q1.4 He et al. on active perception is missing

We will cite this paper. Compared to their method:

• They require GT semantic masks; we use pseudo labels,which is more practical in real-world settings.
• Their planning involves long-horizon, dense sampling; ours uses adaptive step-wise planning.
• They retrain NeRF after each planning iteration; we optimize keyframes online without storing the full history.
• They report no semantic metrics, limiting direct comparison.

In contrast, ActiveSGM emphasizes semantic-guided, memory-efficient, online refinement.

Q1.5 Clarification on the Active Mapping task

Unlike passive mapping or SLAM, active mapping introduces autonomous decision-making into the mapping process. SLAM and passive mapping systems may operate on pre-recorded trajectories or under externally controlled by a human operator or predefined script. These systems passively consume data without influencing where the sensor moves or what information is acquired. In contrast, active mapping empowers the agent to plan its own trajectory in real time, actively selecting the most informative viewpoints and deciding when to stop exploration. The motivation behind active mapping is to achieve more efficient and targeted scene understanding by coupling perception with action, rather than relying on exhaustive or unguided data collection.

Q1.6 [ReplicaSLAM]: Clarification on comparison protocol

We appreciate the reviewer’s detailed questions and would like to clarify the setup and intent behind the ReplicaSLAM experiment (Yellow Band in Table 1).

Baselines: The results are directly copied from SGS-SLAM [7]. These baselines operate under passive SLAM settings, taking RGB-D inputs with ground-truth semantic labels, and evaluate performance on the observed (training) frames. They rely on manually captured sequences and do not include any active exploration components.

Our Setup: In contrast, our method operates under an active mapping setting, where the system autonomously selects which views to observe. We use posed RGB-D as input, and semantics are predicted on-the-fly using a finetuned OneFormer model. As our system plans its own trajectory, it does not observe the same frames used by the baselines—those frames are considered novel views to ActiveSGM.

Evaluation Protocol: To ensure fairness and use the same evaluation testbed, we evaluate ActiveSGM on the same frames used by the baselines, even though our system has not seen these views during exploration. This allows a direct comparison in reconstruction quality while keeping the test set consistent.

Purpose of This Experiment: This experiment is designed to highlight the effectiveness of our semantic mapping pipeline, particularly:

(1) Our sparse semantic representation achieves accurate reconstruction even without access to ground-truth semantic labels.

(2) Our semantic-guided exploration criterion helps select informative views, which improves both geometric and semantic reconstruction—despite relying on noisy pseudo labels.

We hope this clarifies that the goal of the ReplicaSLAM experiment is not to compare exploration strategies directly, but rather to showcase the strength of our semantic representation and active observation selection under a more realistic, challenging setting.

Q1.7 [ReplicaSLAM]: Clarification on "step", "training", and "data usage"

We appreciate the reviewer’s questions regarding the meaning of “steps” and “training” in our experimental setup. The term “steps” is used in two contexts:

1. For OneFormer, it refers to the number of fine-tuning iterations.

2. In SLAM and active mapping pipelines, “steps” refer to the number of observation frames used for optimizing the map representation. Typically, each observation frame results in one optimization step—thus, the number of steps is approximately equal to the number of observed frames.

The term “training” also has two meanings:

1. For OneFormer, it refers to the offline fine-tuning process before deployment.

2. For SLAM/mapping, it refers to optimizing the scene representation from observations, which is commonly called “training” in radiance field literature.

When we mention “training on 1/3 of the data,” we mean that ActiveSGM only observes and optimizes its scene representation using one-third the number of RGB-D frames, which may not necessarily correspond to the same observations used by the baselines. We will revise the text to clarify the terminology.

Q1.8 [ReplicaSLAM]: Performance of ActiveSGM Trained on All Views vs. Active Pipeline 

Due to the limitation of pages, we cannot show all the experiments in the main paper. Please also see experiments in the supplement Table S.4, in which we also train ActiveSGM in a passive maner. As clarified in Q1.6, SGS-SLAM is trained and evaluated on the same trajectory. For a fair comparison, we construct a novel trajectory and evaluate both ActiveSGM and SGS-SLAM on it. The results show that even training on all views of the SLAM trajectory yields lower performance compared to our active pipeline. This supports the core motivation of ActiveSGM — actively selecting informative trajectories leads to better reconstruction and semantic understanding than passively using all available observations.

Q1.9 [Replica (Novel Views)]: Experimental result interpretation (Sparse Semantic Representation)

Yes, the interpretation is mostly correct. ActiveSGM without exploration is the same as using a predefined trajectory and doing Semantic SLAM (w/o tracking).

As we shown in Figure 3, the color-coding ambiguity issue is inherent in RGB-encoding used by SGS-SLAM. ActiveSGM maintains a global semantic distribution rather than relying on RGB features, which avoids this problem and leads to a better semantic understanding. We will clarify this in the revised version.

Q1.10 [MP3D] Reasons for improved performance 

The improved performance comes from both the 3D Gaussian representation and the semantic exploration criterion — unlike baselines in Table 1 (blue) that use hard semantic assignments (each voxel can only have one class label), our soft semantic representation (distribution of most likely classes) combined with active view selection leads to better semantic and geometric reconstruction, as shown in Tables S.6 and S.7.

Q1.11 Differences between baselines and ActiveSGM

We refer the reviewer to Q1.2..

Q1.12 Usage of ground-truth semantic labels

We did not use ground-truth semantic labels in ActiveSGM, as our focus is on the more realistic scenario of handling noisy observations; instead, we aim to improve reconstruction performance using only imperfect pseudo labels.

Dear Reviewer,

Thank you again for your detailed feedback and for taking the time to review our submission. We hope our rebuttal has addressed your concerns clearly. If you have any additional questions or thoughts, we would greatly appreciate the opportunity to clarify further during the discussion period.

Please let us know if there is any aspect of the paper or our response that you’d like us to elaborate on.

Best regards,

Authors