Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory

Thanks for your thoughtful comments, questions and insightful suggestions. We provide our detailed responses below.

1 - About Architectural Similarity with CUT3R

Our architecture seems similar to CUT3R mainly due to the use of cross-attention for the memory-image interaction, which we think is a general pipeline for information exchange. However, for streaming 3D reconstruction methods, we think the core challenge lies in how to design the memory instead of the interaction architecture. Existing methods like CUT3R rely on fully implicit feature storage and may suffer from the information loss of previous frames. To address this, we propose an explicit pointer memory that directly associates globally aligned information with its real-world locations. This dynamic explicit memory design is fundamentally different from that of CUT3R which uses a fixed-size memory.

2 - Performance Comparison with CUT3R

We acknowledge that on some specific datasets the improvement over CUT3R is not obvious. However, note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior. Our model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. This demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively. We think this training cost reduction is also valuable for promoting research in this field.

Also, our core contribution is to improve the trade-off between information preserving and efficiency for long-term reconstruction with the explicit modeling of memory tokens. We verify our advantage on this setting in the next response (3 - More Comparisons with CUT3R and VGGT).

Furthermore, we have also observed the following during our research:

• For the data-driven and end-to-end paradigm, more diverse training datasets and more extensive training generally lead to better performance.
• During training, adjusting the sampling ratio of different types of datasets for each epoch (e.g., increasing the sampling ratio of indoor data while decreasing that of outdoor data) causes the model performance to improve on some while decreasing on others.

Therefore, we think there is still room for further improvement with larger training costs.

3 - More Comparisons with CUT3R and VGGT

In the paper, we use sparsely sampled inputs to evaluate the 3D reconstruction performance to demonstrate that our explicit pointer memory (associated with 3D spatial locations) is effective and does not rely on similarity or continuity between input frames. However, we agree that your suggestion to evaluate on longer sequences is very reasonable. Therefore, we provide comparisons on longer sequences sampled from the 7-scenes and NRGBD datasets. Specifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). For each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames). As shown in the table below, the results demonstrate the superiority of our method in long-sequence reconstruction, which benefits from the explicit pointer memory that effectively stores information from past frames.

As noted by the official NeurIPS guidelines ("papers that appeared online after March 1st, 2025 will generally be considered 'contemporaneous'"), we considered VGGT as a contemporaneous work (public on arXiv on March 14th) and therefore only discussed but not compared intensely in the paper. Our model targets the streaming frame-by-frame online reconstruction setting while VGGT targets one-shot global reconstruction with all inputs, which usually causes a significant drop on the runtime efficiency. We use the following table to compare the peak runtime memory usage of our method and VGGT under different numbers of input frames.

We hope the above response can help address your concerns. We are happy to answer any additional questions you may have.

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1 - Accuracy of camera poses

Thanks for the question. We agree that global optimization methods generally produce more accurate pose estimation. However, they usually require much more time for optimization, and the motivation of memory-based streaming methods is to eliminate the dependency on global optimization to improve the runtime efficiency. Optionally, we can add a global optimization step as post-processing to obtain more accurate pose estimation when time permits.

We further compare the runtime efficiency between our method and DUSt3R (with global optimization) in the table below.

This verifies the advantage of our method in efficiency.

We will address your concerns more thoroughly using the answers given below (About the error accumulation problem and the impact of the camera orders on the reconstruction quality, Comparison with VGGT and MegaSAM).

Q1 - About training data without metric

Sorry for the confusion. As stated in Line 197-198, we only use the scale from the ground truth when the data is metric. For training data without metric, we apply their own scale normalization factors for the groundtruth and the predictions accordingly. In this no-metric setting, we expect the model to learn a geometry distribution which is invariant to scale, thereby mitigating the influence of potentially inaccurate or inconsistent annotated scales with respect to the real world.

Q2 - More details about the pose token $z$

Sorry for the confusion. For each input sequence, we use a learnable token (which is initialized within the model) to serve as the pose token $z_{0}$ of the first frame. We also initialize a set of learnable tokens to serve as a lightweight memory module specifically designed for pose retrieval. After processing each frame, we update this set of tokens (the "pose memory") using the pose token $z_{t}^{'}$ of the current frame which has participated in interaction. When adding the pose token for the next frame, we leverage its image tokens to "retrieve" an initial value for this frame’s pose token from this "pose memory". In other words, the pose tokens $z_{0}, ..., z_{t-1}, z_{t}$ of different frames are connected through this shared token set. We apologize for the confusion and will include these details in the revised version.

Q3 - About the error accumulation problem and the impact of the camera orders on the reconstruction quality

Thanks for the valuable question. Streaming methods targeting real-time incremental reconstruction inevitably suffer from the problem of error accumulation, especially in large scenes with long camera trajectories. The mitigation of this issue motivates us to store past information in the 3D space which the model can directly access when processing the current frame to reduce the influence of frame-to-frame cascading dependencies.

To verify this, the table below compares our performance with CUT3R under a long-sequence reconstruction setting. Specifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). For each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames).

This validates that explicit memory storage alleviates the error accumulation issue.

Our method is also compatible with global alignment as a post-processing step to further reduce the impact of error accumulation when necessary. However, since this would significantly compromise runtime efficiency, we still target the feedforward end-to-end streaming reconstruction without global alignment primarily.

We also evaluate the robustness of our model to the order of camera registration. We observe from the table below that our method still achieves good reconstruction performance from shuffled inputs. To be specific, for each scene in 7-scenes, we sample a sequence with a frame interval of 20 (each sequence contains 25-50 frames). For each scene in NRGBD, we sample a sequence with a frame interval of 40 (each sequence contains 20-40 frames). Then we disrupt the input sequence after the sampling to evaluate the reconstruction results.

Q4 - About the 3D position embedding

Our method is mainly designed for continuous exploration and streaming reconstruction in real-world application scenarios, and there is often a certain continuity between the streaming inputs. Therefore, we use this 3D position embedding to serve as spatial priors. However, when there is a significant shift between the current and previous frames, we expect the data-driven training to teach the model to correct the potential deviations of this spatial prior. This is supported by the reconstruction results with unordered inputs discussed in the previous response (About the error accumulation problem and the impact of the camera orders on the reconstruction quality), and also the reconstruction results shown in our paper where the input frames have low spatial overlap.

Q5 - Comparison with VGGT and MegaSAM

On the streaming setting, VGGT needs to use all the past frames and conducts the all-to-all attention to register the latest frame, which causes a significant drop in runtime efficiency. The following table compares the per-frame runtime and peak memory usage of our method and VGGT.

We do not use all the 3D points to register the new inputs. To prevent an unreasonable surge in the pointer count as the number of frames increases, we dynamically adjust the threshold used for nearest-neighbor search during the memory fusion module based on the spatial extent of the currently reconstructed 3D scene (details of this threshold computation are provided in the supplementary material). Based on our statistics across all test sequences on 7-scenes and NRGBD datasets (each sequence contains 50–100 frames), this dynamic threshold adjustment leads to a stabilization of the total number of pointers at around 1000–2000 after a certain number of frames. Consequently, the per-frame processing time also stabilizes at approximately 0.2 seconds (as shown in Figure 4 in the main paper).

Dynamic SLAM methods like MegaSAM accumulate a sufficient number of keyframes to initialize the exploring system, and often require external monocular depth priors. They can achieve more consistent depth estimation across the sequence after obtaining calibrated poses. In contrast, our method is a more fundamental end-to-end model that jointly estimates dense geometry and camera pose for each frame using only past observations, without requiring global optimization or full-sequence access. This makes it more practical and flexible for real-time applications. Importantly, our design remains compatible with global alignment techniques if needed, enabling further improvement in accuracy without sacrificing online performance.

We hope the above response can help address your concerns. We are happy to answer any additional questions you may have.

Appreciated for the feedback of the authors. For streaming 3D reconstruction applications, such as embodied robotics, it remains unclear whether Point3R demonstrates a clear advantage over MegaSAM in terms of camera pose accuracy. In particular, all supplementary experiments were conducted on fewer than 50 images, which is insufficient to convincingly demonstrate the method’s superiority in practical applications. Additionally, error accumulation cannot be effectively mitigated simply by using explicit 3D points, as they are also subject to drift.

Thank you for the author's reply. Most of my doubts have been resolved, and I maintain my original rating.

Thanks for your thoughtful comments, questions and insightful suggestions. We provide our detailed responses below.

1 - Computational Efficiency Degradation over Time

Thanks for the question. There is a trade-off between long-term information preserving and efficiency, and our core motivation is to improve this trade-off. To preserve more information, a straightforward solution is to keep all past memories instead of replacing them. However, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. This motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by fusing tokens that are close in the 3D space.

Specifically, we dynamically adjust the threshold used for nearest-neighbor search during the memory fusion module, based on the spatial extent of the currently reconstructed 3D scene. Based on our statistics across all test sequences on 7-scenes and NRGBD datasets (each sequence contains 50–100 frames), this dynamic threshold adjustment leads to a stabilization of the total number of pointers at around 1000–2000 after a certain number of frames. Consequently, the per-frame processing time also stabilizes at approximately 0.2 seconds (as shown in Figure 4 in the original paper).

Furthermore, as a promising future direction, we believe that introducing a sliding window mechanism to selectively filter the memory pointers involved in interaction could potentially further improve this efficiency. During the real-world long video captures, the newly captured regions are usually adjacent to those of the most recent frames. Therefore, only using the pointers stored from a number of recent frames may be sufficient for effective and efficient interaction.

2 - More Experimental Results for Unordered Input

Thanks for the suggestion. We think that in real-world application scenarios, online exploration and reconstruction of the current environment exhibit a certain degree of continuity and thus did not include results based on unordered inputs. Still, we agree that evaluating 3D reconstruction performance under unordered inputs is important for demonstrating the robustness of our method. According to the guidelines, we cannot include images to present the visual results (which we will add to the paper) in the rebuttal, we therefore only provide the quantitative results in the table below. Specifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 20 (each sequence contains 25-50 frames). For each scene in NRGBD, we sample a sequence with a frame interval of 40 (each sequence contains 20-40 frames).

We observe that our method still achieves good reconstruction performance after the input sequence is shuffled. This verifies the robustness of our explicit pointer memory to handle the discontinuity of the inputs.

3 - Suboptimal Performance on Certain Benchmarks

We acknowledge that on some specific datasets our method does not achieve state-of-the-art and there is room for improvement. We have observed the following during our research:

• For the data-driven and end-to-end paradigm, more diverse training datasets and more extensive training generally lead to better performance.
• During training, adjusting the sampling ratio of different types of datasets for each epoch (e.g., increasing the sampling ratio of indoor data while decreasing that of outdoor data) causes the model performance to improve on some while decreasing on others.

Also note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior. Our model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. This demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively, which still has room for further improvement with larger training costs. We think this training cost reduction is also valuable for promoting research in this field.

4 - Robustness to Poor Initialization Frames

Thanks for the suggestion. The use of the initial frame as the reference leading to possible sensitivity to poor initialization frames is a common issue faced by the "-3R" series of work. We therefore evaluate the robustness of our method on such scenarios. We select Scene Kitchen in the NRGBD dataset and obtain 37 input images by sampling at the interval of 40 frames. To simulate a poor initialization, we set the 23rd sample (corresponding to frame 920 in the original dataset, which mainly contains large areas of the low-texture wall and floor with no other objects) as the first frame and randomly shuffled the order of the remaining frames. The comparison of results between our method and CUT3R under these two input settings is shown in the table below.

We observe that our method is more robust to the effect of the initialization. Although the performance slightly degrades when the initial frame has significantly degraded content, the drop is acceptable considering the difficulty of such a scenario. We think this is because our explicit pointer memory is aligned with 3D structures of the current scene, which can minimize the impact of initialization as much as possible.

5 - Explicit vs. Implicit Spatial Memory for Challenging Cases

From the experimental comparisons provided in the previous response, it can be observed that our explicit pointer memory is more robust than implicit memory representations when faced with poor scene initialization. Still, we think our pointer memory is also flexible and scalable to a certain extent. Each pointer is directly associated with a 3D position in the scene, and this position is not fixed. Also, as mentioned in the answer to the first question (Computational Efficiency Degradation over Time), the number of memory pointers is directly related to the structural complexity of the current scene. The ability to dynamically and flexibly adjust the memory results in that our method relying less on the specific scene initialization and reaching a stabilization of the total number of memory tokens, thus offering high flexibility and scalability.

We hope the above response can help address your concerns. We are happy to answer any additional questions you may have.

Dear Authors,

Thanks for your work.

In the future, do you think an explicit 3D memory bank is necessary? It seems that scalable data, combined with an implicit memory bank that encodes spatial information, could solve the problem of long-term 3D information storage.

Why would it be necessary? Please provide your reasons. Why would it be unnecessary? Please provide your reasons. Alternatively, from what perspectives should we consider its necessity or lack thereof?

For example, Google's recent model, Genie, appears to achieve excellent geometry consistency through scalability without explicit 3D constraints. Does this imply that for a task like online 3D/4D reconstruction, an explicit 3D memory bank is also unnecessary?

If we were to argue for the necessity of an explicit 3D memory bank, could it (e.g., Point3r) be understood as a form of trade-off compression between an implicit 3D memory bank (e.g., CUT3r) and a fully explicit 3D representation (e.g., Mast3r-SLAM)?

Dear Authors,

Thank you for your contribution to streaming 3D reconstruction with the Point3R framework.

The paper contrasts Point3R with other memory-based approaches, highlighting the benefits of its explicit 3D memory. However, it is important to consider that state-of-the-art SLAM systems (e.g., a hypothetical MASt3R-SLAM, which leverages a powerful front-end) also maintain a form of explicit 3D memory. This memory typically exists as a keyframe-based pose graph with associated 3D map points, and more crucially, it explicitly ensures long-term global consistency through backend optimization (e.g., global Bundle Adjustment after a loop closure is detected).

Point3R, in contrast, employs a learned, continuous fusion mechanism to integrate new information into its unified memory on a frame-by-frame basis. While this may offer greater flexibility for handling local scene changes, minor errors can inevitably accumulate over extended trajectories.

This leads to my question:

1. Compared to a traditional SLAM system like MASt3R-SLAM, which enforces global consistency by explicitly correcting accumulated errors through non-local, global optimization, what is the fundamental merit of Point3R's learned, frame-by-frame fusion approach in ensuring long-term consistency for large-scale scene reconstruction? Could you further elaborate on its core value compared to MASt3R-SLAM by considering various aspects?

2. Besides, when a scene is large enough and a trajectory is long enough for loop closures to occur, how does Point3R's architecture address the accumulated drift that a classic SLAM system would resolve through global optimization? In what specific scenarios or tasks do you believe Point3R's "continuous fusion" paradigm would demonstrate a decisive advantage over the classic "track-map-optimize" paradigm?

3. Please provide a comparative analysis of MASt3R-SLAM and Point3R, assessing their mapping quality, tracking accuracy, and computational efficiency on short, medium, and long sequences. Your report should also include the methodology, detailing how these sequence lengths are defined and how they were generated from the source data.

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

1 - Novelty of 3D hierarchical positional embedding

The transformer architecture exhibits a lack of order among tokens, necessitating positional embedding to incorporate relational priors. Therefore, the design of a 3D positional embedding lies in the core of our method for explicit spatial modeling. However, the extension of RoPE is not straightforward and poses some challenges.

• Unlike the discrete positions used by RoPE, 3D spatial priors usually require more accurate continuous positions.
• We use a variety of training datasets that exhibit large scale variations in the input coordinates.

To address this challenge, we design our 3D hierarchical positional embedding with the following intuitions:

• We directly encode continuous 3D coordinates.
• We employ hierarchical frequency bases to make our positional embedding more flexible and adaptive to different scales. By rotating tokens under multiple frequency bases and averaging the results, we ensure that at least one frequency-aligned subspace reliably encodes the relative positional relationship, regardless of the coordinate scale.

We think the novelty of a positional embedding method lies in the underlying intuitions of the design instead of the implementation (which usually turns out to be simple yet effective).

We apologize for the confusion in Table 5. "w/o 3DHPE" means we remove this 3D hierarchical position embedding in our pointer-image interaction during the training and evaluation. To further demonstrate the effectiveness of our 3D hierarchical positional embedding, we conducted more comprehensive ablation studies. Specifically, we implemented the 3D position embedding using a single frequency base (i.e., removing the hierarchical design), retrained the model, and evaluated its 3D reconstruction performance ("w/o hierarchical freq" in the table below). As the results show, applying no embedding in the pointer-image interaction, using a single-frequency embedding, and adopting our 3D hierarchical embedding lead to progressively improved reconstruction performance.

2 - Main advantage & discussion on efficiency

Sorry for the confusion. To be more accurate, the main motivation is the efficiency to preserve more information when handling long sequences, resulting from the explicit modeling of memory tokens. CUT3R employs a fixed-size memory and replaces the previous one with the updated one (implicitly obtained by the most recent observation), which may suffer from the loss of early-frame information when processing long sequences. A straightforward solution is to keep all past memories instead of replacing them. However, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. This motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by merging tokens that are close in the 3D space.

The results below show the efficiency improvement when using our explicit memory fusion module by comparing the per-frame runtime and the peak memory usage.

Furthermore, to better valid the advantage of our explicit memory to deal with the loss of early-frame information during long-sequence reconstruction, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames) for each scene in 7-scenes, and sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames) for each scene in NRGBD to compare the long sequence reconstruction performance with CUT3R.

Also, due to the use of explicit spatial priors, our model shows higher training efficiency, i.e., our model achieves comparable performance with significantly lower training cost. Our model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. We think this training efficiency is also valuable for promoting research in this field.

3 - Memory points drifting

Thanks for this valuable question. We agree that the memory points may drift after each update since we update their positions. However, note that we fuse both the 3D coordinates and also the features. The updated memory points then represent the information of the updated position. The memory points drifting is therefore an intended phenomenon for dynamic management of the memory.

We hope the above response can help address your concerns. We are happy to answer any additional questions you may have.

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

1 - More analysis about the explicit pointer memory

Thanks for the suggestion. We store features in an explicit position to enable spatial-aware memory management to improve the trade-off between information preserving and efficiency. CUT3R employs a fixed-size memory and replaces the previous one with the updated one (implicitly obtained by the most recent observation), which may suffer from the loss of early-frame information when processing long sequences. A straightforward solution is to keep all past memories instead of replacing them. However, as the number of frames increases, such storage becomes undoubtedly redundant and inefficient. This motivates us to explore how to effectively fuse these redundant tokens, and we achieve this by merging tokens that are close in the 3D space. This requires the storage of each token in an explicit position. We then design a simple yet effective token merging strategy (nearest neighbor searching and average pooling), which can keep the total number of memory tokens within a reasonable range. The obtaining of the memory token in our model is also data-driven and flexible, the explicit position modeling and memory updating mechanism are mainly for managing the memory size while preserving the most important information.

To further support this, we compare our performance with CUT3R under a long-sequence reconstruction setting as shown in the table below.

Specifically, for each scene in 7-scenes, we sample a sequence with a frame interval of 10 (each sequence contains 50-100 frames). For each scene in NRGBD, we sample a sequence with a frame interval of 20 (each sequence contains 40-80 frames). We see that our model outperforms CUT3R in this setting.

Also note that our model achieves comparable performance with significantly lower training cost due to the explicit spatial prior. Our model was trained for 7 days on 8 GPUs (while CUT3R required around a month) and used fewer than half of the training datasets used in CUT3R. This demonstrates the advantage of our proposed explicit pointer memory to enable the model to learn and generalize more effectively.

2 - Comparison with VGGT

Thanks for the suggestion. As noted by the official NeurIPS guidelines ("papers that appeared online after March 1st, 2025 will generally be considered 'contemporaneous'"), we considered VGGT as a contemporaneous work (public on arXiv on March 14th) and therefore only discussed but did not compare intensely in the paper. Also, our model targets the streaming frame-by-frame online reconstruction setting while VGGT targets one-shot global reconstruction with all inputs, which usually achieves better performance at the cost of efficiency.

In this rebuttal, we therefore compare with StreamVGGT [1] (a streaming version of VGGT) to analyze the advantage of our method in the streaming setting and only provide the results of VGGT as a reference.

[1] Zhuo D, Zheng W, Guo J, et al. Streaming 4D Visual Geometry Transformer[J]. arXiv preprint arXiv:2507.11539, 2025.

3 - More ablation studies

Thanks for your suggestion. Table 5 in the paper compares the 3D reconstruction results with and without the 3D hierarchical position embedding ("w/o 3DHPE" means we remove this hierarchical position embedding in our pointer-image interaction during the training and evaluation). To further demonstrate the effectiveness of our 3D hierarchical position embedding, we conducted more comprehensive ablation studies for this module. Specifically, we implemented the 3D position embedding using a single frequency base (i.e., removing the hierarchical design), retrained the model, and evaluated its 3D reconstruction performance ("w/o hierarchical freq" in the table below). As the results show, applying no embedding in the pointer-image interaction, using a single-frequency embedding, and adopting our 3D hierarchical embedding lead to progressively improved reconstruction performance.

To further demonstrate the advantages of our explicit pointers, we provide additional ablations. Specifically, after encoding the current frame into new features, we directly stored them into the memory (removing any information related to explicit coordinates), retrained and evaluated the model ("No fusion" in the table below). Additionally, we applied KNN directly on the implicit features to fuse them with the features in the memory after encoding the current frame, retrained and evaluated the model ("Implicit fusion" in the table below). The results below can further demonstrate the effectiveness of our explicit pointer memory when conducting streaming reconstruction.

We hope the above response can help address your concerns. We are happy to answer any additional questions you may have.