PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-forward Planar Splatting

We sincerely thank you for your positive feedback on our manuscript, especially your recognition of our model’s strong performance and comprehensive experiments. We greatly appreciate your constructive comments and address your questions in detail below.

1. On the Chamfer distance and F-score calculation

Your understanding is correct, we uniformly sample points from the planar surfaces for this evaluation.

2. Comparison with RANSAC-based plane fitting baseline

We supplement a new RANSAC-based plane fitting baseline here. Our method achieves much better performance.

3. How we sample pairs as training set

Your understanding is correct—thank you for the insightful observation. In our training set (totally 4M image pairs), we include approximately 0.57M image pairs with no overlap, while the remaining 3.43M pairs are randomly sampled from nearby frames (mainly within the next 10 frames). We observed that incorporating the 0.57M non-overlapping pairs helps the model learn strong geometric priors, enabling it to infer relative pose and scene geometry even when the two views have little or no overlap. Below, we provide a quantitative analysis to show the impact of including the 0.57M non-overlapping image pairs.

4. Performance analysis on pair’s overlap

We conduct an additional ablation study to evaluate the impact of incorporating the 0.57M non-overlapping image pairs on model performance during training. In addition, we analyze how the overlap ratio affects performance. Specifically, we define the overlap ratio as the maximum percentage of pixels in one image (either the first or the second) that have direct correspondences in the other.

As shown in the table below, including these non-overlapping pairs significantly improves the model’s overall performance. Furthermore, we observe that as the overlap ratio in the test set decreases, the model’s accuracy consistently degrades.

ScannetV2 (wo/ExtData means training without 0.57M non-overlapping image pairs):

MP3D (wo/ExtData means training without 0.57M non-overlapping image pairs):

We hope these clarifications and supplement results effectively address your feedback. Thank you again for your valuable insights, which have significantly strengthened our manuscript.

We sincerely thank the reviewers for recognizing the training strategy and application value of our Plana3r method, which enables end-to-end real-time reconstruction and pose estimation without requiring plane annotations. Below, we address your concerns with detailed clarifications and planned revisions.

1. On the Conceptual Novelty of Plana3r

The feedforward modeling of multi-view geometry (MVG) gained traction with the introduction of Dust3r/Mast3r in early 2024, followed by other “3R” family methods addressing longstanding challenges in classical MVG through point map representations and large vision transformers, including but not limited to multi-view aggregation (VGGT [1]) and dynamic scene reconstruction (Cut3r [2]). While these methods have improved 3D reconstruction performance, point map representations are often redundant and inefficient. All existing 3R methods must output a large number of 3D points to characterize a scene. Thus, pursuing compact and efficient representations in feedforward 3D reconstruction is both urgent and impactful.

Plana3r addresses this by introducing a compact planar 3D representation into the 3R framework, integrating the strengths of feedforward modeling and planar primitives. We build upon PlanarSplatting, which provides an efficient rendering scheme from planar primitives. However, unlike PlanarSplatting, our method does not require precomputed camera poses. From that perspective, Plana3r is a non-trivial technical advancement. Our key contributions are as follows:

• Sparse Planar Representation: Unlike Dust3r/Mast3r, which use a DPT head for dense point regression, and PlanarSplatting, which requires dense depth maps for primitive initialization, Plana3r directly regresses sparse planar primitives from $H/16 \times W/16$ patches. Our novel HPPA module adaptively adjusts the number of primitives, ensuring a sparse yet sufficient representation that captures scene geometry. This reduces storage demands while preserving detail in textureless regions. Devising a new 3D vision foundation model using the planar representation has unique application value.

• Metric Reconstruction and Pose Estimation: Unlike most related work, Plana3r supports metric reconstruction and pose estimation. Leveraging the good data distribution of indoor data, our model achieves high data utilization and accurate metric estimation, offering new insights into 3D vision foundation model design.

• Optimized Training Strategy: Training with differentiable rendering in PlanarSplatting presents challenges due to the randomness in initialization at the start of training—many primitives are initially placed outside the camera’s view, leading to unstable gradients. To mitigate this, we adopt a two-phase training strategy designed to stabilize the training.

2. On the motivation of Hierarchical Primitive Prediction Architecture (HPPA)

We believe there may be a slight misunderstanding regarding the motivation behind HPPA. As stated in our paper (Section 3.1, line 143 to line 145, Page 4):

“To balance efficiency and representational fidelity, we propose a hierarchical primitive prediction architecture (HPPA) to fit the scene using planar primitives, enabling compact and efficient modeling of scene geometry with sparse primitives.”

Our primary goal with HPPA is to achieve a more compact and efficient geometric representation using fewer primitives. Here, the term "efficiency" specifically refers to the number of primitives required to represent a scene, rather than the inference runtime. While we do not claim that HPPA improves inference speed, we do provide a runtime and FLOPs breakdown to demonstrate that the additional computational overhead introduced by the image texture mask is minimal. As the calculations are conducted on $H/16 \times W/16$ and $H/8 \times W/8$ patches, the extra cost is negligible in practice. If the term “efficiency” has caused any confusion, we will revise the wording in Section 3.1 and Section 3.2 to clarify that it refers to representational efficiency (i.e., fewer primitives) rather than runtime efficiency. Below is the detailed runtime analysis tested on NVIDIA RTX 3090:

encoder:               12.1 ms  
decoder:               32.8 ms  
upsample:               0.4 ms  
head_prediction:       18.0 ms  
hppa_low_view1:         2.0 ms  
hppa_high_view1:        1.5 ms  
hppa_low_view2:         1.0 ms  
hppa_high_view2:        0.9 ms  
rendering:              0.9 ms  
-------------------------------
Total HPPA overhead:    around 5.0 ms  
Total compute:         ~1.54 TFLOPs  
Memory footprint:      ~7.8 GB  
Model params: 540M (DUSt3R is 571M, MASt3R is 689M)

3. On the multi-view reconstruction with more than two input images

Plana3r currently supports multi-view reconstruction in a pairwise manner, but does not support a single forward pass for inputs with three or more views. Given $N$ input images, we construct $N-1$ image pairs and perform $N-1$ separate forward passes. The planar primitives predicted from each pair are then merged into a common coordinate system. To evaluate this capability, we tested Plana3r on 50 eight-view samples, sampled every 20 frames from the ScanNetV2 dataset. For fair comparison, we employed MASt3R also in a pairwise manner as the baseline. The quantitative results of the estimated camera trajectories are summarized below:

RRA (Relative Rotation Accuracy) and RTA (Relative Translation Accuracy)

This paper focuses on two-view reconstruction and relative pose estimation—core problems in 3D vision with broad applicability. Our key contribution lies in designing a 3D vision foundation model with a new representation and proposing a new perspective for end-to-end metric indoor scene reconstruction. Currently, we train with two-view inputs at $512 \times 384$ resolution, which already imposes high GPU demands (Section 4.1). While we have not yet conducted end-to-end multi-view training, Plana3r supports multi-view reconstruction at inference time with multiple feed-forwards. Extending to full multi-view training is straightforward—e.g., by scaling the data and following approaches like VGGT [1]. We appreciate the suggestion and agree that one-pass multi-view reconstruction is a promising direction. We plan to explore this when we have sufficient computational resources in the future.

4. Failure case analysis:

As shown in Figures 3, 4, 5, 7, and 8, our method will not fail in low-texture areas and can yield good plane segmentation and planar reconstruction. For the complex clutter scene, if there are many non-planar areas and very complex textures in the scene, such as potted plants, etc., it is difficult for our method to achieve high-quality reconstruction. However, the motivation of this paper is for the man-made, structured indoor scene.

[1] VGGT: Visual Geometry Grounded Transformer (CVPR 2025)

[2] Continuous 3D Perception Model with Persistent State (CVPR 2025)

We hope these clarifications address your concerns. Thank you again for your valuable feedback, which has strengthened our manuscript.

Dear Reviewer CYmQ,

We sincerely appreciate your thoughtful comments on our paper.

As the reviewer-author discussion period will conclude on August 6, we would like to kindly check if there are any remaining concerns that were not fully addressed in our rebuttal. If you have any additional feedback or suggestions regarding our submission, we would be very grateful to receive them.

Thank you again for your time and consideration. We truly value the review process and your input in helping us improve the quality of our work.

Best regards,

PLANA3R Team

We sincerely thank you for your positive feedback on our paper. We are especially grateful that you found the research topic intriguing and recognized the novelty and technical strengths of our proposed method. Your insightful comments have helped us refine our work further. Below, we provide detailed responses to your suggestions and questions, along with the revisions we plan to make.

1. On framework figure

We appreciate your suggestion to enhance Figure 2 and the descriptions of high-resolution and low-resolution primitives. In the revised manuscript, we will update Figure 2 to better illustrate these concepts and provide more detailed explanations of high-resolution and low-resolution primitives in the manuscript to improve clarity and accessibility.

2. On sensor ground-truth depth

We acknowledge that the sensor-provided ground-truth depth is relatively sparse. During training, we apply masks to filter out invalid values (e.g., NaNs) in the depth data. These masked regions are excluded from the loss computations in Equations (3) and (4).

3. On pseudo ground-truth normals

Thank you for raising this insightful point. Following your suggestion, we generated pseudo ground-truth normals from the sensor depth data. Compared to the normals generated by Metric3Dv2, these pseudo normals contain more invalid regions and exhibit less smoothness. To evaluate their impact, we trained a version of our model using these pseudo normals as supervision, as shown in the table below. It can be seen that the model performs better when using the normal provided by Metric3Dv2. Interestingly, using a poor normal not only reduces the performance of 3D reconstruction but also the accuracy of pose estimation.

ScanNetV2 dataset

MP3D dataset

4. On MP3D RGB images

The RGB images in the MP3D dataset are rendered from the officially provided mesh, and thus exhibit lower quality and certain distortions compared to real camera-captured images. Nonetheless, these inputs further underscore the strong generalization capability of our Plana3r model. This result highlights the effectiveness of planar primitives as a robust intermediate representation: despite the degraded image quality, Plana3r consistently predicts high-quality planar primitives, enabling accurate reconstruction of planar surfaces.

We hope these clarifications and planned revisions effectively address your feedback. Thank you again for your valuable insights, which have significantly strengthened our manuscript.

We sincerely thank you for your thorough and insightful feedback, which has significantly enhanced the quality of our manuscript. We greatly appreciate the recognition of the value and performance of our approach. Below, we address your suggestions and questions with detailed clarifications and outline revisions.

1. On PlanarSplatting introduction

Thank you very much for your suggestion. We will include an introduction to PlanarSplatting in our revised manuscript to improve clarity and make the paper more self-contained. Specifically, we will add the following description in Section 3.1:

'' PlanarSplatting's rasterization is a core component of our Plana3r model. PlanarSplatting is an ultra-fast and accurate method for reconstructing planar surfaces in indoor scenes from multi-view images. Rather than detecting, matching, or tracking planes in 2D or 3D, it directly operates on a set of 3D planar primitives, "splatting" them into dense depth and normal maps via differentiable, CUDA-accelerated rasterization.''

2. Symbol definitions and coordinate frames

We apologize for the insufficient explanation of symbols in the manuscript. Your interpretation of $q_{\pi}^{i,j}$ is correct—it denotes the quaternion of a planar primitive generated from image $I^i$, represented in the coordinate frame of camera $j$. In our training setup, we fix $j = 1$, treating $I^1$ as the reference frame. Accordingly, the ground-truth camera pose of $I^1$ is set to the $4{\times}4$ identity matrix. The notation $r_{\pi}^{i,j}$ refers to the radii array of planar primitives associated with image $I^i$. Since the radii represent side lengths and are independent of camera $j$, this notation may be misleading. We will revise it to $r_{\pi}^{i}$ for clarity. Similarly, $d_{\pi}^{i,j}$ was a typo and should be corrected to $d_{\pi}^{i}$, which denotes the depth map of the planar primitive centers associated with image $I^i$. We will revise the Equation (1) in manuscript to clarify the notation and coordinate system accordingly as:

Input: $(I^i, K^i)_{i=1,2}$

Output of Plana3r: $(d_{\pi}^{i},\ r_{\pi}^{i},\ q_{\pi}^{i,j})$, $j{=}1$, and $P_{\text{rel}}$.

Then, the primitive centers $c_{\pi}$ from both $I^1$ and $I^2$ are transformed into the coordinate frame of camera $j=1$ using the corresponding intrinsics $K^i$, depthmap $d_{\pi}^{i}$ and camera poses (page 4, line 127 to line 128). Therefore, throughout training, the primitive centers $c_{\pi}$, quaternions, and camera poses of both $I^1$ and $I^2$ are consistently expressed in the coordinate frame of camera $j = 1$.

3. Differentiable planar primitive rendering in training stage

We appreciate the reviewer’s question regarding the phrase: “As a result, directly applying the differentiable planar primitive rendering...” Here we provide further clarification:

- What to render:

Our Plana3r model takes as input two unposed images and their camera intrinsics, and outputs a set of planar primitives in the coordinate frame of the camera $j{=}1$. During training, we use the differentiable rasterizer from PlanarSplatting to render dense depth and normal maps at the original resolution ($H \times W$) with sparse primitives predicted from low-resolution image features (Page 15, Figure 9 (b)).

• For $I^1$, rendering is done using the $4{\times}4$ identity matrix as its camera pose.
• For $I^2$, rendering is done using the ground-truth relative pose from $I^2$ to $I^1$.

The rendered depth $D^{\text{render}}$ and normal $N^{\text{render}}$ for each image are computed in their respective camera coordinate systems. Thus, while the inputs to the renderer (camera poses and primitives) are defined in the coordinate frame of $I^1$, the rendered outputs (depth and normal maps) are expressed in each image’s own coordinate frame.

- Target of rendering loss:

Note that the primitives are initially predicted from low-resolution image patches and are supervised using patch-level losses at resolutions of ($H/16 \times W/16$) and ($H/8 \times W/8$), as described in Equation (3). This patch-level loss does not use the radii information of planar primitives, but only supervises the depth and quaternion of predicted planar primitives. To capture accurate geometry of the scene and let the gradient flow optimize the radii of planar primitives, we further supervise the model by computing losses between the rendered and ground-truth depth and normal maps (as shown in Equation (4)) using PlanarSplatting's rasterization, at the original image size ($H \times W$). This encourages the model to predict accurate primitive shapes (radii), depths, and orientations, thereby improving the geometric fidelity of the reconstructed scene. We will incorporate this clarification into the manuscript.

4. On warm-up training phase

Thank you for highlighting the need for clarity regarding the warm-up training phase. As noted in Section 4.1 (page 6, line 205 to line 206), the warm-up phase includes the pose loss. We will ensure this is more prominently explained in the revised manuscript to avoid any oversight.

We hope these clarifications address your questions effectively. Thank you again for your valuable reviews.