AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos

Thanks for the comments. All responses below will be put into revision.

[W1: break down of time budget and analysis]

Below is a breakdown of the time budget for the optimization process of a single scene:

As shown above, the merge and rendering pipeline is relatively efficient. The initialization process (which includes converting every superpixel to a initial tablet, and texture initialization) consumes a significant amount of time. This is due to the current naïve implementation for demonstration, where tens of thousands of Python loop is called. We will improve the implementation to enable parallelized initialization in the future work. Furthermore, the NVDiffRast renders more than ten layers to perform alpha composition every forward pass, but most of the scene's structure is single-layered, results in a substantial backward computation burden during training. We regard this as another potential area for considerable optimization in the future work.

[W2: detail of tablet parameters]

Thanks and we appreciate the opportunity to clarify these details:

1. Pixel Range (ru, rv):

• Each tablet's geometry is located in 3D space, while its texture is stored in 2D.
• The pixel range represents the resolution at which the texture is stored.
• For initial tablets, the pixel range is derived directly from the range in the source image.
• For merged tablets, the pixel range is calculated as the average of all corresponding initial tablets.

2. Distance Ratios (λu and λv):

• These ratios establish the relationship between the 2D texture resolution and the 3D size of the tablet.
• For initial tablets, the distance ratio is calculated by dividing the camera's focal length by the average initial distance of the tablet.
• For merged tablets, the distance ratio is the average of all corresponding initial tablets' ratios.

3. Alpha Channel: Yes, the alpha channel is a learnable single-channel map with the same shape as the texture map.

[W3: Initial tablet merge]

During initialization, each superpixel is initialized as a tablet using the estimated depth and surface normal. Using SLIC superpixel on ScanNet 1296x968 resolution image results in around 10k superpixels for each keyframe, leading to a large number of initial tablets. As detailed in Appendix Sec A.1, we introduce an init merge to create larger initial tablets, which helps to improve the speed. Further efficiency improvements could be achieved through fewer superpixel initial number (by controlling the SLIC hyper-parameters) and more aggressive init merge.

[W4: reconstruction accuracy]

The difference in 3D accuracy (termed as Acc in Table 1 of the main paper) between our method and PlanarRecon on the ScanNet dataset can be attributed to several factors:

1. Scope of Reconstruction:

• PlanarRecon often only reconstructs large planar regions. This allows for easier localization and high accuracy on these specific areas, but it limits overall coverage and performance.
• Our method enables more comprehensive reconstruction, including smaller planar regions, which can impact the accuracy metrics but provides a more complete representation of the scene.

2. Ground Truth Coverage:

• It is worth noting that the 3D ground truth planes in ScanNet only partially cover the scene within the camera's view. Even after excluding areas too distant to be relevant using the camera frustum, significant portions remain uncovered.

  • PlanarRecon learns to exclude distant reconstructions during its training stage, leading to improved accuracy metrics.

  • Our method, however, is capable of identifying planar regions for all visible areas. This is evident in Figure 2 of the attached PDF, where most of the red regions highlight this phenomenon.

  • While these uncovered areas affect the evaluation accuracy, they should not necessarily be considered a negative outcome. Our method provides a more complete reconstruction of the scene, including areas that are not represented in the ground truth data.

Please refer to Figure 2 of the attached PDF for a qualitative illustration of this issue.

[W5: depth loss and model]

To clarify, we use Metric3D v2 for predicting monocular depths and Omnidata for surface normals.

Yes, there is a trade-off when using depth loss due to the depth estimation errors. To address this issue, we incorporate multiple loss terms, including photometric, normal and distortion losses, etc., to optimize and regularize the tablets. The depth loss could be viewed as a data term in optimization which constraints the results not too far from the initializations. In practice, we observed that the depth loss helps with the final reconstruction, and it is a common practice in 3D reconstruction to employ it. Further incorporation of uncertainty or better depth estimation methods could be beneficial, which we will put into future work.

Once again, we thank the reviewer for the valuable comments that helps improve our paper.

Thanks for the efforts of the authors again. I have some more questions about the paper in the following:

1. How many scenes are used to evaluate on the ScanNetv2 datasets in practice?
2. Will the authors release the code?
3. I noticed a recent work for planar reconstruction called AirPlanes in CVPR2024. It seems that a good dense reconstruction + RANSAC is still a strong baseline for planar reconstruction. Can the author further discuss the advantages of plane-based/primitive-based optimization for this task? The authors only need to discuss and do not need to compare with Airplanes.

Thanks for the comments. All responses below will be put into revision.

[W1:Baseline setup]

Thanks and we appreciate the opportunity to clarify:

1. Seq-RANSAC: For 3D volume-based methods including Atlas, NeuralRecon, PlanarRecon, and Metric3D with TSDF fusion, we followed PlanarRecon to use their enhanced version of Seq-RANSAC. We refer to PlanarRecon for detailed descriptions. For point-based methods such as SuGaR, since PlanarRecon’s Seq-RANSAC requires 3D TSDF volume as inputs and cannot be easily adapted to points or meshes, we use the classical vanilla Seq-RANSAC, which iteratively applies RANSAC to fit planes. Here we used Open3D plane RANSAC implementation https://www.open3d.org/docs/latest/tutorial/Basic/pointcloud.html#Plane-segmentation for each iteration. The hyper-parameters are carefully tuned for optimal performance.

2. Metric3D: We used the official Metric3D v2 implementation and pre-trained weights (v2-g) from https://github.com/YvanYin/Metric3D. The Metric3D is run on each keyframe to get depth maps, followed by TSDF fusion (widely adopted implementation from https://github.com/andyzeng/tsdf-fusion-python) to fuse into 3D volume. Finally, PlanarRecon’s Seq-RANSAC is applied to the 3D TSDF volume to get the planar results.

3. SuGaR: We adopted the original implementation from https://github.com/Anttwo/SuGaR. During the COLMAP pre-processing, we feed ground-truth camera poses into the pipeline, which provides better initial sparse points. After optimization, SuGaR outputs the mesh model, and we uniformly sampled 100k surface points and applied vanilla Seq-RANSAC on top of sampled points to get the 3D planar results.

We will include these details and add references in the revision to indicate that quantitative results for other baselines were taken from PlanerRecon.

[W2:Geometric evaluation with alpha mask]

To clarify, our point sampling process considers the alpha mask during evaluation. Specifically, we first perform a weight check (as described in Appendix Sec.A.1) using alpha channels to mask out transparent or unseen regions from all views. Then we follow the PlanarRecon to convert the remaining visible plane regions into mesh using Delaunay triangulation. Points are then evenly sampled on this generated mesh. We refer to PlanarRecon for more details.

[W3:blurry results]

Since our method focuses on reconstructing 3D planar surfaces, it has limitations when dealing with geometrically complex, non-planar regions, as discussed in the limitation section. Also, our current AlphaTablets do not consider view-dependent effects, leading to visual quality affected. As a potential solution, the introduction of a hybrid representation that combines our planar AlphaTablets with other methods, such as 3D Gaussians, may be promising for representing complex, non-planar geometries, which we will discuss in future work in the revision.

[W4,5:figure&related work]

Thanks and we will revise the text as suggested.

[Q1:Details and init of SuGaR baseline]

We appreciate the opportunity to provide more details on our setup and results:

• Baseline Setup: We utilized all available views when training SuGaR. We adopted the official implementation, and enhanced COLMAP sparse reconstruction with ground-truth camera poses, which provides better initial sparse points.
• Performance discussion: The ScanNet dataset presents significant challenges like numerous blurry and textureless regions, which are especially problematic for Gaussian-based methods like SuGaR when reconstructing clear geometry. Also, SuGaR heavily relies on COLMAP reconstruction to initialize, but the COLMAP reconstruction on ScanNet is sometimes noisy, affecting the final performance.
• Initialization Experiment: We experiment on our ablation subset to compare the COLMAP initialization with Metric3D’s dense depth-based initialization similar to our method. For Metric3D init, we use the same keyframes as our method and randomly sample a total of 100,000 points as initial points. The results are shown in the table:

The Metric3D init method does indeed enhance the reconstruction quality of SuGaR (qualitative results in Fig.1 in attached PDF file), but the overall reconstruction quality remains constrained, with noticeable jitter and challenges in accurately delineating planar regions, leading to an inferior performance to our approach.

[Q2:breakdown of time budget]

Please refer to response to W1 of reviewer Ku7J.

[Q3:Tablet-camera assignment]

To clarify, (1) we maintain affiliations between the initial tablets and the current (merged) tablets (as stated in Sec A.1); (2) We keep track of the camera index that initially generated each initial tablet; (3) When tablets are merged, we count the number of each camera indices corresponding to all affiliated initial tablets, and assign the most frequently occurring camera to the newly merged tablet. We will include these details in the revision.

[Q4:tablet number plot]

Thank you for the great suggestion. We plot the numbers in Fig.3 of the attached PDF file. The number of tablets decreases rapidly in the early merging stages and gradually converges into several hundreds. Notably, the final tablets contain a large portion of small tablets representing non-planar regions, while the primary planar scene structure is adequately represented with fewer tablets.

[Limitations]

Thanks and we will revise the paper with a detailed runtime analysis and potential improvements as discussed above.

Once again, we thank the reviewer for the valuable comments that helps improve our paper.

Thanks for the comments. All responses below will be put into revision.

[W1: Notion unclear]

We will revise Figure 1 to include a more intuitive display with 3D effect that demonstrates how multiple layers of AlphaTablets are composed to render each pixel in the final image.

[W2: Detailed caption for figures]

We will revise the captions of figures with more details and comprehensive explanations.

[W3: Tablet Storage]

As a 3D planar representation, AlphaTablets are designed to be lightweight. For example, the demo scene requires only 20MB storage, including all the texture maps in a lossless PNG format. Similar to mesh, advanced techniques such as texture atlas and compression can also be applied to further improve the storage efficiency.

Once again, we thank the reviewer for the valuable comments that helps improve our paper.