PointRecon: Online 3D Point Cloud Reconstruction via Ray-based 2D-3D Matching

1. Unclear Advantage Over Prior Methods: The benefits of PointRecon over SimpleRecon or VisFusion, which achieve similar reconstruction quality with lower latency, are not fully evident. Further clarity on the advantages or potential benefits of this method would be valuable. In terms of quality, the proposed PointRecon is simialr to SimpleRecon or VisFusion; In terms of speed, PointRecon is slower than the aforementioned prior work. Is there any aspect that the proposed method has stronger potential than prior work?

2. Latency: The method's sampling approach introduces relatively high latency per frame, particularly during scene adjustment and depth prediction.

3. Noise in Output: The absence of post-processing smoothing results in noisier meshes compared to other approaches with more advanced smoothing techniques.

4. Complexity in Implementation: Ray-based matching and multi-level attention mechanisms increase computational complexity, which may affect scalability. For example, if this method is employed for a larger scene, the ray-based matching computation complexity will increase as the number of rays in the scene increases. Could author test this by running this method on a larger scene, e.g. multi-room environment instead of a single room scene?

5. Limited Justification for Ray-Based Matching: Although an ablation study highlights the value of key components, the core concept of "ray-based matching" could benefit from further justification. More comparisons with alternative methods, such as point-based or traditional epipolar line matching, would strengthen the argument for this approach. The authors could run on the same dataset and test the alternatives by switching the matching module.

• The paper would benefit from a high-level overview explaining (1) how the method is initialized on the first frame and (2) how the global representation is iteratively refined with each new frame before detailing individual steps.

• Depth updates rely on a single new image at each step, which contradicts with most multi-view 3D reconstruction methods that integrate multiple views simultaneously. Using only one view at a time has potential drawbacks: 1) Reduced robustness in homogeneous regions compared to multi-view approaches; 2) Limited co-visibility, impacting point matching quality; 3) Suboptimal performance in extreme depth ranges, as the baseline (i.e. the distance between two cameras) is fixed. Could the authors clarify this design choice? I wonder if the less smooth reconstructions observed in the experiments relate to this limitation.

• Since the method relies on stereo feature matching, the view-independent color jittering may negatively impact matching quality.

• While the point cloud is lightweight, the final 3D reconstruction depends on an algorithm to convert the 3D point cloud to the underlying 3D surfaces. The authors use TSDF Fusion, which inherits its limitations in accuracy and resolution.

1. Although this method improves reonstruction accuracy compared to baseline methods, it is $6\times$ more time-consuming, with a processing time of 0.6 s/frame, which limits its feasibility for online reconstruction.

2. In Line 243, the authors mention adjusting only visible points, which can lead to gaps at the edges between visible and non-visible regions. Although scene normals are supervised during training, there is no test-time guarantee to avoid this issue.

3. Individually predicting an offset for each point in the scene adjustment introduces local noise, as seen in Figure 5.

4. There is a lack of detailed ablation studies and analysis on the ray-based matching method (comparing with point-based) and the relaxation of epipolar geometry constraints.

5. As an online reconstruction method, it is necessary to provide hardware testing information and comparisons of memory usage.

6. The paper is somewhat hard to read and follow.

Major comments:

• The 2D-3D matching approach appears highly dependent on precise camera pose estimation. Sampling along a ray and projecting back to the camera to confirm matches assumes a high degree of pose accuracy, as even minor angular deviations can significantly affect the rays, especially for distant points. Achieving such accuracy is difficult in incremental or real-time systems without extensive bundle adjustment rounds. Consequently, this method can only be applied once pose estimation (and potentially a full reconstruction) is complete. If so, the rationale for incremental processing and online capability in the proposed pipeline is unclear, as an offline method would first need to process the entire sequence. Moreover, image matching with known poses is well-explored, with numerous relevant publications that the authors seemingly overlook. Without being exhaustive, here are a few examples [m1-m4]. Actually, a large portion of the multi-view stereo literature addresses matching, making the proposed appear excessively complicated without clear justification (especially, since the main reason, i.e. being incremental, does not seem to make sense, as I write earlier).
• The experiments are very weak. Showing results on a single dataset, while all other baselines work well on others as well, is clearly insufficient. Also NICER-SLAM [j] is missing that also only use RGB.
• Table 1: Throughout the paper, the authors describe their method as "online", but it runs at ~1 FPS, which does not truly qualify as online. Additionally, its accuracy and precision are lower than methods that are nearly an order of magnitude faster.
• Table 2: The same observations apply here as for Table 1. In many metrics, the proposed method is underperformed by significantly faster alternatives.
• Conclusion: "Experiments show that our approach achieves state-of-the-art performance." This statement is inaccurate. While some metrics may show strong performance, others reveal that it lags behind baseline methods.
• The assumption of known camera poses should be stated upfront (e.g., in the abstract). The current wording suggests that the authors address both geometry and pose estimation by using the term "reconstruction" which is not true.

Minor comments:

• Experiments: Although the authors opt not to use the depth channel, it would still be informative to show comparative results with methods that do, such as [a,b,c,d], since the ScanNet dataset includes this data. This comparison would help readers understand how RGB-only performance currently compares to RGB-D methods.
• L053: Missing related work on volumetric methods: [a,b,c,d].
• L066: Missing related work: [e,f].
• Paragraph at L141: Not all volumetric methods require a predefined grid [c].
• Several methods for image ray to 3D point matching are entirely ignored: [g,h,i].
• Fig.2: The image is very dark; consider using a different image or adjusting the visualization for better clarity.
• L130: "volume ." -> "volume."

[a] Oleynikova, H., Taylor, Z., Fehr, M., Siegwart, R. and Nieto, J., 2017, September. Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1366-1373). IEEE.

[b] Grinvald, Margarita, et al. "Volumetric instance-aware semantic mapping and 3D object discovery." IEEE Robotics and Automation Letters 4.3 (2019): 3037-3044.

[c] Zheng, J., Barath, D., Pollefeys, M. and Armeni, I., 2025. Map-adapt: real-time quality-adaptive semantic 3D maps. In European Conference on Computer Vision (pp. 220-237). Springer, Cham.

[d] Miao, Y., Armeni, I., Pollefeys, M. and Barath, D., 2024. Volumetric semantically consistent 3d panoptic mapping. IROS 2024 [e] Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B. and Revaud, J., 2024. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20697-20709).

[f] Leroy, V., Cabon, Y. and Revaud, J., 2024. Grounding Image Matching in 3D with MASt3R. ECCV 2024

[g] Chen, B., Parra, A., Cao, J., Li, N. and Chin, T.J., 2020. End-to-end learnable geometric vision by backpropagating pnp optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8100-8109).

[h] Zhou, Q., Agostinho, S., Ošep, A. and Leal-Taixé, L., 2022, October. Is geometry enough for matching in visual localization?. In European Conference on Computer Vision (pp. 407-425). Cham: Springer Nature Switzerland.

[i] Wang, S., Kannala, J. and Barath, D., 2024. DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20881-20891).

[j] Zhu, Z., Peng, S., Larsson, V., Cui, Z., Oswald, M.R., Geiger, A. and Pollefeys, M., 2024, March. Nicer-slam: Neural implicit scene encoding for rgb slam. In 2024 International Conference on 3D Vision (3DV) (pp. 42-52). IEEE.

[m1] Goesele, M., Snavely, N., Curless, B., Hoppe, H. and Seitz, S.M., 2007, October. Multi-view stereo for community photo collections. In 2007 IEEE 11th International Conference on Computer Vision (pp. 1-8). IEEE.

[m2] Žbontar, J. and LeCun, Y., 2016. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17(65), pp.1-32.

[m3] Scharstein, D. and Szeliski, R., 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International journal of computer vision, 47, pp.7-42.

[m4] Furukawa, Y. and Ponce, J., 2009. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(8), pp.1362-1376.