Beyond 2D Representation: Learning 3D Scene Field for Robust Monocular Depth Estimation

Q3: Can I achieve comparable or even better performance with TSF-Depth by adding a simple refine module on the existing network.

A3: To evaluate whether comparable or even better performance to TSF-Depth can be achieved by adding a simple refine module on the existing network, we provide an ablation experiment, as shown in Table below: |Method| Sq Rel(↓)| Abs Rel(↓)| RMSE(↓)| RMSE log(↓)| δ<1.25(↑) | δ<1.25² (↑) | δ<1.25³ (↑) |#Params | |------------|:------:|:------:|:------:|:-------:|:-----:|:--------:|:------:|:-------:| |TSF-Depth (Baseline)| 0.746|0.102|4.464|0.176| 0.897|0.965|0.983|27.2M| |TSF-Depth (Refine) |0.741|0.101|4.461 |0.176| 0.897|0.965|0.983| 29.6M| |TSF-Depth|0.692|0.096|4.335 |0.173| 0.903|0.967|0.984| 29.8M|

For the baseline (row 1), we remove the decoder $\theta_I$ branch and positional encoding branch of TSF-Depth and then train it on the KITTI dataset. Therefore, the pipeline and training strategy of the baseline are almost completely similar to existing self-supervised methods. For the Refine framework (row 2), we remove only positional encoding branch of TSF-Depth, and then use $\theta_F$ as a refine module to optimize the splicing features of the encoder $\theta_E$ and the initial depth. The last row reports the depth results for the full model. It can be seen that the refine strategy has only a slight improvement compared to the baseline. The possible reason is that it still relies on 2D features representation. In contrast, our TSF-Depth has a similar number of parameters as the Refine framework, yet achieves significant improvements due to the 3D geometric representation.

Q4: How the network handles sky areas and provide the visualization results of the intermediate depth maps and point cloud.

A4: During the training phase, most of the sky areas will be masked since we follow Monodepth2 [1] using a masked photometric loss. In this loss function, an auto-masking strategy is designed to generate a per-pixel mask, which is then applied to photometric loss to filter out pixels whose appearance does not change from one frame to the next in the sequence. Therefore, since the sky area has a highly consistent appearance in adjacent frames, it will be removed by mask photometric loss in our framework.

As shown in Fig. 10 (b) in the Appendix, we present the visualization results of the depth map at six scales (2nd column) and the corresponding point clouds without (3rd column) and with (4th column) color values of the target image. In addition, we also present the reconstructed results (6th and 7th columns) after the predicted depth map is upsampled to the real image resolution, as shown in Fig. 10 (c) in the Appendix. As can be seen in Fig. 10 (c) in the Appendix, the reconstructed point clouds at different scales are almost identical, which suggests that our generated depth maps and reconstructed point clouds at different scales are geometrically consistent.

Q5: Please state the experimental setting of the ablation experiment more clearly (Effects of 3D Scene Field on 3D Geometric Representation)

A5: Our TSF-Depth aims to achieve robust depth estimation based on the 3D scene representation. In order to evaluate the effectiveness of 3D scene fields for 3D geometric representation, we provide a ablation experiment (e.g., Effects of 3D Scene Field on 3D Geometric Representation) refer to lines 484-529. The depth estimation results on the KITTI dataset is reported in Table 6, where contains three experimental setting: Baseline (row 1), TSF-Depth based 2D geometry features (row 2), and TSF-Depth based on the 3D geoemtry features (row 3). We remove the decoder $\theta_I$ branch and positional encoding branch of TSF-Depth as a Baseline framework and then train it on the KITTI dataset. Therefore, the Baseline relies on extracting front-view 2D features for depth estimation. Note that we denote the encoder of Baseline as $\theta_E^{PT}$. Then, to evaluate the effectiveness of $\theta_E$ in learning 3D representation under our end-to-end self-supervised paradigm, we replace the $\theta_E$ of TSF-Depth with the trained encoder $\theta_E^{PT}$, and then train other modules. In this setup, the Tri-plane feature field is constructed using 2D representation. As for the last row in Table 6, it uses our complete training. Compared to the Baseline (row 1) and the complete TSF-Depth (row 3), we observe that the results of TSF-Depth based on 2D geometry features are worse than both them, due to the subsequent 3D-to-2D mapping requiring 3D geometric representation rather than 2D representation. Therefore, our TSF-Depth can learn 3D geometric representations during the training phase for robust depth estimation.

Reference

[1] Godard, Clément, et al. "Digging into self-supervised monocular depth estimation." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

W1: Add the keyword "self-supervised" to the title.

A: Thanks for your suggestions. We have added the keyword "self-supervised" to the title to improve the clarity of the manuscript.

Q1: There are no constraints between different plane features. Is it possible to get 3D representation capability just by using different image coordinate encodings?

A1: Although there are no explicit constraints between different plane features, they are implicitly constrained by our 3D-to-2D mapping strategy and end-to-end self-supervised paradigm. Referring to lines 277-306, for the generated Tri-plane Feature Field, we project each 3D points onto three orthogonal feature planes to retrieve the corresponding feature via bilinear interpolation, and then aggregate these multi-view geometric feature map via concatenation to predict the depth map. Since the projected geometry is embedded in this mapping process, different plane features can be learned for 3D representation in end-to-end training.

The image coordinate encoding is treated as a scene priors and is used to enhance the 3D representation by providing spatial cues (see lines 239-242). Moreover, the detailed information of the Tri-plane Feature Field is mainly provided by the scene features extracted from the input image. Therefore, it can not get the 3D representation capability simply by only using different image coordinate encodings.

Q2: The prior is not always right, and does TSF-Depth still applicable to challenge cases such as side view and cropped image?

A2: In our work, the scene prior is considered as an auxiliary clue to enhance the 3D representation, while the detailed information of the Tri-plane Feature Field is mainly provided by the scene features extracted from the input image.As you correctly pointed out, these priors may not always be accurate. To address this, we employ relative positional encoding instead of absolute image coordinates as the scene prior (see lines 221-224). This design shifts the focus of our method from fixed pixel or region positions to their relative relationships, enabling robust depth estimation even in scenarios where the absolute spatial layout undergoes significant changes, such as in side views or cropped images.

As presented in Table below, we compare depth estimation performance of the Baseline and our proposed TSF-Depth on the KITTI dataset. The evaluation is conducted using top cropping and removing the sky area from the image at different ratios. The crop ratio is reported in the first column, where 0 means no crop. As the cropping ratio increases from 0 to 1/3, progressively removing more of the sky region, the baseline's error metric (Sq Rel) rises from 0.746 to 0.802, representing a 7.51% increase. In contrast, our proposed TSF-Depth shows a much smaller increase, from 0.692 to 0.704, with only a 1.73% rise in error. This demonstrates that the priors employed in our method are less sensitive to spatial layout variations. Moreover, the integration of these priors with 3D geometric representation enables our proposed TSF-Depth to achieve robust depth estimation.

|Crop Ratio|Method|Sq Rel(↓)| Abs Rel(↓)|RMSE(↓)|RMSE log(↓)|δ<1.25(↑)|*δ<1.25²(↑)|δ<1.25³(↑)| |:----------------:|--------------|:------------:|:-------------:|:----------:|:--------------:|:------------:|:-------------:|:-------------:| | 1/3 | Baseline| 0.802| 0.104| 4.559| 0.181| 0.896 | 0.964 | 0.982 | | | TSF-Depth| 0.704 | 0.098 | 4.382| 0.176 | 0.900 | 0.966 | 0.983 | | 1/6 | Baseline | 0.796 | 0.103 | 4.546 | 0.179 | 0.899 | 0.965 | 0.983 | | | TSF-Depth| 0.699 | 0.097 | 4.370| 0.175 | 0.902 | 0.967 | 0.984 | | 0 | Baseline | 0.746 | 0.102 | 4.464 | 0.176 | 0.897 | 0.965 | 0.983 | | | TSF-Depth| 0.692 | 0.096 | 4.335| 0.173 | 0.903 | 0.967 | 0.984 |

In addition, we also present the visualization results of the depth maps corresponding to different cropping ratios in Fig. 11 in Appendix. We observe that the Baseline produces discontinuous and inaccurate depth results when the spatial layout of the image changes. In contrast, our method generates consistent results across all three cropping ratios, demonstrating its applicability to challenging cases such as cropped images.

The second evaluation focuses on directly testing the trained depth estimation models, including TSF-Depth, on datasets with the top region removed at different pixel ratios. Following your suggestion, we also report results for cropping 100 pixels, a common setting in the field of depth completion. The results, shown in the following table, compare Monodepth2, Lite-Mono-8M, TSF-Depth, and TSF-Depth*. TSF-Depth uses the relative coordinates of the original image size before cropping, while TSF-Depth* recalculates the relative coordinates based on the cropped image. The "Pixels&Ratio" column indicates the number of pixels cropped from the top of the image and their corresponding ratio to the total image height. Compared to Monodepth2 and Lite-Mono-8M, both TSF-Depth and TSF-Dept* show minimal performance degradation as the cropping ratio increases, and are always at the leading level. Importantly, in all cases, while the overall performance of TSF-Depth* slightly lags behind TSF-Depth, the performance gap is small. This consistent performance further highlights the robustness of our method in handling spatial layout changes, demonstrating its adaptability and reliability under diverse cropping scenarios.

| Pixels&Ratio | Method| Sq Rel | Abs Rel | RMSE | RMSE log | δ < 1.25 | δ < 1.25² | δ < 1.25³ | |:-----------------:|--------------|:----------:|:-----------:|:--------:|:------------:|:-------------:|:-------------:|:-------------:| | 125 (1/3)| Monodepth2|1.833| 0.152 | 6.517 | 0.231 | 0.834| 0.937| 0.970| || Lite-Mono-8M| 1.410| 0.138| 5.680| 0.214| 0.849 | 0.943 | 0.976| || TSF-Depth | 1.245 | 0.130| 5.397| 0.206 | 0.860| 0.950 | 0.977| | | TSF-Depth*| 1.232| 0.132| 5.380 | 0.209 | 0.856 | 0.949| 0.977 | | 100 (4/15) | Monodepth2 | 0.973| 0.125 | 5.070 | 0.202| 0.861| 0.955| 0.980| | | Lite-Mono-8M| 1.051| 0.122| 4.966| 0.197| 0.869| 0.955| 0.980| || TSF-Depth | 0.983| 0.117| 4.881 | 0.193 | 0.875| 0.958| 0.981| | | TSF-Depth*| 0.958| 0.118 | 4.833| 0.195| 0.874| 0.958| 0.980| | 62 (1/6)| Monodepth2| 0.867 | 0.117 | 4.867| 0.195| 0.871| 0.958| 0.981| | | Lite-Mono-8M| 0.870| 0.113| 4.617| 0.187 | 0.883 | 0.962| 0.983| || TSF-Depth| 0.784| 0.105 | 4.488| 0.181| 0.890| 0.965| 0.983| | | TSF-Depth*| 0.763 | 0.105| 4.416 | 0.183 | 0.891| 0.965| 0.983 | | 0 (0/375)| Monodepth2 | 0.903| 0.115 | 4.863| 0.193 | 0.877| 0.959| 0.981| || Lite-Mono-8M| 0.729| 0.101| 4.454| 0.178| 0.897| 0.965| 0.984| | | TSF-Depth | 0.692| 0.096 | 4.335| 0.173| 0.903 | 0.967 | 0.984| || TSF-Depth*| 0.692| 0.096| 4.335| 0.173| 0.903| 0.967| 0.984|

Based on the above analysis, we can conclude that our proposed TSF-Depth is still applicable to image resolution variations. This robustness stems from the detailed information of the Tri-plane Feature Field is mainly provided by the scene features extracted from the input image. Additionally, the use of relative positional encoding, instead of absolute image coordinates, as the scene prior enhances the 3D representation. This design enables our method to achieve robust depth estimation even in scenarios where the absolute spatial layout undergoes significant changes.

Dear Reviewer Qiro,

We would like to sincerely thank you for your thoughtful and valuable feedback on our submission. As the discussion period comes to a close, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns or if there are any additional points you would like us to clarify. If our responses have satisfactorily addressed your concerns, we kindly ask you to consider reflecting this in your score.

Once again, thank you for your time and thoughtful input throughout this process.

Best regards,

Authors of submission 243

Q1: How about the performance improvement of our approach when using the same encoder as Lite-Mono?

A1: The following Table presents the depth estimation results on the KITTI dataset when using Lite-Mono-8M as the encoder, comparing the existing method Lite-Mono with the proposed TSF-Depth framework. The table evaluates the performance based on both error metrics (lower is better) and accuracy metrics (higher is better), while also considering model complexity and inference speed.

The Lite-Mono method achieves a relatively low computational cost with 11.2 GFLOPs and 6.3ms inference time. However, its error metrics, such as Sq Rel (0.832) and RMSE (4.597), indicate suboptimal accuracy. The accuracy metrics ($\delta<1.25$: 0.895) are also slightly lower, suggesting room for improvement in capturing depth-related geometric features. Notably, the Lite-Mono results presented here are reproduced by us based on the official code and experimental settings described in the original paper. However, as discussed in several issues on the official repository (see issues: #72, #118, #124, #128 #131, #141, #143, etc.), there are differences between the reproduced results and the ones reported in the paper. Our reproduced results are highly consistent with those provided by other researchers in these discussions.

In contrast, the TSF-Depth framework, when combined with the same Lite-Mono-8M encoder, significantly improves performance. The Sq Rel error is reduced from 0.832 to 0.717, and the RMSE is reduced from 4.597 to 4.355, reflecting better depth prediction accuracy. Moreover, the $\delta<1.25$ accuracy increases from 0.895 to 0.900. These improvements come with a moderate increase in computational cost (16.5 GFLOPs) and a slightly slower inference time (4.4ms).

In addition, the extra parameters mainly come from $\theta_I$, which predicts point clouds to sample 2D features with 3D geometry from the 3D scene field, rather rather directly providing the features required for depth estimation. Therefore, the depth improvement is due to the 3D representation rather than by increasing network parameters.

| Method | Encoder | #Params | GFLOPs | Speed | Sq Rel(↓) | Abs Rel(↓) | RMSE(↓) | RMSE log(↓) | δ<1.25(↑) | δ<1.25² (↑) | δ<1.25³ (↑) | |:------------:|:---------------:|:-------:|:------:|:------:|:------:|:-------:|:-----:|:--------:|:------:|:-------:|:------:| | Lite-Mono | Lite-Mono-8M | 8.7M | 11.2G | 6.3ms | 0.832 | 0.104 | 4.597 | 0.182 | 0.895 | 0.964 | 0.982 | | TSF-Depth| Lite-Mono-8M | 11.2M | 16.5G | 4.4ms | 0.717 | 0.098 | 4.355 | 0.176 | 0.900 | 0.966 | 0.984 |

W1, Q1: Design a challenging test set to assess the robustness of the proposed approach and provide more qualitative comparisons.

A1: Thanks for your suggestions. In general, low-texture regions exhibit smooth variations with gradient values typically close to zero, while reflective regions often display extremely high luminance values near saturation. In the Appendix, we present a challenging sample (Fig. 8 (a)) containing extensive low-texture areas (e.g., roads, buildings, bushes, etc.) and reflective areas (e.g., cars, buildings). To analyze these characteristics, we also present its gradient magnitude (Fig. 8 (b)) and the V channel in HSV space (Fig. 8 (c)). We observe that the gradient the gradient of low textures is close to 0, while the brightness of reflective areas approaches saturation, which aligns well with common perceptions.

Leveraging these properties, we can effectively quantify the low-texture and reflection levels of individual pixels by calculating their gradient magnitudes and brightness values, respectively. Furthermore, the overall difficulty of an image can be assessed by integrating its gradient and brightness scores. To achieve this, we utilize the classical Sobel operator to compute gradient scores and the V channel of HSV space to compute brightness scores, subsequently aggregating them as the basis for selecting the challenging test set.

Specifically, given an image, its challenging score can be defined as $F(I) = T_{\text{low-texture}} + R_{\text{reflection}}$, where $T_{\text{low-texture}} = \frac{\text{Count}(\nabla I < 2)}{\text{Total pixels}}$, and $R_{\text{reflection}} = \frac{\text{Count}(\nabla V > 0.6)}{\text{Total pixels}}$. The $\nabla I$ is the gradient amplitude of $I$ calculated based on the Sobel operator, and $\nabla V$ is the $V$ channel in the HSV space. To evaluate the performance of TSF-Depth under challenging test sets of different difficulty, we compute the maximum score $F_{max}$ and minimum score $F_{min}$ of the entire test set and then select challenging test sets $C =${I, F(I) > \{F_{min}} + \alpha \cdot (F_{max} - F_{min})} based on the threshold $\alpha$. We set six thresholds and compared with the classic Monodepth2 and SOTA Lite-Mono-8M on the KITTI dataset, as shown in Table below: |α | Selected/Total | Method| Sq Rel (↓)| Abs Rel (↓)| RMSE (↓) | RMSE log (↓) | δ < 1.25 (↑) | δ < 1.25² (↑) | δ < 1.25³ (↑) | |:------:|:----------------:|----------------|:--------:|:---------:|:-------:|:----------:|:----------:|:-----------:|:-----------:| | | | Monodepth2|1.554|0.144|6.212 |0.211|0.837|0.946 |0.978| |0.9|14/697| Lite-Mono-8M |1.523|0.132| 5.852 | 0.201| 0.863| 0.954| 0.979| | | |TSF-Depth| 1.193| 0.122| 5.441 |0.188|0.865|0.961|0.983| | | | Monodepth2|0.997|0.120| 5.111|0.191|0.891|0.959|0.983| |0.7 |86/697| Lite-Mono-8M |0.848 |0.107| 4.679|0.175| 0.894| 0.965| 0.984| | | |TSF-Depth|0.741| 0.100| 4.170| 0.168|0.902|0.970 |0.986| | | | Monodepth2| 0.906|0.115| 4.963|0.194|0.896|0.959|0.981| |0.5|291/697|Lite-Mono-8M|0.763| 0.103 | 4.564 | 0.180 | 0.894| 0.965| 0.984| | | |TSF-Depth|0.711|0.099|4.419 |0.175|0.902|0.967|0.984 | | | | Monodepth2|0.916|0.115|4.909 |0.194|0.877|0.959|0.981| |0.3|627/69|Lite-Mono-8M|0.733|0.102|4.507|0.180| 0.889|0.965|0.984| | | |TSF-Depth|0.702|0.097|4.382 |0.174|0.902|0.967|0.984 | | | | Monodepth2|0.906|0.114|4.435|0.193|0.877|0.959|0.981| |0.1| 693/697|Lite-Mono-8M| 0.733|0.101|4.400 |0.179|0.889|0.965|0.984| | | |TSF-Depth|0.692|0.096|4.335 |0.175|0.903|0.967|0.984| | | |Monodepth2|0.903|0.115|4.863|0.193|0.877|0.959| 0.981| | 0 |697/697|Lite-Mono-8M|0.733|0.101|4.400|0.179|0.889|0.965|0.984| | | |TSF-Depth|0.692|0.096|4.335|0.175|0.903|0.967|0.984|

As the threshold increases, all methods show different performance degradation due to the increasingly challenging test set, while our TSF achieves the best performance on all test sets. This observation highlights the robustness of our approach in dealing with difficult scenarios.

We provide more qualitative results on a subset of tests with threshold of 0.9, as shown in Fig. 9 in the Appendix. As can be seen that these samples contain many low-texture or reflection regions, and slender structure, and existing methods are unable to estimate accurate depth results, whereas our method generates robust depth maps. In addition, we also provide more qualitative results on the Make3D, NYUv2 and ScanNet datasets, as shown in Fig. 5, Fig. 6, and Fig. 7 in the Appendix. As we can see from these visualization results, our proposed TSF-Depth generates more accurate depth maps, particularly in these region with low texture, slender structure, shadow occlusion and reflections.

Currently, I cannot conclude that the shift does not affect the depth and distortion. From the affine-invariant depth formulation, all previous works have demonstrated that the shift will cause distortion. Maybe the proposed method can dynamically adjust the shift. However, the paper does not theoretically prove this. Furthermore, the paper does not provide mutli-view reconstructed point clouds for visually evaluation. Authors could follow Marigold's teaser image for multi-view visualization.

Thank you for your detailed feedback and thoughtful comments. We appreciate the opportunity to address your concerns about the robustness and generalization of our method, particularly regarding reflective regions, model convergence, and generalization across datasets.

Reflective regions, such as bus and car windows in Fig. 9 or the window in Fig. 4, indeed pose significant challenges for monocular depth estimation methods due to their inherent ambiguity and the mismatch between visual appearance and scene geometry. These areas create depth ambiguities that make accurate predictions difficult for all existing approaches, including ours. While our method is not completely flawless in these regions, it is important to highlight that in most cases our approach produces depth maps with relatively complete structure in these regions, which is vital for overall scene understanding. Additionally, It is also important to emphasize that our approach is not limited to addressing challenges in reflective regions. Instead, we take a holistic view of monocular depth estimation and address a variety of key challenges that are equally important, including low-texture regions, thin structures, and shadow occlusions. As shown in Figs. 4–7 and Fig. 9 of the manuscript, our method can smoothly predict low-texture areas, better preserves the geometry of fine details, and more effectively handles shadow occlusion regions. Moreover, our method also has significant advantages over existing methods in challenging scenarios (see table in A1). By considering these multiple aspects simultaneously, our method demonstrates superior overall performance across diverse scenarios.

In qualitative comparison, we selected influential methods in self-supervised monocular depth estimation, including Monodepth2, Lite-Mono-8M, PLNet, and StructDepth. Monodepth2 is a widely recognized baseline in the field of monocular depth estimation and has been extensively used as a benchmark in many recent works. Lite-Mono-8M represents a SOTA method for outdoor scenarios, specifically optimized for real-world driving datasets such as KITTI. PLNet and StructDepth focus on indoor scene depth estimation and, while not the latest works, are highly representative due to their open-source implementations and consistently competitive performance in indoor datasets like NYUv2 and ScanNet. These methods cover a broad spectrum of scenarios, from outdoor driving datasets to complex indoor environments, making them suitable benchmarks for evaluating both robustness and generalization.

Although the quantitative results in Table 1, Table 2, Table 3, and Table 4 of the manuscript strongly support that our model has converged well, we will carefully revisit our training pipeline to ensure that no potential improvements have been overlooked.

In terms of generalization and robustness, our method has been rigorously evaluated on four datasets with significant domain differences: KITTI, Make3D, NYUv2, and ScanNet. These datasets cover a wide variety of scenarios, including outdoor driving scenes, reflective surfaces, low-texture regions, and complex indoor environments. Across all datasets, our method consistently outperforms existing self-supervised methods (as shown in Tables 1-4 of the manuscript) and baselines (please see Table 5 in the manuscript, and Table in A4) in both quantitative and qualitative metrics. This demonstrates our method’s robustness and generalization capability across diverse settings.

In summary, while reflective regions remain difficult for all existing methods, our approach has the ability to produce coherent global structures. More importantly, our method goes beyond this single challenge to tackle other significant issues, such as low-texture regions, thin structures, and shadow occlusions, resulting in a robust and generalizable solution. We sincerely appreciate your valuable feedback and hope this response adequately addresses your concerns. We also respectfully hope that you recognize the overall contributions and significance of our work.

Thank you for raising this insightful concern regarding the potential impact of depth shift in our method. We would like to clarify that the shift is not critical in our approach due to the specific design of our framework, which inherently reduces the influence of shift. Firstly, our method employs a learnable tri-plane feature field, where the flexibility of the feature representation allows it to adapt to potential shifts in the point cloud. When the point cloud experiences a shift, the tri-plane feature field can dynamically adjust, ensuring that robust geometric features are sampled for each point. The joint learning between the point cloud and the feature field helps to mitigate the impact of potential shift, enabling the model to achieve stable and accurate depth predictions.  Furthermore, by combining multi-scale point clouds with multi-scale tri-plane feature fields, our method further enhances robustness to shift. Moreover, the multi-scale consistency loss ensures that depth predictions remain stable and geometrically consistent across different resolutions, effectively reducing the impact of global or local shift. These design choices eliminate the need for explicitly modeling or minimizing the shift during training.

Therefore, while depth shift is indeed a critical challenge in affine-invariant depth estimation, the specific design of our framework inherently mitigates its impact. Additionally, our quantitative and qualitative results on multiple datasets (KITTI, Make3D, NYUv2, and ScanNet), along with ablation studies on the KITTI and NYUv2 datasets, validate the effectiveness of our proposed method for depth estimation.

We sincerely thank your valuable feedback and hope this clarification addresses your concerns. Please do not hesitate to reach out if further details or analyses would be helpful.

Thank you for your constructive comment regarding the visualization of the reconstructed point clouds. The visualizations of the reconstructed point clouds from various perspectives are presented in the Fig. 12 of the manuscript. We observed that under the three viewing perspectives, the overall structure of point clouds corresponding to different scales is roughly consistent. Although distortion is observed in some areas of the point cloud from the side view, as we responded to shift, the distortion of the point cloud does not affect the sampling of robust geometric features for depth estimation.

Dear Reviewer ypGr,

We sincerely thank you for your valuable feedback and insightful suggestions. As the discussion period is coming to a close, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns or if there are any additional points you would like us to clarify. If our responses have satisfactorily addressed your concerns, we kindly ask you to consider reflecting this in your score.

Once again, thank you for your time and thoughtful input throughout this process.

Best regards,

Authors of submission 243

W1, Q1: Add notations to Fig. 2.

A1: Thanks for your suggestions. We have added all notations including $F^s$, $E^s$, $\bar F^s_{xy}$, $\bar F^s_{xz}$, $\bar F^s_{zy}$, $\tilde F^s_{xy}$, $\tilde F^s_{xz}$, $\tilde F^s_{zy}$ to Fig. 2, which improves the clarity of the manuscript.

W2, Q2: Provide more qualitative results on the remaining three datasets and the reasons for the performance improvements.

A2: Thanks for your suggestion. The visualization results on the remaining three datasets(i.e., Make3D, NYUv2, ScanNet) are presented in Fig. 5, Fig. 6, and Fig. 7, respectively, in the Appendix. In addition, we also provide more qualitative results on the KITTI dataset. The visualization results are presented in Fig. 4 in the Appendix. We highlight challenging areas with white dashed boxes. From these results we can see that the compared methods produce incomplete depth maps (first two rows), inconsistent depth maps (middle two rows), and discontinuous depth maps (last two rows), while our proposed TSF-Depth generates more accurate depth maps, particularly in these regions with low texture, slender structure, shadow occlusion and reflections.

To evaluate the depth improvement brought by the 3D representation, we provide an ablation experiment (i.e., Effects of 3D Scene Field on 3D Geometric Representation) refer to lines 484-529 of the manuscript. The depth estimation results on the KITTI dataset is reported in Table 6, where contains three experimental setting: baseline (row 1), TSF-Depth based 2D geometry features (row 2), and TSF-Depth based on the 3D geoemtry features (row 3). We remove the decoder $\theta_I$ branch and positional encoding branch of TSF-Depth as a baseline framework and then train it on the KITTI dataset. Therefore, the baseline relies on extracting front-view 2D features for depth estimation. Note that we denote the encoder of baseline as $\theta_E^{PT}$. Then, to evaluate the effectiveness of $\theta_E$ in learning 3D representation under our end-to-end self-supervised paradigm, we replace the $\theta_E$ of TSF-Depth with the trained encoder $\theta_E^{PT}$, and then train other modules. In this setup, TSF-Depth based 2D geometry features has same network parameters as TSF-Depth based on the 3D geoemtry features, and the Tri-plane feature field is constructed using 2D representation. As for the last row in Table 6, it uses our complete training. Compared to the baseline (row 1) and the complete TSF-Depth (row 3), we observe that the results of TSF-Depth based on 2D geometry features are worse than both them, due to the subsequent 3D-to-2D mapping requiring 3D geometric representation rather than 2D representation. Therefore, the depth improvement comes from the 3D representation.

To evaluate the depth improvement is not due to an increase in the number of network parameters, we report the parameter complexity of the baseline, TSF-Depth, and some other compared methods in Table 9 of the manuscript. As can be seen from this table, TSF-Depth achieves the best performance with a similar number of parameters to the baseline and most existing depth estimation methods. Therefore, the depth improvement is due to the 3D representation rather than by increasing network parameters.