Dear Reviewer 9uAR:

Thanks for your positive feedback and valuable questions. We address all your comments as follows.

Weakness 1: Sensor and Modality

We specify the modality of depth models and datasets.

(1) Models. Similar to previous monocular depth models, SDDR takes a RGB image as input, without other modalities.

(2) Data. Depth prediction and refinement use various data to train and evaluate, with RGB as input and depth as ground truth. Depth maps are annotated by varied techniques, e.g., CG rendering (TartanAir, IRS), stereo matching (HRWSI, VDW), LiDAR (IBims-1), and Kinect (DIML). These modalities are not used as input for models. We combine these data for experiments as in lines 561-588.

Weakness 2: Two Noise Components.

We illustrate the source and significance of local inconsistency noise $\epsilon_{cons}$ and edge deformation noise $\epsilon_{edge}$ with the analysis and experiments below.

(1) We illustrate the source of our noises in 3 aspects.

Our noises are motivated by vital problems of the task. Prior arts broadly recognize that depth blur and inconsistency are two key problems for the task. E.g., as in GBDF, "depth maps are usually blurry with inaccurate details such as planes and object boundaries." PatchFusion discusses that "we discover that BoostingDepth suffers from scale inconsistencies." The two problems could stem from resolution and receptive field. Low-resolution inference loses detail, while high-resolution prediction leads to inconsistent structures due to limited model receptive field. Our noises model the two dominant degradations between predicted and ideal depth.

Our noises generally conclude failures in prior arts. Previous methods try to solve the above two problems intuitively, e.g., by selecting and merging patches. In contrast, we model refinement by edge deformation and local inconsistency noise. Derivations in Eq.1-4 and Sec.3.1 generally reveal reasons for blur and inconsistency in prior arts. E.g., fusing patches as Boost produces higher $\epsilon_{cons}$ with inconsistent structures. Filtering as GBDF fails to suppress $\epsilon_{edge}$, yielding blurred details. Our noises conclude previous failures and guide our design.

Our noises accurately depict the aforementioned problems. In Fig.F of the rebuttal PDF, we further provide visual results of our noises. Simulated by regional affine transformations, $\epsilon_{cons}$ represents disrupted depth structures. With position-constrained Gaussian distributions (line 538), $\epsilon_{edge}$ accurately depicts missing or blurred edges. Depth errors can be decoupled with our noises. E.g., $\epsilon_{edge}$ is prominent in low-resolution predictions, while $\epsilon_{cons}$ occupies larger ratios for high resolution.

(2) We show the significance of our noises by 3 experiments.

In lines 545-552, PSNR between predictions and ideal depth with our noises exceeds 40 dB across all samples of Middlebury2021, quantitatively proving the significance of our noises among all errors.

In Fig.2, adding $\epsilon_{cons}$ and $\epsilon_{edge}$ accurately depicts the discrepancy between depth prediction and ground truth. Combining the noises broadly covers depth prediction errors.

Fig.D of the PDF proves that the lower the noise, the better the depth quality. Edge errors and noise levels exhibit a positive correlation. The low noise levels of SDDR bring fine-grained depth.

Weakness 3 and Question 1: Motivation Elaboration

Thanks for your insightful advice! We further discuss motivations of our designs, which will be added to our revised paper.

(1) Noise Modeling. As in Weakness 2, our noises model two key problems of the task.

(2) Poisson Fusion. In lines 107-109 and 40-46, Poisson fusion integrates the value and gradient domain of two inputs. In lines 205-207 and 465-475, low-resolution depth tends to be consistent but blurred, while high-resolution depth involves accurate edges but inconsistency. Poisson fusion is potentially well-suited for refinement, merging consistent depth with meticulous edges. We implement the Poisson fusion operator as a learnable refinement network, without relying on external fusion mask and complex parameter setting (line 130). Our method produces consistent and fine-grained depth with strong generalizability.

(3) Self-distillation. In Eq.2, the optimization objective of Poisson fusion serves as the training loss of refinement network. Low-noise edge labels are necessary to guide the model but are unavailable in diverse natural scenes. Given that the refinement network inherently reduces noises and restores details, self-distillation can naturally form. Depth edge representations are generated as labels by coarse-to-fine refinement. When depth maps are better refined, the labels become more noise-free.

Weakness 4: Comparisons with Previous Loss

Our edge-guided gradient loss differs from prior losses [1, 2] in 4 aspects.

(1) Supervision Paradigm. Previous losses[1, 2] work in a fully-supervised manner with edges from depth ground truth. Our loss serves for self-distilled learning with depth edge representations. As in lines 33-36 and 91-93, in some natural-scene data, edges from ground truth are unreliable. Our loss establishs accurate edge guidance for self-training.

(2) Classification vs. Regression. Wang et al.[1] adopt cross-entropy loss with hard edge labels, classifying whether a pixel belongs to edge area. Our loss guides the model to learn soft edge representations by regression, enforcing both edge area and intensity precisely.

(3) Global vs. Local. Qiao et al.[2] match gradients globally, while our loss works on high-frequency local regions $P_n$ (line 197). Our model improves details in $P_n$ and preserves consistency in flat areas.

(4) Scale and Shift Alignment. Qiao et al.[2] ignore depth scale ambiguity, whereas we perform scale and shift alignment to maintain consistency.

Dear Reviewer DKoq:

Thanks for your valuable feedback. We address all your questions as follows.

Weakness 1: Dependence on Initial Depth

SDDR achieves strong robustness regarding the quality of initial depth, noticeably improving depth edges and details, even when faced with low-quality initial depth. Our robustness is brought by the self-distillation paradigm. GBDF relies on initial depth with filtering as labels, inheriting errors and noises. SDDR conducts coarse-to-fine refinement for accurate edge representation in self-distillation, acquiring strong robustness to correct and refine.

Fig.7 and lines 274-276 of the paper have proved our robustness by the curves of $\delta_1$ with higher noises of initial depth. Here, we conduct three more experiments for further demonstration.

(1) Simulation Experiment. In Fig.A, we gradually degrade initial depth. Despite increasingly noisy initial depth, SDDR maintains meticulous edges and details, as proved by edge-specific ORD metric and visual results. When initial depth degrades, SDDR only exhibits negligible fluctuations, bringing higher relative improvements.

(2) Real Samples. In Fig.B, GBDF inherits noises and errors from low-quality initial depth, while SDDR robustly corrects the errors, restores the structures, and improves the details.

(3) More Advanced Depth Predictors. SDDR can work with various predictors. Using more advanced predictors, e.g., the Depth-Anything, brings improvements without extra effort. In Fig.C, with better depth of Depth-Anything-V2, SDDR further boosts edges and details.

Weakness 2: Smoothing Out Details in Noise Reduction

In all our experiments, SDDR does not smooth out edges. Across various depth predictors and data, SDDR robustly enhances edges and details, as proved by visualizations and edge metrics (ORD and D3R) in Tab.1-4, 6-9, Fig.1, 3-6, 8, and 14-15 of paper and appendix.

The lower the local inconsistency noise $\epsilon_{cons}$ and edge deformation noise $\epsilon_{edge}$, the better the edges and details. Fig.D proves this point. We represent noise levels by adding the noise intensity of $\epsilon_{cons}$ and $\epsilon_{edge}$ as lines 537-544 of appendix. In Fig.D, edge errors and noise levels exhibit a positive correlation. We achieve lower noise level and better edge quality.

Weakness 3: Comparison with Depth-Anything-V2

SDDR noticeably improves edges and details with Depth-Anything models as depth predictors. As mentioned by Reviewer DKoq, refinement differs from recent Depth-Anything. SDDR is a plug-in module to refine initial depth, with much fewer FLOPs and params than depth predictors. Thus, in Fig.C, we adopt Depth-Anything models as predictors. SDDR also produces finer edges and details. Fig.D quantitatively shows our better edge quality and lower noise level.

Question 1: Iteration Number S

In Tab.A, S=4 and 5 only produce subtle improvements than S=3. Edges and details are sufficiently refined after S=3. Thus, edge metrics ORD and D3R are saturated for S=4 and 5. More iterations lead to higher time costs. We adopt S=3 for trade-off between efficiency and performance.

Question 2: Edge-specific Measurements

We follow Boost, Kim et al., and GBDF to adopt the common edge-specific measurements ORD and D3R for depth refinement, as stated in line 241. We report the metrics on 5 benchmarks throughout our tables. The metrics are further depicted in lines 606-613 of appendix.

Limitation 1: Different Noises and Degradations

Our local inconsistency and edge deformation noise are general and effective representations of depth prediction errors in diverse scenarios, rather than being specific to particular cases. Reasons are presented in four aspects.

(1) Domain Knowledge. Prior arts broadly recognize that depth blur and inconsistency are two key problems for the task. E.g., as in GBDF, "depth maps are usually blurry with inaccurate details such as planes and object boundaries." PatchFusion discusses that "we discover that BoostingDepth suffers from scale inconsistencies." However, prior arts only try to solve the problems intuitively, e.g., by selecting and merging patches, leading to unsatisfactory results and limited generalizability.

(2) Theoretical Derivation. In contrast, we model refinement by noisy Poisson fusion with edge deformation and local inconsistency noise. Eq.1-4 and Sec.3.1 of our paper provide derivations of our method, which generally reveal reasons for blur and inconsistency of prior arts. E.g., fusing patches as Boost produces higher local inconsistency noise, leading to inconsistent structures. Filtering as GBDF fails to suppress edge deformation noise, yielding blurred details. Noisy Poisson fusion and the noises generally conclude failures of prior arts and guide our design.

(3) Experiments. In Fig.2 of the paper, combining $\epsilon_{cons}$ and $\epsilon_{edge}$ accurately depicts depth error. In lines 545-552 of appendix, PSNR between predictions and ideal depth with our noises exceeds 40 dB across Middlebury2021, showing that $\epsilon_{cons}$ and $\epsilon_{edge}$ generally and accurately represent depth error.

(4) Model Generalization. SDDR is designed under the guidance of Noisy Poisson Fusion, aiming to suppress the 2 noises. Our state-of-the-art performance on 5 datasets, with synthetic and real-world, indoor and outdoor, dynamic and static scenes, shows our strong model generalizability, which also proves that our noises are broadly effective and not limited to particular cases.

Limitation 2: Corner Case

We show corner cases in Fig.B, involving complex details, challenging structures, and light overexposure. All compared methods cannot produce completely satisfactory results, e.g., noisy leaves with blurred edges and inconsistent roof with a black hole. However, SDDR still significantly improves edges and details over LeReS and GBDF, refining initial depth effectively.

Dear Reviewer Ca4W:

Thanks for your positive feedback and valuable questions. We address all your comments as follows.

Weakness 1 and Limitation 1: Applications and Limitations

(1) Applications. SDDR produces accurate depth with meticulous edge and consistent structure, suitable for various applications such as style transfer, bokeh rendering, and 3D reconstruction. In Fig.E, we achieve better detail and structure over GBDF for the applications.

(2) Limitations. For complex and meticulous structures in Fig.E and Fig.B, all compared methods cannot produce perfect depth, e.g., some blurred leaves in the first row, leading to incomplete structures in generated images. However, for the depth maps and applications of these cases, SDDR still achieves significantly better edges and details than prior arts. This also underscores the necessity of the depth refinement task to refine edges and consistency of depth prediction models.

Weakness 2 and Question 8: Highly Noisy Edges

SDDR achieves strong robustness against noises, noticeably improving depth edges and details, even when faced with highly noisy initial depth. Our robustness is brought by the self-distillation paradigm. GBDF relies on initial depth with filtering as labels, inheriting errors and noises. SDDR generates accurate edge representation in self-distillation, acquiring robustness to correct and refine.

Fig.7 and lines 274-276 have proved our robustness by the curves of $\delta_1$ with higher noises of initial depth. Here, we conduct two more experiments for further demonstration.

(1) Simulation. In Fig.A, we gradually degrade initial depth. Despite increasingly noisy initial depth, SDDR maintains meticulous edges and details, as proved by ORD metrics and visual results. When initial depth degrades, SDDR only exhibits negligible fluctuations, bringing higher relative improvements.

(2) Real Samples. In Fig.B, GBDF inherits noises and errors from noisy initial depth, while SDDR robustly corrects the errors, restores the structures, and improves the details.

Weakness 3: More Detailed Ablation of Each Component

As per your advice, in Tab.B, we further ablate and prove the contribution of each component within SDDR. Since depth refinement aims to improve depth edges, following previous Boost, Kim et al., and GBDF, edge-specific metrics D3R and ORD are our main focus.

(1) By Components. Starting from depth predictor MiDaS, in each row, we involve one more component and discuss the relative performance compared to the previous row. The self-distillation with depth edge representation and coarse-to-fine refinement reduces the edge error D3R by 8.4% and 5.0%. The edge-guided gradient loss and edge-based fusion loss also reduce D3R by 4.8% and 3.2%.

(2) In Total. Combining all components, SDDR reduces edge errors D3R and ORD by 23.0% and 8.2% than MiDaS. For overall depth quality, SDDR decreases depth error REL by 6.2%. This is noteworthy and proves our strong efficacy, since edges only occupy a very small proportion of images.

Question 1: Smoothing the Motivation (Line 107)

Thanks for advice. We will include discussions below to smooth our motivation for Poisson fusion.

Poisson fusion integrates values and gradients of two inputs. For depth refinement, low-resolution depth is consistent but blurred, while high-resolution depth involves accurate edges but inconsistency. Poisson fusion is potentially well-suited for our task, merging consistent depth with meticulous edges.

Question 2: Edge Deformation Noise

We apologize for causing misunderstanding of the noise, which actually does not represent deformed objects. Instead, it showcases misalignments of predicted and ideal edges, e.g., missing, broken, or blurred depth edges. In Fig.F, we visualize our two noises separately. Edge deformation noise accurately depicts missing or blurred edges, while local inconsistency noise represents inconsistent depth structures. We will revise the paper with the illustrations to avoid misunderstanding.

Question 3: Degradation in Depth-Anything-V2

In Fig.C, Depth-Anything models also produce blurred edges or missing structures. SDDR further refines their predictions with better depth edges and details.

Question 4: Synthetic and Real-world Data

Differences between synthetic and real data are in lines 28-36 and 87-98. Similar to insights in prior arts, real-world data contains diverse scenes but suffers from sparse, blurred, or inaccurate depth. Synthetic data is rendered under full control with accurate depth but limited scenes. Thus, MiDaS, DPT, and DepthAnything combine the two types to balance accuracy and generalization. SDDR is not limited by data formats, since it generates accurate edge representation for self-distillation.

Question 5 and Question 7: Significant Performance, Training Randomness, and Parameters

(1) As proved throughout our figures and tables, SDDR significantly improves depth edges and details over prior arts, which is further demonstrated in Weakness 3 by each component.

(2) Our superior performance is not caused by training randomness. In Tab.C, we train SDDR 12 times. Our performance is stable only with minimal fluctuations.

(3) SDDR is not sensitive to parameters. For training and method parameters, in Tab.C, SDDR remains stable with varied learning rates, loss ratios, training epochs, iteration numbers, overlapping ratios, etc. For other issues, e.g., optimizer and initialization, we follow the prior Boost and GBDF.

Question 6: Training Time

SDDR and prior GBDF have similar training time (12 hours for 3 epochs on a A6000 GPU).

Question 9: Real-time Processing

Pursuing highly fine-grained depth, current refinement methods can't be real-time (25 fps). However, in Fig.1(b) and Tab.5, our SDDR achieves the best efficiency among all compared methods. For real-time applications, pruning and quantization can be used to further improve efficiency.

Dear Reviewer Ca4W,

We would like to express our sincere gratitude for the time and effort you dedicated to reviewing our manuscript. We truly appreciate that you have positively recognized the strengths of our work, including the “Innovative Approach”, “Robust Framework”, “Comprehensive Evaluation”, “Higher Efficiency”, and “Strong Generalizability”.

Besides, to carefully address all your questions, we have conducted comprehensive experiments and analyses in the rebuttal. The raised questions help us to further improve the comprehensiveness and clarity of our paper. Through the comments of Reviewer DKoq and 9uAR, we are glad to know that our rebuttal solves all their concerns.

We also hope that your valuable questions can be answered and resolved properly. If you have further questions, please use the official comment to propose and discuss. You can also present your rating after evaluating our responses. Thanks again for your meticulous review and suggestions on our paper.

Best Regards,

Authors of Paper 1051