Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers

Thank you for your positive feedback and recognition of our detailed theoretical and experimental validation, innovative approach, and practical potential. We've addressed all your comments in detail below.

Q1: Filtering depth.

Thank you for the thoughtful question. While post-processing techniques, such as filtering depth based on image gradients, may offer some mitigation, they are often ineffective in handling severe flying-pixel artifacts. Moreover, such techniques typically smooth or replace flying pixels using neighboring values, which can result in edge blurring or structural distortion.

In contrast, our model addresses flying pixels directly during the prediction stage, enabling it to preserve sharp object boundaries and produce structurally accurate and reliable depth maps, without relying on heuristic post-hoc fixes.

Q2: Monocular metric depth estimation.

Our method can be extended to metric depth estimation using metric depth normalizing techniques proposed in Metric3D [Yin et al., ICCV 2023] and Depth Pro [Bochkovskii el al., ICLR 2025]. Specifically, our method could serve as backbone network to predict metric depth normalized by focal length, obtaining metric depth without flying pixels. This could be one of the interesting future works.

Q3: Performance in extremely detailed regions.

Our method performs well in fine-detail regions such as hair structures, which is one of the core strengths of diffusion-based generative models, as they are particularly effective at modeling high-frequency details. We have visualized hair details similar to those in Depth Pro. Due to the limitations of the rebuttal format, we are unable to include these qualitative visualizations here, but we will ensure that they are incorporated into the final version of the paper.

Thank you for your very positive feedback! We truly appreciate you highlighting the well-motivated design of our method, its novel solution to the flying pixel issue, and the significant contribution of our new edge-aware evaluation metric. We've addressed all your comments in detail below.

Q1: Point cloud visualizations for ablation analysis.

Thank you for the valuable suggestion. We have visualized the corresponding point clouds for the ablation studies (with and without SG-DiT) to assess the impact on flying-pixel artifacts. Our SG-DiT not only effectively reduces flying pixels but also better preserves the global structure of the scene. Due to space limitations in the rebuttal, we are unable to include these figures here, but we will ensure that these visualizations are included in the final version of the paper.

Q2: Ablation studies on edge-aware evaluation metric.

Thank you for the insightful suggestion. We have already included ablation studies using the proposed edge-aware evaluation metric, as shown in the table below.

Experimental results demonstrate that our proposed components consistently reduce flying pixels and enhance the structural integrity of the reconstructed point clouds. Specifically, SG-DiT improves the Chamfer Distance from 0.13 to 0.09, while the full model further improves it to 0.08, confirming the effectiveness of our design in mitigating boundary errors.

Thank you for your positive feedback and recognition of our work’s novelty, simplicity, and effectiveness. We address your questions as follows.

Q1: Simple geometry integration achieves SOTA, it is unclear if more efficient or better alternatives than normalize + concat exist.

We observed that geometry features (semantic features) have much larger magnitudes than diffusion features, which caused training instability and frequent NaNs when fused directly. To address this, we normalize the geometry features to match the scale of diffusion features. This simple strategy stabilizes training and achieves strong performance. We also experimented with more complex alternatives, such as learnable scaling or adaptive fusion weights, but they did not bring clear performance gains.

Q2: Improvement over Depth Anything v2.

The main goal of our work is to address the flying pixels problem commonly observed in discriminative models such as Depth Anything v2. As shown in Figures 1 and 4 and Table 4 of our paper, our model significantly reduces flying pixels. While standard benchmarks and metrics (e.g., Table 1 in our paper) typically fail to assess flying pixels at object edges, we propose an edge-aware point cloud metric to quantify flying pixels specifically at edges. This is explained in Section 4.4 of the paper. As illustrated in Table 4 of our paper, our method achieves a Chamfer distance of 0.08, which is a substantial improvement compared to 0.18 from Depth Anything v2.

Q3: Analysis of efficiency.

We provide a quantitative comparison of inference speed against other representative models. For a fair comparison, all models are evaluated on an RTX 4090 GPU with an input resolution of 512×512. As shown in the table below, our model is slower than Depth Anything v2, primarily due to the multi-step nature of diffusion-based inference. To offer a clearer view of the computational overhead, we report the runtime of each component: the semantics encoder takes 10 ms, and each diffusion iteration takes approximately 32.5 ms. With 4 iterations, the total inference time sums to 140 ms.

Q4: Deeper theoretical insight.

Our work is grounded in a principled analysis of the flying pixels problem commonly found in discriminative models such as Depth Anything v2 and Depth Pro, which tend to predict averaged depth values at depth-discontinuous edges due to the objective of minimizing regression loss. In contrast, our pixel-space diffusion model estimates pixel-wise depth distributions, allowing it to preserve sharper edges and avoid flying pixels. Unlike existing diffusion-based methods that rely on latent-space Stable Diffusion and suffer from VAE-induced flying pixels, our model avoids latent compression entirely.

While our model effectively eliminates flying pixels, it introduces new optimization challenges due to the increased difficulty of learning in pixel space. To improve convergence, we integrate geometry priors into the diffusion process. This innovative solution significantly enhances convergence and contributes to the strong performance of our model.

Our design reflects the underlying modeling challenges and contributes new insights into how diffusion can be effectively applied to structured prediction tasks such as depth estimation.

Q5: Inference time for 512x512 images.

The inference time for a 512×512 image is 140ms, as shown in Table 2 of the paper.

Q6: Normalizing both geometry and diffusion features.

We conducted an ablation study to evaluate the effect of normalizing both geometry and diffusion features. As shown in the table below, doing so yields no clear performance improvement. In fact, normalizing geometry features alone already provides stable training and effective integration with diffusion features.

We sincerely thank you for recognizing the originality, methodological rigor, and practical significance of our work. We address your questions as follows.

Q1: Computational overhead.

We will release both the pretrained models and code publicly, allowing users to directly use our model without retraining. In addition, we will release a smaller variant (PPD-Small, as shown in the table below) of our model that can be trained on consumer-grade GPUs (e.g., 4×4090), making it accessible to small labs.

Q2: Training dataset limitations.

Benefiting from the semantic priors provided by the vision foundation model, our method achieves strong generalization even when trained only on the synthetic Hypersim dataset. Similarly, prior work such as Marigold has also shown that diffusion priors can enable strong generalization when trained on the synthetic dataset alone.

Q3: Multi-step diffusion leads to slower inference.

To significantly reduce inference time, we have developed a smaller variant (PPD-Small) of our model presented in the paper (PPD-Large). As shown in the table below, PPD-Small achieves significantly faster inference than Depth Pro and is only slightly slower than Depth Anything v2.

As shown in the table below, PPD-Small achieves comparable accuracy to the larger PPD-Large model presented in our paper.

Q4: How does the model perform with depth data from consumer-grade devices?

Our model performs well on images captured by consumer-grade cameras, as demonstrated in Figure 5 of our paper using the NYUv2 dataset, which was collected with a Kinect camera. Since our model only requires a single RGB image as input, it naturally generalizes to a wide variety of consumer-grade RGB cameras. We will include additional results from other consumer devices, such as iPhone cameras, in the final version of the paper.

Q5: The paper notes flickering depth in video applications. What strategies are planned to integrate temporal consistency?

Our method can be extended to produce temporally consistent depth maps by incorporating strategies such as the temporal alignment technique used in Video Depth Anything (Chen et al., CVPR 2025) or the video diffusion priors introduced in DepthCrafter (Hu et al., CVPR 2025). These strategies can help our model mitigate flickering artifacts and produce temporally consistent depth without flying pixels. This could be one of the interesting future works.

Q6: Are there plans to implement techniques like knowledge distillation or token caching to reduce inference time for edge deployment?

Yes, we have already distilled a lightweight model, PPD-Small, to facilitate faster inference for edge deployment. While our original model, PPD-Large, adopts the DiT-Large architecture, PPD-Small is built upon the more efficient DiT-Small architecture. As shown in the table below, PPD-Small significantly reduces inference time while maintaining competitive accuracy compared to PPD-Large.