Depth-Guided Self-Supervised Learning: Seeing the World in 3D

Figure 1 lacks details. For instance, it is unclear what "2D augmentation" is being performed in PixDepth. It would also be good to visually represent your SSL objective.

We thank the reviewer for these suggestions. 2D Augmentation refers to the standard set of augmentations applied on the 2D image. For instance, random crop, flip, color jitter etc. We don’t show the SSL objective here explicitly because we are using several standard methods for SSL and the SSL algorithm itself is not the focus of our work. Our focus is on extending input with a depth map and using augmentations that correspond to different (inferred) 3D views.

Depth is dropped from the input to encourage the network to not over-rely on it. However, depth is used to compute the error metrics in the results table. In this case, the comparison could be considered unfair, as the previous methods do not have depth as input. As depth is obtained from a supervised network, this method is not a pure SSL method.

We thank the reviewer for this interesting question. We did run an ablation to analyze the performance of the model when depth is not available during inference. An excerpt from the paper (Section 4.5) follows:

“Table 6 reports these results on ImageNette dataset with BYOL. Interestingly, we find that even with the absence of depth information, the accuracy of the model is higher than the baseline BYOL. This indicates that the model has implicitly learned some depth signal and captured better representations. It can also be seen that the performance on IN-3DCC is 1.5% higher than BYOL.”

Most improvements come from adding the depth channel, which I could not consider an important contribution.

We note that 3D Views and the ensemble of Depth + 3D Views also improve performance over state-of-the-art baseline SSL methods, allowing us to argue for the broader claim that depth signals are valuable early in the vision pipeline. Perhaps the reviewer considered the contribution of a depth channel as unimportant because it is such a minor change to the architecture. We agree it is a minor architectural tweak, but wish to emphasize that it is a significant conceptual change: very little work in the field considers computations to extract depth from 2D images to precede object recognition.

What is the reason behind the statement of improvements of SwAV in Table 1 when no results for SwAV are provided?

We do not show results with SwAV + 3D views because the multi-crop setting used in SwAV makes it non-trivial to be adapted to 3D Views. We have mentioned this in the paper but we will update the paper to clarify.

For the results on imagenet-100 no improvement is achieved, it is actually the opposite. This is not clearly reflected in the text in a tricky way. Why are there no results with the combination method (depth + 3D)?

We are not sure if we understand the question correctly. We report results with depth + 3D in Table 2 on ImageNet-100. We do not report results Depth + 3D on ImageNet-1k due to compute constraints. Apologies, we should have mentioned this fact in the article text.

I am afraid that the added robustness to corruptions in ImageNet-C and ImageNet-3DCC comes from the robustness of the depth estimation network (trained with depth GTs over a considerable amount of data).

We begin by emphasizing that the depth-estimation network need not be used during inference. As Section 4.5 (Table 6) indicates, when depth is unavailable at inference, the proposed method is still better than the corresponding baseline SSL method. Regarding the reviewer’s point that the depth-estimation network confers some robustness benefit to the learned embeddings, we certainly agree, although the benefit is indirect through the SSL method. Our investigation was focused on the empirical question of whether such an indirect benefit would indeed be obtained.

Using depth signals as augmentation is not new in SSL. Previous works, e.g., DepthContrast, have explored it thouroughly.

We thank the reviewer for pointing us to DepthContrast [1]. As we understand it, DepthContrast requires ground truth depth information in the form of point-cloud data and relies on format-specific encoders (like PointNet++ for point clouds and UNet for Voxels). In contrast, our method takes a standard 2D RGB image input and assumes only that depth maps can be estimated from a separate module, allowing our method to be seamlessly integrated into standard SSL pipelines. More importantly, the motivation behind the DepthContrast and our method is very different.

We have discussed comparisons with other related work in the Appendix A.

The proposed method lacks generalizbility. Though it can be adopted to any SSL frameworks, the adopted depth estimation model is supervised trained on several datasets. The performance of depth estimation can not gurantee in scenarios that have a huge domain gap compared to the trained datasets.

Depth-Prediction Transformer (DPT) [2] is robust to various natural image datasets as shown in the paper via zero-shot cross dataset transfer. It is due to these reasons that various works in 3D reconstruction [3, 4] and view synthesis [5, 6] use DPT to estimate depth maps.

Also, due to the utilization of the supervised depth-estimation model, it is questionable to claim the proposed method as a SSL framework.

While the reviewer is technically correct, we urge the reviewer to appreciate our underlying premise. We argue that biological agents obtain depth information for free by continuous-time interaction with their environments, and so should a complete, continual learning AI agent. But rather than developing a complex multi-module AI system, we start with the premise that depth maps are available and then ask how they can benefit self-supervised learning. The means by which depth is obtained is not central to our investigation; however, the fact that depth is obtained via extraction from a 2D image (as the human brain does) is key. This work provides insights into and benefits of how to effectively utilize this depth information.

Experiments are all conducted on the subset of Imagenet or the modification of Imagenet. Results on more datasets are expected.

We have reported results on the ImageNet-1k dataset (Table 3) consisting of 1.2 million real-world images. It is common in the SSL literature to report the pretraining results only on ImageNet-1k [7, 8, 9, 10]. We also emphasize that SSL in general is computationally expensive.

[1] Zhang, Zaiwei, et al. "Self-supervised pretraining of 3d features on any point-cloud." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun. "Vision transformers for dense prediction." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[3] Wu, Chao-Yuan, et al. "Multiview compressive coding for 3D reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Liu, Ruoshi, et al. "Zero-1-to-3: Zero-shot one image to 3d object." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Han, Yuxuan, Ruicheng Wang, and Jiaolong Yang. "Single-view view synthesis in the wild with learned adaptive multiplane images." ACM SIGGRAPH 2022 Conference Proceedings. 2022.

[6] Jiang, Yutao, et al. "Diffuse3D: Wide-Angle 3D Photography via Bilateral Diffusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[8] Yeh, Chun-Hsiao, et al. "Decoupled contrastive learning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[9] Shi, Yuge, et al. "Adversarial masking for self-supervised learning." International Conference on Machine Learning. PMLR, 2022.

[10] Dwibedi, Debidatta, et al. "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

The proposed method is evaluated only in small-scale datasets (e.g., ImageNet-100, ImageNet-1k). The proposed method is evaluated only in small-scale models (e.g., ResNet-18, 50).

We have reported results on ImageNet-1k which consists of 1.2 million real-world images. It is a bit shocking to see the reviewer classify this dataset as a small-scale dataset. Even in recent SSL papers [1, 2], Imagenet-1k is the only dataset used for evaluation with a single backbone (ResNet-50)

[Clarity] I recommend the authors to narrow down the scope of the proposed method from general SSL to a specific SSL method. The current backbone tasks RGB-Depth image as inputs, so targeting downstream tasks for RGB-D inputs (e.g., RGB-D segmentation, 3D recon, 3D view synthesis, human pose estimation) is a more reasonable choice to claim the effectiveness of the proposed method. The current claim is too general and insufficient to support the claim.

We respectfully disagree with the reviewer here. As we pointed out to other reviewers, the goal of SSL is to learn representations that are useful for various downstream tasks. It is common in SSL literature to primarily report linear classification accuracy. In our paper, we focused more on understanding the various components of our method (including depth channel dropout, quality of 3d views) rather than evaluating on a range of downstream tasks.

[1] Bardes, Adrien, Jean Ponce, and Yann LeCun. "Vicregl: Self-supervised learning of local visual features." Advances in Neural Information Processing Systems 35 (2022): 8799-8810. [2] Yeh, Chun-Hsiao, et al. "Decoupled contrastive learning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.