DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation

Thanks for your effort for reviewing our paper and giving some insightful suggestions. We hope the following responses could solve your concerns.

Q1. Discussion about different depth estimation methods.

In the Sec. A of our appendix, we have conducted experiments to explore the effect of the depth maps generated by different methods. Specifically, we adopt the Adabins (trained on 26K pairs on KITTI) and more advanced Omnidata (trained on 4M RGB-D pairs, adopted in MultiMAE), where the Omnidata uses more training data and obtained more advanced zero-shot capacity. The estimated depth maps of the two methods are shown in the Fig.10. After finetuning, the DFormer-B that pretrained with different depth maps achieves the same performance, i.e. 55.6. This indicates the quality of depth maps generated by recent depth estimation methods has little effect on the performance of DFormer. Besides, we will also explore replacing the depth with other modalities and analyze the influence of different estimation methods on our model in the future.

Q2. About the open source of DFormer.

In the supplementary materials, we have provided the source code for the finetuning. Moreover, all estimated depth maps, pretrained weights, and source code will be made publicly available.

Q3. Observation to support the argument within the introduction.

As stated in [1], the mean and variance of the BN layer can reflect the overall distribution center of input data and the distribution breadth of input data, respectively. Fusing RGBD features within the RGB-pretrained models would bring in drastic changes in the feature distribution, which makes the previous statistic of batch normalization incompatible with the input features. To verify whether this phenomenon exists, we visualize the statistics of batch normalization in a random layer of the DFormer in Fig. 11 in the new revision. For the RGBD-pretrained DFormer, the running means and the variance of the BN layer only have slight changes after finetuning, illustrating the learned RGBD representation is transferable for the RGBD segmentation tasks. While, for the RGB-pretrained DFormer, the statistics of the BN layer are indeed drastically changed after finetuning, indicating that the encoding is mismatched. The situation causes difficulties in optimizing the RGB-pretrained weights for the RGBD scenes. Following your suggestion, we also visualize some features of the DFormer that use the RGB and RGBD pretraining in Fig. 12 in the new revision.

[1] An empirical study of adder neural networks for object detection. In NeurIPS, 2021.

Q4. Comparison to ACNet.

Thanks for your suggestion. We add this method in Tab.1 of the new revision.

Q5. Discussion on the generalization of DFormer.

In this paper, the DFormer is endowed with the capacity to interact the RGB and depth during pretraining. To verify whether the interaction still works when replacing the depth with another modality, we apply our DFormer to some benchmarks with other modalities, i.e., RGB-T on MFNet (IROS, 2017) and RGB-L on KITTI-360 (TPAMI, 2022), as shown in the below table. As can be seen, RGBD pretraining still improves the performance on the RGB and other modalities, but the improvement is limited compared to that on RGBD scenes. To address this issue, a foreseeable solution is to further scale the DFormer to other modalities, and there are two ways worth trying, i.e., synthesize the pseudo modal data, and separate pretraining on a single modality dataset. For the former, there are some generation methods to generate other pseudo modal data. For example, Pseudo-lidar[2] proposes a method to generate the pseudo lidar data from the depth map. And N-ImageNet [3] obtains the event data on the ImageNet. Besides, collecting data and training the modal generator for more modalities, is also worth exploring. For the latter one, we can separately pretrain the model for processing the supplementary modality and then combine it with the RGB model. We will attempt these methods to bring more significant improvements for DFormer on different multimodal scenes.

'*' means the result is not provided and we implement them via their official code.

[2] Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. CVPR, 2019. [3] N-imagenet: Towards robust, fine-grained object recognition with event cameras. CVPR, 2021.

Thanks for appreciating our paper and raising your rating. Following your suggestions, we added the analyses of the RGB-D pretraining to other methods and the results on MFNet/KITTI360 to the main paper of the new revision.

Q6. Apply the RGB-D pretraining to other methods.

Following your suggestion, to make the comparison more fair, we pretrain the CMX (MiT-B2) on RGB-D ImageNet and it obtain about 1.4% improvement on the mIoU. We also provide the RGB pretrained DFormers to provide more insights. Under the RGB-only pretraining, DFormer-L still outperforms the CMX (MiT-B2) by about 1.1% with less computation cost. For the RGB-D pretraining, compared to the CMX (MiT-B2), the improvement of DFormer-L is further enlarged from 1.1% to 1.5%, which we attribute to that the pretrained fusion weight within DFormer can achieve better and more efficient fusion between RGB-D data.

Q7. More detailed ablation towards the block.

In Tab.5~8, we have provided the ablation experiments about the components of our RGB-D block, the pooling size of our GAA, as well as the fusion manners in GAA and LEA. We notice that the modules that only encode RGB features may be not well ablated. Here we provide more results for the modules that encode RGB features. Due to limited time and computation resources, we use a short pretraining duration of 100 epochs on the DFormer-S. In the future, we will add this ablation experiment of full pretraining duration to the main paper. Note that the results in the below table only ablates the structure within the modules (gray part in Fig.11) that only process the RGB features.

Reviewer zqyp has raised an interesting question about the setting using RGB pretrain + D initialization for DFormer. I am also curious to know that. ​

Besides, I agree that it would be interesting to investigate the effects of using different depth estimation methods under various measurement noises.

Thanks for your effort for reviewing our paper and giving some kind suggestions. We are happy to see your positive comments on writing, experiments and method. We hope the following responses could solve your concerns.

Q1. Discussion about different depth estimation methods.

In the Sec. A of our appendix, we have conducted experiments to explore the effect of the depth maps generated by different methods. Specifically, we adopt the Adabins[1] (trained on 26K pairs on KITTI [2]) and more advanced Omnidata[3] (trained on 4M RGB-D pairs, adopted in MultiMAE), where the Omnidata uses more training data and obtained more advanced zero-shot capacity. The estimated depth maps of the two methods are shown in the Fig.10. After finetuning, the DFormer-B that pretrained with different depth maps achieves the same performance, i.e. 55.6. This indicates the quality of depth maps generated by recent depth estimation methods has little effect on the performance of DFormer. Besides, we will also explore replacing the depth with other modalities and analyze the influence of different estimation methods on our model in the future.

[1] Adabins: Depth estimation using adaptive bins. In CVPR, 2021.

[2] Vision meets robotics: The kitti dataset. IROS 2013.

[3] Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV 2021.

Q3. Pretraining the CMX.

Thanks for your suggestion. To make the experiments more comprehensive, we use the pretraining manner of DFormer to pretrain the CMX (MiT-B2) and then finetune the model via their official code. As shown in below, the pretraining manner in this paper brings about a 1.4% improvement for the CMX (MiT-B2), but our DFormer-L still outperforms CMX (MiT-B2) that adopts the same pretraining manner by a large margin. We attribute this to the well-designed building block and the efficient encoding design of our DFormer.

Q4. DFormer with RGB pretrain + D initialized

To provide more insights, we pretrain the RGB part via RGB-only pretraining and randomly initialize the depth part of the DFormer, and the results are shown in the below table. Under the RGB-only pretraining, DFormer-L still outperforms the CMX (MiT-B2) by about 1.1% with less computation cost. For the RGB-D pretraining, compared to the CMX (MiT-B2), the improvement of DFormer-L is further enlarged from 1.1% to 1.5%, which we attribute to that the pretrained fusion weight within DFormer can achieve better fusion between RGB-D data.

Q5. DFormer with different decoders.

Considering that the RGB-D interaction has been achieved within each block of our DFormer encoder, there is no need to specifically design extra fusion modules within the decoder. Thus, we follow an RGB segmentation method SegNeXt (NeurIPS, 2022) to use the Ham head as the decoder. To better understand the effect of our DFormer, we further provide the results of DFormer-B with different decoder heads in the below table.

Q6. About the feature fusion?

In our DFormer, the interaction between the RGB and depth features is performed within each building block of the encoder. As stated in SegNext (NeurIPS, 2022), the features from Stage 1 contain too much low-level information and cannot bring performance improvement， as shown in the table below. Therefore, we only use the features from the last three stages for the prediction.

In Table 1, the authors compared the proposed method against methods with open code implementations.

Regarding [4] UCTNet, while it reaches high accuracy using uncertainty-aware self-attention, it does not release its source code and models. I think it is fine to only compare against methods with open code. The authors are suggested to discuss UCTNet in the related work section.

Thanks for your effort for reviewing our paper and giving some kind suggestions. We hope the following responses could solve your concerns.

Q1.About the architectural design

The main goal of this paper is to provide a new scheme for the acquisition of transferable representations for RGB-D segmentation tasks. Thus, the focus of our architectural design is how to achieve such RGB-D pretraining in an efficient way, instead of building complicated modules. To this end, we take the interaction between RGB-D into the attention mechanism to achieve efficient fusion within each building block. Besides, we observe that the depth information only needs a small portion of channels to encode, rather than a whole RGB pretrained backbone as done in previous works. Based on these designs, even with a simple decoder head, i.e., Ham head, MLPs heads, our DFormer can achieve state-of-the-art performance, as shown in Q3.

Q2.Comparison with more methods

In the main paper, we have conducted comparative experiments with 13 RGB-D semantic segmentation methods and 11 RGB-D salient object detection methods to verify the effectiveness of our RGB-D pre-training manner. Thanks to your suggestions, we add the mentioned three RGB-D SOD methods, namely VST, RD3D, and BPGNet in Tab.2 of the new revision. Our DFormer still outperforms these methods by a large margin. Note that we fail to compare with UCTNet (ECCV,2022) due to its unopened source code, which is also pointed out by reviewer KRMX.

Q3.Why not replace the backbone with the DFormer in previous methods

The previous works mainly focus on using the interaction between RGB-D data within the encoder and decoder, e.g., CMX interacts the RGB-D features at each stage in the encoder and further fuses them at the decoder. In contrast, our DFormer only performs the efficient interaction between RGBD modalities in each building block of the encoder, which not only reduces the computation burden but also achieves better information fusion. We conduct experiments on DFormer-B with different decoder heads in the below table. We also apply our DFormer-B to the CMX decoder, improving the performance by about 1.1% on mIoU and reducing nearly half the computation cost. MLPs head achieves comparable performance with CMX decoder, further proving there is no need to fuse RGB-D features by designing fusion modules.

Q4. Discussion on pretraining methods

This paper focuses on the RGB-D pretaining in a supervised manner, thus the discussion of the two self-supervised works you mentioned is out of the scope of this paper. Besides, their performances are far behind the state-of-the-art supervised pretrained methods like CMX. As a result, we do not take these two methods into account.

Q5. More RGB-D SOD benchmarks

This paper focuses on the acquirement of transferable representations for RGB-D segmentation tasks, rather than refreshing state-of-the-art performance on all the RGB-D benchmarks. To this end, we conduct experiments on several popular RGB-D segmentation benchmarks and RGB-D SOD benchmarks following the settings of recent works like HiDANet(TIP,23)[1], SPSN(ECCV,22)[2],and SPNet (ICCV,21)[3].

Thanks to your suggestions, we further conduct experiments on the mentioned RGB-D SOD datasets in Tab.19 of the revised appendix. We report the brief results in the following table, and our DFormer still outperforms other methods significantly.

'-' means the results are not provided in the paper. We will reproduce these missing results in the next revision to make the comparison more comprehensive.

[1]HiDANet: Rgb-d salient object detection via hierarchical depth awareness. TIP,2023.

[2]Spsn: Superpixel prototype sampling network for rgb-d salient object detection. ECCV,2022.

[3]Specificity-preserving rgb-d saliency detection. ICCV,2021.

[4]Depth-induced multi-scale recurrent attention network for saliency detection. ICCV,2019.

[5]Learning selective mutual attention and contrast for rgb-d saliency detection. TPAMI,2021.

[6]Saliency detection on light field. CVPR, 2014.

Dear Reviewer HRuY,

Thank you very much for your valuable feedback. We hope that we have addressed your concerns through more comprehensive comparisons for more RGB-D SOD methods and benchmarks, the application of our DFormer to different decoders like the decoder of CMX, and the revision of the paper according to your suggestions.

The discussion period is soon coming to an end. If you still have any further reservations or suggestions, please don't hesitate to share them. Your insights are invaluable to us, and we're keen to address any remaining issues.

Best regards!

Authors