UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

This paper studies pre-training for 3D perception in the context of autonomous driving. They present UniPAD, a novel self-supervised learning paradigm applying 3D volumetric differentiable rendering. UniPAD implicitly encodes 3D space, facilitating the reconstruction of continuous 3D shape structures and the intricate appearance characteristics of their 2D projections. The flexibility of their method enables seamless integration into both 2D and 3D frameworks, enabling a more holistic comprehension of the scenes. They manifest the feasibility and effectiveness of UniPAD by conducting extensive experiments on various downstream 3D tasks.

The paper introduces a 3D representation pre-taining framework that makes use of 3D volumetric differentiable rendering decoder. This technique is not contrastive, and allows for continuous reconstruction. It also is easily adaptable to many modalities, e.g. 3D LiDAR point clouds, and multi-image 2D views.

For point clouds, the corresponding encoders would be VoxelNet, while images are convolutional. These modality specific features are transformed into a voxel space, followed by a projection layer to enhance voxel features. These voxel features then undergo block-wise masking. That is, region removal for point-clouds and sparse convolution for images.

UniPAD then convert these modalities to a unified 3D volumetric representation to preserve as much of the original information as possible. The model then introduce a novel use of neural rendering to flexibly incorporate geometry or textural cues into the learned voxel representation. UniPAD uses Signed Distance Function (SDF) to integrate color and geometry (sampled depth along the ray) features.

To alleviate computational rendering requirement, the model applies memory-efficient ray sampling (dilated, random, and depth aware from LiDAR information). This optimizes precision of neural rendering by concentrating on most relevant segments within scenes.

Overall the model is trained on color loss and depth loss.

UniPAD is tested on NuScenes against a few STOA techniques. It is able to improve the baseline on 3D object detection and semantic segmentation when compared to different image or point cloud pre-training techniques, transform strategies, and input modalities. In the ablation study, it is shown that the depth-aware sampling is the most effective, and that the projection layer is critical.

Overview: UniPAD provides a self-supervised training framework for autonomous driving neural networks. It requires a dataset containing camera images, depths, and/or lidar data.

Method: Images and/or pointclouds are block-wise masked, then passed through a feature extractor, and then projected/mapped into a unified voxel grid space. A neural network renderer then renders the voxel grid features into rgb and depth images. The renderer works by marching along the ray for each pixel, while trilinearly interpolating the voxel grid features at each step, and passing them to a SDF MLP and an RGB MLP. The SDF is used to decide the opacity value for the rendering equation. Finally, an L1 loss is applied on sampled rays from the rendered RGB and the rendered depth.

Results: The authors show that on nuscenes 3D object detection and 3D semantic segmentation, pre-training with UniPAD noticeably improves mAP and NDS metrics and surpasses state-of-the-art numbers. They also show that UniPAD outperforms several other forms of pretraining, and that it consistently improves results regardless of the view transform, input modalities, and backbone size. They also show in their ablation studies how mask ratio, decoder depth, decoder width, choice of renderer, sampling strategy, projection, and using pre-trained components impacts UniPAD's performance.

In this paper, the authors propose a self-supervised learning approach for 3D object detection and 3D semantic segmentation on LiDAR point clouds and multi-view images. The main idea is to i) first mask out certain voxelized LiDAR point clouds or multi-view images, and ii) reconstruct the masked regions for representation learning.

The mask encoder adopts block-wise masking to obtain masked inputs, where the masked points and pixels are removed or ignored during SparseConv processing. To establish a unified representation for both point clouds and images, the authors propose to convert both modalities to the 3D volumetric space.

To embed the geometrical information in the voxel fields, the authors resort to neural rendering, where a scene is represented as an implicit signed distance function (SDF) symbolizing the 3D distance between a query point and its nearest surface. The major implementations of such an approach were inspired by [R1], while concern about memory consumption was raised since the objective of the above SDF is to render a driving scene of a relatively large range. To reduce memory consumption, three ray sampling strategies are proposed, which conduct dilation, random, and depth-aware samplings, respectively.

The authors conduct some experiments on the nuScenes dataset for self-supervised with point clouds, multi-view images, and both. The experiments show that the proposed approach can bring certain degrees of improvement over the randomly initialized baselines.