DiffSF: Diffusion Models for Scene Flow Estimation

Question 1: Additional visualizations.

We propose additional qualitative results in the attached PDF in the common comment section above. Figure 3 shows the visualization comparison between GMSF and DiffSF on the KITTI dataset. The results show that DiffSF is more robust than GMSF when the point clouds are sparse which are thus more likely to be noisy. This is consistent with our quantitative results that DiffSF shows a large improvement in all the evaluation metrics, especially in robustness ($ACC_S$, $ACC_R$, $Outliers$). Figure 2 shows a visualization comparison between GMSF and DiffSF on the FlyingThings3D dataset. Similarly, DiffSF performs better when the point clouds are sparse (column 1), and when occlusion happens (columns 2 and 3). However, when the occlusion is too severe that the object completely disappears, both models can not give accurate predictions (column 4). Similar results are found in the Waymo dataset (see Figure 4 top row). In the bottom row we show that the proposed DiffSF is robust against various road types on the Waymo dataset, which is a very challenging dataset compared to KITTI, with complicated long sequences, and fast vehicle motions.

Question 2: Which datasets are used in the generation of results for Figures 3 and 4?

The datasets used to generate Figures 3 and 4 are FlyingThings3D. We will add this to the figure caption.

Question 3: Limitations in the main paper.

Thanks for the comment. We will include the limitations in the main paper.

Question 1: Consider providing more visualization of the reverse process other than the only one GIF attached.

Thanks for the comment. We plan to add more visualizations of the diffusion process other than the visualization in Figure 1 in the main paper. However, due to the page limitation of the rebuttal process, here we only provide one additional visualization in the attached PDF in the common comment section above. More will be added to the supplemental document of the final version.

Question 2: The citation notation seems to be wrong. citing of papers should be "[]" instead of "()".

Thanks for the comment. We will revise the citing in the paper.

Question 3: What are the meanings of bold and underlined numbers in Tables 1,2 and 6. This should be explained.

Thanks for noting that the meanings of bold and underlined numbers was not explained in the tables. The following explanation will be added to the tables: The bold numbers represent the best performance, and the underlined numbers represent the second best performance.

Question 4: Definition of "uncertainty".

Good catch! We will include the following definition in the paper: With uncertainty we refer to the epistemic uncertainty, which reflects the confidence the model has in its predictions. In our case, we predict an uncertainty for the prediction of each point. That is, if we have a predicted scene flow vector field of size $N \times 3$, the predicted uncertainty will have a size of $N \times 1$. In practice, each uncertainty is predicted by taking the standard deviation of 20 hypotheses with different initial randomly sampled noise.

Question 1: Limited contribution.

Optical flow and scene flow share similar ideas, i.e. estimating object movement from sensors' data. However, the sensors are completely different and the generated data has different modalities. For RGB cameras, the output images are usually on a regular grid, with RGB values always from 0 to 255, enabling the generation of dense feature maps. On the contrary, for LiDAR sensors, the output point clouds are unstructured and sparse, and the point clouds between consecutive frames could even have different sizes. Due to these inherent differences in data structures, approaches for optical flow estimation cannot be directly transferred to scene flow estimation. For example in [1], the method takes the noisy label and one or more conditioning images as input, the noisy label and the images are concatenated together and then sent into a U-Net. The output of the U-Net is the learned optical flow. Point clouds cannot be processed with CNNs (unless they are densified, which is a separate CV problem) and due to the varying number of points, concatenation over the feature dimension cannot be performed as in [1].

Question 2: The visualizations of source points with added noise at different time steps should be included.

Thanks for the comment. We plan to add more visualizations of the diffusion process other than the visualization in Figure 1 in the main paper. However, due to the page limitation of the rebuttal process, here we only provide one additional visualization in the attached PDF in the common comment section above. More will be added to the supplemental document of the final version.

Question 3: The study is limited to the GMSF model architecture.

We chose GMSF as our baseline as it is the currently most powerful scene flow estimation model, more than 50% better than the second best (DifFlow3D) on $F3D_o$ dataset in every metric. Note that instead of taking GMSF directly, we also made modifications to the GMSF model to make it integrate into the diffusion models better. For more information about the importance of the architecture modifications, please refer to the reply to reviewer gpDD (Question 3). The table shows that by improving the architecture, the $EPE_{3D}$ on $F3D_o$-all improves from 0.061 (line 2) to 0.036 (line 4). The $EPE_{3D}$ on $KITTI_o$ improves from 0.054 (line 2) to 0.029 (line 4), which demonstrates our claim that the modifications we made to the architecture allow for better integration into the diffusion model.

To investigate whether the proposed diffusion model works for self-supervised scene flow estimation, instead of adapting other models into our proposed method, we adapt the proposed method by simply changing the supervision from fully-supervised to self-supervised. The self-supervision consists of two parts: the Chamfer loss and the smoothness loss, which are two common losses employed in self-supervised scene flow estimation.

The Chamfer loss is defined as

$$L_{Chamfer} = \sum_{x_i\in\mathcal{X}} \min_{y_j\in\mathcal{Y}} |\hat{x_i}'- y_j|^2 + \sum_{y_i\in\mathcal{Y}}\min_{x_i\in\mathcal{X}} |x_i - \hat{y_j}'|^2,$$

where $x_i$ and $y_j$ are the points from source and target, $\hat{x_i}'$ and $\hat{y_j}'$ are transformed points.

The smoothness loss is defined as

$$L_{Smoothness} = \sum_{x_i\in\mathcal{X}}\frac{1}{N}\sum_{y_j\in\mathcal{N}(x_i)}\|SF(x_i) - SF(y_j)\|^2,$$

where $\mathcal{N}(x_i)$ denotes the nearest neighbors of $x_i$. $N$ is the number of points in the local region. $SF()$ is the predicted scene flow.

The results are shown in the following table, which shows that our proposed approach also works in the context of self-supervised scene flow estimation. An initial try without parameter tuning already gives us a huge performance improvement on the KITTI dataset. The result on FlyingThings3D is also comparable to state-of-the-art.

Table 1: Ablation study on shared weights on $F3D_o$.

Question 4: Adding noise to the embeddings instead of the vector field.

We thank the reviewer for the novel perspective that we have not considered so far. While VAE models are commonly used in image generation with great success their application to scene flow estimation is not commonly studied. Latent representations of data are often global, making them suitable for higher-level tasks like image generation [5], point cloud generation [6], classification [7], etc. However, scene flow estimation is a rather low-level task that focuses on how each point moves instead of understanding the whole scene. This difference in nature might make using a VAE to encode features of the point cloud to get high-level information not favorable for scene flow estimation. However, We are happy about the suggestion and will consider the option in future research.

Additional References

[4] PointPWC-Net: A Coarse-to-Fine Network for Supervised and Self-Supervised Scene Flow Estimation on 3D Point Clouds

[5] High-Resolution Image Synthesis with Latent Diffusion Models

[6] LION: Latent Point Diffusion Models for 3D Shape Generation

[7] Robust Latent Subspace Learning for Image Classification

Question 1: The organization of the technical part can be improved.

The original idea of the organization was to decouple the introduction of the standard diffusion process and our contribution. We first recap the basics of diffusion models for the audience in Section 3.1. In Section 3.2 we introduce an overview of formulating scene flow estimation as diffusion process, including the training and sampling algorithms. Then we introduce our contribution and give the detailed model architecture and the training loss we employed in Section 3.2 and 3.3. But thanks to the reviewer for pointing out that this part can be difficult to follow. We will clarify the organization in the main paper.

Question 2: Will the shared weight in Feature Extraction and linear layers in the Transformer negatively affect the training?

DiffSF uses a global matching model to predict the noise in the reverse process. Since the core of the global matching approach is to match similar points, it makes sense to use shared-weights feature extractors. However, we agree that the distribution of the (noisy) source point cloud and the (clean) target point might be different, thus, we follow the reviewer's recommendation and try feature extraction without shared weights. The results are given in Table 1 which shows that, while having a lower number of parameters, the model with a shared weight feature extractor performs better than the same model without shared weights.

Table 1: Ablation study on shared weights on $F3D_o$.

Question 3: Which part contributes to the performance improvement, the architecture or the denoising process?

Both the improved architecture and the denoising process contribute to the performance improvement. In Table 2 we improved Table 7 in the supplementary material to make the contributions of different design choices more clear. The architecture is modified to allow for a better integration into the diffusion model. Table 2 shows that when fitting the original architecture of GMSF directly to the diffusion models, there is a performance drop ($EPE_{3D}$ from 0.039 to 0.061). By improving the architecture, the performance of GMSF improves $EPE_{3D}$ from 0.039 to 0.037, and the performance of DiffSF improves $EPE_{3D}$ from 0.061 to 0.036, which is significant and demonstrates that the improved architecture integrates with the diffusion models better. The numbers in parentheses mark the improved percentage of DiffSF compared to GMSF with the same architecture. On the $F3D_o$ dataset, the result shows a more than 10% improvement of $ACC_S$, $ACC_R$, and $Outliers$ if we evaluate all the points. An even higher improvement is shown if we evaluate only the non-occluded points. For the $EPE_{3D}$, there is an improvement of 2.7% over all the points and 6.3% over the non-occluded points. Given the fact that FlyingThings3D is close to saturation, we argue that the relative improvements are more reasonable measurements than absolute improvements. A similar trend is found when testing the generalization ability on the $KITTI_o$ dataset. Note that the models are only trained on $F3D_o$ and then generalized to $KITTI_o$. The new training scheme on $F3D_o$ (number of points 4096 and batch size 24) causes some generalization performance degradation on $KITTI_o$ (from $EPE_{3D}$ 0.033 to 0.089). From Table 3 we can tell that both the improved architecture and the diffusion process contribute to the improvement of the performance. Due to the limited resources, we didn't do an additional ablation study on the Waymo dataset, where we will have to additionally train three different models.

Table 2: Ablation study on $F3D_o$.

Table 3: Ablation study on $KITTI_o$.

Question 4: Overclaimed statements.

By this sentence we mean that we are the first to apply diffusion models to estimate the full scene flow vector field directly from two point clouds instead of a refinement plug-in module, as in DifFlow3D. However, we agree that this sentence might be misinterpreted and we will therefore rephrase it as: To the best of our knowledge, we are the first to propose using diffusion models to estimate the full scene flow directly from two point clouds.

Question 5: Negative societal impacts.

Potential negative societal impacts might be: As any other tracking algorithm scene flow estimation can be used in surveillance scenarios, which might raise privacy concerns and ethical issues. From an ecological perspective, training of deep learning models usually takes time and resources, thus environmental impact should be taken into consideration when training and applying such compute-intensive models. However, in our opinion, the positive impact of our proposed approach in practical applications such as robotics and autonomous driving outweighs the negative impact in the applications we consider.