We appreciate your acknowledgment of our novel contribution about our problem setting, model design, and multi-level controllable generation. We try to address your concerns as follows.

Q1. Theoretical foundation for cross-modal consistency.

Our theoretical derivation is formulated with the following steps:

(1) The multi-modality joint generation requires the mutual cross-modality condition in the denoising process of each modality (Eq.5).

(2) The cross-modality alignment mainly relies on the condition on the corresponding local information from the other modality.

(3) Our proposed epipolar-line cross-modality condition module can align the geometrical space (Eq.6,7) between LiDAR and camera modalities, which allows it to find the cross-modality local correspondence between LiDAR and camera feature maps. We agree that there is no rigorous mathematical proof, but it is consistent with intuition and should be enough to 1) formulate the multi-modality joint generation problem and 2) motivate the design of our epipolar-line cross-modality condition module. Our paper is more related to application instead of diffusion model theory, so the rigorous mathematical proof is out of our scope. This format is similar to previous ICLR/NeurIPS publications about multiview camera image generation with cross-view consistency claims [1,2,3].

[1] MagicDrive: Street View Generation with Diverse 3D Geometry Control (ICLR 2024)

[2] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024)

[3] Mvdream: Multi-view diffusion for 3d generation (NeurIPS 2023)

Q2. Epipolar geometry may not fully resolve 3D space ambiguities.

We agree that pure epipolar geometry may not fully solve the problem. However, diffusion models themselves also have strong ability to model the distribution of each single-modality data, especially the image diffusion model pretrained on large-scale datasets. Therefore, the epipolar-line cross-modality condition only needs to provide some extra meaningful guidance about the 3D spatial cues, which is enough to guide the generation of powerful diffusion models in an aligned manner for each modality. We show the performance gain from epipolar geometry in our ablation studies that it can significantly improve the cross-modality alignment.

Q3. Applying existing control methods.

This our main contribution focuses on multi-modality joint generation. We do not focus on fancy control signals different from prior works. In contrast, our contribution of controllable generation lies in the aligned control between images and point clouds. For example, in Fig.6, we can insert or delete objects simultaneously in the point clouds and multiview images, which cannot be achieved by previous works. It's worth mentioning that X-Drive is a flexible framework that can be easily combined with other single-modality controllable generation methods.

Q4. Data utility in downstream tasks.

We refer you to Reviewer vwWu Q3 where we show the usefulness of synthetic data in the training of multi-modality 3D object detection model.

Q5. Advanced transformer architectures/T5 text encoder/video generation.

Thanks for these useful suggestions! We position this work as the first attempt of LiDAR-camera joint generation, and we are willing to incorporate these advanced techniques in the future.

Q6. Higher resolution.

We follow the image resolution in MagicDrive, and the point clouds density is constrained by the LiDAR sensor in nuScenes dataset (32-beam). Our cross-modality condition is not the main bottleneck for increasing the resolution, so we can easily extend our framework to higher resolution following single-modality state-of-the-arts in the future.

Q7. Quantitative metric for cross-modality consistency.

Since there is no existing metric, we propose a novel DAS metric for this goal in the paper. We refer you to Sec. 5.1 and Appendix A for more details. Our results show the great gain in DAS metric brought by our X-Drive framework, verifying the ability of X-Drive to improve cross-modality consistency.

Q8. Extra ablation studies.

We refer you to Reviewer Tmsf Q2 for additional ablation studies, including the epipolar line component, two-stage training strategy, and quantitative sampling number along epipolar lines. Due to the limited time of rebuttal, we will include more ablation studies in the final version.

Q9. Code release.

We will release the code and all the documents upon the acceptance of this paper.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.

We thank the reviewer for the detailed response. We are glad to see that our response can address your concerns. We will incorporate your suggestions in the final version and future work!

We thank for your recognition of our insight, and we try to address your concerns as follows.

Q1. Lack of quantitative comparison with baselines.

Thanks for your suggestions. We will split the table in the final version. For image generation, we will include more baselines. For point clouds generation, the three baselines in the table are the only ones reporting quantitative results on nuScenes dataset, so we include them as baselines (We have to use nuScenes due to Reviewer wkMY Q2). For multi-modality generation, we can incorporate one more baseline that takes the same textual prompt and box conditions for point clouds and multiview image generation, which is exactly the "w/o cross-modality" item in our ablation studies (Tab.5). X-Drive can still notably outperform this vanilla baseline.

Q2. Lack of ablation studies.

Thanks for pointing it out. We include two additional ablation studies: (1) w/o two-stage training that directly trains the point clouds branch and multi-modality modules together by merging the second and third training stages. (2) w/o epipolar line that randomly sample the same number of points on the other modality feature map without epipolar line for cross-modality condition. (3) half sampling points along epipolar lines that reduces the number of sampled points along epipolar lines in the cross-modality condition module by half from [24, 24, 12, 6, 12, 24, 24] to [12, 12, 6, 3, 6, 12, 12] for each block.

Our results show that the two-stage training can notably improve the quality of point clouds (JSD metric). The epipolar-line is important for the alignment between LiDAR and camera modalities (DAS metric). More sampling along epipolar lines are also critical for the reliance of cross-modality conditions (DAS metric). All ablations result in notable performance drops in comparison with our full model.

Q3. Marginal improvements.

As we mentioned in Q2 of Reviewer vwWu, the goal of our paper is to enhance the cross-modality alignment instead of improving single-modality generation. In this case, we do not aim to significantly outperform prior work in the quality of each single-modality data. Our camera branch largely follows MagicDrive, and the LiDAR branch is similar with RangeLDM, since the single-modality models are not our focus. Thus, we only expect performance comparable with these prior works. We highlight the joint generation framework and cross-modality condition modules as our main contributions, which bring high cross-modality consistency reflected by the significant performance gain in DAS metric.

Q4. Typos.

Thanks for pointing them out. We will fix these typos and proofread again in the next version.

Q5. Distribution at the bottom of Fig.1(b).

This is just an illustration of our idea without quantitative referring. We want to show that our cross-modality condition can align the distributions of LiDAR point clouds and multiview camera images in the multi-modality generation to describe the same driving scene.

Q6. How is the generation of the LiDAR point cloud achieved?

After obtaining range-view representation with range and intensity $(r,i)$ for each coordinate $(\theta,\phi)$ in the range view, we can recover the corresponding point $(x,y,z)$ by inversing Eq.3 as follows. $z=r\sin(\phi),\ y=r\cos(\phi)\sin(\theta),\ x=r\cos(\phi)cos(\theta)$ We will clarify it in the next version.

Q7. Latency of our method. Additional cost for the designed module.

We measure the latency for each module of our model. For the entire X-Drive model, it takes 7.2 seconds and 17GB GPU memory for the synthesis of each sample. For camera branch only, it takes 4.7 seconds and 12GB GPU memory. For LiDAR branch only, it takes 1.4 seconds and 4GB GPU memory. Thus, we think our cross-modality condition would not bring much extra cost.

We hope our response will clarify the reviewer's confusion. If you have any questions, please feel free to let us know during the rebuttal window. And we sincerely hope to obtain support from the reviewer.

Dear Reviewer Tmsf,

We would like to kindly remind you that the response period will end within a few hours, and we have not yet received your feedback. Given the importance of your feedback to us, we would greatly appreciate it if you could take the time to share your thoughts in the remaining hours.

Thank you for your time and consideration, and we look forward to hearing from you.

Best regards,

The Authors

We apprciate your recognition of our problem setting, model design, and paper writing. We try to address your concerns as follows:

Q1. Point clouds 3D representation or range-view representation.

We agree with you on the strengths of 3D point clouds representation. However, the goal of our work is to synthesize sensor data instead of reconstructing the 3D senes. We hope the synthetic point clouds can simulate the real data captured by the LiDAR sensor. If there are occlusion or distant sparsity in real LiDAR sensor data, we would expect the same patterns in synthetic data to avoid the domain gap between real and synthetic data. In this case, range-view representation is naturally compatible with the data collection process of LiDAR sensor (a unique point along each beam). Besides, since we focus on the cross-modality alignment instead of single-modality generation, we prefer to follow previous state-of-the-arts in point clouds generation. Our main contribution, the epipolar-line cross-modality condition, can be easily extended to 3D point clouds representation by sampling local features along the camera rays from the 3D point clouds representation.

Q2. Quality of point clouds generation.

We appreciate your careful observation. Before addressing your concrete concerns, we emphasize the following keypoints.

1. Previous point clouds generation works mainly report results for 64-beam LiDAR sensor (SemanticKITTI dataset) but we are working on 32-beam LiDAR point clouds (nuScenes data) with reasons explained in Reviewer wkMY Q2. The 64-beam data is naturally easier for model training and looks better due to its higher density. For fairness, we only compare with 32-beam results of prior works, which are only reported in LiDM (qualitatively in Fig.4 [2]) and RangeLDM (qualitatively and quantitatively in Fig.6 and Tab.4 [3]).
2. Our paper mainly focuses on cross-modality alignment. We roughly follow RangeLDM [3] in the design of point clouds branch except our extra box conditions. As a result, we expect a comparable rather than much better point clouds quality compared to RangeLDM since we do not focus on notably improving the single-modality quality.
3. Our multi-modality framework and epipolar-line cross-modality condition are flexibly compatible with various single-modality diffusion models since it only needs to sample points from feature maps. It can be easily combined with new single-modality state-of-the-arts in future.

Additionally, we clarify the realism issues in the generated scenes as follows:

• Pattern realism: In rainy weather, there should be some breaks in the concentric circles of LiDAR points (1st and 4th examples in Fig.8) due to the water on the ground, which is also witnessed in real point clouds (https://drive.google.com/file/d/1DTRSEKDvxX4UhAzgMkJgjJE2erwCEILQ/view?usp=sharing). This reflects the ability of our model in the generation of extreme weather. For sunny weather, the concentric circle patterns are also witnessed in point clouds generated by X-Drive (other examples in Fig.8).
• Object realism: We show some qualitative comparison between X-Drive and RangeLDM/LiDM on 32-beam point clouds (https://drive.google.com/file/d/1DTRSEKDvxX4UhAzgMkJgjJE2erwCEILQ/view?usp=sharing). X-Drive shows comparable or even better object realism in comparison with LiDM and RangeLDM on 32-beam data. We agree that there are some gaps compared to real data for all these three methods, but our main goal is to improve the cross-modality consistency instead of single-modality point clouds quality. We will show in Q3 that current synthesis can already help downstream tasks.
• Pedestrian realism: We acknowledge the improvement space for pedestrian point clouds synthesis, but this also exists in all existing works including LiDM and RangeLDM. Since the goal of our paper is not to improve the point clouds generation, we aim to achieve a single-modality performance comparable with previous work, but with a better cross-modality alignment of the entire scene.

Q3. Can the synthesis help model training?

We conduct experiments on multi-sensor object detection with SparseFusion model [7]. We train a model with the combination of 1.4k real scenes (5% nuScenes training set) and 1.4k synethtic scenes, which is compared with a baseline trained only with 1.4k real scenes. Results show the helpful role of our synthetic data in the training of downstream multi-sensor perception models.

We also note that most existing point clouds generation works (like LiDM and RangeLDM) do not conduct the experiment about synthetic data for model training. As an orthogonal exploration of LiDAR-camera joint generation, we believe X-Drive can also benefit from the future development of single-modality algorithms to further boost the usefulness of synthetic data for downstream model training.

[7] Xie, Yichen, et al. "Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection." ICCV 2023

Q4. Multi-modality OOD generation results.

Fig.6 shows the results of OOD generation including lighting, weather, object insertion and object removal. For object insertion and removal, we already show the results of both point clouds and multiview images. For lighting, LiDAR sensor would not be affected by lighting condition like daytime or night. For weather, rainy weather would interfere the relectance of laser, which is reflected by the sparse and messy point clouds shown in https://drive.google.com/file/d/1DTRSEKDvxX4UhAzgMkJgjJE2erwCEILQ/view?usp=sharing and also reflected in the 1st and 4th examples of Fig.8.

Q5. Baseline setting for RangeLDM+MagicDrive.

There is nothing guaranteeing the alignment between the different modalities from the two models RangeLDM and MagicDrive. However, we have no choice except this weak baseline since RangeLDM is the only point clouds generation algorithm that we have access to either code or results on nuScenes dataset (The authors kindly share with us the nuScenes generation results). As a stronger baseline, we refer you to the "w/o cross-modality" item in our ablation study Tab.3, where the point clouds and multiview images are generated independently based on the same bounding box and textual description conditions as a constraint of alignment. X-Drive also notably outperforms this strong baseline.

Q6. Decision of studying this problem in range-image space.

Please refer to our response to Q1.

Q7. ICP metric.

This is a very good suggestion. We will explore it in our future work.

Q8. Computational cost.

X-Drive does not require extra encoders except the light-weight epipolar-line module (Fig.3 or violet parts in Fig.2). The cross-modality condition is directly based on the feature maps from the diffusion models of each modality. We measure the latency and memory cost for each module of our model. For the entire X-Drive model, it takes 7.2 seconds and 17GB GPU memory for the synthesis of each sample. For the camera branch only, it takes 4.7 seconds and 12GB GPU memory. For the LiDAR branch only, it takes 1.4 seconds and 4GB GPU memory. Thus, we think our cross-modality condition would not bring much extra cost.

Q9. Effect of zero initialization.

We made a trial without it in the beginning. In the early stage of multi-modality training, the lack of zero initialization made the model converge very slowly and we also noticed the bad quality of each modality generative results, so we incorporated the zero-initialization strategy in all following experiments.

Q10. Sequential image generation and then image-to-point-clouds generation.

This is feasible with our framework. People can first generate images guided by some conditions using other existing image generation algorithms like MagicDrive. Then, they can use the synthetic images as condition to generate point clouds through our camera-to-LiDAR conditional generation mode. This strategy can even allow more conditions supported by state-of-the-art image generation frameworks (like HD-map for MagicDrive), but it may lead to higher latency than joint multi-modality generation.

Q11. Discussion about GenMM.

Thanks for sharing this related work. They actually follow a sequential pipeline by firstly inpainting the video then inpainting the point clouds based on the video. However, their video-to-LiDAR step relies on some geometrical solutions, which is different from our diffusion model. The geometrical strategy has some limitations: Firstly, the precision relies on the high resolution of 3D volume, which makes it hard and very slow for from-scratch large-scale scene generation. Secondly, it is impossible to generate the reflectance intensity of LiDAR sensors.

Q12. Typos.

Thanks for pointing them out. We will fix these typos and proofread again in the next version.

We wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments!

Dear Reviewer vwWu,

Thank you for the detailed response. We are glad that our response can solve some of your concerns. We try to address your follow-ups as below:

Point clouds vs range-image:

Our goal is to synthesize sensor data. Unlike 3D object scans, LiDAR point clouds essentially construct a 2.5D scene from a single viewpoint instead of a full 3D point clouds [8,9]. The range image perfectly represents the LiDAR sweep in the spherical coordinate. Each pixel on the range image corresponds to a laser beam. Since each beam would only be reflected at one certain position, there would only exist one unique value at each range image pixel. The occlusions result from the laser beam reflection itself instead of either point clouds or range-image representations.

In our implementation, since nuScenes provides point clouds, there is a minor loss in the conversion to range images. The vertical resolution of the range image is determined by the number of laser beams (32 for nuScenes), so there is no quantization loss in the vertical dimension. However, our horizontal resolution (1024) may not exactly match the rotation of the LiDAR sensor (about 1080 per sweep), which results in minor information loss due to horizontal quantization (about 5%).

However, for point clouds representation, it is hard to mimic the 2.5D scene of LiDAR point clouds, i.e. it is difficult to ensure one unique point along each laser beam. We believe this would pose more challenges to the data synthesis than the minor quantization loss. We are trying to solve this challenge of point clouds representation in our future work.

Dataset with LiDAR + multi-camera data:

Thanks for pointing out the two datasets. For Waymo Open Dataset, they only provide front and side cameras without rear view cameras for the ego-vehicle, so there is a mismatch in the LiDAR and camera sensor data. For Argoverse, it also uses 32-beam LiDAR sensors and the dataset scale is similar with nuScenes, so it would not bring great difference compared to nuScenes. Since nuScenes is much more popular on various tasks, that's why we chose nuScenes as our dataset. We will include Argoverse as a future work.

Qualitative evaluation:

Object realism: We improve the layout of the shared file) to make it more clear. In the shared file, we show the synthetic object point clouds from RangeLDM, LiDM, and X-Drive. We do not know the exact categories for RangeLDM and LiDM objects, since their models cannot handle object-level category condition. For X-Drive objects, we have denoted the object categories in the shared file. The 1st and 2nd object are distant, but their orientations are still clear. The 3rd and 4th objects clearly show the shapes and orientations of two trucks. The 5th object is clearly the left side of a car, where we can identify its overall contour.

Pattern realism: Even in the real point clouds, the distant rings are not so circular (updated in the shared file). It is possibly due to the lack of motion compensation in the same LiDAR sweep. Besides, the L2 loss of diffusion model is not sensitive to small position errors. This can be relieved by longer training schedules, but we cannot afford many computational resources in this part since we only have four GPUs.

General notes: We agree with you in the importance of object-level quality, but these rely on orthogonal efforts. The point clouds realism mainly relies on the point clouds branch instead of the cross-modality condition. In our ablation studies, we show that our cross-modality condition would not hurt single-modality qualities (w/o cross-modality in Tab.3). At the time of our ICLR submission, there is no public code of point clouds generation model on nuScenes, so we have no choice but to develop the point clouds branch based on RangeLDM by ourselves which is trained in our first two training stages. For our main contribution, we develop a plug-in cross-modality condition module that can be injected between the feature maps of any SOTA multi-view images and LiDAR point clouds generative models. It can stand on the shoulders of single-modality giants due to the orthogonal nature. If someone releases a new SOTA point clouds generation model, we can easily replace our point clouds branch with their algorithm and delete our first two training stages, and all the point clouds realism issues can be solved naturally in this case.

Thanks again for your time and efforts! Feel free to raise additional questions if you still have any concerns.

[8] Meyer, Gregory P., et al. "Lasernet: An efficient probabilistic 3d object detector for autonomous driving." CVPR 2019.

[9] Hu, Peiyun, et al. "What you see is what you get: Exploiting visibility for 3d object detection." CVPR 2020.

Dear Reviewer vwWu,

We are glad that our response can help. We agree that there is some inevitable loss in the conversion process. After observing the "point clouds -> range-image -> reconstructed point clouds" conversion process of ground-truth LiDAR data, we find it would lose about 5% of points, but it only has minor effect on the appearance of reconstructed point clouds. We will also explore the point clouds representation in our future work.

We really appreciate your efforts in reviewing our paper and all the insightful feedbacks. If our response can address most of your concerns, can we humbly request you to raise your rating?

Best regards,

Authors

We appreciate your recognition of our motivation, method design, and paper writing. We try to address your concerns as follows:

Q1.Existing application of epipolar line in multiview diffusion models.

Thank you for sharing these related works. However, their methods are largely different from our X-Drive. For [1,3], they construct an explicit 3D feature volume from multiview image features. This limits their methods to single object since feature volumes for large-scale outdoor scenes demand infeasible memory costs. In contrast, we do not need explicit 3D scene representation. For [2], it applies a joint global self-attention to aggregate multiview image features, which also has high memory cost, but we adopt an efficient local condition based on epipolar-line cross-attention.

Besides, our contribution lies in the cross-modality alignment, which reflects a different motivation and insight compared with previous multiview diffusion model works. We will include these related works with discussions in our revision.

[1] Liu, Yuan, et al. "Syncdreamer: Generating multiview-consistent images from a single-view image." arXiv preprint arXiv:2309.03453 (2023).

[2] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).

[3] Yang, Jiayu, et al. "Consistnet: Enforcing 3d consistency for multi-view images diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q2. Experiments are only conducted on nuScenes dataset.

We conduct experiments on nuScenes dataset because it is the only mainstream dataset equipped with both LiDAR sensor and multi-view surrounding cameras. Such paired sensor data is necessary for our multi-modality experiments. In contrast, for other popular datasets, although they all have LiDAR data, SemanticKITTI only has front view cameras, KITTI360 uses fisheye cameras, and Waymo Open Dataset lacks back view cameras. We will extend our work to the robust case with missing sensors in future work.

We also refer to previous publications about LiDAR-camera multi-modality 3D object detection. For the similar reason, most of them like [4,5,6] only provide experiments on nuScenes dataset.

[4] Yang, Zeyu, et al. "Deepinteraction: 3d object detection via modality interaction." NeurIPS 2022

[5] Xie, Yichen, et al. "Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection." ICCV 2023

[6] Yan, Junjie, et al. "Cross modal transformer: Towards fast and robust 3d object detection." ICCV 2023

Q3. GPUs and training/inference time.

We are using four NVIDIA RTX A6000 GPUs for training, which is fewer than typical 8-16 GPUs for prior multi-view images or point clouds generation works. The first stage of training (point clouds VAE) takes about 1 day. The second stage (point clouds diffusion model) takes about 6 days. The third stage (multi-modality diffusion model) takes about 6 days. For inference, it takes about 7.2 seconds for each scene (point clouds + multiview images) with a single A6000 GPU.

Q4. Question about Eq.5.

This equation means the denoising of either LiDAR or camera modality is conditioned on both modalities. Could you specify your question if there is anything unclear? Thank you so much.

Thanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window.