To Reviewer 5c6P

Question 1: General, accurate, and efficient 3D representations

The author asserts that "General, accurate, and efficient 3D representations are three crucial requirements for 3D generation." However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient.

Response 1

We are grateful for the reviewer's feedback for our missing of citation. The statement "General, accurate, and efficient 3D representations are three crucial requirements for 3D generation" is based on the common understanding in the 3D generation community, and cited from [1].

• General: A 3D representation should be able to represent a wide range of 3D objects, scenes, and shapes [2]. For example, voxel grids are general representations to any 3D shape, but they are computationally expensive.
• Accurate: A 3D representation should accurately capture the geometry, appearance, and structure of 3D objects. For example, point clouds are accurate representations that capture the exact 3D points of an object, but they lack connectivity information [3].
• Effcient: A 3D representation should be efficient in terms of memory usage, computational cost, and training time. For example, 3DGS [4] are efficient representations that can generate high-quality 3D shapes with low memory and computational cost.

Question 2: Synthesizing a complete interior mesh

The author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments.

Response 2

We appologize for missing a detailed description to the concept of inside surface. More results in "Figure 5: Quantitative Comparison in Image-to-3D Generation" have illustrated the effectiveness of our method in synthesizing a complete interior mesh. Here are the explaination:

• In the GSO dataset shown in Figure 5, our method is able to capture the inside surface of the shoe, while other methods fail to detect it and hide the welt;
• Also, the OmniObject3D dataset in Figure 5 showcases our method's ability to capture the inside surface of objects such as cupboards and bowls. Other methods fail to accurately predict the inside surface under the input view.

Question 3: Complete Internel Mesh Dataset.

The authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within.

Response 3

For datasets obtained through 3D scanning using multi-view images or depth sensors, it is true that the inside of the objects appears empty. However, the Objaverse dataset contains a variety of 3D models with and without interior meshes, which is why we conducted experiments on this dataset. Our X-Ray method outperforms the state-of-the-art method TripoSR by a large margin on this dataset. We acknowledge the reviewer's concern regarding the difficulty in obtaining datasets with interior details. However, as 3D modeling techniques advance, we anticipate that future datasets will include more detailed 3D models contains the inside mesh. Furthermore, we have found that indoor scenes and 3D CAD models are now accessible for training, providing abundant interior details that can be effectively learned by the X-Ray. As a result, we are able to synthesize complete scene-level 3D rooms or buildings with intricate interior details. We will include the results of these datasets in the revised paper.

Question 4: Ablation studies

The ablation study is insuffcient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity.

Response 4

• We appreciate the reviewer's suggestion for additional ablation studies. Due to limited computational resources, we were unable to conduct extensive ablation studies on arbitary X-Ray frame. However, previously, we first selected 16 X-Ray frames to represent each 3D object and then found that using only 8 frames achieved very close performance.

• Additionally, we conducted an experiment to analyze the Encoding-Decoding Intrinsic Error, as shown in Figure 4 of paper. This is why we chose to use 8 frames in our experiments. The previous experimental results are as follows:

  Method	CD(L1) ↓	FS@0.1 ↑
  X-Ray w/ 16 frames	0.053	0.982
  X-Ray w/ 8 frames	0.056	0.973

• Besides, we attempted to use the image diffusion model as our backbone for the lightweight computation. However, it did not produce satisfactory results, which is because the image diffusion model only considers the 2D spatial relation and cannot effectively capture the relationship between different frames. Consequently, we made the decision to switch to an off-the-shelf Stable Video Diffusion method for generating X-Ray frames. This method is better suited for addressing the sequential generation problem in 3D reconstruction. The previous experimental results are as follows:

  Backbone	CD(L1) ↓	FS@0.1 ↑
  Stable Diffusion	0.114	0.651
  Stable Diffusion + Frame Attention	0.062	0.936
  Stable Video Diffusion	0.056	0.973

Reference

[1] Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, Ying Shan. "Advances in 3D Generation: A Survey". arXiv:2401.17807.

[2] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, Bernhard Schölkopf. "Ghost on the Shell: An Expressive Representation of General 3D Shapes". ICLR2014.

[3] Mescheder, Lars and Oechsle, Michael and Niemeyer, Michael and Nowozin, Sebastian and Geiger, Andreas. "Occupancy Networks: Learning 3D Reconstruction in Function Space". CVPR2019.

[4] Kerbl, Bernhard and Kopanas, Georgios and Leimk{"u}hler, Thomas and Drettakis, George. "3D Gaussian Splatting for Real-Time Radiance Field Rendering". ACM Transactions on Graphics.

To Reviewer wtWj

We appreciate the reviewer's highly positive rating to our paper! We address the reviewer's further concerns and questions in the following responses.

Questions 1: Performance of Computed Tomography

Considering Computed Tomography, it is expected that the performance will significantly increase as the number of layer L increases. However, the experiments analyzed in Section 5.2 show t performance improvements are very minimal when L is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is but L is extremely high (not mandatory).

Response 1

We understand reviewer's concern about the performance when using all frames without omitting any surface. Indeed, conducting experiments with extremely high layers using Diffusion Models is not feasible for GPU memory limitation. However, this issue can be resolved via autoregressive Large-Large Model (LLM) to generate varying numbers of surfaces for each ray, ranging from 0 to even 100+, depending on the actual surface layers for each ray. With LLM, we can flexiblely model both indoor and outdoor scenes using X-Ray. We are going to release the excited research work in the future!

Questions 2: Training Resources

The entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.

Response 2

We appreciate the reviewer's concern about the training pipeline. The training pipeline is relative long and memory-intensive, which is a common issue in training large-scale 3D models. For example, previous state-of-the-art method, Open-LRM, trained their model on 64 NVIDIA V100 GPUs for 5-6 days. Given our limited GPU resources, we made efforts to optimize and minimize the computational requirements as much as possible.

Questions 3: Normalization

When training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper.

Response 3

The reviewer is correct. We did apply a preprocessing step to normalize the object, ensuring that the rays from a specific camera position can capture the entire object. The normalization code for arbitrary 3D objects is as follows:

def normalize_object():
    bbox_min, bbox_max = scene_bbox()
    scale = 1 / max(bbox_max - bbox_min)
    for obj in scene_root_objects():
        obj.scale = obj.scale * scale
    # Apply scale to matrix_world.
    bpy.context.view_layer.update()
    bbox_min, bbox_max = scene_bbox()
    offset = -(bbox_min + bbox_max) / 2
    for obj in scene_root_objects():
        obj.matrix_world.translation += offset
    bpy.ops.object.select_all(action="DESELECT")

Questions 4: GPU Memory

How much GPU memory is required during inference?

Response 4

During inference, the GPU memory required is 4.8 GB for X-Ray diffusion model and 2.5 GB for X-Ray Upsampler. Thank you for the question, we will include this information in the revised paper.

To Reviewer EY92

Question 1: Multiple viewpoints Extension.

How to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality.

Response 1

we aim to show the advantage of the X-Ray by providing a simple baseline for single-view 3D reconstruction. While extending this to multiple viewpoints is interesting, it might not be necessary for this task. To address the reviewer’s concern, we encoded 1000 randomly selected 3D meshes into X-Ray from different viewpoints, like front and top views, and then decoded them back into 3D meshes. The standard deviation of reconstruction error from different views is negligible (CD < 1e-3). This demonstrates the robustness of the X-Ray to multiple viewpoints and its ability to reconstruct 3D meshes from any viewpoint. We will include this discussion in the revised paper.

Question 2: Limitation of finite and nonuniform intersection points

One limitation of X-ray is that the limited number of intersection points makes it difficult to encode complex shapes like hairs. Another limitation is nonuniform number of intersection points makes the presentation inefficient.

Response 2

• In the experimental results (Figure 4 (a)/(b)), using 8 or even 6 frames is sufficient to cover most of the surfaces, as most 3D models are not highly detailed. However, for complex shapes such as hairs, this limitation can be addressed by increasing the number of frames, to encode more intricate details.
• To further overcome the nonuniform number of surface layers limitation, autoregressive Large-Language Model (LLM) can be employed as the generator to handle surfaces with a dynamic number of layers. This approach allows for rays to generate varying numbers of surfaces, ranging from 0 to 100+, depending on the actual surface layers for each ray.

Question 3: Dataset

Carefully designed dataset is difficult to get and if existed would increase L making the method inefficient.

Response 3

Obtaining a dataset with interior details is a challenge, as mentioned by the reviewer. However, we are optimistic about the increasing availability of fine-grained 3D datasets. With advancements in 3D modeling, more detailed 3D models will likely be included in future datasets. Additionally, we have access to indoor scenes and CAD models, which contain rich interior details that can be effectively learned by the X-Ray. This allows us to synthesize complete scene-level 3D rooms or buildings with intricate interior details. As described above, LLM is our next version of generator to handle surfaces with a dynamic number of layers, which will be more efficient in generating complex shapes.

Questions 4: Related Work: DUSt3R

It is a interesting to connect and mention DUSt3R (CVPR2024).

Response 4

We will mention the connection to the work of DUSt3R in our revised paper as it has made significant contributions to multi-view depth and pose estimation.

Question 5: Representation Operation

Describe the representation operation in more detail.

Response 5

We have described the process of obtaining X-Ray from a 3D mesh in the Appendix PDF and source code. The "hit" value is determined by checking if the depth is greater than zero. We first identify the intersected mesh face index and then query its properties as X-Ray. Below is the pseudocode we provided:

from trimesh.ray.ray_pyembree import RayMeshIntersector

def ray_cast_mesh(mesh, ro, rd):
    intersector = RayMeshIntersector(mesh)
    index_faces, _, _ = intersector.intersects_id(
        ray_origins=ro, rd=rd,  multiple_hits=True)
    return index_faces

def Mesh2XRay(mesh, Rt, K):
    # get camera center and ray direction
    ros, rds = get_camera_ray(Rt, K) 

    XRay = []
    for ro, rd in zip(ros, rds):
        index_faces, = ray_cast_mesh(mesh, ro, rd)
        depth, normal, color = extract_features(mesh, index_faces)
        # calc hit
        hit = depth > 0
        xray = torch.cat([hit, depth, normal, color], dim=-1)
        XRay += [xray]
    XRay = torch.stack(XRay) # (H, W, L, 8)
    return XRay

Question 6: Photo-metric Quality

How is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)?

Response 6

Novel view synthesis and single-view reconstruction are two distinct tasks. Evaluating the quality of 3D reconstruction based on photometric metrics may not be appropriate. For example, methods like NeRF and 3DGS can achieve high scores in novel-view synthesis at the photometric level, however, their extracted 3D shapes and textures often suffer from really poor quality. Ours focus is on accurately reconstructing the 3D ground-truth shape as the evaluation. We appreciate the reviewer's observation regarding photometric quality. We will predict additional 3D Gaussian Splatting parameters to synthesize high-quality novel view images in the revised paper.

Question 7: Normal Prediction

How significant is the effective of the predicted normal?

Response 7

Point normals can be estimated from the point cloud, but determining their orientation (inside or outside the surface) is challenging. To solve this, we generate additional point cloud normals for consistent orientation. We tested the importance of normal prediction for rendering methods like LRM and our method. By fine-tuning OpenLRM with normal supervision, we found that adding ground-truth normals did not significantly improve its reconstruction metric and is still not comparible with ours. However, normal prediction is crucial for our method due to Poisson surface reconstruction. This ablation study will be included in the revised paper.

To Reviewer xHzv

Question 1: Comparison to Existing Techniques

A thorough comparison to existing techniques like depth peeling, multi-view depth images, MPI, and the PI3D.

Response 1

• We discussed multi-view images and MPI in the related work. Furthermore, Multi-view depth cannot sense the inside surface and might records the redundancy surfaces into nearby views. MPI divides the object into planes with fixed distances, while our X-Ray stores a dynamic number of surfaces. Refer to the following Table 1 for efficient comparison.
• We apologize for not being familiar with depth peeling before submission. Our X-Ray and depth peeling serve different purposes. Depth peeling is mainly for rendering transparent surfaces, while X-Ray transform any 3D object in video format. Besides, our main contribution is using video diffusion as generator to generate objects.
• The PI3D is an interesting approach that uses diffusion models for 3D generation. Since we do not have access to source code, to ensure a fair comparison, we will reimplement the PI3D method and include it in the revised paper.

Question 2: Efficiency of the X-Ray

The paper should provide more evidence and analysis to support its claims about the efficiency of the X-Ray.

Response 2

We compared the efficiency of different representations using 500 3D meshes from ShapeNet dataset. The results showed that both point cloud and X-Ray were highly efficient, with lower memory, faster encoding & decoding times. However, the X-Ray had the advantage of being reorganizable as a video format for diffusion models, leading to better performance.

Table 1. Comparison with other 3D representations in Efficiency.

Question 3: Ablation Studies

The paper should explain how the unique properties of the X-Ray are utilized in the diffusion process and whether other sequential representations could achieve similar performance.

Response 3

In section A.3 of the Appendix, we have conducted ablation studies to examine various aspects of the X-Ray model, including the impact of the hit/miss indicators. After exploring different formats, we transitioned to an off-the-shelf Stable Video Diffusion method, which proved to be more effective in addressing the sequential generation problem in 3D reconstruction.

Question 4: Evaluation Metrics

The evaluation should include a broader range of categories to assess the generative capabilities of X-Ray. Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?

Response 4

We chose to focus on cars as they have complex internal and external 3D surfaces, making them an ideal category to showcase the benefits of our method. Objaverse has over 1000 categories. Previous state-of-the-art methods only used reconstruction metrics for evaluation. Thank you for reminding us. We included generative metrics for the single-view 3D reconstruction experiments on GSO in the table below. Since our method can sense the inner surfaces, we have achieved overwhelming superiority. Any additional metrics will be included in the revised paper.

Question 5: Additional Suggestions

The evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.

Response 5

1. SDS-based methods like Zero-1-to-3 and DreamGaussian rely on diffusion loss to optimize NeRF or 3D Gaussian splatting. However, these methods are time-consuming, taking several minutes to generate a single 3D mesh from an image. This makes them impractical for evaluation on large datasets like GSO and OmniObject3D. They all did not report their 3D reconstruction metrics. In contrast, all rendering-based methods and our X-Ray can generate 3D meshes from images in just a few seconds. Additionally, SDS-based methods also struggle to generate inner surfaces, limiting their performance on datasets of GSO and OmniObject3D.

2. We have add more visualizations in the Appendix (Sec A.4). Specifically, Figures 6 and 7 showcase the arbitary 3D objects generated by our proposed method. We appreciate your feedback and are planning to further enhance the performance on the 3D CAD model dataset (abundant inside and outside meshes) in the revised paper.

3. We have included a dedicated section in the Appendix (Sec A.5) and Figure 8 to analyze failure cases, and we will enhance this section in the revised paper.

Question 6: Solution to Limitation

How does this sparsity affect generation? How can it be addressed?

Response 6

To further overcome the sparse related limitation, an autoregressive Large-Language Model (LLM) can be employed as the generator, allowing for each ray to generate varying numbers of surfaces, ranging from 0 to 100+, depending on the actual surface layers.