LAM3D: Large Image-Point Clouds Alignment Model for 3D Reconstruction from Single Image

Thank you for your insightful questions and suggestions. We have provided our responses below:

Q1. Texture:

We acknowledge the reviewer's concern about LAM3D's current limitation in generating textured meshes, as mentioned in the paper. While our primary focus was on achieving accurate geometry reconstruction, we fully agree that texture is a crucial component in generating realistic 3D content. Our approach was motivated by the need to resolve geometric distortion issues caused by the lack of explicit 3D priors in recent large reconstruction models. We believe that recovering high-quality geometries is foundational for both textured and non-textured 3D reconstruction. We appreciate the suggestion to use a multi-view diffusion model like Zero-1-to-3 to generate textures, and we recognize this as a promising direction to effectively complement LAM3D’s geometry output. Furthermore, we are committed to exploring such solutions in future work to enhance the capabilities of LAM3D.

Q2. More Comparison

We appreciate the reviewer highlighting these recent papers, particularly Michelangelo, which also adopts an alignment-based approach for 3D reconstruction. As discussed in response to Q1 of Reviewer 1gWL and in our main paper, Michelangelo's alignment strategy has two key limitations: (1) the contrastive loss used in Michelangelo enhances linear separability in the latent space, which is beneficial for discriminative tasks but problematic for 3D reconstruction that requires a continuous latent space to capture the morphable nature of 3D objects; (2) the choice of a vector-shaped latent representation falls short in preserving the spatial information necessary for accurate 3D reconstructions. We address these issues with our diffusion-based alignment strategy, which promotes continuity in the latent space, and a tri-plane structured latent representation that retains spatial information. To validate our approach, we constructed a baseline model based on ULIP [58], a framework similar to Michelangelo that unifies 3D, image, and text modalities. Our experiments demonstrated that while Michelangelo's strategy performs well at the ShapeNet scale, it fails to generalize effectively to the large-scale and category-agnostic Objaverse dataset. These results are detailed in Figures 1, 4, and Tab. 2 of our main paper. Additionally, we recognize the promising direction presented by SV3D, which employs large-scale video diffusion models for 3D reconstruction, and we will discuss this in the revised related works section.

Q3. KNN+PointNet v.s. FPS+Transformer

The method mentioned by the reviewer, using KNN for point collection followed by a PointNet encoder, was initially our preferred approach due to its simplicity and efficiency during the algorithm development stage. However, our experiments revealed that the KNN+PointNet combination struggled to fully capture the input geometry, often resulting in oversmoothed reconstructions and low-quality details. This limitation likely arises because the projection operation aggregates information from a broader range of spatial locations, whereas KNN+PointNet is constrained to local operations. This finding motivated our shift to using a Transformer-based approach, which better integrates long-range interactions between local patches, and our experiments demonstrated the effectiveness of this method in improving reconstruction quality.

Thank you for your insightful questions and suggestions. We have provided our responses below:

Q1. Comparison

We thank the reviewer for highlighting these related works. Shape-Image-Alignment is not a novel concept, and has been explored in works like Michelangelo and ULIP [58]. The essence of both Michelangelo and ULIP is training a 3D autoencoder with a CLIP-aligned latent space. However, the experiments of Michelangelo were limited to the ShapeNet scale. To examine the effectiveness of this approach on a larger scale, we built a baseline model using the 3D encoder of ULIP, as described in lines 293 to 304, and found notable limitations when scaling to the Objaverse dataset with 140k category-agnostic objects. Firstly, the contrastive loss used in Michelangelo and ULIP enhances linear separability in the latent space, which is beneficial for discriminative tasks. However, this approach is problematic for 3D reconstruction, as it requires a continuous latent space to capture the morphable nature of 3D objects. While Michelangelo succeeded within the relatively small and dense ShapeNet dataset, this alignment approach failed to generalize at the Objaverse scale, leading to poor 3D reconstruction quality as demonstrated in Fig. 1 and Tab. 2. Secondly, the constraint of encoding 3D shapes into a vector feature representation of size $\mathbb{R}^{512}$ to align with pre-trained CLIP features proved inadequate for capturing the rich diversity of 3D geometries, as seen in our comparisons in Fig. 4. In contrast, our diffusion-based alignment strategy fosters a continuous latent space, while our tri-plane latent representation effectively preserves spatial information that is often lost in vector-based latent representations, thereby enabling more accurate and scalable 3D reconstructions. Regarding the 3DGen method, it treats the image as a condition for the diffusion model and uses a CLIP image encoder to produce a vector-shaped feature for the diffusion process. However, as we discussed earlier, this approach can lead to a loss of spatial information. In contrast, our method utilizes DINO as the image encoder, generating per-patch encodings of size $\mathbb{R}^{1025 \times 768}$, which allows the subsequent diffusion alignment module to transform these into a tri-plane representation, preserving crucial spatial information. Additionally, our diffusion model differs fundamentally from 3DGen’s approach, which we discuss further in the response to Q2.

Q2. Parallel Diffusion

We appreciate the reviewer's concern about the potential loss of 3D information. Previous methods often handle tri-plane structures through either roll-out operations, as seen in 3DGen, or channel concatenation, as discussed in response to Q3. However, both approaches have significant drawbacks. The roll-out method involves convolution operations that span adjacent planes, but the values at the edges of these planes are not spatially continuous, leading to interference and inaccuracies during the learning process. Similarly, channel concatenation aligns features at the same spatial location across planes, but without an explicit relationship between them, applying a single convolution kernel can cause interference in the regression of each plane. To address these issues, our method leverages the planar properties of the tri-plane structure by processing each plane independently. This approach effectively resolves inter-plane interference and results in better reconstruction outcomes. Furthermore, the three planes can be viewed as orthographic projections of the 3D object. We explicitly decouple features across planes by introducing a latent tri-plane loss, which preserves the tri-plane structure in the latent space, as shown in Fig. 6 of the appendix. Although our three independent diffusion models may lack direct inter-plane interaction, this design diminishes inter-plane interference and ensures that 3D information is not lost, as the plane features are decoupled and maintained through the latent tri-plane loss during the compression stage.

Q3. Channel Concatenation and Parallel Diffusion

As explained in our response to Q2, using channel concatenation results in the regression of values based on all three planes at the same spatial location, despite there being no explicit relationship between them. This lack of spatial relationship leads to interference during the learning process. To address this, we constructed a baseline model with a single diffusion network that concatenates the planes channel-wise. Our experimental results, presented in Fig. 5 and Tab. 3, demonstrate that the parallel diffusion approach outperforms this single UNet variant, offering better reconstruction quality and visual effects. The parallel diffusion design mitigates the inter-plane interference, leading to more accurate and effective 3D reconstructions.

Q4. More Evaluation

We appreciate the reviewer's suggestion to include results on real-world data. We evaluated our method on the Google Scanned Objects dataset, which includes a diverse range of real-world household objects, and our approach outperforms previous methods as shown in Tab. 1. In response to the reviewer's comment, we have now included additional comparisons on more diverse sources of images in Fig. 3 of the global response.

Q5. DiT

Due to limited computational resources during the development stage, we focused on validating our approach with a parallel UNet design, which delivered state-of-the-art results. However, DiT is indeed a promising direction, and we plan to explore it in future work.

Q6. Open Source

We appreciate the reviewer's interest in open-sourcing of LAM3D. We are actively seeking institutional approval to release both the training and inference codebase to the public. Additionally, we plan to provide an online demo to facilitate quick and easy access to our model for the community.

Thank you for your insightful questions and suggestions. We have provided our responses below:

Q1. Model Capacity

As shown in Tab. 1 of the global response, we provide parameter size comparisons between our model and SOTA methods. It is worth noting that recent large reconstruction models are designed with high capacity to leverage large-scale training data, enhancing their generalizability in reconstructing novel objects. Our method aligns with this design philosophy.

Q2. Inference Time

As shown in Tab. 1 of the global response, our method achieves better inference time. Specifically, our time breakdown is as follows: approximately 0.02 seconds for encoding an image with DINO, 4.52 seconds to align the image feature to latent tri-planes using 50 diffusion steps, 0.24 seconds to upsample the latent tri-plane to the raw tri-plane space with the plane decoder and refiner, and 1.05 seconds for mesh extraction using Marching Cubes.

Q3. Missing Details

We have described the design of each module in the method section. We mainly focus on the conceptual aspects in our paper, and we acknowledge that the implementation details are somewhat limited due to page limitations. The inference pipeline is outlined in the caption of Fig. 2 and in Section 4.1, lines 249 to 251. We will provide additional explanations to clarify any potential misunderstandings.

Q4. Single Dataset Evaluation

Our evaluation protocol was precisely adopted from CRM (ECCV 2024) [51]. A common practice in large reconstruction models is to train on the Objaverse dataset and evaluate on the Google Scanned Objects (GSO) dataset. This approach was also used in TGS (CVPR 2024) [63] and One-2-3-45 (NeurIPS 2023) [24]. In response to the reviewer's feedback, we have included additional evaluation results in Fig. 3 of the global response. These results encompass reconstruction outcomes for images from ImageNet, MVImageNet, AI generated images, and images collected from the Internet.

Q5. Clarity

We appreciate the reviewer highlighting areas for improvement. We acknowledge the concerns regarding the clarity of the figure and the input description in our manuscript. To clarify, our pipeline consists of two stages during training: point clouds are used as input for the first stage, while images are used for the second stage. During inference, only images are required as input. Thus, the performance gains achieved are attributed to the effectiveness of our proposed modules and strategies, rather than the use of additional input modalities. We will revise the figure and manuscript to clearly distinguish between the training and inference phases, ensuring that the role of each input is clearly communicated.

Q6. Presentation

We appreciate the reviewer’s feedback on the presentation and clarity of the flow chart. We recognize that the current chart may not clearly indicate the inputs. We will revise the flow chart to explicitly show that point clouds are used in the first training stage, images in the second training stage, and only images are needed during inference. We will ensure that the inputs are clearly highlighted at each stage of the pipeline to improve overall clarity.

Q7. Lack of Details and Reproducibility

In our manuscript, we focused on the conceptual aspects and design choices of each module to provide a clear understanding of the overall framework and its motivations. We acknowledge that this approach did not cover the implementation details of each component in depth. To address this, we will include additional implementation details in the appendix. Additionally, we are actively seeking institutional approval to release code and pre-trained models to the public for easy reproduction of our results by the community.

Q8. Inference Process

We apologize for any confusion caused by the description of the inference process in lines 249-251. To clarify, our model requires only a single-view image of an object during inference. The process starts by encoding this image into a feature representation using the DINO image encoder. This image feature is then aligned to a latent tri-plane representation through three independent diffusion UNets, where the latent tri-plane represents a compressed tri-plane shape in the latent space. The latent tri-plane is subsequently upsampled to a high-resolution tri-plane using the plane decoder and refiner from the first training stage. We then employ the SDF MLP to decode the signed distance for any query position on the tri-plane, as described in lines 173-175, which allows us to use the Marching Cubes algorithm to reconstruct a mesh from the tri-plane. We will revise the manuscript to include a more detailed explanation of this inference process to enhance clarity.

Q9. Concept and Gaps

We appreciate the feedback. We will expand the related work to include a comprehensive review of [24,25,63,51,35]. Specifically, we will highlight that these previous methods primarily employ a rendering-based loss for supervision, which overlooks the use of available explicit 3D priors. This absence of explicit 3D priors often results in 3D geometric distortions, examples of which are demonstrated in Fig. 1(b) for LRM [12] and Fig. 1(c) for CRM [51]. In contrast, our method leverages 3D point clouds as explicit guidance during the training stage. We explore the approach of achieving accurate single-image 3D reconstruction by aligning image features with explicit 3D features, marking a novel advancement in the domain.

Q10. Typo

We apologize for the oversight. This typo will be corrected in the revised manuscript.

Thank you for your insightful questions and suggestions. We have provided our responses below:

Q1. Omission of Texture and Texture Extension

We appreciate the reviewer's observation regarding the omission of texture mapping. Our research primarily focused on addressing the limitations of previous methods that rely solely on volumetric rendering for training large-scale 3D reconstruction models, often neglecting the integration of explicit 3D prior knowledge. By aligning image domain features with the 3D domain, our approach significantly improves geometry reconstruction capabilities.

While texture mapping is left for future work, our method can be easily extended to reconstruct both texture and geometry. This would involve modifying the first stage of training by adjusting the autoencoder to accept a colored point cloud, then introducing an additional MLP to decode the interpolated tri-plane features and regress an RGB vector. In this extended approach, the model would decode both the signed distance and the color at corresponding positions. The inference pipeline remains unchanged, as we can use Marching Cubes to extract both geometry and texture. Another potential approach, as suggested by Reviewer vD6Z, involves unprojecting multi-view images generated from models like zero-1-to-3.

Q2. Evaluation Fairness

We appreciate the reviewer’s concern regarding fairness. We believe our comparison is fair because our primary aim is to address the geometry distortion problem caused by the lack of explicit 3D prior knowledge in large reconstruction models. Our method tackles a fundamental aspect of 3D modeling that is crucial for both textured and non-textured models. Additionally, we have included a comparison with the Slice3D model in our global response (Fig. 2 and Fig. 3) to ensure our evaluation covers methods with a similar geometric focus. While D$^2$IM-Net is a pioneering work in disentangling detail in implicit fields, it was trained on ShapeNet, a smaller and less diverse dataset than the large-scale Objaverse dataset we used, making a direct performance comparison challenging. We will discuss both Slice3D and D$^2$IM-Net in the related work section.

Q3. Constrained Generalization Ability

We appreciate the reviewer's observation regarding the requirement for point cloud data during training. Unlike previous rendering-based reconstruction models like LRM and CRM that utilize both 3D assets and multi-view images for training, our approach is constrained to train solely on 3D assets. Despite this, our method demonstrates strong generalization capabilities. While LRM uses 730k 3D assets and 220k videos and CRM employs 376k 3D objects for training, our model is trained on a smaller, category-agnostic dataset of 140k 3D assets. Nonetheless, we achieve state-of-the-art performance on the GSO dataset, which contains over 1,000 3D-scanned real-world items not included in our training data. This highlights the effectiveness of our method in generalizing across varied objects. We argue that although rendering-based methods benefit from larger datasets, their effectiveness is often limited by the lack of explicit 3D knowledge, which can reduce data efficiency. In contrast, our approach, with direct access to 3D representations, enables more efficient learning and improves our ability to generalize across diverse object categories.

Q4. Diffusion

Our motivation for using independent diffusion UNets is outlined in lines 51-53 of the paper, and we have experimentally demonstrated that the design leads to improved alignment quality (Tab. 3 and Fig. 5 of our paper). The UNet-based diffusion process is inherently designed to process 2D planar data, while tri-plane are composed of three 2D orthographic projections of a 3D object. Previous methods, such as 3DGen [11] and NFD [43], either roll out the tri-plane as a continuous plane or concatenate channels based on spatial positions, both of which have significant drawbacks. The roll-out method involves convolution operations that overlap between planes, yet the values at the edges of adjacent planes are not spatially continuous, leading to interference. Conversely, while channel concatenation aligns features at the same spatial location, there is no explicit relationship between them, and applying the same convolution kernel to process them results in interference with the regression of each plane. These issues and their impact on convolution operations are illustrated in Fig. 1 of the global response.

To address these problems, we leverage the tri-plane structure’s property of being three orthographic representations of an object and employed independent diffusion processes for each plane to effectively align image features with planar features. Our ablation study, detailed in lines 305 to 314 of our paper, further supports the effectiveness of this approach. We will also include this additional analysis of the independent diffusion method in the revised paper.

Q5. Various Categories

Unlike methods trained on ShapeNet in a category-specific manner with a limited number of object categories, our approach is category-agnostic and does not rely on predefined categories during training. This strategy enables our model to handle a broader range of objects by focusing on general geometric features rather than specific types. Similar to other large reconstruction models, our training data includes a wide variety of object shapes and structures. To validate our model's generalization capability, we conducted evaluations on the GSO dataset, which is composed of objects not seen during training, as shown in Fig. 3 of our paper. Our model not only successfully handled these unseen objects but also outperformed existing methods, demonstrating robustness across different categories. We also provide supplementary evaluations on data from various sources in Fig. 3 of the global response.