More mathematical symbols and equations

Thank you for pointing this out. We will follow your suggestion to include more mathematical symbols and equations in our revision when introducing the method.

More implementation details

We will follow the suggestion to include more implementation details in our revision, such as the specifics of the network architecture and the weights of the loss terms.

Inference time and memory usage

We followed your suggestion to include a comparison of inference time and memory usage in the rebuttal PDF (please refer to Table 3). We found that while our method generates meshes at the highest resolution (512^3, 8x larger than 256^3), we still maintain a competitive speed and memory footprint.

Normal maps of the mesh (after post-processing) vs. normal maps predicted by the model

Unfortunately, the post-processing we used cannot guarantee that the mesh normals will be completely aligned with the predicted normal maps after processing. This is because the algorithm operates in local space and avoids large vertex movements. Additionally, the predicted normal maps may contain errors or conflicts, such as inconsistent neighboring normals, which cannot be perfectly matched. The adopted algorithm is an iterative numerical optimization method and does not compute an analytic solution.

However, we have quantitatively verified that the post-processing module can significantly improve mesh normal consistency with the predicted normal map. For example, before post-processing, only 26.4% of mesh vertices had a normal angle error of less than 2 degrees. After post-processing, this number increased to 40.8%. For a 10-degree threshold, the ratio increases from 78.8% to 86.4%. For more details, please refer to Table 4 in the rebuttal PDF.

We have also included qualitative examples to illustrate the importance of this post-processing module in recovering sharp geometric details and reducing noisy artifacts induced by the marching cubes algorithm. Please check out Figure 5 of the original paper.

Geometry supervision for real-world training datasets

We agree that image supervision is easier to add when extending to real-world training datasets. However, it is not impossible to obtain corresponding depth maps and even meshes for real-world RGB images, such as through depth sensors or Structure from Motion (SfM). Given the depth map (and even meshes), we can still apply direct 3D geometry supervision. If full 3D shapes are not captured, we can also apply partial supervision to the visible views (only supervising the visible points) while generating the full shape. We acknowledge that this may require more advanced techniques and designs, and we leave it as a promising avenue for future work.

Table 3, row (a):

Yes, for Table 3, row (a) of the paper, we remove the normal input but preserve the normal output and normal supervision.

2D diffusion models drastically slow down the reconstruction speed

We currently use the normal ControlNet of Zero123++ to generate the multi-view normal inputs. It tiles the multi-view RGB images as a single input condition and generates the tiled normal maps in a single diffusion process, which takes approximately 4.1 seconds on an H100 GPU. For the application of single-image to 3D, generating tiled multi-view RGB images takes about 3.6 seconds. The total time on the 2D diffusion model side is only around 7.7 seconds, which is still acceptable.

No real-world images tested?

We would like to clarify that one of our main testing datasets, OmniObject3D, is a real-world scanned 3D dataset. In addition, we also include some qualitative examples with real-world input in our rebuttal PDF (see Fig. 3), where MeshFormer performs quite well.

The term 'open-world' means that MeshFormer differs from many previous methods (such as A, B, C listed by the reviewer). Those methods are trained on datasets with a limited number of object categories (e.g., tens of categories in ShapeNet) and cannot generalize to novel categories. Unlike those methods (e.g., 3D native diffusion), MeshFormer takes as input sparse-view images and normal maps generated by 2D diffusion models, and demonstrates much stronger generalizability. MeshFormer is thus not limited to the training 3D dataset and can handle arbitrary object categories.

Ablation study of 2D model errors

We would like to clarify that, in Tab. 3 of the paper, we have analyzed the influence of errors from 2D normal models (rows f and g). Additionally, we provide some qualitative examples in Fig. 8.

For the effect of multi-view RGB, please compare Tab. 1, row ‘Ours’ (predicted RGB) and Tab. 3, row f (ground truth RGB) of the paper.

Claims about technical novelties

Thank you for pointing this out. We will cite these prior works and discuss the differences in our revision. We fully understand your concerns and would like to address the potential misunderstanding in detail.

Point 1

The reviewer summarized that the "primary contribution lies in adopting a voxel-based 3D representation and employing a network architecture …." We respectfully disagree with this argument. Our main claim is that by proposing a 3D-guided reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision, MeshFormer can significantly improve both mesh quality and training efficiency. Our main findings/contributions include:

• (a) Using normal images as additional input in feed-forward reconstruction models greatly enhances the prediction of geometric details.

• (b) Proposing to learn and output a 3D normal map, which can be used for further geometric detail enhancement.

• (c) Combining SDF loss and rendering losses in training feed-forward reconstruction models enables a unified single-stage training process. In contrast, concurrent works rely on complex multi-stage "NeRF-to-Mesh" training strategies to export high-quality meshes (e.g., MeshLRM, InstantMesh) and struggle to generate high-quality geometry.

• (d) Explicitly leveraging 3D native voxel representation, network architecture, and projective priors fosters faster convergence speeds and significantly reduces the training requirement.

We would like to emphasize that all four points are crucial to MeshFormer. We thus do not agree that our primary contribution lies solely or primarily in adopting 3D representation and network architecture. For example, without points (a), (b), and (c), our mesh quality would be significantly compromised.

Point 2

We totally agree that the utilization of 3D voxel representation is common in general 3D generation. However, all works listed by the reviewer (A-E) focus on 3D native diffusion, one of the paradigms in 3D generation, which differs from the route of MeshFormer. There are some common limitations of this line of work. For instance, all of A-E focus on geometry generation only and cannot predict high-quality texture directly from the network. Also, due to the limited amount of 3D data, 3D native diffusion methods typically struggle with open-world capability and focus on closed-domain datasets (e.g., ShapeNet) in their experiments (A, B, C).

In MeshFormer, we aim to achieve direct high-quality texture generation and handle arbitrary object categories. We are thus following another route: sparse-view feed-forward reconstruction, instead of the 3D native diffusion. In this specific task setting, many of the works provided by the reviewer are not suitable for comparison in our experiments. More comparable works are recent LRM-style methods (e.g., InstantMesh, MeshLRM, LGM, TripoSR, etc). However, most of them only utilize the combination of triplane representation and large-scale transformers.

In our paper, we do not claim to be the first to use voxel representation in 3D generation. Instead, we would like to share our findings:

• In this specific task setting (open-world sparse-view reconstruction with feed-forward textures), we are not limited to the triplane representation. 3D native structures (voxels), network architectures, and projective priors can facilitate more efficient training and significantly reduce the training resources required (from over one hundred GPUs to only 8 GPUs).
• In this specific task setting, we require a scalable network to learn a lot of priors. However, not only triplane-based transformers can be scalable. When marrying the 3D convolution with transformer layers, it can also be scalable.
• In addition to using 3D native representation and networks in 3D native diffusion, we can also combine them with differentiable rendering to train a feed-forward sparse-view reconstruction model with rendering losses.
• For image conditioning, C and E only take a single image as a condition. D first employs max pooling across multi-view features and then uses cross-attention across the pooled multi-view features and the voxel feature. The max pooling is typically affected by occlusion (voxels are not visible in all views) and thus becomes less effective. Instead, we propose using cross-attention across all multi-view projected image features and voxel features, which can implicitly leverage structure and visibility priors to focus only on visible regions. We demonstrate that this strategy is more efficient than the pooling strategies used in D as shown in Tab. 3 of the paper and Tab. 1 of the rebuttal PDF.

Experiments tried/ablated but did not show significant differences

We are happy to follow the reviewer's suggestions to include more discussions about the experiments we have conducted in our revision, such as:

• the difference between joint training and separate training of the dense model and sparse model;
• the difference between max-pooling, mean-pooling, and cross-attention in projection-aware feature aggregation;
• the comparison of inference time and memory consumption with baseline methods;
• mesh generation quality over training time;
• quantitative analysis of the geometry enhancement module;
• more qualitative examples on real-world images.

Please let us know if there are any additional experiments you are interested in.

Mesh generation/reconstruction quality over training time

Our MeshFormer can be trained efficiently with only 8 GPUs, generally converging in roughly two days. We have followed the suggestion to include a quantitative analysis of our mesh generation quality over training time. As shown in Table 2 of the rebuttal PDF, the performance quickly improves and nearly converges with marginal changes after the two-day training period.

Thin structures

We would like to clarify that the loose thread of the toy was not displayed due to a slight pose mismatch when visualizing the results. In fact, the loose thread is reconstructed by our MeshFormer. We have included additional views of our generated results (see Figure 1 of the rebuttal PDF), where the loose thread can be observed. You can check the video on our website.

MeshFormer generates high-resolution (512) SDF volumes, which are sufficient to preserve very thin structures, as shown in Figure 2 of the rebuttal PDF.

The name "VoxelFormer"

The meaning of "VoxelFormer" is two-fold. On the one hand, it refers to combining voxel representation and transformer layers, in contrast to recent LRM-based methods that rely on triplane representation and transformers to achieve scalability. On the other hand, it can also be interpreted as Voxel-Form-er, meaning the module that generates the voxels. We are open and happy to consider other names if the reviewer has more suitable suggestions.

Occlusion-aware feature aggregation

We agree that occlusion-aware feature aggregation is very important. This is the primary reason we use a cross-attention layer to aggregate the projected multi-view features instead of using a simple average or max pooling method like previous approaches. We hope the cross-attention layer can implicitly utilize prior knowledge of coarse structure and visibility to focus on the visible views. Our experiments also verify its superiority over mean pooling aggregation (Table 3 (d) of the paper). While we could explicitly filter out some views according to the predicted occupancy from the first stage, as the reviewer suggested, we would like to point out that the occupancy band has some thickness, and accurately determining the visibility of each voxel can be quite challenging. This may require more advanced techniques, and we leave this as a promising future direction.

Table 3 (d)

We followed the suggestion to add an ablation variant using the max-pooling aggregation mechanism (please refer to Table 1 of the rebuttal PDF). We found that while the max-pooling aggregation performs slightly better than average pooling, it is still significantly inferior to our projection-aware cross-attention mechanism.

Yes, we follow the "skip-connection" scheme of the typical UNet to concatenate the voxel features before the bottleneck with the voxel features after the bottleneck. If this is not what you are asking, please let us know.

Shared backbone and Plücker embedding

Yes, we use a shared backbone (trainable DINOv2) for both RGB and normal images. We did not use Plücker embedding in our experiments. Instead, we leveraged the camera poses to unproject 2D image features into 3D feature volumes.

Detailed description of the sparse VoxelFormer architecture

Yes, we will follow your suggestion to include more details about the Sparse VoxelFormer architecture in our revision, such as the number of sparse convolution layers used at each resolution.

Joint training

Yes, we began our experiments by training the dense model and sparse model separately. However, we found that this approach leads to a domain gap for the occupancy field during inference, as the predicted occupancy by the dense model may be imperfect, while the sparse model is only trained with ground truth occupancy. Joint training can mitigate this gap and reduce artifacts during inference.

Interpolation from $256^3$ to $512^3$

Thank you for pointing this out. We will add the missing description in our revision. Specifically, we generate a sparse feature volume with a resolution of 256, and then trilinearly interpolate it to a sparse feature volume with a resolution of 512. This interpolated sparse features are then fed into the SDF decoder for predicting SDF values, which are subsequently used to compute the loss against the 512-resolution ground truth SDF.

Multi-view normals for the teaser

Yes, we use GT multi-view normals for the teaser. We will make this clearer in our revision.

XCube

Thank you for pointing this out. We will cite XCube and discuss it in our revision. We agree that XCube shares many high-level ideas with us, such as hierarchical sparse voxel representation and coarse-to-fine generation. However, we would like to point out that they follow the paradigm of the 3D native diffusion, which can only generate geometry but fails to directly predict texture from the model. In contrast, we follow the paradigm of the feed-forward sparse-view reconstruction and incorporate differentiable rendering into the pipeline, which enables the network to directly generate high-quality texture. Additionally, we combine 3D (sparse) convolution with the transformer layer to increase capacity and scalability of the network, while they mainly rely on 3D convolution only and may be limited in both capacity and scalability.