Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images

W-1: Does the proposed approach rely heavily on the quality of back-view generation, and are there empirical studies or ablations to address its impact on the fusion process?

Your concern seems to be about conducting ablation studies on the back-view generation method. In our original paper, we used Zero-123, Stable-Zero123, and Zero123-XL, and our experiments showed that the 3D generation results were fairly similar across these methods. In fact, the resolution of the back-view images (224224) generated through back-view generation methods is much lower compared to the front-view images(9601200). Therefore, the performance of Tailor3D does not heavily rely on the quality of back-view generation.

W-2: Would incorporating more views increase the complexity of the proposed approach?

No, it won’t. We fuse the triplanes generated from different views, allowing us to directly utilize the pretrained LRM models. As a result, we only need to fine-tune the module responsible for fusing the triplanes, which does not increase the computational complexity.

W-3: Equtation 4

The meaning of Equation 4 is as follows: The initial triplane embedding is denoted as $f_{init}$. For both the front and back views, we set the camera extrinsics as $E_{t}$. The $'$ symbol represents the edited images. Here, $F_{f}^{'}$ refers to the features of the edited front-view image, which are used as input to generate the triplane $T_{f}$, the triplane derived from the front-view image. Similarly, $F_{b}^{'}$ refers to the features of the edited back-view image, which are used as input to generate the triplane $T_{b}^{f}$. However, $T_{b}^{f}$ needs to be rotated by 180 degrees to become $T_{b}$, which can then be fused with the triplane $T_{f}$.

Thank you very much for your response. Let me address some of your questions below.

W-1&2: Some typo about the paper.

Thank you so much for reading so thoroughly; we will make sure to address and correct this in future versions.

W-3: Why does inconsistency still exist between the front and back views if Dual-LRM aims to address multi-view editing inconsistencies?

Our explanation might not have been clear enough—multi-view inconsistency actually has two distinct aspects.

• The first aspect is multi-view inconsistencies during the reconstruction process. If the front and back images are inconsistent (e.g., the cameras are misaligned), Dual-LRM is capable of handling this issue.

• The second aspect is multi-view inconsistencies during editing. Editing multiple views simultaneously is challenging—for instance, if I want to change someone's outfit, it’s quite difficult to ensure that all views are updated consistently to reflect the change. By using only the front and back views, we can largely avoid this issue.

W-4: The paper lacks sufficient comparisons, as methods like InstantMesh and CRM also use multi-view images for 3D generation.

First, our method is not aimed at enhancing the quality of 3D object generation but rather at providing an editing approach. Additionally, both CRM and InstantMesh use Zero-123++ to generate six views from a single image before reconstruction, so the setting is somewhat different.

W-5：The technical novelty is limited, and a comparison with InstantMesh using edited two-view inputs is recommended.

Our initial intention was to avoid re-pretraining. We aimed to achieve results by simply fine-tuning the model parameters of LRM. In contrast, methods like Instant3D and InstantMesh require re-pretraining.

Additionally, InstantMesh requires six fixed-view inputs, which do not include the front and back views. As a result, this approach cannot be directly applied to InstantMesh. However, as you mentioned, multi-view image editing is indeed a very critical direction for future work.

Q-1: What are the advantages of single-view editing over multi-view diffusion models, given the potential for inconsistencies even with front and back views?

As mentioned in the paper, methods using multi-view diffusion primarily optimize for each individual object or scene rather than serving as a general framework. Similar to NeRF, these approaches require optimization for each object to be edited. In contrast, the LRM-based method is a general framework that uses a feed-forward approach. Additionally, as shown in Figure 8, we provide a comparison with MVEdit (multi-view diffusion). Moreover, multi-view diffusion currently only supports text-driven editing and cannot modify specific parts of an object.

Q-2: What are the advantages of the proposed LoRA Triplane Transformer over training an LRM with two-view inputs, beyond memory cost, and how does its performance compare to directly training a Dual-side LRM from scratch?

The main purpose of using LoRA here is to reduce memory usage. The original pretraining structure was overly redundant and required 64 A100 GPUs for pretraining. After adopting the LoRA structure, our model can be fine-tuned on a 4090 GPU. I forgot to mention this in the paper because we used A100 GPUs at the time to speed up the experiments.

We greatly appreciate your response. Regarding your question, we have conducted additional experiments, and the results are presented in the PDF of the supplementary materials. In Figure 1, we show the back-view editing process of InstantMesh as you mentioned, along with our own editing process. The results demonstrate that InstantMesh has very high requirements for viewpoint consistency. After editing, when the images are reconstructed using InstantMesh, they fail to produce correct results. For example, the sofa shows holes and missing parts. In contrast, Tailor3D successfully avoids these issues and produces a more reliable reconstruction.

In Figure 2, we present more examples. It is important to note that these examples are all from the original paper and are not cherry-picked. In addition to the failures caused by viewpoint inconsistencies, InstantMesh can even alter the original shape of the object. For example, in the case of Captain America, the original input image shows that the figure's legs are short and thick, but after processing with InstantMesh, the legs become long and thin. Tailor3D, on the other hand, better preserves the features from the original input image.

W-1: What is the key technical contribution, given that the method combines existing approaches and uses a new front-and-back view data format?

As you mentioned, our core idea focuses on a dual-view image editing paradigm and the method for fusing triplanes obtained from different perspectives, rather than pursuing the highest generation quality.

W-2 & W-3: The generated 3D results are poor, highlighting concerns about the reliance on low-quality 3D generative models for real-world applications.

Tailor3D is not about achieving the highest generation quality but is designed to be compatible with any 3D generation method based on triplanes. Therefore, it can be extended to most 3D object generation frameworks. Additionally, this structure can be fine-tuned on a personal computer without the need for large-scale pretraining. For these reasons, we believe Tailor3D holds practical value for real-world editing applications.

May I ask why you changed it from 6 to 5? Did my response not meet your expectations?

Since my initial response needed to address all of your questions comprehensively, some details may have been overlooked. Here, we will focus on addressing the first question in detail.

You're absolutely right, using two views does seem a bit unusual. By saying it is a intial exploration, we does not mean that the proposed scheme is built on a non-realistic foundation. Instead, we think it is a simple, but indeed reasonable (enough for reconstruction using double-side information) and efficient (requires less information compared with other multi-view schemes).

It is impossible to fuse image features and generate a single triplane through fine-tuning alone; pretraining is required.

The structures of LRM and DMV3D (which can also be considered a type of MV-LRM) are significantly different. Please note that in LRM, the camera parameters are input into the triplane transformer in a modulated form, whereas DMV3D and Instant3D do not use this approach. This difference makes the two frameworks independently pretrained frameworks. Therefore, as you suggested, directly concatenating image features to generate a single triplane would require re-pretraining, which would take at least 64 A100 GPUs for a week on Objaverse. In contrast, Tailor3D can achieve results similar to DMV3D by simply fine-tuning on top of LRM. (Moreover, neither DMV3D nor Instant3D has been open-sourced.)

Tailor3D is a fast and efficient module that extends LRM or any single-view reconstruction framework into a dual-view editing framework. Our core contribution lies in proposing this pipeline, which can be applied to extend any single-view reconstruction into a dual-view editing framework. Dual-view editing is already sufficient to meet the requirements of most 3D editing tasks.

Thank you so much for reading our paper so thoroughly. Your insights and reflections on the work even surpass my own as the author! These are incredibly constructive comments!

W-1(1): Why is memory usage a critical concern, and is LoRA truly necessary for the model?

The purpose of using LoRA is to reduce memory usage. Tailor3D can be fine-tuned on an RTX-4090. Tailor3D is essentially a fine-tuned version of the LRM framework. However, to accelerate the experiments later on, we used an A100.

W-1(2): Why not directly fine-tune the LRM model using features from both images to generate a single triplane, as it seems more natural and straightforward?

Concatenate the image features of the front and back views, or even more views (such as the four views in DMV3D), and then use a transformer to generate a triplane. This is indeed the approach adopted by most papers currently.

While working on this project, the third-party reproduction code for LRM was the only available open-source implementation, as none of the other methods had been made publicly accessible. We also aimed to extend the LRM from single-view to multi-view efficiently. However, as illustrated in Figure 10, a fundamental difference exists between LRM and Instant3D (MV-LRM) in how they handle camera parameters.

• LRM injects the camera parameters into the image-to-3D triplane process.
• Instant3D integrates the camera parameters into the image backbone.

These structural differences make it infeasible to directly fine-tune LRM to produce Instant3D’s outputs.

So we considered an alternative approach: generating two triplanes from two images and then combining these triplanes at the 3D level to integrate the information. This design allows us to leverage the pretrained LRM model parameters and achieve two-view results with minimal fine-tuning. Additionally, no prior work has explored fusing multi-view information directly at the triplane level, which motivated us to attempt this approach.

W-1(3): Is the dual-side image setting truly necessary, or could using four views with fine-tuning for edits be a more flexible solution?

You're absolutely right—using two views does seem a bit unusual. The reasons we chose this paradigm are as follows.

• Initial exploration: Tailor3D is an initial exploration, and focusing on the front and back views is likely the simplest scenario.

• Sufficiency: Editing with just the front and back views is sufficient to meet the editing needs of most objects. Achieving highly accurate editing across four views is very challenging due to the lack of multi-view consistent image editing approaches.

• Experimentation: We attempted generating four triplanes from four images (front, back, left, and right) and combining them into a single triplane through rotation, but this approach proved unsuccessful.

Currently, rotating triplanes only works for a 180-degree rotation (i.e., between the front and back triplanes). Rotating the left and right triplanes by 90 degrees is not feasible, as the features become misaligned after rotation.

W-2: How does the proposed method achieve single-image generation and why does it outperform others despite relying on dual-side inputs?

For Tailor3D, the front-view image is used as input, while the back-view image is generated using multi-view image generation algorithms like Zero-1-to-3[1]. These front and back views are then fed into the model. Similarly, other methods in the table, such as TriplaneGaussian, Wonder3D, and LGM, follow a similar approach: a single front-view image is used to generate 4-6 additional views, and these generated images are then used for reconstruction. While this approach may seem somewhat unusual, it is indeed categorized as a single-image method.

W-3: Could higher-level ablations, such as training LRM to directly output a triplane from two image embeddings, better support the design decisions in the paper?

You make an excellent point. Our motivation at the time was to fine-tune LRM to achieve our current results. If we were to merge 2D features, it would require large-scale pretraining from scratch as mentioned in W-1(2), which would be extremely costly. Do you think this experiment is necessary? If you feel it is, I can consider adding this ablation experiment.

[1] Zero-1-to-3: Zero-shot One Image to 3D Object.

I just want to say that, regarding the response of why dual-side is necessary, I think using "this is an initial exploration" is NOT really convincing and weird response... If a setting is not a valid and reasonable setting, then I think this setting should be discarded during initial exploration and work on a more reasonable setting... Overall, I am not saying in this case the two view setting is really a bad idea, just try to point out that the response makes me even unsure whether the setting is reasonable or not...