Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling

We sincerely appreciate the detailed feedback you provide for our work! Below, we provide the following clarifications to address your concerns:

---

W1 & Q1: Which 3D representations are more suitable?

Thanks for the valuable comment! Since our tracking is performed on rendered 2D images from the 3D assets, the performance relies on the quality of the rendered images, rather than solely depending on the type of 3D representations (e.g., NeRFs, 3D Gaussians, 3D mesh). Basically, better cross-view consistency of the generated 3D assets would lead to better performance in tracking, because they have less floating artifacts that can distract the tracking model.

For the performance of the whole pipeline, it is better to adopt the 3D generation methods that can generate decent geometric shapes of the subject. The reason is that our pipeline is designed mainly to fix the unnatural appearance issue to make the generated assets match the identity of the given source view, with little modification on the geometry and shape of the subject (there are geometric fixes from our pipeline, as demonstrated in Fig. 6 in the main paper and Fig. I in the appendix, but they are not major changes to the geometry). Therefore, starting from a 3D method that can produce decent geometry and shape is more favorable for our method to achieve optimal performance.

We have prepared interesting visual comparisons for ablations on different methods with different 3D representations. However, due to the policy of the rebuttal not allowing for PDF or other types of media, we choose to describe the qualitative comparisons in detail to the reviewer. The qualitative comparisons requested for the following questions are addressed in the same way.

Description of Visual Comparisons

If solely focusing on the tracking performance, methods like Wonder3D with 3D mesh as representation produces slightly better tracking results than methods like LGM with 3D Gaussians and SV3D with NeRFs, because 3D mesh generally has fewer floating artifacts than NeRFs and 3D Gaussians, which makes it easier for tracking.

However, for producing a better final generated 3D asset, we empirically find that LGM has an overall better capability of producing realistic shapes and geometry of the subjects, better handling the depth of the subject along with the spatial relationship between the objects. Although the tracking performance on LGM may not be as accurate as Wonder3D (but not far away from Wonder3D), we observe that our method can offset the effect in the following Resplat stage when rectifying the multi-view geometry, generally being robust towards the minor inaccuracies from the tracking stage.

Therefore, using LGM helps us yield the most pleasant results for the final generated 3D assets, which is the reason for our current implementation to use LGM as the initial 3D generation method in the pipeline.

---

W2 & Q2: Normal tracking v.s. backward tracking

Thanks for the question! In our method, backward tracking is to stack the frames from the target view to the source view to form a video, and then use the video tracking tool CoTracker to track the pixels in the target view. If the tracking result of the pixel gets lost or falls outside the valid regions of the source view, it means the pixel needs to be inpainted in the following Inpaint stage. The only difference between the videos for forward tracking and backward tracking is the camera movement direction. Therefore, the video tracking tool CoTracker can well handle the videos with reversed-order frames as normal.

For comparing normal tracking (forward tracking) and backward tracking, backward tracking is almost always a better choice than normal tracking, both from the design principle and the experimental experience. For the underlying design principle of backward tracking, we can make an analogy to image warping to better understand this. There are two warping solutions: (1) Forward mapping, and (2) backward mapping (reverse mapping). In forward mapping, the algorithm iterates all the pixels in the source image $I_1$ to map their values to the target image $I_2$. However, this can lead to "holes" in the target image because many pixels in $I_2$ may not get mapped from a certain pixel in $I_1$. In comparison, when performing backward mapping, any pixel in $I_2$ that should be mapped to a certain pixel in $I_1$ can successfully find its corresponding pixel in $I_1$. The problem of backward mapping, in the image warping context, is that pixels in $I_2$ which are occluded or do not exist in $I_1$, cannot get its value during the mapping process. However, in our context of backward tracking, it is exactly what we are searching for! The goal of the Track stage is exactly to identify the pixels in the target view that are occluded or missing in the source view. Therefore, our backward tracking design perfectly matches our need for figuring out which parts in the target views need to be inpainted. Additionally, from our experimental experience (Fig. 3 as an example), backward tracking is indeed a more suitable solution for our pipeline, which matches the design principle of our backward tracking algorithm.

---

W3 & Q4: Figure 1 design

Thanks for the suggestion! We will revise the figure to remove the icon of the thinking person, and reorganize the components of the figure to better convey the broad picture of our method.

---

Q3: The correlation between the progressive design and loss design, and ablations on degrees of progressiveness

Thanks for the comment! If we understand correctly, the reviewer would like to know about the correlation between our adopted inpainting loss (Eq. 7) and the general diffusion-based inpainting loss in works like RealFill [a]. The loss calculation steps between these two are the same, and the difference is that we are performing an innovative progressive inpainting strategy. This is because our task poses additional challenges compared to the original RealFill setting, which requires the model to infill the viewpoints that have significant variance from the source viewpoint. If we only train the inpainting model with the source view, the model will not be competent enough to fill in the textures for the side and back views that have large variations.

For the ablations on the degrees of progressiveness, we have also prepared informative visual comparisons for the reviewer, and the description of the qualitative comparisons is shown below:

Description of Visual Comparisons

Basically, for the degrees of progressiveness smaller than the current choice (current choice is $20^\circ$, and we choose $10^\circ$ for ablation), the visual results are very similar to the current results, but we need additional steps to calculate the inpainting masks for the increased number of training stages. The total training time can roughly stay the same, because we can half the iterations needed for each stage and still get comparable results. For the degrees of progressiveness larger than the current choice (we choose $30^\circ$), we could observe that the regions (especially for those that are far away from the reference view, e.g., the regions that are close to the back view for the target views of $\pm 90^\circ$) are inpainted worse than the current implementation. Our current choice of $20^\circ$ of progressiveness balances between the number of frames that need for tracking, and the difficulty of inpainting the unseen regions that are far away from the source view.

If the reviewer still has questions regarding the correlation between the current progressive design and loss design, please feel free to further clarify this question and we are happy to answer!

[a] Tang et al. RealFill: Reference-Driven Generation for Authentic Image Completion. ACM TOG 2024.

---

L1: Error accumulation with the multi-stage solution

Thanks for the comment! It is true that multi-stage solutions have the disadvantage of error accumulation. However, our three stages are well orchestrated, with the earlier stage striving to make the task of the later stage easier. At the same time, the later stage is designed to make up for the errors which are caused by the inaccuracies from the earlier stages. For example, one of the purposes to use backward tracking is to produce inpainting masks that are more suitable for the following Inpaint stage. The Resplat stage refines the multi-view consistency with mask-aware latent updates to mitigate the error coming from the prior Track and Inpaint stages. Therefore, the three-stage solution is cohesively integrated with each other, instead of just functioning as three independent algorithms.

---

L2: Limitation on progressive rotation restoration

Thanks for the insightful comment! Subjects with complex patterns are indeed challenging for our current model, which are also the hard cases for other 3D generation models. However, we believe that our work can provide inspiration on how to utilize the powerful 2D tools like tracking and inpainting for solving this challenge. For the segmentation lines, it can be largely suppressed with using soft masks during inpainting. Basically, at the mask boundaries we apply feathering to blur the mask, then use the blurred soft mask for integrating the original image and the inpainted image. With this simple operation, the segmentation line issue can be significantly addressed.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns.

We sincerely appreciate the constructive feedback you have given for our work! Below, we provide the following clarifications to address your concerns:

---

W1: More clarification on the task setting

Thanks for the comments! It is true that our setting is a bit different from the traditional personalization setting of using generative models to generate subjects with novel text prompts. In comparison, our setting aims at generating 3D/4D assets that correspond to the identity of the subjects when observing from different views. Both settings are generating something of the specific subject in certain conditions. The traditional personalization setting is "under certain text prompts", while our setting is "under certain viewing directions", which is the reason for us to name our task "subject-driven 3D/4D generation". We will make it more clear in the final version of our paper.

---

W2: More diverse samples

We indeed have evaluations on in-the-wild data as mentioned in Lines 247-249, and the results are shown in Fig. F and the vase example in Fig. G (though only the vase is a non-animal object). The results seem to be biased towards animals because animals are the most common objects that are used for dynamic scenes, as for the examples shown in the L4GM website, animals also take the majority of their chosen objects. However, our work can definitely work on other subjects beyond animals. First, many of our examples are scenes of animals interacting with certain objects like apples and violins, which demonstrates that our method is applicable to other objects to certain extent. Second, we have prepared interesting visual comparisons and want to show the reviewer of how our method is capable of enhancing identity preservation for more general objects. Due to the policy of the rebuttal not allowing for PDF or other types of media, we choose to describe the qualitative comparisons in detail to the reviewer.

Description of Visual Comparisons

We would like to describe two cases for demonstrating the capability of our method. Both cases are objects from in-the-wild images. The first case is a house covered with snow on the roof. One obvious issue for other 3D generation methods like SV3D and LGM is that they fail to preserve the property of the covering snow on top of the house, while our method is capable of inpainting the snow pattern on the roof from the back view. The second case is a traffic cone with red and silver stripes. Other 3D generation methods like SV3D and LGM mess up the stripe patterns and produce unnatural colors. With the assistance of our method, the texture and appearance of the traffic cone get well preserved.

---

W3: 4D results

We do have 4D reconstruction results in the paper. We would like to kindly point out to the reviewer about our 4D results as follows. Figs. 5, 6 in the main paper and Figs. F, I, J in the appendix show the snapshots of different timesteps from different views for the 4D assets. Also, the user study is conducted in the 4D setting, as presented in Sec. A in the appendix. Participants are shown with a series of videos showing the comparisons of the 4D scenes for scoring. For animable demonstrations, there are also examples in the video in the supplementary material (03:15 - 04:12) for reference. Therefore, we do have abundant 4D results and demonstrations for the paper.

---

W4-1: Evaluation on identity preservation

Thanks for the suggestion from the reviewer! After checking the VBench paper, we get inspired to use vision-language models (VLMs) for evaluating the identity preservation capability of the models. We follow the setting of DreamBench++ [a] for evaluation, and use a variety of VLMs with different architectures and sizes to ensure the reliability and soundness of the evaluation. The results are shown as follow:

We can observe that (1) our method consistently outperforms the state-of-the-arts TRELLIS and Hunyuan3D-2.5 in the quantitative evaluation regarding identity preservation and generation quality; and (2) different VLMs share consistent scoring rank for different models, demonstrating the robust generalization of the adopted evaluation protocol.

[a] Peng et al. DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation. ICLR 2025.

---

W4-2: User study on identity preservation

Identity preservation is actually already taken into account in the user study. In the instruction of the user study, as shown in Sec. A in the appendix, we can see that there is the instruction "Whether different viewpoints of the generated results look alike the subject in the given reference image" (Lines 623-624) that informs the participants that whether the identity of the generated novel views match the given source view is a crucial factor for the scores. Therefore, we do ask the participants to focus on identity preservation, it is just we avoid to explicitly ask them to specifically focus on this point.

The reason for us to not explicitly emphasize this point is that we want to foster the fairness of the user study. If we explicitly mention that our project is about identity preservation, the outcome will be obvious because our method is designed to fix this issue. It will make participants consciously feel in preference of our method. Therefore, our current user study setting is more fair, and can reflect the overall quality of all the methods.

---

W5: Diffusion model details in Section 3.1

Thanks for the suggestion! The original reason for us to have a relatively long Sec. 3.1 is because we want to present the equations for diffusion models, especially Eqs. 4 and 5, that are needed for reference when introducing the inpainting loss (Eq. 7) and mask-aware multi-view refinement (Eq. 8). We will definitely shorten the preliminary for diffusion models and present more interesting qualitative results in the main paper!

---

Q1: Initial 3D model generation

Thanks for pointing out! In our current implementation, we use LGM for lifting the image from 2D to 3D. Actually, the method for lifting from 2D to 3D can be arbitrarily chosen, because our tracking and inpainting infill the unobserved regions with only 2D models. Therefore, for any chosen method to lift the image from 2D to 3D, we can always render multi-view observations from whatever 3D representation is used, and apply our algorithm afterwards.

---

Q2: Prompts used during progressive inpainting

We thank the reviewer for the valuable question! The inpainting process can actually work well without the class name of the subject! The justification for us to choose this implementation is to follow the solution in RealFill [b], which demonstrates that the inpainting process is still feasible without having the object name or the scene description in the text prompts. We keep this setting in our work, so it serves as a more general and simpler solution for inpainting the scene. The reason for the model being able to inpaint without the guidance of the text prompt is that the special token $\hat{V}$ (or sks) is supposed to be optimized for capturing the content of the scene, instead of only representing the identity of the object. This is because the only task the model needs to focus on is to inpaint this specific scene, instead of generating the object in novel text prompts like the traditional setting mentioned in your previous comment of W1: More clarification on the task setting. Therefore, the model still works well without mentioning the class name or concrete contents of the scene.

Nevertheless, your comment is indeed very inspiring, as including the scene content in the text prompt would definitely have the potential to further improve the inpainting performance!

[b] Tang et al. RealFill: Reference-Driven Generation for Authentic Image Completion. ACM TOG 2024.

---

Q3: Quantitative evaluation

The original quantitative evaluation in the paper is to showcase the limitation of the existing evaluation metrics in subject-driven generation. Now, with your valuable suggestion, we are delighted to present a more reliable and sound metric as mentioned in the response to your question W4-1: Evaluation on identity preservation, and it shows the superiority of our method on identity preservation compared with current state-of-the-arts.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!

We are sincerely grateful for your constructive feedback! We provide the following clarifications in response to your concerns:

---

W1 & Q1: Ablation without inpainting

Thanks for expressing the concern! Without inpainting, the 3D assets would largely be unready, which are caused by mainly two reasons:

(1) The color component from Wonder3D mismatches with the underlying geometry of the 3D assets, which results in incorrect color fulfilling and messy textures for many parts of the 3D assets.

(2) The color from Wonder3D is only for guidance and still away from good identity preservation quality. Even for the most recent 3D methods like Hunyuan3D-2.5, the identity preservation still needs to be improved.

For this ablation, we have prepared very interesting qualitative results for the reviewers. However, due to the policy of the rebuttal not allowing for PDF or other types of media, we choose to describe the qualitative comparisons in detail to the reviewer.

Description of Visual Comparisons

We use the "dog eating apple" example shown in Fig. 3 for easy reference. Without inpainting, there are many obvious flaws from the results. For example, on the right view of the scene, the dog's nose has a large grayish patch. The dog's body is also covered with messy dark brown patterns. Therefore, it is still far away from a good 3D asset for decent identity preservation. After the inpainting is done, the large grayish patch on the dog's nose gets infilled with reasonable color, and the textures on the dog's body gets more realistic and coherent with the given source view.

Therefore, the inpainting part indeed plays an indispensable role for achieving good results with decent identity preservation. We will present this ablation result in the final version of our paper.

---

W2-1 & Q2: Inconsistencies introduced during inpainting and temporal consistency across frames

Thanks for the insightful comments! Your intuition is mostly correct! With our designed solution of progressive inpainting, we indeed manage to avoid the inconsistencies to a great extent. The remaining inconsistency is mainly caused by the inaccurate tracking results from the tracker. Once the tracker yields a false negative (a position in the target view should be tracked to a valid position in source view, but the tracker gets lost), the inconsistency emerges. Basically, we fail to propagate the pixel values from the source view to the positions of the false negative pixels in the target view. Consequently, since the pixel loses track from the source view, it will be wrongly marked as "inpainting needed", which results in inconsistent inpainted appearance.

The false negatives are most commonly seen at the boundaries of the objects. As we can observe from Fig. 6 in the main paper and Fig. I in the appendix, the inconsistent geometry is most obvious around the object boundaries.

---

W2-2 & Q3: Temporal consistency for video-to-4D

In this work, we are mostly focusing on identity preservation of the generated 3D/4D assets, which is often a static attribute of the subjects regarding their appearance. Therefore, we do not have special designs tailored for temporal consistencies, but we observe that it can be decently handled with the base 4D generation models we adopt. The multiview-to-4D lifting model that we are using, L4GM, enforces temporal consistency from its training process. Specifically, for a given video sequence from the source view, together with the multi-view observations for the first frame generated by our method, the base 4D generation model is capable of generating multi-view dynamic videos with decent temporal consistency.

Essentially, what we propose is a unified solution for both 3D and 4D generation, while for 4D generation we only need to generate multi-view observations for static assets, and pass the tasks of handling temporal consistency to the 4D base models. This unified and general solution grants our method more compatibility with the ever-evolving feed-forward 4D generation models (e.g., Splat4D [a], 4D-LRM [b]) in the future, and can benefit from their strong capability of maintaining temporal consistencies.

[a] Yin et al. Splat4D: Diffusion-Enhanced 4D Gaussian Splatting for Temporally and Spatially Consistent Content Creation. SIGGRAPH 2025.

[b] Ma et al. 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time. arXiv:2506.18890.

---

W3 & Q4: Clarification on denoising schedule

Yes, the understanding from the reviewer is correct! 30% of the denoising schedule corresponds to setting the strength parameter in the inpainting pipeline to 0.3 in practice. We will clarify this explicitly in the final version of the paper. Thanks for the suggestion to make our paper more clear!

---

W4: Text prompts for inpainting

Thanks for the thoughtful comment! We believe that it is definitely a good idea to extract a general class name for guiding the inpainter. The reason for us to not have the general class name in our current version is that we follow the exact solution in RealFill [c], which also did not extract a general class name for guiding the inpainting process. We find this solution also generally works well in our setting, so we keep this simple and general solution in our work. Nevertheless, your comment is indeed very inspiring, as including the scene content or class name for the text prompt would definitely have the potential to further improve the inpainting performance! For the notation of the rare token, we will revise the corresponding text in our final version.

[c] Tang et al. RealFill: Reference-Driven Generation for Authentic Image Completion. ACM TOG 2024.

---

T1: Typo

Thanks for the careful reading and spotting the typo! Yes, in Eq. 7, a squared norm should be included in the loss term!

---

We would like to sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!

We sincerely appreciate the detailed feedback from you for our work! We provide the following clarifications in response to your concerns:

---

W1 & Q1-1: Significance and the scope of the paper

We would like to point out that the problem we are solving is not specific for certain models of LGM/L4GM. This issue still exists for other models, and even for the most recent 3D methods like Hunyuan3D-2.5. The blueish effect, as shown in Fig. 1, is just an example of how the generated views can look different from the identity of the source view. Solving the dataset bias, as suggested by the reviewer, is actually very challenging to accomplish. We have some arguments in Lines 651-655 about the infeasibility of enumerating every bias issue during dataset filtering.

Take the most recent method Hunyuan3D-2.5 as an example, they already have careful dataset preprocessing steps for optimizing the appearance for back views and side views. However, if playing with the Hunyuan3D-2.5 demo with some in-the-wild images, we can observe the issue of the generated 3D assets having unnaturally smooth and textureless appearances, along with a whitish tone. It may be a result of the data filtering strategy used in Hunyuan3D-2.5, as the data cleaning algorithm filters out "bad" data according to certain rules. At the same time, it inevitably gets biased towards other issues that are the opposites of the data properties being filtered out.

We have prepared intriguing qualitative results for the reviewers. However, due to the policy of the rebuttal not allowing for PDF or other types of media, we choose to describe the qualitative comparisons in detail to the reviewer. The qualitative comparisons requested for the following questions are addressed in the same way.

Description of Visual Comparisons

For the most recent advancement of Hunyuan3D-2.5, we place 5 samples generated by the model side by side, and it becomes obvious to see that for the side views and back views of the object, although the shape and color tones looks generally reasonable, the appearance is over-smoothed and looks unrealistic. This could be the consequence of using certain rules for data filtering and preprocessing, and the rest of the data gets biased towards the opposite direction.

Besides the dataset bias issues which are already very challenging to solve as mentioned above, there are other issues that result in identity mismatch of the generated back views and source views. For example, (1) the prior or bias introduced in the optimization target when training the 3D/4D generation model, and (2) the inaccurate boundaries of the reference view of the object, when segmenting the object out from an in-the-wild image/video. Therefore, resolving the issue of unfaithful generated novel view appearance is very challenging. Our paper provides a "use-as-needed" solution that can be applied whenever the user feels that the back and side views of the generated assets need to be enhanced. Thus, we believe that our paper can be broadly useful for various types of 3D generation methods to enhance their quality regarding identity preservation.

---

Q1-2: Can our model serve as a general solution to fix the problem?

Yes! This is actually a strength of our proposed method, as our solution is representation-agnostic. It is because our progressive texture infilling is completely performing in 2D, leveraging the powerful 2D tools for the 3D task. Therefore, for handling any 3D generation methods with any 3D representations, our pipeline is the same, which demonstrates the general applicability and broad scope of our method. More specifically, we will first render views from the 3D representation, and then use Track to identify the infilling mask, Inpaint to infill the unseen regions, and Resplat to unproject the 2D observations back to 3D. Specifically, we choose Wonder3D which uses 3D mesh as the representation to demonstrate the general capability of our model. The description of the qualitative results is shown below:

Description of Visual Comparisons

We use the "dog eating apple" example shown in Fig. 3 for easy reference. For the results from Wonder3D, the back and side views have weird dark brown patterns mixed with the orange color of the dog. We render from different viewpoints and apply our method to progressively inpaint the scene. Actually, 3D mesh as representation would even make the Track stage easier as 3D mesh generally has less floating artifacts than other representations like NeRFs or 3D Gaussians (more discussions on this point can be referred in Reviewer AB6n's W1 & Q1: Which 3D representations are more suitable?). After our method is applied, the weird dark brown pattern issue gets mitigated, and thus improves the identity preservation of the generated asset.

---

W2: Originality

The originality of our paper comes from our innovation to leverage the powerful 2D models like tracking and inpainting tools for 3D generation tasks. 2D models are trained on massive images and videos, which can come to rescue for the 3D field that has relatively scarce data. Also, as acknowledged by other reviewers, the utilization of strong 2D generative models (Reviewer aG8d) is novel and interesting (Reviewer HPe1). Specially, our creative backward tracking (Reviewer AB6n) and progressive personalized inpainting is a solid and interesting idea that can also be useful in other contexts (Reviewer aG8d).

Also, seamlessly combining these methods is a non-trivial job. For example, the Track stage not only identifies the regions that need to be inpainted, but also propagates the corresponding pixels along the tracking path to novel views to mitigate the difficulty of the following Inpaint stage, and our backward tracking is designed for better inpainting quality in the Inpaint stage. Therefore, our three-stage pipeline is delicately designed and well orchestrated.

---

W3 & Q2-1: Quality comparison with existing methods

Thanks for the suggestions! Also suggested by Reviewer HPe1, we follow DreamBench++ [a] to adopt MLLM-based evaluation on identity preservation and the generation quality. Besides GPT-4o which is adopted in DreamBench++, we also use five other vision-language models (VLMs) with different architectures and sizes to foster the reliability of the evaluation. The results are reported below with 184 view samples:

We can observe that (1) our method consistently outperforms the state-of-the-arts TRELLIS and Hunyuan3D-2.5 in the quantitative evaluation regarding identity preservation and generation quality; (2) different VLMs share consistent scoring rank for different models, demonstrating the robust generalization of the adopted evaluation protocol.

[a] Peng et al. DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation. ICLR 2025.

---

Q2-2: Comparison with SV4D

The reason for us to exclude SV4D in the user study and Tab. 1 in our original manuscript is that the official code of SV4D has only released the multi-view video generation part with 21 pre-defined viewpoints, without the following 4D optimization step. Therefore, only a certain number of selected views can be rendered with its official code, so we choose to pick the closest viewpoints only for qualitative comparison in the appendix. To resolve the concern, we also evaluate the DINO-I scores in the same setting as Tab. 1 based on these closest viewpoints.

We can see that SV4D still has inferior performance than our method regarding identity preservation. For qualitative comparison with SV3D/SV4D, we have also prepared informative visual results for the reviewer, and the description of the visualizations is shown below:

Description of Visual Comparisons

From a group of SV3D/SV4D results on side views and back views placed side by side, we can observe that they often suffer from whitish tone and blurry texture at the generated viewpoints, which may still originate from the imbalance object properties of the dataset used for training the model, or the bias from the network design. It indicates that the problem still exists for SV3D/SV4D.

---

We would like to sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!