Q1: I still wonder about the specific meaning of "4D texturing" for the object and how this differs from a 3D model that is first textured and then animated using skinning techniques. Even for dynamic textures, one could also generate the dynamic texture for a 3D model and then animate the character through skinning. This approach seems useful if the mesh is also significantly dynamic, such as with topology changes.

A1: We thank the reviewer for these insightful comments. "4D texturing" in our work refers to the process of generating temporally and spatially consistent textures for dynamic mesh sequences over time. This differs from traditional pipelines that generate static 3D textures first and subsequently animate the mesh through skinning techniques. While traditional methods can handle static textures effectively, they fall short in scenarios requiring dynamic, time-varying textures that reflect changes such as lighting variations, motion-specific deformations (e.g., wrinkles), or stylized animations. In our additional results, the snowman case exhibits significant dynamic changes. We suggest the reviewer kindly refer to our updated paper (Fig. 13) and supplementary video.

---

Q2: Could one first perform 3D texturing and then render it with a generative background video? Clarify how the proposed method handles dynamic meshes with topology changes, if applicable. Compare the method with a pipeline of 3D texturing followed by rendering with a generative background video, highlighting any benefits of the presented "integrated" approach. Could you provide examples where dynamic 4D texturing is essential and cannot be achieved through traditional methods?

A2: We thank the reviewer for providing this insightful suggestion. Please kindly refer to General Response (A) for the discussion with traditional textured mesh animations. We suggest the reviewer go through our updated supplementary video for the dynamic results and comparison with textured mesh animations.

---

Q3: I also have concerns about the novelty of the paper. The entire pipeline can be seen as a depth-conditioned CTRL-Adapter for mesh sequence texturing with UV space aggregation, which feels like a straightforward composition of existing models. Provide a clearer definition and motivation for "4D texturing" to help readers understand its significance in the context of their work.

A3: We thank the reviewer for the suggestion to provide a clearer definition and motivation for "4D texturing" and we have added these in our paper (updated in Abstract (L034-045) and Introduction (L057-060, 085-087)). For the consideration of novelty, our method is the first to handle the task of 4D scene texturing and demonstrate the capability of video diffusion model in offering temporal variations with mesh guidance. In addition, we discuss our motivation in General Response (B), and you can kindly refer to it.

---

Q4: I understand the difficulty in evaluating the results, but it would be helpful and necessary to conduct an evaluation of Appearance Quality, Spatio-temporal Consistency, and Consistency with Prompt via a user study - The quantitative evaluation is insufficient.

The following action can be taken: Conduct a user study evaluating the specific aspects, e.g., Appearance Quality, Spatio-temporal Consistency, and Consistency with Prompt, and compare the proposed method with previous models.

A4: We have already conducted the user study with these metrics in the initial edition (Appearance Quality, Spatio-temporal Consistency, and Consistency with Prompt). Please kindly refer to Table 1 and 5.2 Quantitative Evaluation (L503-509) for details.

---

Q5: Limitations and Future Works should be included.

A5: Thanks for the suggestion. We provide a "Limitation and Discussion" section in the appendix. We outline the potential limitation in scene-level texturing due to the limited dataset, and leave this for our future work. In addition, we discuss the computation time compared with the 3D texturing method and we believe the computation time may be shorten with the advancement of video diffusion models.

Dear Reviewer CcR5

We sincerely appreciate your reviews and comments on our paper. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all your concerns. Please let us know if you have further questions after reading our rebuttal.

We hope to address all the potential issues during the discussion period. Thank you once again for your time and effort in reviewing our submission.

Submission#2974 Authors

Q1: The importance of this pipeline should be further clarified, such as by comparing it with pipelines based on 2D poses or textured meshes. The paper should include more comprehensive comparisons to highlight the contribution of the pipeline. For example, is it a reasonable pipeline to first generate textured meshes and then use animated meshes for video generation? Would using textured meshes yield better outcomes?

A1: We appreciate the reviewer’s insightful comment. To address the concern, we have added visual comparisons of traditional textured mesh animations in Fig. 10, accompanied by a detailed video comparison in the supplementary materials. Additionally, our pipeline is designed to handle general characters, not limited to human figures, where 2D poses may not always be available. This flexibility broadens the scope of our approach. To further support this, we have included more visual results in Fig. 13 and the supplementary videos, showcasing the versatility and contributions of our method in comparison to existing pipelines. We also compared our method with the traditional textured mesh animations (based on Text2Tex), please kindly refer to General Response (A).

---

Q2: Animation can drive the mesh. Are the position and rotation of the mesh manually controlled, such as in the second example on the first page? How is the animated mesh obtained?

A2: We note that our work assumes animated mesh sequences as inputs and focus on generating plausible dynamic textures, instead of mesh animation. Specifically, we obtain the animations from Mixamo and Sketchfab websites. The Mixamo assets include skeleton rotations, rigging, and skinning weights. We animate the mesh by linear blend skinning based on the pre-defined skeleton hierarchy. The Sketchfab provides complete animations and we extract the vertices and faces using blender software.

---

Q3: We observe some temporal changes in Figure 5. Is this one of the contributions of your paper? What are the advantages of generating videos with untextured meshes compared to textured meshes?

A3: Yes, capturing temporal changes is one of the contributions of our paper. Our method is to capture the temporal changes (e.g., lighting controls, appearance transformations) using video diffusion model, which is hard to obtain with textured mesh animations, please kindly refer to General Response (B) for our discussion.

---

Q4: How do you distinguish between temporal changes and temporal inconsistencies, as there are some temporal inconsistencies in your results?

A4: Temporal changes and temporal inconsistencies are not necessarily in conflict but represent different aspects of temporal dynamics. Temporal changes refer to deliberate and meaningful variations over time, such as lighting shifts, surface deformations (e.g., wrinkles), or stylistic transformations, which are essential for capturing realism and dynamism in animations. Temporal incosistency usually refers to sudden unrealistic changes caused by limitations of the texture generation methods. We suggest the reviewer kindly refer to Fig. 13 and our supplementary video for these results.

Dear Reviewer jsAN

We sincerely appreciate your reviews and comments on our paper. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all your concerns. Please let us know if you have further questions after reading our rebuttal.

We hope to address all the potential issues during the discussion period. Thank you once again for your time and effort in reviewing our submission.

Submission#2974 Authors

Q1: In the paper, the authors mentioned that the mesh texture could be significantly influenced by the background, providing an example with a white background. I’m curious how the generated texture might look if a non-white background was used, especially one that contrasts strongly with the foreground object. How would such a background affect the consistency and quality of the generated texture?

A1: We appreciate the insightful question. In Fig. 8, we have included additional experiments to demonstrate the effect of varying background conditions. Specifically, we tested an alternative background noise shuffle used in SyncMVD (a) and a highly contrasting background noise initialization (b). The results indicate that the texture tends to be influenced by the high-contrast background latent initialization, leading to deviations from the intended textual prompt. This suggests that while our method can generate textures under various backgrounds, strong contrast can negatively impact both the consistency and alignment of the texture with the desired description.

---

Q2: The generated textures do not blend seamlessly with the background, creating a disjointed appearance that resembles separate foreground and background elements stitched together.

A2: We agree with the observation regarding the disjointed appearance between the foreground and background elements. This issue primarily stems from the lack of a comprehensive scene-level dataset for 4D texturing, which constrains our approach. Currently, we use a shared background mesh across different views, which can disrupt overall visual consistency. Addressing the challenge of seamless integration in scene-level 4D texturing is an open problem, and we intend to explore potential solutions in future work to improve coherence between foreground and background elements. We have updated the Limitation and Discussion section in our appendix.

---

Q3: Some of the compared methods, such as TokenFlow and Text2Video-Zero, do not utilize mesh or depth inputs, making direct comparisons less equitable.

A3: We acknowledge the concern regarding the fairness of comparisons with methods like TokenFlow and Text2Video-Zero, which do not utilize mesh or depth inputs. Since our work is the first to address 4D scene texturing, there are no existing methods that directly align with our setup. We have endeavored to provide a comprehensive comparison by evaluating current text-to-image (T2I) and text-to-video (T2V) methods under various conditions, including depth, mesh, DDIM features, and DensePose features. While depth and mesh conditions are closest to our scenario, other conditions are also relevant and widely discussed in the context of controllable video generation.

---

In addition, we kindly encourage the reviewer to view our updated supplementary video, which illustrates the advantages of our method in achieving consistent character generation compared to traditional textured mesh animation methods, as discussed in General Response (B).

Dear Reviewer MURz

We sincerely appreciate your reviews and comments on our paper. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all your concerns. Please let us know if you have further questions after reading our rebuttal.

We hope to address all the potential issues during the discussion period. Thank you once again for your time and effort in reviewing our submission.

Submission#2974 Authors

Q5: While using UV textures for each mesh can enhance multi-view consistency, this approach seems more like an averaging of multiple viewpoints to produce a smoother result. Can the authors elaborate on how this averaging mechanism ensures true multi-view consistency?

A5: The UV space serves as a global reference and we hope to merge the information from different views to the UV space, which ensures the global consistency. Inspired by some mesh texturing methods (e.g., SyncMVD, Meta TexureGen), we simply uses the weighted aggregation to merge latents observed from different views.

---

Q6: Given that the current method requires rendering V views for each mesh in the sequence, which may be computationally intensive, could the authors discuss the efficiency of the method? Details on the time required to process a sample would help assess the method's practicality.

A6: We have included average computation times in the appendix for clarity. Our method requires approximately 30 minutes per sequence, which is comparable to static texture generation methods like Text2Tex (22 minutes for a static texture). The computation time primarily depends on the foundation model (CTRL-Adapter), which takes approximately 5 minutes to generate a video with 24 frames. We anticipate significant efficiency improvements with advancements in conditioned video diffusion models, further enhancing the practicality of our approach.

---

Q7: It would be beneficial to include video visualizations or comparative examples to further illustrate the method's performance and effectiveness.

A7: We have provided additional experiments highlighting temporal changes using textual prompts such as flashed a magical light, dramatic shifts in lighting, cyberpunk style in Fig. 13 and included these results along with video visualizations in the supplementary materials. Also, we have updated the main paper with Fig. 7~Fig. 13 for comprehensive comparisons.

Q1: Additional qualitative and quantitative comparisons in the ablation study. With only one demonstration, it is difficult to convincingly assess the effectiveness of the proposed method.

A1: We follow Reviewer coJ5 and MURz's suggestion and include one more example for UV reference module ablation in Fig. 9 and more ablation experiments on backgrounds in Fig. 8 (alternative noise shuffle strategy used in SyncMVD (a), highly contrast with foreground (b), another case with Ironman $c$). Our findings suggest that the background initialization indeed affects the appearance of the texture and the background shuffling strategy used in static texture generation may not be applicable in our pipeline.

---

Q2: The authors suggest that video diffusion models struggle with multi-view consistent texturing for 3D mesh sequences due to a lack of 3D geometry awareness. However, the approach already uses a depth-aware video diffusion model, which inherently includes some geometric awareness. Why does this straightforward combination not achieve the desired consistency? Does this imply that depth-aware video diffusion models alone cannot guarantee multi-view consistency even with depth information? If so, could the authors provide performance metrics or visual comparisons showing results when using only the depth-conditioned video diffusion model as a prior? Additionally, for a single viewpoint, does the video diffusion model produce temporally consistent results? If not, visual examples would help clarify.

A2: We appreciate the reviewer’s observation regarding the role of depth-aware video diffusion models in achieving multi-view consistency. While depth maps indeed provide some geometric information and allow consistent results within a single viewpoint (as demonstrated by the CTRL-Adapter), they are inherently view-dependent and lack global spatial information encoded by the 3D mesh, such as the absolute position of points in a world coordinate system. This limitation prevents consistent character generation and texture generation across different views. In Fig. 11, we have added three experiments to demonstrate that depth-conditioned diffusion models alone fail to ensure a global consistent texture. The results highlight how depth maps, despite their localized geometric guidance, cannot resolve inconsistencies across different viewpoints. In contrast, UV maps act as a global constraint by mapping 3D spatial positions to consistent 2D UV coordinates $(x, y, z)\_{xyz} \rightarrow (x’, y’)\_{\text{UV}}$, leveraging the point correspondence of the same points on the object in different views, thus encouraging 3D consistency.

---

Q3: Since a mesh input is available, a straightforward approach could be to texture the mesh on the first frame using methods like Text2Tex or SceneTex, then animate the textured mesh. This method might improve efficiency and naturally maintain multi-view consistency across frames. How does this alternative approach compare in terms of both methodology and performance? An in-depth discussion of these differences would be beneficial.

A3: We appreciate the reviewer's suggestion for in-depth discussion of textured mesh animation, using methods such as Text2Tex for comparison. Please kindly refer to General Response (A) for the discussion with traditional textured mesh animations (Text2Tex) and General Response (B) for the motivation of 4D scene texturing.

---

Q4: The authors mention that for each predefined viewpoint, a sequence of K rendered meshes is used as input and individually textured by the depth-guided diffusion model. Could the authors clarify the motivation behind this setup? Since the videos are generated separately for each view, multi-view inconsistencies are expected. Why introduce this setup if it inherently leads to consistency issues at the start?

A4: We aim to texuring 4D scenes and capturing temporal variations (e.g., lighting and wrinkles) within mesh sequences to produce vivid visual result, which is widely expected in downstream tasks (e.g., generating a consistent character). Depth maps alone, even when used with video diffusion models, fail to achieve this goal due to their lack of global consistency as shown in Fig. 13. Instead, we use UV space as a global reference which is an off-the-shelf attribute of mesh and ensures consistent texturing across all views. Our method merges the appearance from different views and aggregate the information in UV space, and render the latents from the latent UV for each view to achieve the consistency. This strategy provides a robust solution for multi-view consistency while capturing dynamic temporal details, although the initial views may not be well-aligned across different views.

Dear Reviewer coJ5

We sincerely appreciate your reviews and comments on our paper. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all your concerns. Please let us know if you have further questions after reading our rebuttal.

We hope to address all the potential issues during the discussion period. Thank you once again for your time and effort in reviewing our submission.

Submission#2974 Authors

Thank you for your suggestions and detailed feedback. We would like to further clarify the points you raised.

1. We appreciate your suggestion regarding additional examples. We will provide more visual examples and include quantitative results from our user study in the appendix in our revision.

2. (and 4) We think the main concern from the reviewer's side is about the plausibility of the latent aggregation strategy. There are some prior works have demonstrated the effectiveness of generating high-quality textures for static 3D meshes in the T-pose using the texture aggregation in the latent space, such as SyncMVD [1] and Meta 3D TextureGen [2]. Our method denoises the latents in temporal batches and integrates with the multi-view aggregation strategy to maintain the consitency. In addition, we have experimented the direct latents blending strategy in Fig. 3, which causes the oversmooth problem because of the "variation shifts problem" discussed in L305-309. We overcome this issue by rewriting the denoising formula. As shown in the qualitative results, our method produces detailed texture sequences (Fig. 7) and consistent visual appearances (Fig. 12 and Fig. 13).

3. Thanks for the suggestion. We have included a video comparison in our supplementary material and since these examples are generated using the same assets, they already appear in Fig. 1 (1st case) and Fig. 13 (2nd and 3rd cases), respectively. As such, we refrained from duplicating these comparisons in the main text.

Hope these clarifications address your concerns. Thank you again for your detailed review and for engaging with our work.

---

References

[1] Yuxin Liu, Minshan Xie, Hanyuan Liu, Tien-Tsin Wong. Text-Guided Texturing by Synchronized Multi-View Diffusion. SIGGRAPH Asia. 2024.

[2] Raphael Bensadoun and Yanir Kleiman and Idan Azuri and Omri Harosh and Andrea Vedaldi and Natalia Neverova and Oran Gafni. Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects. arXiv preprint arXiv:2407.02430

Dear AC and SAC,

We have provided answers and explanations to reviewers' questions. As some reviewers have not engaged in discussions despite several follow-up emails, can you send an email to urge them to participate before the deadline on Dec 2?

Thank you,