Path-Tracing Distillation: Enhancing Stability in Text-to-3D Generation by Mitigating Out-of-Distribution Issues

• Oversaturated results: The results demonstrate significantly less appearance realism than previous works such as ProlificDreamer. The quality looks similarly cartoonish as LucidDreamer and it is not clear where this is coming from. Most results look like paintings, e.g., in Fig. 1 the mouse playing a touba or the duck swimming look very oversaturated.
• Sparse evaluation: There are only some CLIP scores provided which mainly measure text alignment from a single 2D rendered image. There are no 3D metrics used to evaluate the approach. The bare minimum is to conduct a user study as in previous works.
• Two-stage process: Relying on a two-stage process where SF3D is used as an initialization kind of breaks the story of the paper. How does it work without that? Since the model will be mainly determined by feed-forward generation methods and now rather looks like a texture boosting method. Previous SDS methods generate assets from scratch without external models. The geometry of the final scene is basically determined by the feed-forward model SF3D, and these feed-forward models are mainly trained on synthetic data. Hence, they do not have good prompt alignment and are limited in their output which will project into this SDS line of work. The main advantage of SDS works is that they mainly use 2D priors and have much more potential for text alignment, scene complexity, and appearance quality. But this is somehow broken through using SF3D as an initialization.
• Requirement of multiple checkpoints: In the first stage of optimization many checkpoints are saved to fit the rendered images, this provides a large overhead. I do not fully understand why this can not be done on the fly without saving a lot of model checkpoints to disk, as it also breaks continuity and introduces discretization artifacts by setting a hyperparameter of how many checkpoints are feasible to store.
• No video results: For 3D tasks it is very important to include renderings in the form of videos for readers to judge the results. It is difficult to evaluate the method based on the paper alone.

1: The paper's literature review could be more comprehensive, especially regarding recent studies on distribution-related or initialization -related issues with SDS. A more thorough discussion of related work and clearer differentiation from existing approaches would enhance the paper's contribution. Relevant works include:

DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping

Rethinking Score Distillation as a Bridge Between Image Distributions

Connecting Consistency Distillation to Score Distillation for Text-to-3D Generation

2: While the reported CLIP scores show improvements over some baselines, they are only comparable to those achieved by ProlificDreamer. This raises concerns about the practical significance of the proposed method. The paper lacks additional evidence, such as a user study or an anonymous demo project page, to demonstrate a clear advantage in terms of visual quality or perceptual realism.

3: The rationale for using SF3D for 3D Gaussian initialization is not fully convincing. SF3D utilizes DMTet as its 3D representation, and extracting point cloud from it to initialize 3DGS seems indirect and potentially inefficient. Alternative initialization strategies, such as directly using feed-forward Gaussian models, could be explored and compared. The paper lacks a thorough justification for the chosen approach.

1. There is no theoretical support on the OOD issue as found out by the authors, though it is likely that the rendered images from initial 3D models are out of distribution of the high-quality images used to train the score model.
2. It is not clear whether the OOD issue during early stage of the training will affect the final results or not, at the late stage of the training, the 3D model will render better quality images and therefore align with the high-quality images, the OOD is not an issue anymore, the final result should not affect too much by the OOD in the early stage.
3. The experimental results is ok but not convincing, for example, Table.1 Prolific Dreamer outperforms yours though the authors claim the former is 5 times over yours, since the authors claim their approach can be integrated to most of existing methods, the best way to compare is for each methods in Table.1, you may come out two results, one for original method, the other for original method integrated yours. for another example, ablation study does not add any value, to me, it seems the authors just selected the best combination to achieve the best result, the approach may not general to various cases.
4. Minors: Figure 5 appears somewhat blurred; consider using a vector image. There are also several typos, such as "the the" throughout the paper. The terms "CLIP similarity" and "CLIP score" suddenly appear in the abstract, etc.

• The experimental details for understanding the out-of-distribution issue are missing. For example, the number of samples used to calculate $s_{\text{approx}}$ and the specific text prompts are not provided. Additionally, while the authors mention that the difference between $s_{\text{pretrain}}$ and $s_{\text{approx}}$ is more pronounced when $t$ is small, including plots showing this difference would better establish the motivation for the method.

• The authors claim their method outperforms other state-of-the-art methods; however, models trained with PTD are initialized by SF3D [1], while other methods are either randomly initialized or use Point-E [2]. SF3D is a stronger initialization, making it unclear whether the performance gain is due to the initialization or the path-tracing approach itself.

• To understand the specific contribution of path-tracing, the authors should compare PTD against LucidDreamer initialized by SF3D, which would help isolate the impact of PTD from the initialization effects.

• There are very few qualitative comparisons with other state-of-the-art methods. More qualitative examples would provide stronger visual evidence of PTD’s effectiveness.

References

[1] Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. Sf3d: Stable fast 3d mesh reconstruction with uv-unwrapping and illumination disentanglement, 2024. URL https:// arxiv.org/abs/2408.00653.

[2] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.

• Sections 4.1 and 4.2 appear inconsistent. The final method in Section 4.2 seems to only save the last checkpoint and interpolate parameters to generate intermediate score networks. If this is the case, the path checkpointing scheme introduced in Sec. 4.1 and Alg. 1, where checkpoints are saved at regular intervals, may be unnecessary. Additionally, it would strengthen the paper to include an experiment comparing interpolated LoRA with path checkpointing at fixed intervals.

• The comparison with LucidDreamer seems not fair; currently, the proposed method benefits from initializing the 3D representation with a pre-trained generator. To ensure fairness, LucidDreamer should be evaluated using the same initialization.

• The ablation studies are limited. The choice of interval score distillation is not sufficiently justified, particularly given other possible distillation methods such as SDS, VSD, and related approaches [1, 2, 3].

[1] Katzir et al., Noise-free Score Distillation

[2] Wang et al., SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity

[3] Wang et al., Taming Mode Collapse in Score Distillation for Text-to-3D Generation

• Although the paper claims faster convergence in both the abstract and introduction, there is no empirical evidence provided to support this claim.