Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching

1. The theoretical foundation presented in this paper is not robust and sound. Specifically, the naive distillation objective in Eq. (7) lacks intuitiveness and requires further explanation in the context of this paper. Additionally, the proof of Proposition 1 appears to be incorrect, as it directly compares vectors (instead of their norm lengths). Without a complete proof, it is difficult to accept that TSM can effectively reduce the "errors." Figure 3 does not clarify the differences, and the main benefit of TSM seems to be the reduction of gradient variance, which is not well justified by quantitative measurements.

2. The motivation for reducing prediction error is inadequately explained. The inconsistencies shown in Figure 2 are unconvincing, as both ISM and TSM produce mostly identical results with only minor differences. The advantages of TSM in terms of reduced error are not clearly demonstrated, and when the difference between $\boldsymbol{\epsilon} _\phi(\boldsymbol{x} _t)$ and $\boldsymbol{\epsilon} _\phi(\boldsymbol{x} _\mu)$ is small, the impact of classifier-free guidance may be amplified. Moreover, if the goal is merely to reduce error, setting $t = \mu$ would eliminate the "error" to 0. Reducing the tunable parameter $\delta_T = t - s$ in ISM could also achieve error reduction. An experiment comparing ISM with TSM using the same $t - \mu = \delta_T$ is necessary.

3. There is a lack of fair comparison. The comparisons presented appear unfair, as the authors use two different base diffusion models in Table 1, with TSM using SDv2.1 showing only minimal improvements over ISM. This improvement may also originate from the increase in the impact of CFG as noted in weakness 2. As shown in figure 5, the differences between ISM and TSM is also minor. A fair comparison utilizing the same diffusion model is essential.

1. Although the paper proposes TSM to reduce the accumulated error in ISM, it remains to be further validated whether TSM can fully address the issue of accumulated errors, especially when dealing with more complex 3D models where the impact of accumulated errors might be more significant.
2. This paper use the powerful base model SDXL for comparison with the SD model-based methods is unfair because this paper is employing a stronger prior base model.
3. This paper utilizes the SDXL model, which is significantly slower than SD2-1 in 2D generation, suggesting it would be extremely slow for 3D generation. The paper does not discuss comparative experiments on time consumption.
4. The paper briefly mentions limitations, such as issues with light handling, but does not provide a detailed analysis of when the model fails or its limitations in complex scenarios.

The motivation for using the dual paths appears somewhat ad-hoc.

In Fig.2, notable inconsistencies still exist for the proposed TSM, such as variations in hair, face color, and neck.

Comparisons between TSM and ISM are shown only in Figure 5, which is insufficient to demonstrate the effectiveness of TSM.

There are no video demos showcasing the generated 3D results.

• Unclear motivation. This paper's states that the main problem in previous text-to-3D framework (ISM) is "pseudo ground truth inconsistency" that derives from accumulation of error within the DDIM inversion process. However, the term "pseudo ground truth" is never clearly defined within the paper, failing to convince the reader of the necessity of their task.
• Lack of justification for the method. Furthermore, the paper fails to clearly deliver critical failure modes caused by ISM due to the error accumulation caused by DDIM inversion: in order for its motivation to be meaningful, cases of failure or degradation resulting from ISM-based method, from which TSM improves upon, must be given. Such comparisons are crucial to the credibility of the paper, but comparison of results with those of ISM are not given within the paper, making the reader question the necessity of their contribution.
• Lack of experiment. The paper gives two contributions, TSM and pixel-level gradient clipping, but I believe the ablation studies to be lacking, with the results given in Appendix showing marginal performance improvement at best. In fact, most qualitative results in the paper show very little to marginal improvement from the results given in LucidDreamer (ISM).
• Clarification on performance gain from using SDXL. Table 1 shows quantitative evaluation and comparison to previous methods: however, major improvement in performance in the given table seems to be resulting from changing of diffusion model from SD 2.1 to SDXL, which is given and not related to any form of contribution stated in the paper. I believe all compared works utilize SD 2.1 or earlier version of Stable Diffusion, and in this setting TSM seems to show very marginal quantitative improvement.