Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillation

Thank you for your valuable and constructive review! We try to address your concern below.

W1. Problematic figure placement.

Thank you for your suggestion! But now it's hard for us to adjust this, and we would really like to reorganize the placement before the next version.

W2. Inaccuracy on related work. And should demonstrate on other representations like 3DGS.

Thank you for your reminder! We have included these papers and modified our description in the introduction. Also, we provide more results generated in the second and third stages in the appendix. The comparison of the geometry and texture refinement can prove the effectiveness of our method. Due to the time limit, we are very willing to demonstrate on 3DGS-based method like GaussianDreamer before the next version!

Q1: Lookahead will bring over-saturated then why lookahead is important.

Thanks for your question!

Firstly, NeRF converges faster indeed in L-VSD than VSD. Secondly, we want to argue that in Figure 2b, it's obvious that the results have a much clearer shape with edges though still over-saturated. From the last figure of Figure 2b, we find that we can get better results by lowering the learning rate, then we come to the inference that "Lookahead" is the effective direction to investigate, while "lookahead" only is not effective enough. We also claim that only by lowering the learning rate is very unstable. We provide massive failed results in Figures 16 and 17 in the appendix. So lookahead is important and effective but not sufficient.

Q2: About the necessity of LoRA steps.

Thank you for pointing out that! Sorry for making Section 3.1 so confusing. We include Section 3.1 of investigating the impact of more LoRA steps in order to show a more complete map of possible factors. Also, as many statements in L-VSD are related to the convergence of LoRA, we believe including this part can help people have a better overview of the process. Now we have modified the section title to express more clearly.

Q3: Contradictive number with CLIP similarity.

Thank you for your careful review! We have a typo in the original paper. Actually, we calculate the arccos angle of the CLIP similarity. The smaller the angle is, the higher the original CLIP similarity is. We have modified this sentence about the evaluation metrics in the revised PDF.

Thank you for your active response! But we still want to make some discussions here. We hope it can address your concern to some extent.

1. Actually the collasion of L-VSD doesn't lead to the conclusion of repetitive LoRA finetuning. L-VSD and our method takes the same LoRA finetuning times as VSD, once only. And we also design experiments in section3.3. to demonstrate that repetitive LoRA finetuning is unnecessary.

2. We don't focus on the repetitive LoRA finetuning, but we focus on the approximation distribution $q^{\mu}$ proposed by VSD, where the root of mismatching problem in VSD lies. Maybe you can refer to our answer to @LcCM W1.

3. We strongly refer you to additional results in Figure.8 with resolution of 256. In Figure.19, VSD generates destroyed car with random red color, connecting destroyed car with a fire but our method generates more purely. And the texture of hand and the bowl in the bottom is also more realistic. In Figure.22, for the limits of computation and high memory demand of high-order correction generation, we can only compare with high-order ablation with resolution of 64. As we can observe, the results of our method generates more complete scene, and we strongly suspect that the geometry quality of this more complex scene is limited by the resolution.

Thank you for your valuable and constructive review! We try to address your concern below.

W1: Section 3.1 seems to be confusing, and irrelevant about the method.

Thanks for your valuable suggestion! Actually, we are trying to show the complete investigation maps. And in terms of discussing the impact of the convergence of "approximated distribution" in the original paper, we thought this section is necessary and helpful for a complete comprehensive study. We have modified the section title to express more clearly. Also, we changed the vibe of the conclusion of Section 3.1, to avoid confusing.

W2: Ablation experiments on the method with high-order terms are missing.

Thank you for your suggestion! We have done this experiment on another two samples and provided the results in Figure 22 in the appendix. We name the method HL-VSD (high-order lookahead VSD) because we use the high-order term instead of the linear term to correct the score. As shown in the figure, the results all collapse and become irrecognizable, which proves the effectiveness and necessity of linearized lookahead.

W3: Over-saturated problems with the generated results.

Yes, we don't claim that we have solved this problem completely, but we believe the results outperform VSD's results. Also, we provide more high-resolution results in the appendix like Figures 18, 19, and 22, and update the images in Figure 8. We believe these will strongly support our proposed method.

W4: The method consumes more time and other methods like CSD and LODS improves the efficiency.

Thank you for your reference!

Firstly, CSD is not a VSD-based method. LODS changes the learning objective of VSD and deviates from the VSD-based method. But as the pipeline is similar, we believe our method can be incorporated into LODS too. We are willing to implement this before the next version.

Secondly, we believe our method is orthogonal with other techniques that accelerate the whole process. As mentioned by YuaJ, we believe our method $L^2$-VSD, which is now implemented on VSD, can be transferred to these accelerated modern pipelines like GaussianDreamer as VSD and SDS, because SDS, VSD, and our $L^2$-VSD are all 3D distillation algorithms, orthogonal to the pipeline design.

W5: A typo in Fig.7

Thank you for your detailed review! We have corrected it.

Q1: Proposed method can't improve the LoRA convergence.

Firstly, through experiments in Section 3.1, we are trying to show that improving the convergence of LoRA is not the key factor. We include this part's discussion to provide a global view.

Secondly, as written in Line 228, the loss of L-VSD with $\gamma=1$ floats in a similar level to VSD with $\gamma=10$, which suggests that lookahead in the LoRA model makes it faster to adapt to the current distribution, thus improving the convergence compared to VSD.

Q2: VSD reduced to standard SDS in an extreme case, which contradicts the intuition behind the proposed methods.

Thank you for your consideration of this subject!

But firstly SDS is a special case of VSD when the particle number equals one. Even if we have a perfect LoRA model, as the timestep in LoRA training is different from the timestep in object training, it will only predict the mean of the added noise.

Secondly, we don't emphasize that improving the LoRA convergence is the only key factor for the performance. Our experiments in Section 3.3 prove that overfitting in the LoRA model is harmful to the performance.

Thank you for the response!

But we think our contribution is (1) finding out an important flaw with a representative method in score distillation 3D generation; (2) design extensive experiments to demonstrate the key factor; (3) propose the linearized-lookahead method to tackle this problem. We don't directly propose a whole new concurrent method compared with others.

In terms of efficiency, we have responsed that by taking last-layer approximation, we can save more time. And our method is orthogonal to other acceleration methods. So we are very optimistic about the future of improment.

More importantly, about the visual results. As we have provided more visual results in the revised PDF like figure.8,18,19 and 22, could you please tell us where or which samples do you have concern with? Also, could you please tell us which reviewer also raise concerns about the visual results in the revised PDF? Reviewer mA7z has concern with figure.22, which is mainly designed for comparison with high-order correction as you demand. We set the resolution as 64 because the huge memory it takes. And reviewer mA7z hasn't respond to us. Could you please share more opinions on it?

As we have spared no effort trying to addressing your concern, we are eager to know what the problem still lies, rather than accepting "oh, someone else also says that", which is not supported with a conclusion of a discussion. Thank you again.

Thank you for your valuable and careful review! We try to address your concern below.

W1. The visual results presented for L2-VSD do not clearly demonstrate an improvement over those generated by VSD. And the CLIP similarity number is misleading.

Firstly, we have provided more high-resolution results in Figures 8, 18, 19, and 22 in the appendix, especially by Figure.8 which is generated with high-resolution 256. The blurry degration of performance in Figure.22 is caused due to low-resolution 64. Figure.22 is provided mainly for comparison with high-order generation, and this process demands high memory. We believe these results can help us demonstrate the effectiveness of our method.

Secondly, sorry for the incomplete metrics description! Actually, we calculate the arccos angle of the usual defined CLIP similarity, so the lower the angle is, the higher the CLIP similarity. We have corrected our description of evaluation metrics in the revised PDF. Besides, we calculate the FID averaged across different prompts while the FID of each prompt is measured across 120 views spanning 360°.

W2. Drawing meaningful conclusions from loss reduction in Fig. 3 and improved quality in Fig. 2a is not advisable.

Thank you for your question! But we never correlate the reduction in loss with the quality improvement. In the original paper, we want to ensure the convergence of LoRA by the loss curve of Figure 3. Besides, as written in Section 3.1 in the original paper, the overall quality of Figure 2a "does not" witness a continual improvement. So we come to the conclusion that better convergence is not sufficient and not the key factor. And that's also why we claim that investigating in lookahead is more important.

W3. About the clearness of clarity.

Thank you for your summarization! Our main points are 1) analyzing two possible factors that cause the gaps of VSD and justify them separately, 2) ensuring the convergence of LoRA is not that helpful and increases the computation budget hugely, 3) correcting the optimization order of VSD has the potential to bring better results, but naively using it doesn't help, and 4) combining the stable but less satisfying VSD and unstable but possibly better L-VSD together and propose the $L^2$-VSD. We recognize the convergence part of Section 3.1 is kind of misleading and have modified the section title! We will try our best to make a better statement to catch the logic.

Q1: Justify the quantitative results in Table.1

The answer is included in the answer to W1.

Q2: Can the method be applied to mesh fine-tuning?

Yes! Our method can generalize to other representations. We conduct experiments in stages 2 and 3 in VSD. And we have provided the results in Figures 18 and 19 in the appendix. This comparison reflects the generalization of our method on other representations. We are willing to demonstrate on 3DGS before the next version.

Thank you for the valuable and careful review! We do appreciate your careful reading. Below, we try to address your concern point by point.

Q1&Q10: How well can the method generalize to other methods like GaussianDreamer, different 3D representation, or subsequent training phase? How to mitigate the long training time problem?

1. In terms of 3D representations and method, from the aspect of algorithm, our method $L^2$-VSD can generalize as SDS and VSD as long as the rendering process of the chosen representation is differentiable. But we do acknowledge that there might exist biases between different representations. So we provide more results in meshes from the second and the third stages in Fig.18 and Fig.19. The results indicate that our method can still generate more realistic results in mesh. Due to time limitations, we promise to experiment on 3DGS before the next version.

2. We agree with Reviewer YuaJ that the computation time of VSD is huge, and now many concurrent methods like GaussianDreamer exist to accelerate the whole process. We believe our method $L^2$-VSD, which is now implemented on VSD, can be transferred to these accelerated modern pipelines as VSD and SDS, because SDS, VSD, and our $L^2$-VSD are all 3D distillation algorithms, orthogonal to the pipeline design.

Q2: Could the mismatching problem be more precise? Are there any metrics?

Thank you for the valuable question. Mathematically, according to the equation.7 in the original paper of ProlificDreamer, the update rule of VSD follows the following ODE:

where $q_t^{\mu_\tau}$ is the corresponding noisy object renderings distribution at diffusion time t w.r.t $\mu_\tau$ at ODE time $\tau$.

In this sense, the real update rule of VSD actually uses $q_t^{\mu^{\tau-1}}$, which causes the mismatching problem. So maybe $D_{KL}(q_t^{\mu_\tau}(x_t|y,c)|q_t^{\mu_{\tau-1}}(x_t|y,c))$ can be used to precisely describe this problem. What is interesting is that, from this perspective, one "possible" solution is to penalize the change of $\mu_\tau$, but definitely resulting in much longer time.

If you think this is helpful, we would be very glad to add this analysis into the final version.

Thank you for the valuable and constructive review! We try to address your concern and argue for the meaning of our work below. Hope to discuss more if you have contrasting opinions!

W1: This work seems to only explore the LoRA training of VSD, which limits the scope of this work. The introduction of LoRA in VSD is already a compromise in implementation, and it doesn't make sense to further analyze its theory-implementation gaps. Also, the improvements seem to be marginal with respect to the additional time.

Thank you for your consideration on this meaningful subject! But we might have different thoughts on it.

1. Firstly, as Reviewer LcCM has mentioned in "Strengths", VSD is a representative score distillation method for 3D generation and has shown a powerful potential to generate more vivid assets. We can basically divide modern 3D score distillation methods into VSD-based and SDS-based ones, and SDS is actually a subclass of VSD with a single particle. So we believe efforts in analyzing these basic frameworks are meaningful for proposing a new framework.

2. Secondly, our motivation is to investigate the gaps in VSD though it already establishes a mature and complete theoretical framework. One of the biggest differences between VSD and SDS is introducing the approximated score of 3D views. So we don't mean to "only explore" the LoRA training for VSD, but have to explore this "approximated score", which is expressed within LoRA.

3. Lastly, we want to correct seriously that "the implementation of VSD is never a compromise but a default recommendation". We strongly refer you to Section 3.2 of the original paper of VSD. Using LoRA rather than a small U-Net can greatly enhance performance. Also, the introduction of LoRA is not the root cause of the gaps we analyze.

W2: Figures 2,3, and 4 use the same prompt case, which weakens the generalizability of the results. For qualitative analysis, experiments should be performed on multiple samples.

Thank you for pointing out that! Actually, we intended to offer a more comprehensive comparison from the global view, so we use the hamburger case as the global illustrative example. To address your concern and provide more convincing signals, we included the average loss curves with multiple samples in Figure 20b of the revised PDF. The conclusion holds the same as in Section 3.1. Also, we provide one sample "crown" other than "hamburger" to augment the proof.

W3: Why LoRA overfitting is harmful, which is obviously consistent with the theory of VSD.

Thank you for your interest. As written in Line 268 in Section 3.3, from the theory of VSD, LoRA is used to estimate the current distribution of 3D views. But current training of LoRA is to overfit on a specific view of a specific possible object. So the distribution is biased. Moreover, overfitting the LoRA model will destroy the prior information with respect to text from Stable Diffusion, which is harmful to the performance evidenced by the original paper of VSD.

W4: Why is it necessary for the LoRA model to look ahead from the results in figure2, that L-VSD does not seem to have any advantage over VSD?

From our observation, we find the geometry edges of these results in Figure 2b are more clear and have a more complete shape. Also as mentioned in Line 228 in Section 3.2, other than visual judgment, we find that the loss level of L-VSD with $\gamma=1$ floats in a similar level to VSD with $\gamma=10$, which suggests that L-VSD is better for fit on the current distribution.

W5: What is the meaning of $r(x)$ in Figure 1?

We are sorry for the confusion. But actually, we want to show the corrective guidance from $r(x)$. It's kind of hard to draw an arrow between distributions of different steps.

Q1: For L-VSD, in a certain iteration step $i$, are the input $x_{t'}$ used for LoRA training (Eq.(6)) and the input $x_t$ used for NeRF training (Eq.(5)) sampled from the same camera view? Do they use different timesteps?

Yes! They are sampled from the same camera view, which is held the same as in VSD. But they use different timesteps, which are also held as the same as in VSD.

Q2: Lowering the learning rate of LoRA seems to be quite effective, so why do we need $L^2$-VSD?

Thank you for your question! As shown in Figure 16 and Figure 17, the strategy of lowering the learning rate is very unstable and uncontrollable. We provide many failure cases of L-VSD in the appendix. Also, we are motivated by this positive sample, believing that a lookahead for the LoRA model is necessary. And then investigate more on the difference between VSD and L-VSD, thus having more stable and controllable $L^2$-VSD.