Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping

Q1: Description for Figure 2 is unclear.

Thanks for pointing it out. For both Stable Diffusion and SDXL, we present images organized in 3 rows and 6 columns. Let’s explain them in detail one by one.

Row1: The setting is training a 2D image using SDS loss with different gradients.

Row2: The visualized gradients of the optimized image as shown in Row1.

Row3: The setting is training a texture field for a fixed mesh using SDS loss with different gradients.

Column1: The setting is training 2D image/3D texture field in latent space as Latent-NeRF.

Column2-6: training them in image space where the VAE encoder will be incorporated in gradient backward.

Column2: Using original gradient backward through VAE encoder. (VAE gradient)

Column3: Using a linear approximation for VAE encoder.

Column4: Using the pixel-wise normalized VAE gradient.

Column5: Using the pixel-wise clipped VAE gradient based on value.

Column6: Using the pixel-wise clipped VAE gradient based on norm.

We have made the caption of Figure 2 more clear in the revision.

Q2: Figure 6 is unclear.

Thanks for pointing it out. Since VAE has two parts: encoder and decoder, we fit linear approximation for both parts. The left three columns are the results of decoder approximation where fitted and GT RGB images are presented. The right three columns are the results of encoder approximation where fitted and GT latent images are presented. We have made a clear description for them in the revision.

Q3: More illustrations to support Section 4.5 (depth controlnet).

We have provided two cases to illustrate the problem in Section 1.2 of the separate supplementary file. Depth controlnet improves the success rate of generating high quality meshes like the bottom of Figure 1. In our observation, only about 40% cases succeed without depth controlnet and the rate increases to 60%-70% by incorporating the depth controlnet. In the rest cases, while the quality is not as high as in Figure 1, our PGC could still enhance over the baseline.

Q4: Typo of the range (0,1).

Yes, it is a typo and should be [-1, 1]. We appreciate your careful reading and have fixed the typo in the revision.

Q5: More insights of the constraint.

Great comment. More specifically, we note that RGB values are normalized to the range of [0, 1] while the absolute value of the image gradients produced by SDS loss can vary from $0$ to $10^{5}$. If we optimize a 2D image directly using these gradients with SGD, the optimized image will quickly explode thereby leading to failure. Although sigmoid function and Adam optimizer are often used to guarantee the RGB constraints and smoother gradients (as all the experiments in the main paper), we think the extremely large and unbalanced gradients still do not make sense.

Q6: Alternative method by clipping $(\epsilon_{\phi} -\epsilon) \frac{dz}{dx}$ rather than the $\frac{dz}{dx}$.

Apologies for this misunderstanding. In our design, we take operations on $w(t)(\epsilon_{\phi} -\epsilon) \frac{\partial z}{\partial x}$ which are the gradients of an image. The term $\frac{\partial z}{\partial x}$ is the Jocobian matrix of size $(128\times 128 \times 4) \times (1024\times 1024 \times 3)$ under SDXL setting. We have revised the relevant expressions in the paper.

Q1: More ablations on parameter-wise NGD and GC.

First we note that when we conduct 2D experiments (i.e. just optimizing a 2D image as the first row of Figure 2), “parameter-wise” design is equal to the “pixel-wise” counterpart because a pixel itself is a parameter to be optimized. For 3D experiments, parameter-wise NGD and GC will allow the extremely large and unbalanced gradients of the rendered image to be back-propagated until the gradients of the earliest parameters of NeRF/DMTet. This causes suboptimal results.

Importantly, we provide the evidence in the Section 1.2 of the separate supplementary file (Figures 4) that parameter-wise NGD and GC cannot produce as detailed results as our PGC under the text-to-texture setting using SDXL.

Q2: More comparisons.

Thanks. We have now provided the results of ProlificDreamer and improved Fantasia3D in the Section 1.3 of the separate supplementary file (Figures 5 and 6).

For ProlificDreamer, we use Stable Diffusion as guidance. We observe that our PGC also stabilizes the training. We cannot test the combination of SDXL and VSD due to the restriction of GPU memory. Overall, our method enables the unleashing of the potentials of the advanced diffusion model successfully.

For improved Fantasia3D, we have already tried the alternative weighting strategy: $w(t)=\frac{1}{\sigma_t^2}$ mentioned in their readme Q7. Since threestudio and stable-dreamfusion have not implemented this strategy, we did not include this for fair comparisons in our submission. In our experiments, we find that our proposed PGC consistently enhances texture quality.

Q3: Related work for gradient-related techniques.

Thanks. We have added extra discussion in the related work following this suggestion. We note an important difference that previous works focus on model parameters while we manipulate the images – the intermediate variables in the forward/backward process.

Q4: Description for Figure 2 is unclear.

Thanks for pointing it out. For both Stable Diffusion and SDXL, we present images organized in 3 rows and 6 columns. We explain them in detail one by one.

Row1: The setting is training a 2D image using SDS loss with different gradients.

Row2: The visualized gradients of the optimized image as shown in Row1.

Row3: The setting is training a texture field for a fixed mesh using SDS loss with different gradients.

Column1: The setting is training 2D image/3D texture field in latent space as Latent-NeRF.

Column2-6: training them in image space where the VAE encoder will be incorporated in gradient backward.

Column2: Using original gradient backward through VAE encoder. (VAE gradient)

Column3: Using a linear approximation for VAE encoder.

Column4: Using the pixel-wise normalized VAE gradient.

Column5: Using the pixel-wise clipped VAE gradient based on value.

Column6: Using the pixel-wise clipped VAE gradient based on norm.

We have made the caption of Figure 2 more clear in the revision.

Q5: Typo.

We appreciate your careful reading. We have fixed the typo in the revision.

Thanks. We set the threshold as 0.1 for all parameters. We have updated Figure 4 in the supplementary file by adding the ablation on clipping value used in parameter-wise GC. The clipping value ranges from $10^{-3}$ to $10^{-1}$. Parameter-wise clipping led to the generation of blurry textures in all settings. This approach appears inadequate in addressing unbalanced gradients as the earliest gradients of model parameters remain correlated with high-magnitude noise in image gradients, potentially overwhelming the lower-magnitude gradients during the backward process. Such occurrences are prevalent in the fp16 mode, where an adaptive scaler is employed to ensure the preservation of small gradients. Previous experiments demonstrated a decrease in the frequency of skip events (wherein the scaler skips an update step and reduces the scale if Inf or NaN exists in gradients) upon the application of PGC, providing evidence that the pixel-wise design is more effective in stabilizing the training process.

Q1: In-depth analysis.

We acknowledge that we find enhancement in the experiments. However, we have tried our best to explain the phenomenon in Section 4 of the main paper. First from the failure of the vanilla method and partial success of latent gradients/linear approximated gradients (Columns (i)(ii)(iii) of Figure 2), we find the gradient issue. Then through the numerical value (Table below) and visualization of the gradients (Row (b) of Figure 2), we establish the noise assumption (Section 4.4.1). Based on the assumption and equations 8&10, we can effectively remove the noise through equation 11. Figure 2 also shows how these gradients affect the generation quality.

To gain further insights, we analyze the distribution of per-pixel gradient norms through a histogram on all the images at different steps during one optimization on 2D experiments using SDXL (as the Row (a) of Figure 2). The table below illustrates our findings, revealing that 1.3% of pixel gradient norms surpass 1,000, predominantly falling outside the typical pixel value range of [0, 1]. Additionally, our investigation indicates that 99.8% of images contain at least one pixel with a gradient norm exceeding 10,000.

Q2: Any side effects?

In the mesh optimization where an initial mesh is provided, we observe a consistent improvement using our method. We did find side effects in NeRF optimization from scratch: sometimes the shape may vanish at the beginning when we use SDS or VSD loss with PGC. However, Stable-DreamFusion and Fantasia3D use (normal+mask) as latent for SDS in the early training. This provides a more stable shape initialization. For other pipelines (e.g. ProlificDreamer), we can also skip the early training stage and directly apply our proposed PGC in the later training.

Q3: More results?

Yes. Thanks. We have now provided more results in the separate supplementary file (Figures 7, 8, 9).