Diffusing Differentiable Representations

Thank you for your comments and review. We will address your concerns in turn.

Experiment details: As mentioned in the paper, generating a NeRF with our method takes around 24 minutes for 199 NFEs. We used as many samples as possible in the 40GB VRAM of an NVIDIA A6000; our experiments were eight samples. Generating a comparable NeRF using SJC takes around 34 minutes for 10000 steps using the code they provide. Generating eight batched SIREN images using our method takes 82 seconds. Generating a SIREN Panorama with our method takes 218 seconds.

As you correctly pointed out, Figure 1 in the paper had some issues (the images were out of order). We have briefly described the problems in the common response and attached the corrected Figure 1 in the accompanying PDF.

We use RePaint for all our experiments, including the SIREN image and NeRF experiments. Without the RePaint method, the results are much less compelling. In fact, the NeRF experiments diverge without it. The INR implicitly defines the constraint, so it is impossible to remove the constraints from the INR. However, as a proxy for the constraint, one could consider using the CFG scale, where a high CFG corresponds to a stronger constraint because it enforces the renders from the INR to look a specific way. On the other hand, low CFG means more flexibility in the kinds of INR we can sample. Figure 3 shows the results of ablating RePaint on SIREN Panoramas with a high and a low CFG.

We thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.

Thank you for your reply and for providing more details. However, I have 3 more further questions:

1. How many iterations per Eular step do you use for optimization?

2. Generating eight batched SIREN images using our method takes 82 seconds. Generating a SIREN Panorama with our method takes 218 seconds.

What is the baseline methods' runtime?

2. My concern regarding limitations has not been addressed yet. Did you discuss limitations in Section 6? Or do you think there is no clear limitation?

Thank you for your comments and review. We will address your concerns in turn.

SJC and Dreamfusion (SDS) are extremely similar in methodology when accounting for the conversion between the denoising model and the score function. The key differences are in the weighting function that the expectation is taken over, the optimization and sampled times in the schedule, and regularizers added for the 3D case. We show how the SJC and SDS objectives are identical up to a weighting term and delineate the major differences between the methods and how they compare to our method in the common response.

Reliable metrics for generative models are primarily image-based. This is why we used the SIREN image diffrep to evaluate our method quantitatively. While it is possible to generate NeRFs and SIREN panoramas using the other methods, comparing them would be infeasible since there are no good techniques to compare distributions of general diffreps yet.

We will make the code public in the final version of the paper.

We have significantly changed the technical description of our methodology to make it simpler and more intuitive. We hope you find this new description clearer.

The score model associated with the noise predictor implicitly defines a distribution on the data space $\mathcal X$, and the reverse time PF-ODE provides a means to sample from that distribution. In this section, we will show how to use the score model and the PF-ODE to sample the parameters of a differentiable representation (diffrep) so that the rendered views look like samples from the implicit distribution.

In SJC and DreamFusion, they derive their parameter updates using the pullback of the sample score function through the render map $f$. Explicitly, this pullback is obtained by using the chain rule $\tfrac{d(\log p)}{d\theta} = \tfrac{d(\log p)}{dx}\tfrac{dx}{d\theta}$. The pullback score is then used as is to perform gradient ascent in the parameter space. However, since $p$ is a distribution and not a simple scalar field, the change of variables formula requires an additional $\log \det J$ for the appropriate score function in the parameter space, where $J$ is the Jacobian of $f$. Carefully examining this approach through the lens of differential geometry reveals even deeper issues.

From differential geometry, recall that it is a vector field (a section of the tangent bundle $T\mathcal X$), not a differential form (a section of the cotangent bundle $T^*\mathcal X$), which defines the integral curves (flows) on a manifold. To derive the probability flow ODE in the parameter space, we must pull back the vector field $\frac{dx}{dt}\in T\mathcal X$ to the parameter vector field $\frac{d\theta}{dt} \in T\Theta$ using the render map $f$. The pullback of the vector field through $f$ is given by $(f^* \frac{dx}{dt})|_\theta = (J^\top J)^{-1}J^\top \frac{dx}{dt}|\_{f(\theta)},$ and thus the pullback of the probability flow ODE is

$

\frac{d\theta}{dt} = -\dot\sigma(t)\sigma(t)(J^\top J)^{-1}J^\top\nabla \log p_t(f(\theta)).

$

We note that this is in contrast to the SJC and DreamFusion score. The confusion arises when comparing the types of elements that are pulled back. While the left-hand side of the PF-ODE $\frac{dx}{dt}$ is a vector field, the score function $\nabla \log p_t(x(t))$ on the right-hand side is a covector field, i.e., it is a differential form. Pulling back the score function as a differential form correctly yields $J^\top\nabla \log p_t(f(\theta))$, the term used in SJC. The problem is that hidden in PF-ODE is the use of the (inverse) Euclidean metric to convert the differential-form score function into a vector field to update the parameters. In canonical coordinates, the Euclidean metric is the identity. As a result, the components of the score function do not change when they are transformed into a vector field by the Euclidean metric. Therefore, we can safely ignore the metric term in the original PF-ODE formulation for $\mathcal X$.

If we want to convert the pulled-back score function into the corresponding pulled-back vector field, we need to use the pulled-back Euclidean metric inverse given by $(J^\top J)^{-1}$. When we use this metric, we get the same pulled-back form of the PF-ODE as given above.

A more informal way of seeing why the correct update must look like the pulled-back ODE we have derived here rather than that given by SJC is to look at the case where the number of input and output dimensions is the same. From the chain rule, we have $\frac{d\theta}{dt} = \frac{d\theta}{dx} \frac{dx}{dt}$. We can compute $\frac{d\theta}{dx}$ using the inverse function theorem $\frac{d\theta}{dx} = \left(\frac{dx}{d\theta}\right)^{-1} = J^{-1}$, where $J$ is the Jacobian of $f$. So we arrive at $\frac{d\theta}{dt}=J^{-1}\frac{dx}{dt}$. When $f$ is invertible, this equation is equivalent to the one given above. For non-invertible $f$, we can interpret the pullback as the solution to the least squares minimization problem $\min\left\|J\frac{d\theta}{dt} -\frac{dx}{dt}\right\|^2$ for $\frac{d\theta}{dt}$.

We thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.

Thank you for your comments and review. We will address your concerns in turn.

We have added a discussion of additional 3D asset generation from image diffusion methods like Zero-1-to-3, HiFA, Magic123, LatentNeRF, and Fantasia3D, which you mentioned. These works build off the Dreamfusion (SDS) / SJC method by adding additional inputs, finetuning, or regularization to improve the generation quality or by expanding the inputs for generating 3D assets. Zero-1-to-3 builds off of SJC using the same form of the score chaining, but with a score function fine-tuned to make use of a single input real image and additional view information. Magic 123 uses Zero-1-to-3 with some additional priors based on the 3D information. Fantasia3D separates geometry modeling and appearance into separate components, using SDS to update both. HiFA adds a different schedule and applies regularization to improve SDS. Finally, LatentNERF also uses SDS but parametrizes the object directly in the latent space of the stable diffusion autoencoder, rendered not to RGB values but instead into the dimensions of the latent space. These methods use the SDS/SJC backbone method for sampling/mode finding. In contrast, our work is focused on providing a more faithful sampling procedure to replace SDS/SJC for 3D generation and broader differentiable function sampling.

Similarity metrics for SJC: Initially, we did not report PSNR, SSIM, and LPIPS for SJC because the generations have little relation to the ones generated by the vanilla diffusion model with the same seed. We have added these in the table below. Notice that they are far, far worse than our method (a better comparison is a measure of distributional similarity like KID, which we report in Figure 1). This is because SJC does not follow the same trajectory as the PF-ODE, so even if it were performing sampling, the samples would not look identical to those produced in the image space. The PSNR, SSIM, and LPIPS scores are used to compare images with identical content but with different compression levels; they are not designed to compare distributions of images.

Table for metrics on SJC

Benchmarking Dreamfusion: SJC and Dreamfusion (SDS) are extremely similar in methodology when accounting for the conversion between the denoising model and the score function. The key differences are in the weighting function that the expectation is taken over, the optimization and sampled times in the schedule, and regularizers added for the 3D case. The SJC paper includes a section discussing the differences between the methods and provides a qualitative assessment of the assets generated by both methods.

Precise details of how RePaint is applied: The most straightforward approach to encourage the pullback of the PF-ODE to produce intermediate samples that are representable by $f$ is to adapt the RePaint method (designed for inpainting) to this consistency constraint. RePaint takes advantage of the complete Langevin SDE diffusion process instead of the PF-ODE we have been considering here. It works by intermixing several forward and reverse steps in the schedule. Since RePaint requires a stochastic process to apply the conditioning, we use the DDIM \citep{song_denoising_2022} sampling procedure with $\eta=0.75$ for both the forward and backward steps.

$\epsilon(t)$ and $\epsilon(\pi)$: Following the pulled-back PF-ODE we can find $\theta_t$ so that $f(\theta_t)$ represents a sample $x(t)\sim p_{0t}(x(t)|x(0)) = \mathcal N(x(t);x(0),\sigma^2(t)I)$. One limitation of this approach is that $x(t)$ is noisy, and differentiable image or 3D representations have a hard time expressing noise. In these settings, $J$ is ill-conditioned, leading to poor performance and long convergence times.

To address this issue, we can factor $x(t)$ into the noiseless signal and noise using the reparameterization of the perturbation kernel $x(t) = \hat x_0(t) + \sigma(t)\epsilon(t)$. If we let $\epsilon(t) = \epsilon$ be constant throughout the sampling trajectory and start with $\hat x_0(T) = 0$, we can update $\hat x_0$ using $\frac{d\hat x_0}{dt} = \frac{dx}{dt} - \dot\sigma(t)\epsilon$. Using this decomposition, we let $f(\theta_t)$ represent the noiseless $\hat x_0(t)$ instead of $x(t)$, which makes it much more likely that $J$ is much better conditioned.

Question: Why is the score not the gradient of a scalar function? Here, we use scalar in the differential geometry sense of being a scalar quantity on the given manifold, not an object that has a value dependent on the chosen coordinate chart. For continuous probability distributions, the probability density function varies not just on the chosen point but also on the given coordinate chart, as made evident by its changing value under a change of variables (where the additional $\det J$ term arises). Informally, this can be seen from the fact that $\int_R p_X(x)dx = \int_R p_Y(y)dy$, but because of the changing volume element $p_X\ne p_Y$. In contrast, a scalar function will have the same value when evaluated at the same point but in two distinct coordinate charts.

We also agree that the method section could also be explained better. We have provided a revised method section in our response to Reviewer wR2k. We hope this more intuitive description of our method improves your consideration of our paper.

We thank you for your time spent reviewing. We have invested substantial effort in addressing your concerns and improving the quality of the paper, and we ask that you consider adjusting your score in light of our response. Please let us know if you have additional questions we can help answer.

Thank you for your positive review. Please see the common response for additional SIREN panoramas. Let us know if there are any remaining concerns we might be able to address.