DDMI: Domain-agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations

We thank your detailed and thoughtful feedback for our work. We will address the concerns below.

1. [W1] Lack of references & comparison with references.

   We appreciate your reference to the recent papers that we have overlooked. We will include all the papers in the manuscript.

   • Regarding the paper ICML 2023.

     Indeed, their work also employs a VAE framework for modeling INRs and utilizes multiple hyper-generators to capture different aspects of signals. However, they still model the weight space of functions with fixed PEs, similar to previous generative models for INRs.

     In our paper, we have proposed generating PEs instead of generating weight space, and have quantitatively shown that it is more effective in generating high-fidelity signals. This is also supported by quantitative evidence presented in the table below, where our model outperforms the VaMoH approach in terms of signal fidelity on the CelebA 64x64 dataset.

     	CelebA 64x64
     VaMoH	66.27
     Ours	9.74

   • Regarding the paper CVPR 2023.

     We want to stress that scale-free image generation demonstrated in our paper differs from the super-resolution task. (CVPR 2023) proposes a specialized model architecture and methodology tailored for super-resolution tasks, which cannot generate diverse images like ours. Therefore, it is not feasible to make a fair comparison with our work.

$\\\\$

2. [W2] More details regarding interpolation/basis functions.

   We will improve the manuscript accordingly within the rebuttal period and notify you after the update.

$\\\\$

3. [Q1] Did you try to train a more powerful VAE latent space instead of training a latent diffusion model?

   During the early stages of our research, we experimented with training more powerful VAEs, including hierarchical VAEs [A]. Nevertheless, we observed that even these advanced VAEs encounter challenges in producing high-quality INRs, primarily due to issues with the 'prior hole' phenomenon. This limitation was found to be less significant in latent diffusion models, which is why we pursued that direction.

   ---

   [A] Vahdat and Kautz, NVAE: A Deep Hierarchical Variational Autoencoder, NeurIPS 2020

$\\\\$

4. [Q2] Posterior distribution used in D2C-VAE.

   We assumed multivariate normal distribution for posterior $q_\phi(\mathbf{z}|\mathbf{x})$ in all domains.

$\\\\$

5. [Q3] Dimension of latent space.

   The dimensions of the latent space $**z**$ are detailed in the supplementary materials, specifically in Table 9. We found that reducing the dimensionality of the latent space compromises sample quality to some extent, but it also results in a more efficient computational budget. This trade-off is consistent with observations from LDMs [B].

   ---

   [B] Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022

Thank you for your reply and support. As you suggested, we have updated the manuscript to provide more details about the basis/interpolation function. In this paper, we have defined basis fields as a set of dense grids (vector fields) consisting of generated basis vectors to be interpolated for generating a positional embedding given coordinates. See Section 3.1 & Section 4 in the main paper and Appendix A & Table 9 in the Appendix, and inform us whether the revised explanation sufficiently addresses any previous concerns.

Thank you for the constructive feedback and suggestions. We will address the mentioned questions below.

1. [W1] Unclear expressions + $\alpha$

   We will address the issues outlined below:

   • Tri-plane in the figure

     • You’re right, for the case of 2D images, the latent type would be a single plane, rather than a tri-plane. We will clearly clarify this in the main figure of the manuscript.

   • More implementation details in the main paper

     • As you suggested, we will add implementation details from appendix into the main paper for better readability.

   • Projection of sparse point clouds

     Below, we will elaborate the orthographic projection process from Conv-ONet [A]:

     1. Feature Extraction: PointNet processes the input point cloud to produce feature tensor of shape BxNxD .
     2. Projection: For each point, we apply an orthographic projection to map its feature onto three canonical planes (xy, yz, zx) of size HxW. Features projecting onto the same pixel are aggregated using average pooling, resulting in three distinct 2D grid features, each with the dimensionality HxWxD.

     Thus, the resulting 2D grids can be processed 2D CNN. We refer Figure 2(a) of Conv-ONet for better understanding.

   We appreciate your considerate comments. We will edit the manuscript accordingly within the rebuttal period and notify you after the update.

   ---

   [A] Peng et al., Convolutional Occupancy Networks, ECCV 2020

$\\\\$

2. [Q1] Why occupancy function over SDF?

   We are aware of the superior expressiveness power of SDF over occupancy functions. However, we favored occupancy functions over SDF for 3D INR due to the significant reduction in preprocessing time and dataset size. For instance, the SDF representation of the ShapeNetV2-chair dataset occupies roughly 620GB, whereas the occupancy function representation requires only about 12GB. Surprisingly, the results suggest that our method with occupancy functions yields shape quality that is comparable or superior to SDF-based methods, both qualitatively and quantitatively. This observation was consistent despite the inherent differences in the representations.

$\\\\$

3. [Q3] Code release

   Of course, we will release our code in the final version.

• Emphasizing other contributions rather than LDM.

  Thank you for your valuable feedback and suggestions. In this paper, we proposed a new meta-architecture for generating INRs given discrete inputs. Our architecture uses a latent diffusion model in the latent space of an asymmetric VAE that learns a joint latent space that bridges discrete data to continuous functions. The LDM in our title/model name describes the overall architecture of the proposed method similar to other models’ names in the literature [B,C,D].

  Indeed, reviewer KC2J raised a great point that our contributions are not limited to diffusion models. Our main contributions (D2C-VAE, HDBFs, CFC) are definitely applicable to other generative models, such as Normalizing Flows and GANs. As suggested, we will provide additional experiments with other generative models other than latent diffusion models to demonstrate the extensionality of our contributions.

  ---

  [B] Liu et al., AudioLDM: Text-to-Audio Generation with Latent Diffusion Models, ICML 2023

  [C] Koo et al., SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation, ICCV 2023

  [D] Stan et al., LDM3D: Latent Diffusion Model for 3D, CVPRW 2023

$\\\\$

• Improving the presentation of D2C-VAE + $\alpha$.

  As suggested, we have updated the manuscript (see sections 3.1, 3.2, 4, and Appendix A). Note that we reflect on your previous feedback in the current manuscript as well (clarification on the main figure, details on projection of point clouds).

We appreciate the reviewer for the valuable feedback. We will address the raised questions below.

1. [W1] Regarding the soundness of eq.(2).

   We would like to clarify the rationale behind our D2C-VAE. The objective of D2C-VAE is to maximize the ELBO of the log-likelihood of a continuous function $\boldsymbol{\omega}$ with the discrete data $**x**$. It is articulated below based on the standard variational inference principles: $$ \log p(\boldsymbol{\omega}) & = \log \int p_{\psi, \pi_\theta}(\boldsymbol{\omega}| \mathbf{z}) \cdot p(\mathbf{z}) d\mathbf{z} \\\\ & = \log \int \frac{p_{\psi, \pi_\theta}(\boldsymbol{\omega}|\mathbf{z})}{q_\phi(\mathbf{z}|\mathbf{x})} \cdot q_\phi(\mathbf{z}|\mathbf{x}) \cdot p(\mathbf{z}) d\mathbf{z} \\\\ & \geq \int \log \left(\frac{p_{\psi, \pi_\theta}(\boldsymbol{\omega}|\mathbf{z})}{q_\phi(\mathbf{z}|\mathbf{x})} \cdot p(\mathbf{z}) \right) \cdot q_\phi(\mathbf{z}|\mathbf{x}) \ d\mathbf{z} \\\\ & = \int q_\phi(\mathbf{z}|\mathbf{x}) \cdot \left( \log p_{\psi, \pi_\theta}(\boldsymbol{\omega}|\mathbf{z}) - \log \left( \frac{q_\phi(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})} \right) \right) \\\\ & = E_{q_\phi(\mathbf{z}|\mathbf{x})}\left[\log p_{\psi, \pi_\theta}(\boldsymbol{\omega}|\mathbf{z}) \right] - D_{KL}(q_\phi(\mathbf{z}|\mathbf{x}) || p(\mathbf{z}))

$

p_{\psi, \pi_\theta}(\boldsymbol{\omega}|\mathbf{z}) & \approx p_{\psi, \pi_\theta}(\boldsymbol{\omega(\mathbf{c})}|\mathbf{z})
\\\\
& = \prod_{c \in \mathbf{c}} p_{\psi, \pi_\theta}(\boldsymbol{\omega}(c)|\mathbf{z}), 

$$ where we assume the coordinate-wise independence. This leads to the approximation in Equation 3 of our paper. Assuming the likelihood as a multivariate normal distribution with isotropic covariance (\sigma$),

We appreciate your thorough and insightful feedback. We will address the raised questions below.

1. [W1, W2] Limited novelty & why dynamic PE?

   Recent INR studies in the literature have evidenced the significance of positional encoding (PE) to achieve high-quality representations. Parametric PEs, such as hash grids and orthogonal planes with learnable parameters [A,B,C], exhibit a superior expressive capacity compared to fixed PE (e.g., Fourier feature mappings [D]).

   The outstanding expressive power of parametric PEs suggests a promising direction for generative models, favoring learning to generate PE over MLP weights. In [E], they have demonstrated the generation of high-fidelity 3D-aware 2D images by generating tri-plane PEs based on GAN. Unlike the prior work, we propose PE generation with the latent diffusion model, enhancing it by our novel modules such as multi-resolution framework (HDBF) and coarse-to-fine conditioning (CFC).

   Indeed, at the early stage of this project, we developed a latent diffusion model for weight generation (with fixed PEs) with a hypernetwork to modulate the MLP weight of INR ($\pi$) with latent from the Encoder. The weight generation model exhibited significantly slower convergence and failed to produce high-quality samples. Due to limited resources, we are not able to provide experimental results to compare weight vs PE generation (without HDBFs and CFC). We will add the comparison in the final version.

   ---

   [A] Muller et al., Instant neural graphics primitives with a multi-resolution hash encoding, ACM ToG 2022

   [B] Cao et al., Hexplane: A fast representation for dynamic scenes, CVPR 2023

   [C] Fridovich-Keil et al., K-planes: Explicit radiance fields in space, time, and appearance, CVPR 2023

   [D] Mildenhall et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, ECCV 2020

   [E] Chan et. al., Efficient geometry-aware 3d generative adversarial networks, CVPR 2022

$\\\\$

2. [W3] Scalability

   Thank you for your comments on scalability. We have evaluated our method on larger datasets such as CIFAR10 and LSUN church, which contain 50K and 126K samples respectively, as detailed in Table 11 of the appendix. While we acknowledge the value of scaling to even larger datasets like Objaverse, our current resources limit this expansion. Nevertheless, we see this as a crucial next step and are exploring opportunities for such large-scale evaluations.

$\\\\$

3. [W4] Significance of domain-agnostic architecture

   Thank you for your insightful comment. Our domain-agnostic architecture, providing a consistent latent type for LDM, enables learning the unified priors across multiple domains, as you mentioned. We believe that our approach paves the way to a large-scale, multi-modal foundation model that encompasses 2D, 3D, or even other data types. Such a model could serve as a new tool for a wide range of cross-domain downstream tasks.

$\\\\$

4. [W5] Domain-agnostic architecture of DDMI

   Our framework proposes a domain-agnostic meta-architecture for INR generation. Also, overall training and generation schemes are the same across multiple domains. This domain-agnostic meta-architecture allows us to use identical diffusion models in the latent space for generating 2D grid latent variables $**z**$.