Hyper-Transforming Latent Diffusion Models

We thank the reviewer for the constructive feedback and encouraging remarks. Your comments helped us significantly improve the manuscript. Below, we address all the concerns raised.

On Our Claims Regarding Unconstrained Resolution

We agree that validating our model’s ability to generalize to unseen coordinates is essential. Following your suggestion, we added new experiments on super-resolution. During reconstruction, test images at the training resolution are encoded, decoded into INRs, and evaluated on denser grids. For sampling, latent codes are drawn from the diffusion prior and decoded similarly. These experiments confirm that $`LDMI`$ captures continuous signals and generalizes to higher resolutions—highlighting a key advantage of INR-based generation.

New quantitative results for ERA5 and ShapeNet

To better support our claims of modality generalization, we now report the PSNR measured on the ERA5 and Chairs datasets:

We omit GASP (not applicable to reconstruction due to their GAN-based framework). $`LDMI`$ generalizes without adaptation and outperforms comparable models.

On CelebA and CelebA-HQ

You are correct—the original submission trained on CelebA, while baselines used CelebA-HQ. We now explicitly distinguish them and retrain $`LDMI`$ on CelebA-HQ at $64 \times 64$ for a fair comparison. To further demonstrate scalability, we train $`LDMI`$ on CelebA-HQ at $256\times 256$, achieving strong results not addressed by prior work.

On related work: NDPs and SimFormer

Thank you for pointing out these complementary works. We agree that approaches such as Neural Diffusion Processes, and SimFormer, are interesting and relevant within the broader context of function-space modeling with diffusion processes. In the following, we highlight the main difference.

NDPs leverage a diffusion model to define distributions over function values at given coordinates. Their architecture explicitly enforces exchangeability and permutation invariance via a bi-dimensional attention mechanism, and their sampling mechanism mimics Gaussian processes and related meta-learning methods such as Neural Processes. Importantly, the function itself is not represented via a neural network whose parameters are generated or learned—rather, the model learns to denoise function values directly, conditioned on inputs.

Simformer is designed for simulation-based inference (SBI), where the goal is to infer unknown parameters of stochastic simulators from observations. It treats both data and parameters as random variables and learns a diffusion model over the joint distribution $p(\boldsymbol{x}, \boldsymbol{\theta})$, allowing for flexible sampling of any conditional (e.g., posterior, likelihood, marginals). Parameters may include function-valued (infinite-dimensional) components, but they are not represented as INRs—rather, they are input variables within the inference pipeline. Simformer excels at amortized Bayesian inference with unstructured or missing data and flexible conditioning.

We added these references and discussion to the Related Work section of our revised manuscript.

On the interpretation of pixel values as integrals

We agree that pixels approximate integrals over regions. Following [SIREN, NeRF, Functa], we adopt the standard Dirac delta approximation by treating pixel values as samples at center coordinates. We’ve made this assumption explicit in the revised paper.

On the core challenge we address

Thank you for raising this point. We have clarified the motivation in the revised manuscript.

Our goal is to overcome the scalability bottlenecks of MLP-based hypernetworks in generative modeling of functions via INRs. While INRs excel at representing continuous signals, prior methods (e.g., Functa, GASP, VAMoH) require hypernetworks with tens of millions of parameters—even for small 3-layer INRs—making scaling impractical.

To address this, we propose the $`HD`$ decoder, a Transformer-based hypernetwork that maps latent samples to INR weights. Its parameter count remains fixed as INR size grows, with only sequence length increasing. We further optimize efficiency with a grouping strategy, enabling generation of large INRs at low cost.

Integrating this design into latent diffusion models allows $`LDMI`$ to model more complex signals and generalize across resolutions efficiently.

We thank the reviewer for the constructive feedback and for highlighting relevant connections to related work. Below, we address each of the raised concerns in detail.

On the relation to HyperDreamBooth

We appreciate your suggestion to consider HyperDreamBooth, which we have now cited and discussed in the revised manuscript. While the goals of our work differ, there are indeed interesting architectural parallels. Both approaches leverage transformer-based hypernetworks to generate weights, motivated by scalability and efficiency.

HyperDreamBooth focuses on personalizing pre-trained text-to-image diffusion models by generating low-rank residuals (via LoRA) from a single image—enabling rapid adaptation to new subjects. In contrast, $`LDMI`$ introduces a generative model over the space of implicit neural representations (INRs), which represent continuous functions across diverse modalities (e.g., images, 3D occupancy fields, and climate data).

Rather than modulating a pre-trained model, our Hyper-Transformer Decoder generates the full parameter set of an INR from latent samples, enabling resolution-agnostic, function-level generation. While both methods rely on Transformer-based hypernetworks, $`LDMI`$ operates in a distinct regime of generative modeling over functions.

On the competitiveness of our method and strengthened evaluation

Thank you for raising this point. In response, we have significantly strengthened our empirical results to more clearly demonstrate the advantages of our approach.

First, we added results on higher-complexity datasets such as CelebA-HQ $(256 \times 256)$, which fall outside the scope of prior baselines. Our model demonstrates strong performance in both generation and reconstruction at multiple resolutions—without retraining—showcasing the benefits of INR-based generation.

Second, we updated Table 1 with new results and included the number of hypernetwork parameters to highlight $`LDMI`$’s key strength: scalability. For example:

• GASP/VAMoH require 25.7M parameters to generate ~50K INR weights.
• $`LDMI`$ uses 8.06M parameters to generate ~330K INR weights for a deeper, 5-layer INR.

Despite having significantly fewer parameters, $`LDMI`$ delivers superior performance. For details, please refer to our response to Reviewer ER44, where the updated table is provided.

To further contextualize this comparison, we highlight some key aspects of the baselines:

• Functa relies on test-time optimization, fitting each test INR with access to ground-truth data. While this explains its high PSNR, it departs from our amortized inference setting and limits fair comparison. We include it for completeness given the task similarity.
• GASP cannot perform reconstructions due to its GAN-based design, limiting its use in conditional tasks.

In summary, $`LDMI`$ offers a compelling trade-off between quality, scalability, and generalization, supported by stronger experiments in the revised version.

On the hyper-training setting and training efficiency

Thank you for this question. In our hyper-transforming setup, we use the publicly available pre-trained LDM from [Rombach et al., 2022], specifically the LDM-VQ-4 variant, trained on CelebA-HQ at $(256 \times 256)$ resolution, and the LDM-VQ-8, trained on ImageNet. We freeze both the encoder and the diffusion-based prior, replacing only the decoder with our $`HD`$ module. While Section 3.5 of the original submission specified the frozen components, we have now revised the text to make this distinction more explicit.

This setup provides a key advantage: significantly faster training. By freezing most of the architecture and training only the decoder, we reduce the number of learnable parameters and avoid the need for a full two-stage training pipeline—common in diffusion models—where the autoencoder and the prior must be learned sequentially. Our hyper-transforming approach thus offers a lightweight, modular alternative that efficiently adapts pre-trained models to new functional decoding regimes.

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the clarity, motivation, and contributions of our work. Your comments helped improve the paper significantly. Below, we address each of the concerns raised in your review.

On the notation used for weights

We appreciate your comments on the novelty of our weight generation strategy and agree that clearer notation helps with readability. In the revised version, we have clarified the notation as follows:

• The bar in $\bar{\mathbf{W}}$ denotes globally shared, learnable parameters (e.g., $\bar{\mathbf{W}}^\text{i}$ and $\bar{\mathbf{W}}^\text{b}$).
• The superscript $i$ in $\bar{\mathbf{W}}^\text{i}$ refers to “input” to the Transformer Decoder. The superscript $\text{o}$ in $\mathbf{W}^\text{o}$ refers to its “output”, which transforms these global input weights by attending to the latent tokens.
• The subscript $l$ in $\mathbf{W}_l$ identifies the target INR layer. We omit this when not needed for clarity.
• The superscript $\text{b}$ in $\bar{\mathbf{W}}^\text{b}$ stands for the “base” weights, which serve as the template modulated by $\mathbf{W}^\text{o}$ via our grouping and reconstruction strategy.

These clarifications are now reflected in the manuscript and Figure 2.

On the correctness Equation 7 and the weight dimensions

We highly appreciate that you pointed this out, you are completely right. We have corrected the following two issues:

• We corrected the typo in line 272 to have $\bar{\mathbf{W}}^\text{b} \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$.
• Now we use the Hadamard or element-wise product $\odot$ for the weight reconstruction.

In Equation (7), each $w_{\lfloor c / k\rfloor}^{\mathrm{o}}$ (we're skipping bold notation here due to rendering issues) is an output token of the Transformer, and also the $\lfloor c / k\rfloor$-th column of the grouped weight matrix $\mathbf{W}^{\text{o}} \in \mathbb{R}^{d_{\text{out}}\times G}$. The length of the Transformer decoder sequences are thus $L\times G$, which is the total number of grouped columns. We can set the embedding dimension of the Transformer to $d_{out}$ or project the tokens using one feed-forward layer, depending on the case.

Additionally, we corrected the typo in line 238 to have $d_\text{in}$. We hope after these corrections, the concern is fully addressed.

On the competitiveness of our method

Our key strength is scalability: we outperform baselines while using fewer hypernetwork parameters. For instance:

• GASP/VAMoH: 25.7M params → 50K INR weights
• $`LDMI`$: 8.06M params → 330K INR weights (5-layer network)

We also extended Table 1 and added experiments on resolution generalization at generation and reconstruction, including CelebA-HQ $(256 \times 256)$, which are not tackled by prior work. We refer to our response to Reviewer ER44, where we include the updated Table. To clarify the context behind other baselines:

• Functa uses test-time optimization, tuning each test sample with ground-truth.
• GASP cannot perform reconstructions due to its GAN-based design.

In sum, $`LDMI`$ offers a strong balance of quality, scalability, and generalization.

On the choice of the weight reconstruction method

Thank you for this insightful question. Initially, we adopted a normalization-based reconstruction (inspired by Trans-INR), which worked well for MLP-based INRs. However, this approach led to instability when generating SIREN weights, particularly due to vanishing gradients—hindering generalization in tasks like super-resolution.

To address this, we introduced a novel scaling-based reconstruction:

$$\left(1+w_{\lfloor c / k\rfloor}^{\text{o}}\right) \odot \bar{w}_c^{\text{b}}.$$

This avoids collapse when $\boldsymbol{w}_{\lfloor c / k\rfloor}^{\mathrm{o}} \approx 0$ and provides stable training for high-frequency signals—leading to the strong results in our revision.

On coordinate input handling in the hyper-transforming setup

Thank you for raising this question. Our goal is to model data as samples from a stochastic process, using our $`HD`$ decoder to map latents into functions. In the hyper-transforming setup, we use convolutional encoders (often pre-trained) that implicitly capture coordinate information from structured grids:

• For image and ERA5 data: ResNet encoders
• For 3D occupancy fields: 3D convolutional encoders

The decoder then produces INR weights, enabling evaluation at arbitrary coordinates and decoupling resolution from representation.

We thank the reviewer for the detailed and thoughtful feedback, as well as for recognizing the novelty and clarity of our work. Below, we address the key concerns raised.

On the scope and nature of the contribution

$`LDMI`$ is not intended to compete with standard diffusion models that operate in pixel space. These models approximate $p(\boldsymbol{y})$ over discrete grids. In contrast, we model $p(\boldsymbol{y}|\boldsymbol{x})$ as a stochastic process, where $\boldsymbol{y}$ is a function represented via INR parameters $\theta$. This defines a substantially richer and more complex generative space—orthogonal to that of the suggested references. To our knowledge, $`LDMI`$ is the first to use Transformers as decoders to generate INR weights from latent samples in this setting.

That said, we appreciate the reviewer’s suggestion and agree that including and discussing these works helps contextualize our contributions. We have added the suggested pixel-based diffusion models [A, B, C] to the Related Work section and explicitly discussed how our problem setting fundamentally differs.

On baseline comparisons

To provide fair comparisons in the INR setting, we benchmark against the closest functional baselines. In addition, to emphasize the differences with respect to the provided references, we have added super-resolution experiments (e.g., samples and reconstructions) to show that LDMI generalizes across resolutions without retraining—a key property enabled by its function-based design.

On scalability of the $`HD`$ Decoder

While our original submission already compared $`LDMI`$ against MLP-based hypernetworks (used in all baselines), we have now further strengthened our scalability analysis. The updated Table 1 includes hypernetwork sizes:

GASP and VAMoH use ~25.7M parameters to generate 50K weights for shallow 3-layer INRs. In contrast, our $`HD`$ decoder uses only 8.06M parameters to generate 330K weights for a deeper 5-layer INR. Despite using 70% fewer parameters, $`LDMI`$ performs better, scaling to complex signals like CelebA-HQ $(256 \times 256)$—not tackled by any baseline.

To show superiority against latent diffusion and MLPs, as suggested, the table below compares $`LDMI`$ using either an MLP or our $`HD`$ on CelebA-HQ:

On the competitiveness of our method

We believe a few clarifications are helpful for contextualizing the results in Table 1.

Functa’s high PSNR arises from test-time optimization: it fits a separate modulation vector per test image using ground truth, unlike our amortized inference approach. This undermines a fair comparison, but we include it for completeness and clarify this in the revised manuscript.

While GASP reports lower FID on CelebA, it cannot perform reconstructions due to its GAN-based design, limiting its use in conditional tasks.

Considering these factors, LDMI provides a balanced solution—competitive in both sampling and reconstruction—while being more scalable and efficient.

Discussion on Diffusion Transformers

We appreciate the suggested references and have added them to the Related Work section. These approaches apply Transformers as denoising backbones in pixel-space diffusion. Our use is fundamentally different: we apply Transformers as hypernetworks to generate INR weights, enabling function-level generation beyond the limitations of discrete grids.