We thank reviewer ZbmH for the valuable feedback.

W1. Qualitative results.

Thank you for the valuable feedback. We agree that earlier layers exhibit more visible influence in Figure 4. This is expected in our hierarchical design—deeper layers naturally introduce finer, subtler variations since prior layers are fixed. Regarding vehicle color in the NeRF dataset: it is not controlled by a specific layer but emerges from all layers jointly. This is because NeRF INRs jointly regress RGB and density at the output, causing color to be decoded at the final stage and entangled with spatial attributes, rather than controlled by a specific layer.

We appreciate your suggestion to adopt Figure 1’s visualization style. However, Figure 4 was designed to showcase different conditional chains per row, and we cannot revise it due to rebuttal constraints. Nevertheless, please note that some layer effects are visible and aligned with Table 1—for example, in the NeRF samples, a spoiler appears in the 3rd and 4th samples of the first row, with layer 3 fixed in the latter, supporting its role in controlling that attribute.

W2. Marginal improvements.

Thank you for the suggestion. Our primary goal is not to optimize reconstruction/generation quality alone, but to introduce a framework that enables hierarchical control—a key capability missing in prior generative INR methods such as mNIF and Functa.

As shown in Table 2, CHINR achieves notable improvements in evaluation metrics (e.g., PSNR/FID/SSIM) over most baselines except mNIF. However, CHINR exhibits significantly less memorization than mNIF, indicating better generalization. Furthermore, we aim to demonstrate that the conditional chain enables controllable generation without sacrificing quality—rather than to show it outperforms all models on every metric.

TW1. How is $h^l$ computed.

$h^l$ is an instance-specific latent at layer $l$, which is mapped to a gating vector that weighted averages the experts at layer $l$. To compute $h^l$, we use meta-learning or auto-decoding (Section 3.2) by initializing it around zero and optimizing it for each data instance.

TW2. Figure 2 meaning.

Thanks for pointing this out. We will make the captions clearer to explain this figure. Specifically, the transparency of noise refers to the noise schedule in the diffusion process, where the noise gradually dominates in the forward process. We shade the color of latents to distinguish latents from different layers. The latent at each layer is mapped to a gating vector that weighted averages the mixture of experts at that layer.

TW3. $||\epsilon,\epsilon_\theta()||^2$ in Line 242.

Thanks for pointing this out. It should be $||\epsilon - \epsilon_\theta()||^2$ as used in diffusion models. We will correct this in the final version.

We thank reviewer aYYY for the constructive feedback.

W1. Lack of predefined interpretability for each layer.

Thank you for the thoughtful comment. CHINR is designed to align the hierarchical structure of INRs with semantic abstraction, allowing each layer to control different levels of detail. While the semantics of each layer are not predefined (they are learned freely in Stage 1), the progressive control in Figure 4 and latent composition in Figure 6 demonstrate consistent layer-wise influence, suggesting that meaningful hierarchical semantics naturally emerge through training. Incorporating attribute supervision into Stage 1 to enforce predefined semantics at each layer is a promising direction to improve interpretability and fine-grained control in practical applications.

W2. Generalize to more complex data.

1. Thank you for your insightful comments. Compared with other state-of-the-art methods such as StyleGAN, the data instance size represented by CHINR is significantly smaller (e.g., $5 \times 64$ latent vector). Additionally, the LoE utilizes a five-layer neural network, which inherently limits its generation quality. The primary advantage of using implicit neural representations (INRs) lies in their ability to represent data at arbitrary resolutions, rather than in achieving high reconstruction fidelity.

2. To generalize to more complex data, two main directions can be pursued: (1) enhancing the capacity of the INR, and (2) refining the meta-learning pipeline. For the first approach, dividing each data instance into smaller patches can significantly reduce the burden on the INR, allowing it to better capture local structures. For the second, existing meta-learning pipelines typically initialize the latent vector from a Gaussian distribution and optimize it over just three steps. This process can be improved by enabling more efficient optimization over a greater number of steps, thereby enhancing performance and adaptability.

W3. Training/inference cost

Thanks for your suggestion. We analyze the resource consumption of CelebA experiments in the following. All experiments were conducted on an RTX 3090 GPU with a batch size of 8 for both training and inference.

Training cost for Stage 1 and 2:

Inference cost of CHINR, generating 1000 samples following HCDM:

We thank reviewer N8d9 for the valuable comments.

W1. Scalability to larger datasets.

Thank you for pointing this out. As discussed in the conclusion, scaling CHINR to larger datasets is a known challenge. While CHINR demonstrates the core idea of hierarchical control through INR parameter modeling, extending it to more complex data can be achieved by enhancing the INR representation capacity and employing more efficient training approaches. For example, localized solutions such as patch-wise or spatial-adaptive modulation methods [1,2,3] can help INRs better capture local structures and improve scalability.

Thanks for pointing out the layout problem of Fig. 4 and 7. We will increase the font size of those figures.

Reference

[1] Wang, Peihao, et al. Neural implicit dictionary learning via mixture-of-expert training. ICML, 2022. [2] Bauer, Matthias, et al. Spatial functa: Scaling functa to imagenet classification and generation. 2023. [3] Park, Dogyun, et al. DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations. ICLR, 2024.

We thank reviewer T7ed for the thoughtful comments.

W1. Out-of-distribution samples in conditional chain.

Thank you for your insightful feedback. We address error accumulation during both the training and inference phases.

In the training phase, we begin by teaching the model to capture inter-layer dependencies, using ground truth layer-wise latents (obtained from Stage 1) as conditions. Subsequently, we fine-tune the model to construct a coherent conditioning chain by progressively incorporating generated latents as conditions.

During inference, we further mitigate potential degradation by adjusting the standard deviation of the input noise in the diffusion process. This reduces the likelihood of generating outlier latents that could compromise output quality.

W2. Binary condition.

We choose to use binary condition for the following reasons:

1. With fewer bits to represent data, the model is less likely to memorize every detail of the training data. Instead, it must extract the core features that are most useful/general across many examples. This acts as a form of regularization.

2. As the model processes the input data, it learns to assign similar items to the same quantized value. Over time, these discrete codes become representative of certain semantic concepts or clusters, which correspond to the inherent structure of the data.

W3. Generalize to data without inherent frequency hierarchy.

Thank you for your interesting question. CHINR leverages the frequency-based representational hierarchy inherent in INR structure, which aligns well with data like images and 3D shapes with spatial frequency hierarchy. The current formulation may not be directly applicable for modalities like text that don't follow such a frequency hierarchy. However, CHINR’s core idea—modeling hierarchical latent dependencies across network layers—may still be adapted by identifying alternative structures (e.g., syntactic or semantic hierarchies) relevant to the modality.

W.4 Reconstruction quality affects Stage 2

The sensitivity of hierarchical control to quality variations is evident in two key aspects:

1. Reconstruction Quality in Stage 1: The effectiveness of hierarchical control in Stage 2 is bounded by the reconstruction quality achieved in Stage 1. If the proposed LoE fails to learn hierarchically structured latents during Stage 1, it limits the controllability during the generation in Stage 2. Our findings show that when the PSNR on CelebA-HQ exceeds 25, no visually noticeable distortions are observed, and hierarchical control performs reliably.

2. Quality of Ground-Truth Data: The controllability is also constrained by the quality of the ground-truth data used in Stage 1. Higher noise levels hinder the learning of conditional dependencies. In the extreme case where the data is entirely noise, the resulting latents lack meaningful structure. As the noise in the ground-truth data increases, both reconstruction and generation quality progressively degrade.

We thank reviewer XvvS for the valuable feedback.

W1. Scalability to larger datasets

Thanks for raising this scalability concern. The CHINR framework focuses on establishing the connection between INR parameters and data semantics for controllable generation. While our experiments use smaller datasets, the core framework is generalizable to larger-scale data as long as this fundamental connection holds. To handle increased complexity, more efficient learning methods should be employed to handle more complex data patterns.

Beyond simply adding more layers or experts, we can adopt localized solutions such as patch-wise or spatial-adaptive modulation, as explored successfully in [1, 2]. This reduces the burden of a single set of INR parameters to represent the whole data and allows the model to capture local structures, improving scalability and convergence without significantly increasing inference cost.

W2. Reliance on meta-learning

Thanks for pointing out this thoughtful concern. We agree that meta-learning requires a shared initialization for fast adaption, which works well for data with consistent structure (e.g., faces) but may struggle with more complex data. This is an intrinsic issue of the meta-learning method. To address this, one can allow more inner-loop updates at the cost of increased training time. It is also possible to incorporate more adaptive initialization or weight update strategies [3, 4] to better handle data variability. Alternatively, we can use auto-decoding as done in the NeRF experiments, which avoids the need for fast adaptation and may offer better generalization in such cases.

W3. Lack of targeted modification mechanism

Thank you for the insightful comment. CHINR is designed to leverage the hierarchical structure of INR parameters for layer-wise semantic control, rather than direct attribute editing. Although it doesn't currently support direct modifications like changing lip color, it can be extended with attribute supervision (in Stage 1) or latent manipulation to enable targeted edits—an exciting direction for future work.

Reference

[1] Bauer, Matthias, et al. Spatial functa: Scaling functa to imagenet classification and generation. 2023.

[2] Park, Dogyun, et al. DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations. ICLR, 2024.

[3] Wang, Ruohan, et al. Structured prediction for conditional meta-learning. Neurips, 2020.

[4] Baik, Sungyong, et al. Meta-learning with adaptive hyperparameters. Neurips, 2020.