Scaling Up Parameter Generation: A Recurrent Diffusion Approach

Q1: How does RPG perform on architectures not observed during training?

A1: Thank you for the insightful comment. RPG targets functional generalization: producing weights for unseen tasks while the network topology remains fixed. Extending RPG to handle architecture changes belongs to structural generalization, a distinct research direction tied to design and hardware considerations. We therefore confine this study to a fixed architecture, where the challenges are already substantial and the practical value clear. We will clarify this limitation and future extension in the revision.

Q2: The total computational overhead is not compared to directly training a model on the target task.

A2: We acknowledge that collecting checkpoints and training RPG requires several hours or days, but once trained, it can generate models for target tasks indefinitely. The advantage of direct training on these tasks decreases as the number of tasks increases.

This is analogous to an image generator: once trained, it can create unlimited images cheaply, and we do not count manually drawing every potential image as part of its training cost.

Q3: Why does the generated parameter performance degrade significantly on unseen tasks?

A3: Thank you for the question. We clarify the details below.

• Unseen tasks are evaluated in a zero-shot setting， so some decline relative to full-shot training is expected.

• The drop is modest: RPG still outperforms random initialization both at the outset and after finetuning.

Consequently, RPG provides faster convergence and higher final accuracy, as shown in Table 18. For your convenience, we reproduce the key numbers below.

• Setting: We compare three approaches: (i) training ViT-Tiny from scratch, (ii) fine-tuning an ImageNet-pretrained ViT-Tiny, and (iii) fine-tuning an RPG-initialized model.

  epoch	from scratch	fine-tune	RPG init + fine-tune
  0	50.0	52.9	94.4
  1	61.4	90.3	96.8
  5	69.1	96.3	97.3
  10	74.9	97.1	97.5
  50	86.7	97.7	97.8

• Conclusion: RPG significantly accelerates training for unseen tasks. These results establish RPG initialization as an effective approach that substantially reduces training costs while maintaining competitive performance.

Q4: In what real-world scenarios would recurrent diffusion-based parameter generation be more advantageous than direct training, and are there measurable benefits that justify its use?

A4: Thanks for the comment. RPG offers several practical advantages in real-world scenarios:

• Efficient initialization: RPG provides superior parameter initialization compared to random weights, significantly reducing training time and computational costs as demonstrated in Table 18.
• Few-shot learning: For tasks with limited training data, RPG-generated parameters serve as informed priors that enable better performance than training from scratch.
• Rapid model deployment: RPG enables near-instantaneous model adaptation for new tasks without requiring extensive training, making it valuable for real-time applications where quick deployment is critical.

Thank you for the detailed rebuttal. However, my primary concerns about the model's scalability to unseen architectures and its high computational overhead have not been fully addressed, so I will be maintaining my original score.

Q1: Lack of statistical significance over the results, showing only the distribution of parameters generated by a single model.

A1: We have conducted two additional repeated experiments (model2 and model3) for Table 1. The original content in Table 1 corresponds to model1 in the updated table. We now present the distribution of parameters generated by 3 independent models as well as the aggregated results across all 3 models (all). When time permits, we will also conduct repeated experiments for other key results such as Tables 2, 7, and 8.

Q2: No code release.

A2: We will release all experimental codes in the revision. We guarantee that all experiments in the paper are reproducible.

Q3: What is the impact of the positional encoding?

A3: We address this question from two perspectives: (1) the necessity of positional encoding, and (2) the choice between 1D and 2D positional encoding.

• 1. Necessity of positional encoding: For Mamba-based recurrent models, positional encoding may be less critical since the model can inherently understand token order through its recurrent nature. However, for transformer-encoder-based models, positional encoding is essential to indicate which parameter position is being generated. For consistency across different architectures and framework flexibility, we uniformly apply positional encoding to all models.

• 2. 1D vs. 2D positional encoding: Under the current paradigm, there is no significant performance difference between 1D and 2D positional encoding. However, we plan to extend our framework to support variable-structure parameter generation in future work. The 2D encoding provides richer structural information about the model architecture, which will be crucial for these planned extensions.

Q4: How are permutation states generated and how do they work?

A4: Thanks for your comment. We explain in detail how permutation states are generated and how they work:

• How permutation states are generated: We assign different permutation states to checkpoints with different symmetric states. Typically, multiple models saved in one training run have similar symmetric states (assigned the same permutation state), and vice versa. However, in this work, we assume that all checkpoints are in different symmetric states, which is actually a more conservative assumption, so we assign different permutation states to each checkpoint.

• How to choose permutation state during inference?

  • First, neural networks with different initialization can have various symmetric structures that perform similarly[1].

  • This symmetry phenomenon doesn't affect performance, but hinders hyper networks' training. It happens when training with different random seeds.

  • Based on this, we devise the permutation state to encode this symmetry and facilitate RPG's learning. That is, assigning different permutation states to checkpoints trained with different random seeds.

  • Previous works such as P-diff [2] collect checkpoints from a single training trajectory, i.e., one random seed. Permutation state may not be vital in this setting.

  • In inference, we only need to randomly assign permutation state.

  • To further explore permutation state's influence, we conduct experiment in Appendix C.1 and put it here for convenience:

    • Setting: we use the ViT-T tuned on CIFAR-10, and test RPG with and without permutation state.

    number of random seeds	original	without permutation state	with permutation state
    1	88.2	88.0	88.1
    3	88.3	fail	88.1
    10	88.5	fail	88.2

    • Findings:
      • RPG w/, w/o permutation state performs similar when trained only on one type of checkpoints.
      • When the number of random seeds increases, RPG without permution soon fails.
      • RPG with permutation works well with large number of different random seeds.
    • Conclusions:
      • Permutation state encodes the parameter symmetries as conditions and works well when trained on checkpoints from multiple training runs.

• Overview of how permutation states work: In the training phase, the permutation state (i.e., a one-hot encoding) passes through an MLP and is added to the positional encoding of the model input to represent the current model's symmetric state. In the inference phase, we typically only care about the model's performance rather than the model's symmetric state. Therefore, we only need to randomly select a permutation state (default is state #1, i.e., 000...001). Alternatively, we can specify a particular state to generate models in a specific basin.

[1] B. Zhao et al. Symmetry in Neural Network Parameter Spaces
[2] K. Wang et al. Neural Network Diffusion

Q1: There are works such as [1,2] that have scalable hypernetworks.

A1: Thanks for the constructive suggestions. We answer the question as follows:

• The papers pose are interesting, [1] achieves large scale generation for meta-learning, [2] cusmotizes NeRFs via transformer hypernetworks.
• We change the statement in line 20 to "hypernetworks that generate parameters from noise dont scale well".
• We apologize for overlooking these literatures at first, and we will add them to Section 5 in the revision.

Q2: There are works such as [2,3,4] that demonstrate hypernetworks can be used in real-world applications.

A2: Thanks the reviewer for the comment and we answer as follows.

• We appreciate reviewer Q57G for pointing out these relevant works. NeRFs are practical models and this serves as a effective indicator for hypernetworks' utility.
• Our work, on the other hand, are trying to explore the generation of large models in recent years, such as vision transformers and LoRA adapter for LLMs.
• We conduct experiment on models with 3 to 200M parameters across image classification, object detection, small language models, and LoRA adapters for text and image generation. Through these explorations, our scope is broadening the application scenarios of parameter generation.
• We highly apprecitate your interest in hypernetworks and the helpful suggestions posed, and we rephrase "This discrepancy not only underscores the challenge of scalability but also confines existing generators to academic demonstrations rather than real-world applicability." $\rightarrow$ "This discrepancy underscores the challenge of scalability and if solved, can strengthen parameter generation's real-world applicability."

We hope these answer can address the concerns, and we appreciate the reviewer Q57G's interest in hypernetworks. We hope to hear more in discussion phase.

Q3: Why is (a) generating parameters from noise more appealing than (b) mapping learnable embeddings to network weights?

A3: Thank you for the question. We answer as below.

• Better scalability and memory use.

• Embedding decoders must output the full parameter tensor in one pass, so memory scales with model size.

• RPG generates weights sequentially: each token is conditioned on previous predictions. This design choice trades a small compute overhead for orders-of-magnitude lower memory.

• Action: This discussion is added to the revision.

Q4: Using linear mode connectivity (LMC) to evaluate model similarity.

A4: Thanks for the comment. We conduct experiment as follows:

• Setting: We use ResNet18 on CIFAR-10 with different training + sampling schemes, and test the performance of linear model weights interpolation:

  Method \ Accuracy (%)	$A$	$0.5A+0.5B$	$B$
  LMC of 2 models sampled by RPG trained on checkpoints from the same basin	95.1	95.0	95.0
  LMC of 2 models sampled by RPG trained on checkpoints from 2 basins and sampled with 1 permutation state	95.0	94.8	94.9
  LMC of 2 models sampled by RPG trained on checkpoints from 2 basins and sampled with 2 permutation states	94.9	18.4	94.9

Q5: Missing related work: A discussion on [1,2,3,4].

A5: Thanks for the comment. We discuss papers [1,2,3,4] in our A1 and A2, and we put the results here for convenience.

Observations:

• Existing approaches tend to specialize: some stress high performance, others emphasize generalization, and few focus on scalability.

• In contrast, our work can deliver strong performance, broad generalization, and efficient scalability simultaneously.

We add this discussion and the table above in the revision.

[1] Babu, Sudarshan, Pedro Savarese, and Michael Maire. "HyperNetwork Designs for Improved Classification and Robust Meta-Learning."
[2] Chen, Yinbo, and Xiaolong Wang. "Transformers as meta-learners for implicit neural representations." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
[3] Babu, Sudarshan, et al. "Hyperfields: Towards zero-shot generation of nerfs from text." arXiv preprint arXiv:2310.17075 (2023).
[4] Lorraine, Jonathan, et al. "Att3d: Amortized text-to-3d object synthesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q6: Is the permutation state essentially a one hot encoding on the trained networks used to train the hypernetwork?

A6: Thanks for the comment.

• We chose one-hot encoding because it offers a straightforward way to represent parameter symmetry.
• However, the method does not depend on this particular choice: any encoding scheme that distinguishes different permuatation states is adaptable to our framework.
• We add this clarification in Section 2.2, line 117 in the revision.

Q1: Training vs inference details are underspecified. How to choose permutation states at inference time remains unclear.

A1: Thanks for the comment.

• How is unnormalization implemented?

  • The weights (scales) and bias (shifts) we used for normalization and unnormalization are NOT calculated independently for each model, but are based on the average over the entire training set.
  • The layer-wise mean and std are calculated by: $\mu_{l}=\frac{1}{N}\sum_{n=1}^{N}mean(W_{n}^{l}), \sigma_{l}=\frac{1}{N}\sum_{n=1}^{N}std(W_{n}^{l})$, where $W_{n}^{l}$ denotes layer $l$ of the $n$-th model, and $\mu_{l}$, $\sigma_{l}$ denotes its mean, variance, respectively.
  • These statics are pre-computed and stored, and utilized in unnormalization process for generated parameters.

• How to choose permutation state during inference?

  • Due to space limitations, please refer to A4 of DHqN.

• Action:

  • We put these results and analysis in Section 3.3.
  • We add more motivation of permutation state line 117, Section 2.2.

Q2: how many RPG parameters are required relative to each target model.

A2: Thanks for the comment.

• We present the details of structures of RPG and number of parameters it can generate in Table 12, Appendix B.3. We put the data here:

  model scales	RPG-T	RPG-S	RPG-B	RPG-L
  num. can generate	up to 50K	up to 10M	up to 50M	up to 200M
  recurrent	1.3M	256M	1018M	3076M
  diffusion	0.3M	17M	69M	273M
  total RPG	1.6M	273M	1087M	3349M

• Findings:

  • The denoising network's parameter scale is close to the number of parameters that RPG can generate.
  • Most of RPG's parameters come from recurrent backbone.

• Conclusions:

  • RPG hass a light-weight denoising network.
  • With lighter backbone, RPG is possible to generate billion parameters.
  • RPG can generate almost infinite number of models once trained, so the efficiency trade-off is worwhile in the long run.

• Action:

  • We add these analysis to Section 3.1.
  • We will explore lighter backbones.

Q3: How many checkpoints are needed to train RPG well.

A3: Thanks for the comment.

• We evaluate RPG's performance on 2 scenarios: close-set (generate models seen in training) and open-set (generate unseen models) generation.

• Close-Set: In Table 21, Appendix C.5, we conduct ablation studies of checkpoints number for training RPG. We use ViT-Tiny on ImageNet.

  • Setting:

  number of checkpoints	5	25	100	300	1002
  acc.	75.1	75.2	75.2	75.2	75.3

  • Findings and Conclusions:
    • Only 1 checkpoint can train RPG well, showing its strong close-set geneation ability with very limited data.

• Open-set: Thanks for the suggestion, we conduct the following ablation:

  • Setting:
    • We randomly select 5, 25, 100, 300, and 1002 (all) checkpoints claimed in Section 4 as training set. All configurations are consistent with Section 4.

  number of checkpoints	5	25	100	300	1002
  acc.	54.1	84.7	86.9	87.8	90.0

  • Findings and conclusions:
    • RPG can generalize to unseen tasks even trained on a smaller number of checkpoints, i.e., 100.
    • More data can improve performance, showing that generalization needs basic amount of training data.

Q4: "Unseen tasks" have the same training data as seen tasks, and the "task" amounts to adapting the weights of the last layer, so it is not a useful result in practice. Furthermore, the relative sizes of training and test sets (1002 and 20 tasks) are not realistic.

A4: Thanks for the comments.

• 1. "Unseen tasks" have the same training data as seen tasks.

• 2. The "task" amounts to adapting the weights of the last layer, so it is not a useful result in practice.

• 3. The relative sizes of training and test sets (1002 and 20 tasks) are not realistic.

• We refer our answers to them as A 4.1, A 4.2, and A 4.3, respectively.

• A 4.1: Thanks for the comment.

  • Parameters for unseen tasks and seen tasks are of same network structures, but they are trained on different binary embedding classification tasks.
  • ALL parameters in unseen tasks are not involved in RPG's training, so the evaluation process is zero-shot.

• A 4.2: Thanks for the comment.

  • We first conduct full-parameter fine-tuning.
  • Therefore, the backbone parameters for the 1022 training samples are not identical.

• A 4.3: Thanks for the comment.

  • As ablation study in A3 suggests, RPG can effectively learn from limited training checkpoints, i.e., achieving 84.7% accuracy with only 25 checkpoints, 86.2 with 100 checkpoints

  • Additionally, our experiments on object detection and larger dataset in Appendix C.2 provide more generalization results: (Note that we change embedding to hex to save words)

  • object detection:

    • Setting: YOLOv8n on PASCAL VOC, using 20-bit 0-1 embeddings.

    • Results:

      unseen tasks	epoch-0	epoch-1	epoch-2	epoch-5
      2010c	0.00 / 0.29	0.01 / 0.30	0.06 / 0.33	0.18 / 0.39
      08026	0.01 / 0.25	0.02 / 0.30	0.05 / 0.32	0.21 / 0.39
      11048	0.01 / 0.28	0.09 / 0.33	0.12 / 0.35	0.32 / 0.38
      20061	0.00 / 0.24	0.04 / 0.25	0.04 / 0.29	0.13 / 0.35
      0a060	0.00 / 0.19	0.03 / 0.22	0.11 / 0.26	0.21 / 0.33

    • Conclusion:

      • RPG offers good initializations, faster adaptation and reduces training costs.
      • RPG can generalize to complex tasks.

  • larger dataset:

    • Setting: ViT-Small on ImageNet-1K (coarse-grained 10 classes).

    • Results:

      unseen tasks	original	RPG
      015	91.5	89.7
      1a1	88.5	87.0
      3f2	89.1	88.1
      30b	89.3	87.3
      2ad	91.4	90.4
      370	88.6	87.2
      2be	92.5	90.1
      1ea	89.5	87.5
      2d1	90.1	87.9
      360	90.2	89.2

    • Conclusion:

      • RPG can also generalize at ImageNet, achieving comparable performance to trained models.

Q5: Some key results are buried in the appendix.

A5: We make the following modifications.

• Move Tables 19 and 20 , and relevant analysis to the main text in the revision.
• Examine other details of our work and refine it accordingly.

Q6: Can you report standard deviations for Tables 1, 2 and 7?

A6: We provide the std below and include them in the revision:

• Note:
  • We present the data in accuracy (std) format.
  • Due to word limit, we only provide Table 1 and will update the rest results once entered discussion phase.
• Table 1:

  arch. \ acc.(%)	ResNet-18	ResNet-50	ViT-Tiny	ViT-Small	ViT-Base	ConvNeXt-A	ConvNeXt-L
  original	70.0(0.28)	79.8(0.05)	74.9(0.13)	81.4(0.08)	84.4(0.08)	75.2(0.07)	85.8(0.10)
  generated	69.5(0.23)	79.5(0.07)	75.3 (0.07)	80.5 (0.14)	84.4 (0.12)	74.4 (0.11)	85.5 (0.19)

Q7: Why do blue subscript of Table 7 vary across entries in the same column?

A7: Some works did not have updated the code with the latest paper, making them difficult to reproduce. Therefore, all the data in Table 7 are taken from the original papers and are not our reproduced results. Since the checkpoints used in the original papers vary across studies, we have added blue subscript markers to indicate this.

Q8: What is the smallest RPG parameter count that can match at least 95 percent of baseline accuracy?

A8: Thanks for the comment.

• We conduct ablation studies Appendix C.3, Table 19 to explore this, and we put the results here for convenience.

• Settings: We generate ViT-Tiny with RPG of various parameter counts and report the accuracy.

• Results:

  # RPG parameter	acc.
  20M	0.3
  81M	70.8
  273M	75.2
  1087M	75.3

• Findings and Conclusions:

  • At parameter count of 81M, RPG generates model with accuracy of 70.8%
  • As in Section 3.1, Table 1, baseline accuracy is 74.9%.
  • The generated model achieves 70.8% ÷ 74.9% = 94.5% (roughly 95%) of the original model's performance.

Q9: Generated models are still below one billion parameters, limiting immediate applicability to large language models.

A9: Thanks for the comment.

• We admit that RPG currently don't support LLM full model generation.
• As in A2, RPG's scaling up bottleneck is its heavy backbone. We are currently working on finding lighter alternatives.
• Moreover, we can generate LoRA parameters, which offers efficient task adaptation of LLMs.
• Solutions for scaling up:
  • Finding lighter recurrent backbones.
  • Improving the Efficiency of tokenization.

Q10: RPG is not permutation invariant; it learns explicit neuron permutations, which increases model size and training time.

A10: Thanks for the comment.

• We admit that permutation invariant is a crucial question in parameter generation, which enables training on checkpoints collected from various sources.
• We introduce permutation state to mitigate this problem, as stated in A1. Results show that without permutation state can cause failure in training on models with different permutations.
• Our method can alleviate the problem, but achieving permutation invariant remains challenging, and we will continue to explore in the future.

Dear Reviewer D95z,

We sincerely thank you for your insightful and constructive feedback. Your comments have highlighted key areas for improvement, and we have developed a comprehensive plan to address them in the revision. Below, we outline the improvements we will implement:

1. Moving Key Results to the Main Text

• We are going to move Table 21 to the main text to better emphasize its importance.
• For Table 12, due to space constraints, we are going to include a condensed version in the main text, summarizing key parameter count information. This will ensure clarity while adhering to space limitations.
• Additionally, we are going to provide detailed explanations linking these results to our core research claims to improve clarity and accessibility.

2. Clarifying Task Definition

• We emphasize that our approach is not a simple label flip. Different groupings of labels require the model to extract distinct features. For example:
  • For four fruits (red apple, green apple, kiwi, cherry), classifying (red apple, green apple) as positive focuses on extracting shape features.
  • Conversely, classifying (red apple, cherry) as positive requires the model to focus on extracting the color red. These represent entirely different feature extraction processes, with no overlap.
• To address your concerns, we are going to:
  • Revise the definition of "unseen tasks" to emphasize parameter distribution differences rather than "label flipping".
  • Include additional experiments demonstrating task variability and its impact on model performance.

3. Presentation Optimization

• We are going to restructure Sections 3 and 4 to enhance the logical flow and make the paper more reader-friendly.
• Key results raised by reviewers will be prioritized and discussed in the main text to better highlight our contributions.

We deeply appreciate your valuable feedback, which has been instrumental in improving our paper. Should you have further suggestions or questions, please do not hesitate to reach out. We are committed to addressing all your concerns and look forward to submitting a stronger revision.

Sincerely,
The Authors