To enhance understanding of our approach, we present a detailed theoretical analysis of the proposed method.

Theoretical analysis of the generalization

Our method learns to model the parameter distribution of specialized pretrained models trained on diverse domains, tasks, and datasets. We formalize this approach through a global function $\mathcal{H}$ that samples from the parameter latent space of a collection of specialized models $\mathbf{F}=\{\mathcal{F}_i\}, \quad, i=1, 2, \ldots, N$.

**Theoretical Framework **

Let $\mathcal{H}: \mathcal{Z} \rightarrow \mathcal{W}$ denote our proposed method, which maps from a shared latent space $\mathcal{Z}$ to the weight spaces $\mathcal{W}$ of the specialized models. Through the diffusion model's smooth sampling process, we guarantee that for any specialized model $\mathcal{F}_i$ with parameters $w_i$, there exists a latent vector $z_i$ such that:

$|\mathcal{H}(z_i) - w_i| \leq \epsilon$

This theoretical guarantee ensures that $\mathcal{H}$ can approximate any specialized model parameters within an error bound $\epsilon$. Furthermore, the continuous nature of the diffusion process enables sampling of intermediate points between specialized models, effectively creating a smooth manifold of task-specific models.

Performance Guarantees

Upon convergence, the global function $\mathcal{H}$ can generate new parameters that match or exceed the performance of the best individual pretrained model. Formally:

$\forall \mathcal{F}_i\in \mathbf{F}, \exists z\in \mathcal{Z} \text{ s.t. } \theta =\mathcal{H}(z) \text{ and } \mathcal{L}(\mathcal{F}_i^{\theta}, \mathcal{D}_i)\leq \mathcal{L}(\mathcal{F}_i, \mathcal{D}_i)$

where $\mathcal{F}_i^{\theta}$ represents the target model initialized with the generated parameters $\theta$, and $\mathcal{L}$ denotes the loss function measuring performance.

** Generalization Bounds**

The generalization capabilities of our method are theoretically bounded by:

$\Delta_G(\mathcal{H}) \leq \min_{i} \Delta_G(\mathcal{F}_i) - \gamma$

where:

• $\Delta_G(\mathcal{H})$ represents the generalization gap of our method
• $\Delta_G(\mathcal{F}_i)$ represents the generalization gap of each specialized model
• $\gamma > 0$ depends on $\mathcal{H}$'s ability to produce task-specific parameters through smooth interpolation

As the number of specialized models increases, $\gamma$ also increases, leading to a decrease in $\mathcal{H}$'s generalization gap. We empirically validated this claim through experiments where we gradually increased the number of pretrained parameters from 5,000 to 20,000, as shown in Figure 5-a.

Conclusion

Our method provides theoretical guarantees for better generalization compared to individual pretrained models, making it particularly well-suited for transfer learning applications. Further formal analysis of these generalization guarantees will be provided in our complete work.

• Computational Costs:: We thank the reviewer for raising these important points about computational resources. We provide a detailed analysis of computational requirements across different phases: Excluding results in Table 6 and 7, all the results have been obtained by training our method on a single Nvidia Titan V100 24GB.

1. Training Phase: While our approach requires storing pretrained weights from multiple datasets, the storage requirements is mostly due to the training data.

   • The primary storage burden comes from maintaining pretrained parameters across diverse datasets, which must be generated through individual model training sessions.
   • The total computational cost scales with (1) the complexity of the base model and (2) the number of datasets used for training.
   • In our experiments, the VAE component requires the most training time, while the diffusion model converges relatively quickly (approximately 1.5 hours for the full model).

2. Post-training Phase: After training our method there no need to keep the pretraind zoo and image datasets.

3. Inference Phase: Except for the LLMs model we text all our models on a single nvidia titan v100 -24GB.

• training time and resource consumption compared to existing methods: : Existing methods [ 1, 2]are not computationally efficient compared to our work because of the following reasons: | Aspect | Our Method | Existing Methods[1, 2] | |---------|------------|-----------------| | Base Requirements | Pretrained models | Pretrained models | | Dataset Handling | Single model for multiple datasets | Separate model per dataset | | Architecture Support | Works with multiple architectures | Limited to single architecture | | Combined Flexibility | Handles multiple architectures and datasets together | Cannot combine architectures and datasets | | Training Time | Faster convergence (≈50% less time) (Table 10) | Longer training despite fewer parameters |

The full parameters count of our model for the experiment\al results in Table 10 is as follows:

The baseline we used required extensive pretrained data but was restricted to a single dataset and a single pretrained architecture. Conducting the experiment in Table 3 with any of the baselines or existing methods would necessitate generating 30 separate parameter models. In contrast, our approach achieves this with a single unified model, showcasing its efficiency and scalability.

• Reliance on Dataset-Specific Encoding:: We acknowledge that D2NWG relies on encoders such as the Set Transformer and CLIP-based encoders to effectively capture dataset characteristics. However, the generalization capability of our approach is primarily influenced by the quality and diversity of the pretrained model zoo used during training. We have demonstrated this both empirically and in our previous discussions, highlighting its importance in achieving robust generalization.

Questions:

• how does the storage and processing overhead of D2NWG compare to traditional pretraining and fine-tuning methods, particularly when scaling to very large models such as LLMs?

Comparison with Traditional Methods
Traditional pretraining and fine-tuning methods involve significant storage overhead due to the need to save the full model or multiple sets of fine-tuned weights for each task. In contrast:

• Storage Efficiency: D2NWG consolidates task-specific parameters into a latent distribution, drastically reducing the need to store multiple pretrained models or full fine-tuned weights. This becomes increasingly advantageous with larger numbers of tasks or specialized models.

• Processing Overhead: By sampling directly from the learned distribution, D2NWG minimizes the need to store by allowing parameter generation on-the-fly. This contrasts with traditional fine-tuning, where the full model or LoRA parameters for each task must be stored.

Scalability to Large LLMs
For large-scale LLMs, where both memory and computation demands are substantial, D2NWG opens the door to optimal parameter latent space Exploration. Instead of storing full weights for every potential specialization, it enables task-specific weights generation and population(particle swarm, evolutionary search) based search by generating the initial population in latent space or weight space. This not only reduces memory requirements but also offers a scalable pathway for adapting large models to diverse tasks without the prohibitive overhead of traditional methods.

Preventing Overfitting and enabling diversity: To mitigate overfitting, we incorporate a beta-VAE in all our experiments. During the training process of the diffusion model, we utilize the VAE in sampling mode. For image datasets, we ensure diversity in the training data by sampling a new conditioned image at every batch. This encourages the model to learn a broader representation of the data distribution. Additionally, we perform fine-tuning even on in-distribution tasks to ensure the sampled weights can be effectively adapted. Furthermore, we employ transfer learning to enable the model to generalize and move efficiently between tasks. For evaluating the weights of LLMs, we assessed the best-performing model by submitting it to the Open-LM leaderboard on Hugging Face. The results of this evaluation are detailed in Table 7. Additionally, we generated text samples to assess the correctness and effectiveness of the proposed method, with the outcomes presented in Table 12.

Additional results on performance improvement of LLMs To further validate our approach, we applied our method on LLaMA-3.2-1B and gpt2-smal and validate them on the OpenLM leaderboard. Our method achieved the top performance among all LLaMA-3.2-1B-Instruct models, outperforming fine-tuned models based on the same architecture. For LLaMA-3.2-1B, we used spectral analysis to select the top 25% of transformer layers. In contrast, for GPT-2-small, we learn the distribution of the entire model, including the embedding layer, and for both we follow the procedures outlined in Tables 6 and 7.

The results, obtained via Huggingface, demonstrate that our method improves specialization in tasks like math and IFEVAL by up to 4%, while maintaining generalization, unlike fine-tuned models, which degrade performance on leaderboard tasks. This showcases our method's ability to enhance task-specific capabilities without sacrificing overall model performance.

Table: Model components and their configuration modes for llma3.2.1B.

settings summary This Table Summarizes the experimental settings for the presented figures and tables. "Min #cls" and "Max #cls" represent the minimum and maximum number of classes, respectively. "# Test Datasets" and "# Train Datasets" indicate the number of test and training datasets, respectively. "#Params" refers to the number of parameters, and "Conditioning" specifies the type of conditioning applied. For example, Tab 3 we learn the distribution of the model pretrained on 30 different datasets with a single D2NWG.

We thank you for your feedback and hope we have provided a comprehensive response to all of your concerns.

multiple architectures multiple datasets parameters generation.

This additional experiment aims to highlight the effectiveness and potential of our method compared to existing approaches

We extended our experiments to analyze 38 pretrained models across 19 architectures, each trained on CIFAR-10 and CIFAR-100. For feature extraction, we used LLaMA-3.2-11-Vision-Instruct to encode dataset features and class descriptions, while LLAMA-3-1.8B-Instruct extracted architecture configuration embeddings. These embeddings conditioned our parameter generation process. The results of learning the joint distribution across these architectures and models are shown below. The choice of vllm was recommended by one of the reviewers.

As demonstrated in the Table above , our approach achieves comparable performance to existing methods while requiring a single generative model instead of 38 specific ones. This experiment demonstrates our claim regarding the superiority of our method against the existing work. The pretrained model architecture and its parameter count are publicly available in a non-affiliated GitHub repository. https://github.com/chenyaofo/pytorch-cifar-models

Sampling Enhances Transfer Learning

We evaluated sampling from a distribution of pretrained models against single-model transfer learning using ResNet56 and our model trained on 19 diverse architectures weights pretrained on (CIFAR-10/100). Three setups were tested: (1) direct evaluation of pretrained models, (2) sampling conditioned on training sets (e.g., STL-10, CIFAR-10), and (3) sampling conditioned on test sets to address data leakage concerns.

Results show our approach consistently outperforms single-model transfer learning, with similar performance between training- and test-conditioned sampling from the same distribution. This highlights the practicality of leveraging diverse pretrained models for robust generalization.

Transfer learning: Our method vs pretrained models

The empirical evidence supports our key claims while highlighting the practical applicability and reliability of our proposed method. which is not possible with existing work.

We thank you for your constructive feedback, which has helped strengthen our work through additional experiments and analyses. We welcome any further questions or concerns you may have.

Thank you for raising these important methodological points. We have conducted additional experiments to address your concerns

Comparing to existing Approaches

Assuming that all approaches—[1], [2], and Our Approach—can generate the same parameters, we summarize the main differences in the following table:

• The 200M parameter claim is supported by our experimental results presented in Tables 6 and 7. In these experiments, we successfully learned the distribution of up to 8 transformer blocks simultaneously, with each transformer containing over 200M parameters. To further validate our findings, we conducted additional experiments on larger language models, including the complete GPT-2 Small architecture

• Regarding meta-learning claim Thank you for the question. When discussing few-shot methods, it's important to note that they still require fine-tuning for in-distribution tasks. This is because these methods are optimized through gradient descent to perform well across several tasks simultaneously, which typically leads to a compromise where the model performs adequately but not exceptionally on any single task. In contrast, our method can access specialized pre-trained weights at any time through sampling, allowing for better task-specific performance

• overcoming fine-tuning: Yes, the proposed method eliminates the need for a fine-tuning stage during sampling for in-distribution evaluation. Our approach maintains high performance on source tasks w.hile enabling effective transfer to new domains, addressing real-world requirements where both capabilities are essential

• Architecture Conditioning (Figure 1): We have included experimental results for architecture conditioning in Table 9. Regarding cross-architecture generalization, we acknowledge that our reported experiment uses class-conditioned generation with a fixed set of known architectures. However, this experiment could be extended to handle unseen architectures by embedding their configurations using instruction-tuned Large Language Models (LLMs) to generate precomputed class embeddings from it configuration file. We will provide additional experimental results on this experiment.

Careful Concatenation: means we sample the larger possible model parameters multiple times and then concatenate them layer-wise to match or exceed the size of the larger target model. The cross-architecture evaluation was inspired by [3]; however, we did not have access to the complete training zoo, which limited our ability to perform a fully fair comparison.

• Regarding Model-wise and Layer-wise Vectorization: Model-wise vectorization experiments concerns all the classifier head adaptation experiments and performance evaluations compared to [3]. Layer-wise vectorization experiments primarily focus on cross-architecture and Large Language Model (LLM) experiments, excluding LoRA base experiment on Glue datasets.

Addressing Concerns Related to dataset Encoding We use CLIP to extract 512-dimensional features from images, grouped by class. Our Set Transformer processes these features (e.g., shape (1,10,5,512) for CIFAR-10 with 5 images per class) to create fixed-size outputs while handling variable inputs. The transformer is trained once with frozen weights and linear probing, enabling efficient adaptation across different datasets. This approach ensures consistent class-aware representations regardless of dataset size while maintaining permutation invariance.

-Regarding dataset encoding :Thank you for the question. Initially, we attempted to train the diffusion model without latent space alignment and relied solely on dataset conditioning, similar to text-to-image generation. However, in our experiments, this approach led to convergence issues, particularly when jointly optimizing the diffusion model with the set transformer. Unlike text-to-image models, the dynamics of weight generation require a closer alignment between the latent space of the weights and the dataset representation.
This choice was ultimately adopted after iterative experimentation and validation, as it significantly improved training stability and performance. We acknowledge the distinction from text-to-image models and will further clarify this rationale in the paper.

• We perform inference without the VAE encoder. During training, the dataset encoder (set-transformer) outputs were used alongside the weight latent representations to guide the diffusion process. At inference, random samples from the target dataset are provided as condition similar to prompt usage in LLMs. This allows the model to map the noise sampled to the appropriate output without requiring the vae encoder.
  Thank you for highlighting this. To clarify, we did not rank layers in the LoRA experiments. The ranking mentioned in Section 3.5 refers specifically to spectrum analysis conducted on LLAMA and other LLM experiments, not in LoRA experiment.

• what is 5 ways-1shot: In meta-learning, "5-way 1-shot" refers to a setting where the dataset contains 5 classes ("5-way") and only one image or sample per class ("1-shot"). For example, in a "5-way 5-shot" setting, there would be 5 classes with 5 images per class, resulting in a total of 25 images. This setup is commonly used to evaluate few-shot learning models by testing their ability to generalize with minimal data. Table 2: D2NWG_CLIP This table presents the results of linear probing the CLIP model's image encoder on 20,000 subsets of ImageNet. Subsequently, it evaluates the sampling performance on CIFAR-10 and STL-10.

What Do You Mean by "Unseen"? In our experiments, "unseen" refers to datasets from tasks that were not used to collect the pretrained weights for training our generative model. This means the generative model was trained without access to data from these specific tasks, ensuring that the weights remain unbiased and generalizable to new, previously unencountered datasets..Unseen dataset/tasks are ew, previously unencountered datasets.

• Sect. 4.2, Yes, the main reason is the presence of very similar images or classes across different categories within the training data. This experiment also demonstrates our method's ability to generate weights based on the similarity between datasets. Note that we obtain this dataset from a public github [4] so we were not aware of such scenario.

• **In Figure 2, why dessert food is so good **: Upon inspecting the dataset, we found that similar images were present in other datasets but labeled under different class names and numbers. Since the model is conditioned on image features for the classification task, it tends to sample based on image similarity.

• Experiments in Section 4.2,: We generated 20,000 subsets of ImageNet and fine-tuned the classifier heads of these models. Subsequently,we trained our method using these pretrained weights. We evaluated our parameter generator on novel datasets such as CIFAR-10 and STL-10 with no fine-tuning.

-Lora Experi,emts: The experiment on LoRA focuses on an in-distribution setting, aiming to demonstrate that the distribution of combined LoRA adaptations can be effectively learned. This is achieved by conditioning on task descriptions, enabling task-specific sampling and model retrieval, as described in [2].

Thank you very much for your insightful comment we are actively working on your comment to improve the overall paper.

[2]Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

We establish the theoretical foundation supporting the generalization capabilities of our method.

Theoretical analysis of the generalization

Our method learns to model the parameter distribution of specialized pretrained models trained on diverse domains, tasks, and datasets. We formalize this approach through a global function $\mathcal{H}$ that samples from the parameter latent space of a collection of specialized models $\mathbf{F}=\{\mathcal{F}_i\}, \quad, i=1, 2, \ldots, N$.

**Theoretical Framework **

Let $\mathcal{H}: \mathcal{Z} \rightarrow \mathcal{W}$ denote our proposed method, which maps from a shared latent space $\mathcal{Z}$ to the weight spaces $\mathcal{W}$ of the specialized models. Through the diffusion model's smooth sampling process, we guarantee that for any specialized model $\mathcal{F}_i$ with parameters $w_i$, there exists a latent vector $z_i$ such that:

$|\mathcal{H}(z_i) - w_i| \leq \epsilon$

This theoretical guarantee ensures that $\mathcal{H}$ can approximate any specialized model parameters within an error bound $\epsilon$. Furthermore, the continuous nature of the diffusion process enables sampling of intermediate points between specialized models, effectively creating a smooth manifold of task-specific models.

Performance Guarantees

Upon convergence, the global function $\mathcal{H}$ can generate new parameters that match or exceed the performance of the best individual pretrained model. Formally:

$\forall \mathcal{F}_i\in \mathbf{F}, \exists z\in \mathcal{Z} \text{ s.t. } \theta =\mathcal{H}(z) \text{ and } \mathcal{L}(\mathcal{F}_i^{\theta}, \mathcal{D}_i)\leq \mathcal{L}(\mathcal{F}_i, \mathcal{D}_i)$

where $\mathcal{F}_i^{\theta}$ represents the target model initialized with the generated parameters $\theta$, and $\mathcal{L}$ denotes the loss function measuring performance.

** Generalization Bounds**

The generalization capabilities of our method are theoretically bounded by:

$\Delta_G(\mathcal{H}) \leq \min_{i} \Delta_G(\mathcal{F}_i) - \gamma$

where:

• $\Delta_G(\mathcal{H})$ represents the generalization gap of our method
• $\Delta_G(\mathcal{F}_i)$ represents the generalization gap of each specialized model
• $\gamma > 0$ depends on $\mathcal{H}$'s ability to produce task-specific parameters through smooth interpolation

As the number of specialized models increases, $\gamma$ also increases, leading to a decrease in $\mathcal{H}$'s generalization gap. We empirically validated this claim through experiments where we gradually increased the number of pretrained parameters from 5,000 to 20,000, as shown in Figure 5-a.

Conclusion

Our method provides theoretical guarantees for better generalization compared to individual pretrained models, making it particularly well-suited for transfer learning applications. Further formal analysis of these generalization guarantees will be provided in our complete work.

We sincerely appreciate Reviewer 4nQa's thoughtful comments regarding the paper's organization. While this alternative organization could offer a different perspective, our current experimental evaluation is organized as follows:

I. Vision Tasks

• Sampling Without Training
  1. In-Distribution Evaluation
  2. Out-of-Distribution Evaluation
• Sampling and Fine-Tuning
  1. In-Distribution Evaluation
  2. Out-of-Distribution Evaluation

II. Language Tasks

• LoRA Weights Generation
  1. In-Distribution Data
• Large Language Models optimal Parameters Space Exploration
  1. Sampling from optimal parameters latent space for known task(In-distribution)
  2. Evaluation on unseen benchmark tasks.

III. Ablation Study(Appendix)

While reorganizing the content around distribution types is an interesting suggestion, we believe that such restructuring would not alter our findings or lead to any scientific insight, as it does not require additional experimental settings. We will reorganized the paper accordingly and hope that Reviewer 4nQa will consider these points in their final decision regarding our work.

We sincerely appreciate the reviewer's detailed and constructive feedback, which will help strengthen our paper.

Weaknesses:

1. High Workload and Tuning Challenges Due to Additional VAE Training:
   While VAE training requires careful tuning, its one-time independent training enables faster and more efficient diffusion model training afterward. Our experiments in Appendix Figure 5b show that removing the VAE and training directly on raw weights leads to poor convergence and lower performance compared to VAE-based approaches. The VAE's effectiveness in matching pretrained model performance justified its inclusion, though we remain open to exploring alternative encoding methods in future work.

2. Ambiguity in Dataset Encoding Choices in Section 3.3:
   Thank you for the feedback. We acknowledge that Section 3.3 could be clearer. Our work uses two main types of encoding:

   • For image datasets, we employ a Set Transformer to capture inter- and intra-class relationships. Due to convergence challenges in joint optimization, we adopted a two-stage process: first training and freezing the VAE encoder, then aligning the Set Transformer outputs to the VAE’s latent space using contrastive loss. During diffusion, only a single dense linear layer adapting the Set Transformer and the diffusion model are optimized for simplicity.
   • For language tasks, we use LLAMA-3.1-8B-Instruct to extract task description features offline, which are then processed by a linear layer during diffusion optimization.

   We will revise Section 3.3 to clearly distinguish these two approaches and their implementation details, ensuring clarity and adaptability for practitioners.

3. Minor Typos and Formatting Issues:
   We appreciate the reviewer for pointing out these issues. We are actively working to improve the writing and will conduct a thorough proofreading to address typos, formatting inconsistencies, and other errors to enhance the clarity and professionalism of the final version.

Questions: We apologize to the reviewer for not explicitly addressing the topic of bias and fairness, as it was not a focus of our research and has not been a well known concept in the context of neural network parameter generation. Nonetheless, we appreciate the concern and will provide a detailed response along with additional experiments to address this important topic. Quantifying and Mitigating Bias in Public Checkpoint Weight Distributions

While prior work on neural network weight generation has not extensively addressed fairness concerns, this is an important consideration for developing robust and equitable AI systems. We propose several approaches for both quantifying and mitigating potential biases:

-Possible Bias Quantification Methods

• Evaluate models initialized with generated weights using established fairness metrics (e.g., demographic parity, equal opportunity, equalized odds)
• Compare performance disparities across different demographic subgroups on standardized benchmark datasets
• Analyze the statistical properties of generated weight distributions in latent space to detect systematic deviations or skews
• Conduct ablation studies to isolate the impact of different architectural components on fairness metrics

Possible Bias Mitigation Strategies

• Distribution Alignment Analysis

• Systematically evaluate whether the diffusion-based weight generation process amplifies or diminishes biases present in the pretrained model distribution (results presented in Table 12)

• Diverse Sampling

• Generate multiple weight configurations and select those maximizing both task performance and fairness metrics

• On the need to address Bias in the generated weight*: We acknowledge this as an important avenue for future work and will consider developing more targeted mechanisms, such as fairness-aware sampling strategies or bias-adjusted validation metrics, in subsequent research.

• Evolutionary Optimization

• Implement a genetic algorithm approach where:
  • The initial population consists of weights sampled using our diffusion method
  • Fitness function incorporates both performance and fairness objectives
  • Selection and crossover operations guide the population toward more equitable solutions

• Constraint Integration

• Incorporate fairness constraints directly into the weight generation process through modified loss functions or regularization terms. such as a conceptual loss on the VAE that ensures that the reconstructed weights aligned with this goal in such a way the diffusion well be forced to learn the latent variable with better bias mitigation.

• On the criteria for Dataset Selection: Our dataset selection was guided by alignment with baselines, practical relevance, diversity, and parameter space exploration:

  1. Baseline Alignment: MiniImageNet and tieredImageNet were chosen to compare with MetaDiff, a diffusion-based meta-learning baseline. Similarly, Tiny-Model Zoo datasets were included for fair evaluation against the only published methods in this field.
  2. Practical Relevance: Classifier head tuning used standard benchmarks to reflect real-world scenarios where pretrained backbones are fine-tuned. In Table 5, we focused on LoRA due to its growing use in task-specific adaptations.
  3. Diversity: MetaAlbum and Figure 2 experiments evaluated generalization across diverse and unseen datasets, surpassing baseline diversity.
  4. Parameter Space Exploration: Table 6 investigated fine-tuned LLMs, leveraging our method’s ability to generate task-specific parameters without random noise, ensuring controlled perturbations in the parameter space.
  5. External Validation: The best models were submitted to OpenLM leaderboard, demonstrating improved performance in an uncontrolled evaluation.

• Evaluating the Impact of Generating All Parameters Versus Only the Classification Head:
  Table 1 benchmarks our method against MetaDiff, a baseline that focuses exclusively on classification heads in its experiments. To ensure a fair comparison, we pre-trained 50,000 models specifically for this evaluation.

  We have already demonstrated that full model generation generalizes better than generating only the classification head. These results are presented in Appendix Table 16, we provided them in the below table. The model used is a mobilenet v3 from OFA models subnets.

Below, we provide experimental results( Table-16) demonstrating that generating full model parameters significantly improves generalization over the clasifier head. The main difference is that it is much easier to create large pretrained dataset with linear probing than full models.

Cross Datasets Transfer Learning

Additional results on performance improvement of LLMs To further validate our approach, we applied our method on LLaMA-3.2-1B and gpt2-smal and validate them on the OpenLM leaderboard. Our method achieved the top performance among all LLaMA-3.2-1B-Instruct models, outperforming fine-tuned models based on the same architecture. For LLaMA-3.2-1B, we used spectral analysis to select the top 25% of transformer layers. In contrast, for GPT-2-small, we learn the distribution of the entire model, including the embedding layer, and for both we follow the procedures outlined in Tables 6 and 7.

The results, obtained via Huggingface, demonstrate that our method improves specialization in tasks like math and IFEVAL by up to 4%, while maintaining generalization, unlike fine-tuned models, which degrade performance on leaderboard tasks. This showcases our method's ability to enhance task-specific capabilities without sacrificing overall model performance.

Table: Model components and their configuration modes for llma3.2.1B.

settings summary This Table Summarizes the experimental settings for the presented figures and tables. "Min #cls" and "Max #cls" represent the minimum and maximum number of classes, respectively. "# Test Datasets" and "# Train Datasets" indicate the number of test and training datasets, respectively. "#Params" refers to the number of parameters, and "Conditioning" specifies the type of conditioning applied. For example, Tab 3 we learn the distribution of the model pretrained on 30 different datasets with a single D2NWG.

We thank you for your feedback and hope we have provided a comprehensive response to all of your concerns.

To enhance understanding of our approach, we present a detailed theoretical analysis of the proposed method.

Theoretical analysis of the generalization

Our method learns to model the parameter distribution of specialized pretrained models trained on diverse domains, tasks, and datasets. We formalize this approach through a global function $\mathcal{H}$ that samples from the parameter latent space of a collection of specialized models $\mathbf{F}=\{\mathcal{F}_i\}, \quad, i=1, 2, \ldots, N$.

**Theoretical Framework **

Let $\mathcal{H}: \mathcal{Z} \rightarrow \mathcal{W}$ denote our proposed method, which maps from a shared latent space $\mathcal{Z}$ to the weight spaces $\mathcal{W}$ of the specialized models. Through the diffusion model's smooth sampling process, we guarantee that for any specialized model $\mathcal{F}_i$ with parameters $w_i$, there exists a latent vector $z_i$ such that:

$|\mathcal{H}(z_i) - w_i| \leq \epsilon$

This theoretical guarantee ensures that $\mathcal{H}$ can approximate any specialized model parameters within an error bound $\epsilon$. Furthermore, the continuous nature of the diffusion process enables sampling of intermediate points between specialized models, effectively creating a smooth manifold of task-specific models.

Performance Guarantees

Upon convergence, the global function $\mathcal{H}$ can generate new parameters that match or exceed the performance of the best individual pretrained model. Formally:

$\forall \mathcal{F}_i\in \mathbf{F}, \exists z\in \mathcal{Z} \text{ s.t. } \theta =\mathcal{H}(z) \text{ and } \mathcal{L}(\mathcal{F}_i^{\theta}, \mathcal{D}_i)\leq \mathcal{L}(\mathcal{F}_i, \mathcal{D}_i)$

where $\mathcal{F}_i^{\theta}$ represents the target model initialized with the generated parameters $\theta$, and $\mathcal{L}$ denotes the loss function measuring performance.

** Generalization Bounds**

The generalization capabilities of our method are theoretically bounded by:

$\Delta_G(\mathcal{H}) \leq \min_{i} \Delta_G(\mathcal{F}_i) - \gamma$

where:

• $\Delta_G(\mathcal{H})$ represents the generalization gap of our method
• $\Delta_G(\mathcal{F}_i)$ represents the generalization gap of each specialized model
• $\gamma > 0$ depends on $\mathcal{H}$'s ability to produce task-specific parameters through smooth interpolation

As the number of specialized models increases, $\gamma$ also increases, leading to a decrease in $\mathcal{H}$'s generalization gap. We empirically validated this claim through experiments where we gradually increased the number of pretrained parameters from 5,000 to 20,000, as shown in Table 5-a.

Conclusion

Our method provides theoretical guarantees for better generalization compared to individual pretrained models, making it particularly well-suited for transfer learning applications. Further formal analysis of these generalization guarantees will be provided in our complete work.

1. We believe that exploring flow-based models and recent advancements in diffusion models could potentially eliminate the need for using a VAE. We leave this avenue for future investigation. Regarding bias we proposed potential solutions because we were not familiar with the topic.
2. We once again thank Reviewer oEWb for their valuable feedback. We acknowledge that the primary scope of our proposed method does not currently focus on bias and fairness, as these are outside the immediate scope of our research. However, your concerns have highlighted intriguing areas for exploration that weight generation methods have yet to address.

To investigate these concerns further, we conducted an additional study to evaluate the extent to which the proposed method could influence a model calibrated for bias mitigation. Specifically, we fine-tuned Google’s flan_t5_small model on the Stereotype dataset. This dataset was chosen due to its simplicity and well-established evaluation procedures.

In our experiment, we learned the distribution of the weights from the last epoch(75) of the fine-tuned model (excluding the embedding layer), sampled weights from this distribution, and compared the results to the original fine-tuned model, referred to as the base model. The following metrics were monitored:

• LMS (Language Modeling Score): Evaluates overall language modeling performance, indirectly reflecting disparities in predictions across different groups.
• SS (Stereotype Score): Measures the degree to which a model reinforces societal stereotypes, focusing on minimizing biased associations.
• ICAT (Ideal Context Association Test): Assesses compounded biases at the intersection of multiple attributes, emphasizing fairness in complex real-world scenarios.

https://github.com/moinnadeem/StereoSet

As shown in the table below, the proposed method—leveraging sampling improves the overall performance across all metrics. Additionally, we can sample multiple weights, each optimized for different tasks. These task-specific models can be integrated into algorithms such as evolutionary search to further fine-tune performance on an evaluation set, as described in our previous response.

StereoSet Benchmark Scores for Base Model and Sampled Model(lower is better)

We will also include comments in the ethics and limitations section to emphasize that the proposed method can either amplify or mitigate stereotypes, depending on its application.

We hope these results provide valuable insight into how our method can enhance performance, particularly in the context of fairness. In our future work, we will rigorously evaluate these topics.

1. Regarding the concept of zero-shot:: We apologize for the misunderstanding. In our work, "zero-shot" refers to not performing any optimization steps on the target dataset, consistent with [4], against which we also evaluate in Table 4. We confirm there is no test data leakage in our evaluation setting. We respectfully argue that dataset-conditioned parameter generation remains relevant, as it enables task-specific priors without optimization. We will clarify these points in the paper to avoid confusion
2. Adressing comparison against pretrained models.: We appreciate the reviewer’s concern regarding cost-effectiveness and similarity to pre-trained models. In Table 3 we demonstrate that a single D2NWG model can encode 30 datasets and generate dataset-specific parameters that achieve performance on par with or surpassing conventional pre-trained models. This significantly reduces the computational cost and complexity of maintaining separate pre-trained models for each dataset or task as the number of pre trained models and datasets increases. Our method can automatically select the right pretrained model based on the dataset's similarity to the pretrained dataset used to train our methods. It should be noted that we only conditioned on datasets or task when we have the models pretrained on at least two tasks or datasets . it can effectively leverage multiple pretrained models from diverse domains without performance degradation. Comparison with Pre-Trained Models:
   To address this concern, we conducted additional experiments using models pretrained on the MetaAlbum dataset (Table 3) and applied them to the unseen datasets shown in the Table below including CIFAR-10 and other datasets. The results show that with a single epoch, D2NWG outperforms both pretrained models and random initialization. This highlights our method’s effectiveness and its capability to generalize better than traditional pretrained models for transfer learning.

compares the accuracy of three methods—RandInit, D2NWG, and ImageNet Pretrained—oone epcoh fine-tuned on CIFAR-10, STL-10, AIRCRAFT-100, AIRCRAFT-30, and PETS.

Observations:

1. D2NWG consistently outperforms RandInit across all datasets and shows competitive or superior performance compared to the Pretrained model, particularly in STL-10 and PETS.

2. In CIFAR-10, D2NWG achieves almost the same performance as the Pretrained model.

3. On the AIRCRAFT datasets, D2NWG shows slight improvement over both RandInit and Pretrained, though overall performance remains low across all methods.

4. On the no evidence that D2NWG can handle diverse tasks and different model architectures simultaneously.: We believe that is a misunderstanding since those results are well reported in the appendix D-1, D-2, We provided the cross-architecture adaptation results in Figure 4 and a mixture of architectures and dataset experiment results are reported in Tables 9 and 10. we clearly show the evidence.

5. Comparing to a Well-Developed Optimizer:
   We respectfully disagree. A traditional optimizer, no matter how well-developed, requires a full training process from scratch each time, even when optimizing the same model on the same dataset. In contrast, our method is fundamentally different—it generates parameters directly from pretrained distributions, bypassing the need for iterative optimization. This makes D2NWG a distinct and complementary approach rather than a replacement for traditional optimizers.

6. On the Impact of Different Components in the Proposed Method:
   While each component can impact generalization, our design choices are empirically justified:

   • (1) Figure 6b demonstrates that direct diffusion on raw weights performs poorly compared to our latent space approach.
   • (2) Our ablation studies show that separate training of components improves stability and convergence compared to joint optimization.
   • (3) The Set Transformer's permutation invariance ensures consistent dataset encoding.

[1]Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning [2] ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation [3] DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion [4] Towards Scalable and Versatile Weight Space Learning

We are grateful for the reviewer's positive assessment and their helpful suggestions for improving the clarity of our work. Below, we address each point of yours concerns:

• Addressing Concerns About Practical Significance: : While we partially agree with the reviewers regarding the computational resources required for large-scale implementation of our method, we believe that these resources do not represent a fundamental limitation. This is akin to how the computational demands of large language models (LLMs) have not hindered their development or deployment. In fact, we argue that the practicality of our approach is underscored by its unique applications, which are highlighted below:

1. Efficient Model Retrieval and Generation: Our method enables generating models on demand without requiring storage of large model collections, as demonstrated in Table 3.
2. Improved Transfer Learning: We provide better initialization strategies, which enhance transfer learning across various tasks (Table 4).
3. Specialized LoRA Parameter Generation: Our approach facilitates the generation of specialized parameters for large architectures, including LLMs, improving scalability and adaptability (Table 5).
4. LLM Enhancement without Training Data Access: By leveraging sampling and latent space exploration, we demonstrate performance improvements in LLMs even without access to their original training data (Table 7).

Furthermore, recent advances in retrieval-augmented machine learning [1, 2, 3] strongly support the practical relevance of our work, especially in scenarios requiring efficient model adaptation and deployment. These advancements further validate the real-world impact of our method.

2. Addressing claim related to generating complete parameters with robust performance: We appreciate the reviewer’s comments on Tables 1 and 2. However, it seems there may have been a misunderstanding regarding the purpose of these experiments. We would like to provide clarification as follows:

• classifier head: This evaluation approach is of practical importance and widely used in transfer learning scenarios. Therefore, we employed linear probing as a straightforward method to assess the performance of our approach.

• Multi-task learning evaluation: Our method is designed to learn from a distribution of pretrained models, each trained on diverse datasets. Consequently, it is natural to compare our approach against established multi-task learning methods, such as meta-learning. The purpose of Table 1 is to demonstrate the effectiveness of our method in comparison to MetaDiff, a diffusion-based meta-learning approach that focuses exclusively on the classifier head.

Thus, evaluating performance at the classifier head level does not reflect an inability to learn the full set of parameters but rather aligns with the comparative framework and objectives of the experiments presented. The choice to conduct these experiments using the classifier head stems from the practical advantage of quickly generating a large set of specialized pretrained models This choice is also motivated by practical considerations related to data collection rather than being a technical limitation of the proposed method.

• We present the full-model experimental results using more complex and diverse datasets, including some of those featured in Table 2, in Tables 3, 14, 15, and 16. These results demonstrate that the sampled weights perform on par with or exceed the performance of the pretrained models in in-distribution sampling.

Below, we provide experimental results( Table-16) demonstrating that generating full model parameters significantly improves generalization.

Cross Datasets Transfer Learning

Additional results on performance improvement of LLMs To further validate our approach, we applied our method on LLaMA-3.2-1B and gpt2-smal and validate them on the OpenLM leaderboard. Our method achieved the top performance among all LLaMA-3.2-1B-Instruct models, outperforming fine-tuned models based on the same architecture. For LLaMA-3.2-1B, we used spectral analysis to select the top 25% of transformer layers. In contrast, for GPT-2-small, we learn the distribution of the entire model, including the embedding layer, and for both we follow the procedures outlined in Tables 6 and 7.

The results, obtained via Huggingface, demonstrate that our method improves specialization in tasks like math and IFEVAL by up to 4%, while maintaining generalization, unlike fine-tuned models, which degrade performance on leaderboard tasks. This showcases our method's ability to enhance task-specific capabilities without sacrificing overall model performance.

Table: Model components and their configuration modes for llma3.2.1B.

settings summary This Table Summarizes the experimental settings for the presented figures and tables. "Min #cls" and "Max #cls" represent the minimum and maximum number of classes, respectively. "# Test Datasets" and "# Train Datasets" indicate the number of test and training datasets, respectively. "#Params" refers to the number of parameters, and "Conditioning" specifies the type of conditioning applied. For example, Tab 3 we learn the distribution of the model pretrained on 30 different datasets with a single D2NWG.

We thank you for your feedback and hope we have provided a comprehensive response to all of your concerns.

For more clarity we provide a theoretical analysis on the generalization of our approach

Theoretical analysis of the generalization

Our method learns to model the parameter distribution of specialized pretrained models trained on diverse domains, tasks, and datasets. We formalize this approach through a global function $\mathcal{H}$ that samples from the parameter latent space of a collection of specialized models $\mathbf{F}=\{\mathcal{F}_i\}, \quad, i=1, 2, \ldots, N$.

**Theoretical Framework **

Let $\mathcal{H}: \mathcal{Z} \rightarrow \mathcal{W}$ denote our proposed method, which maps from a shared latent space $\mathcal{Z}$ to the weight spaces $\mathcal{W}$ of the specialized models. Through the diffusion model's smooth sampling process, we guarantee that for any specialized model $\mathcal{F}_i$ with parameters $w_i$, there exists a latent vector $z_i$ such that:

$|\mathcal{H}(z_i) - w_i| \leq \epsilon$

This theoretical guarantee ensures that $\mathcal{H}$ can approximate any specialized model parameters within an error bound $\epsilon$. Furthermore, the continuous nature of the diffusion process enables sampling of intermediate points between specialized models, effectively creating a smooth manifold of task-specific models.

Performance Guarantees

Upon convergence, the global function $\mathcal{H}$ can generate new parameters that match or exceed the performance of the best individual pretrained model. Formally:

$\forall \mathcal{F}_i\in \mathbf{F}, \exists z\in \mathcal{Z} \text{ s.t. } \theta =\mathcal{H}(z) \text{ and } \mathcal{L}(\mathcal{F}_i^{\theta}, \mathcal{D}_i)\leq \mathcal{L}(\mathcal{F}_i, \mathcal{D}_i)$

where $\mathcal{F}_i^{\theta}$ represents the target model initialized with the generated parameters $\theta$, and $\mathcal{L}$ denotes the loss function measuring performance.

** Generalization Bounds**

The generalization capabilities of our method are theoretically bounded by:

$\Delta_G(\mathcal{H}) \leq \min_{i} \Delta_G(\mathcal{F}_i) - \gamma$

where:

• $\Delta_G(\mathcal{H})$ represents the generalization gap of our method
• $\Delta_G(\mathcal{F}_i)$ represents the generalization gap of each specialized model
• $\gamma > 0$ depends on $\mathcal{H}$'s ability to produce task-specific parameters through smooth interpolation

As the number of specialized models increases, $\gamma$ also increases, leading to a decrease in $\mathcal{H}$'s generalization gap. We empirically validated this claim through experiments where we gradually increased the number of pretrained parameters from 5,000 to 20,000, as shown in Figure-5-a.

Conclusion

Our method provides theoretical guarantees for better generalization compared to individual pretrained models, making it particularly well-suited for transfer learning applications. Further formal analysis of these generalization guarantees will be provided in our complete work. Our theoretical analysis reinforces the previously discussed advantages of our method over a single model, highlighting its practicality and superiority in diverse scenarios.

multiple architectures multiple datasets parameters generation.

We once again provide a practical example demonstrating how our method enables learning the distribution of compressed, multiple pretrained models and architectures. for better transfer learning.

We extended our experiments to analyze 38 pretrained models across 19 architectures, each trained on CIFAR-10 and CIFAR-100. For feature extraction, we used LLaMA-3.2-11-Vision-Instruct to encode dataset features and class descriptions, while LLAMA-3-1.8B-Instruct extracted architecture configuration embeddings. These embeddings conditioned our parameter generation process. The results of learning the joint distribution across these architectures and models are shown below. The choice of vllm was recommended by one of the reviewers.

As demonstrated in the Table above , our approach achieves comparable performance to existing methods while requiring a single generative model instead of 38 specific ones. This experiment demonstrates our claim regarding the superiority of our method against the existing work. The pretrained model architecture and its parameter count are publicly available in a non-affiliated GitHub repository. https://github.com/chenyaofo/pytorch-cifar-models

We thank you for your constructive feedback, which has helped strengthen our work through additional experiments and analyses. We welcome any further questions or concerns you may have.

Additional Results: Practical Significance of Distribution Sampling vs Single Model Transfer Learning

To validate the practical significance of sampling from a distribution of multiple pretrained models versus using a single pretrained model for transfer learning tasks, we conducted comprehensive experiments using ResNet56. Our model was trained to learn the distribution of 19 diverse architectures pretrained on both CIFAR-10 and CIFAR-100. We explored three experimental scenarios:

1. Baseline Evaluation: We directly used the respective pretrained models and evaluated their performance on target test sets.

2. Training Set-Conditioned Sampling: We performed conditional sampling based on:

   • STL-10 training set for STL-10 evaluation
   • CIFAR-10 training set for CIFAR-10.1 evaluation The conditioning included ResNet56 architecture descriptions without specifying the pretraining dataset.

3. Test Set-Conditioned Sampling: We replicated scenario 2, but conditioned the sampling directly on:

   • STL-10 test set
   • CIFAR-10.1 test set

The test set-conditioned sampling experiment demonstrates that our method's performance is independent of test set access, addressing the concerns about potential data leakage.

As shown in the Table below our proposed method significantly outperforms traditional pretrained model transfer learning. Notably, there was no significant performance difference between sampling conditioned on test or training sets, provided both sets were drawn from the same distribution. These results and those reported in the main papers demonstrate the practical utility of our approach for real-world transfer learning applications where it can effectively handle diverse models and pretrained on diverse datasets while achieving better generalization in line with our theoretical analysis.

Transfer learning: Our method vs pretrained models

The empirical evidence supports our key claims while highlighting the practical applicability and reliability of our proposed method.

Please let us know if you have any further questions or require additional clarification.

Network model used in Figure 4 experiment is defined as follows:

import torch.nn as nn from ofa.model_zoo import ofa_net

class MobileSubnet(nn.Module): def init(self, num_classes, pretrained=True): super(MobileSubnet, self).init()

    # Initialize OFA network with the specified configuration
    ofa_network = ofa_net('ofa_mbv3_d234_e346_k357_w1.0', pretrained=pretrained)
    
    # Set the active subnet with kernel size, expansion ratio, and depth
    ofa_network.set_active_subnet(ks=3, e=3, d=2)
    
    # Extract the active subnet and optionally preserve weights
    subnet = ofa_network.get_active_subnet(preserve_weight=pretrained)
    
    # Replace the classifier to match the number of output classes
    subnet.classifier = nn.Linear(1280, num_classes, bias=False)
    
    # Save the number of classes and the subnet
    self.num_classes = num_classes
    self.net = subnet
    
    # Clean up to save memory
    del ofa_network

def forward(self, inputs):
    # Forward pass through the subnet
    return self.net(inputs)

Structure worklow: In our experiments, we structured the workflow by organizing by set of experiments based on available computational resources and storage capacity. This structured approach allowed us to complete experiments incrementally, enabling flexibility in handling convergence issues—a common challenge in building unified models. Dividing experiments into manageable sets proved to be an effective strategy. Even though we built a separate model for each experiment set, our method significantly improves computational efficiency compared to existing methods, which often rely on maintaining a single model for every model-dataset pair. The primary challenge lies in ensuring convergence, which requires careful consideration and optimization. We believe that organizing settings by clusters of objectives is not a limitation but rather a standard and effective practice.

Results of fine-tuning for 10 epochs: Here are the fine-tuned results for 10 epochs for both the proposed and pretrained models. In our experiments, the images were downsampled to $128 \times 128$, rather than $224 \times 224$ or $229 \times 229$ as reported in the referenced paper. Notably, our initially sampled weights demonstrated improved performance compared to the results reported in the main paper. This was intentional to highlight the flexibility and adaptability of our method.

In this table, we report the best of 10 sampled weights. Our method allows for sampling new initial weights at any time, providing flexibility, whereas the pretrained weights remain fixed and unchangeable. While the pretrained models outperformed our method in one fine-tuning scenario, this result should not be generalized, as our method is specifically designed for small-dataset fine-tuning rather than large-scale fine-tuning tasks.

Overall Trends

. Efficiency: Our method achieves better performance than the pretrained model across all epochs for both datasets. . Consistency: While the pretrained model experiences fluctuations and slower improvements, our method exhibits steady and reliable progress. . Scalability: The advantage of our method is particularly pronounced in scenarios with lower initial accuracy (e.g., Aircraft100), showcasing its potential for challenging tasks or datasets.

[A] Towards Scalable and Versatile Weight Space Learning, ICMl 2024

[1] Do Better ImageNet Models Transfer Better? CVPR, 2019.

We hope that our response has provided clarity and effectively addressed your concerns.

Regarding the evaluation protocol

Before addressing your concerns, we would like to provide context for the experiment in Figure 4. The experiments in Figure 4 were designed to evaluate performance against [A]. However, we did not have access to the pretrained checkpoints used in their experiment. As a result, we created our own equivalent model. Additionally, in their experiment, training was conducted for less than 10 epochs.

We appreciate the reviewer's perspective on the evaluation protocol, but respectfully disagree with their interpretation. While training duration is indeed important, our focus on single-epoch performance is intentional and practically relevant for several reasons:

1. Transfer learning's primary goal is achieving strong performance with minimal fine-tuning, making initial convergence speed crucial.

2. The varying dataset sizes actually strengthen our findings - D2NWG's consistent performance across both large (CIFAR-10) and small datasets (Aircraft, Pets, STL-10) demonstrates its robustness and efficiency.

3. Few-shot and limited-data scenarios are common in real-world applications where extensive fine-tuning data isn't available. Our method's superior performance in these conditions highlights its practical utility.

4. The convergence patterns shown in existing literature indicate that models tend to reach similar performance levels with extended training. Our focus on early-stage performance therefore provides meaningful differentiation between approaches.

Moreover, as demonstrated in Table 15, our approach enables joint learning the distribution of models pretrained on ImageNet alongside other datasets without any performance degradation. It is also important to note that, our method allows the generation of diverse sets of initial weights. These can be sampled as needed and evaluated on a validation set to select the best-performing weights, providing an additional layer of flexibility and robustness to our approach.

The comparison with the [1] results is methodologically unsound for several reasons:

1. Different architectures: The cited work[1] uses a fundamentally different model architecture. Comparing performance across distinct architectures provides no meaningful insight into method effectiveness.

2. Incompatible training protocols: The cited results come from a 10-epoch training regime with specific hyperparameters and optimization choices that differ from our single-epoch evaluation framework.

3. Non-reproducible baseline: Without access to their exact model configuration and training setup, establishing their performance as a meaningful baseline is impossible.

The valid approach is comparing methods under identical conditions (as is done in all the baselines) - same architecture, training protocol, and dataset. Our results demonstrate consistent improvements over baselines in these controlled experiments, which is the appropriate metric for evaluating method effectiveness.

Addressing Concerns About Practical Computational Resources

. Trained models by increasing the data volume: The size of the pretrained dataset is not a limitation in our approach. We have demonstrated that both text descriptions and images can be used, with a few images per class encoded into a single representative vector during training. This avoids the need for large-scale dataset processing while maintaining an accurate representation of the dataset.

Instead of focusing solely on increasing model efficiency through fine-tuning—often leading to catastrophic forgetting—we propose training the same model on new tasks while collecting all the pretrained weights. By learning the distribution of these weights, our method enables on-demand generation of weights at any time and in any quantity. This approach highlights the scalability and flexibility of our method.

. Scalability: To scale our work to larger models, we propose a chunk-based encoding technique. As demonstrated in Section 3. Furthermore, chunk-wise encoding enables us to handle diverse architectures efficiently, bypassing the challenges associated with varying numbers of parameters across models. this approach allowed us to encode the full parameters of GPT-2 without any performance lost

[1] Do Better ImageNet Models Transfer Better? CVPR, 2019.

Addressing Concerns on Missing Ablations and Baselines for Components:

1. Missing Ablations: We appreciate the reviewer’s feedback and would like to clarify that the requested ablation studies are presented in Appendix Section D, where we validate key design choices through the following analyses:

• Latent vs. Raw Space Diffusion (Figure 6b): Demonstrates the superiority of latent space diffusion.
• Dataset Encoding Approaches (Figure 7): Highlights the effectiveness of our contrastive learning approach applied to the set-transformer-based dataset encoder.
• Impact of Model Zoo Size on Generalization (Figure 5a): Validates the critical role of model zoo size in improving generalization.

These results collectively substantiate the robustness of our design choices and underline the advantages of our proposed methodologies.

2. 'Addressing concerns on missing Baselines and which one is the main contribution of this work: To address this, we provide a detailed comparison of our method against the suggested baseline in the table below, ensuring a thorough assessment of its significance and effectiveness.

The goal of G.pt is to predict the performance of a given model on the same dataset at different stages of training, analogous to methods that learn to predict the optimization trajectory. Similar to how G.pt generates the final loss in a single step for a target loss, our approach generates the parameters for a given dataset that achieves the desired final performance under in-distribution sampling. However, applying our approach to G.pt would require rethinking their experimental setup—specifically, combining the experiments on CIFAR-10 and MNIST into a unified evaluation framework to better illustrate generalizability. Moreover, excluding the dataset encoder from the model, as suggested, raises a critical limitation: the model would lack the ability to determine the task for which it is generating parameters, leading to task-agnostic sampling. Finally, the larger models used in G.pt experiments range from 60M to 850M parameters, which is disproportionately large for encoding models with fewer than 10,000 parameters.

Our D2NWG represents a significant leap in neural network parameter generation from pretrained distribution. It addresses real-world challenges like task-specific parameter generation, expert checkpoints compression into a single model, and limitations of G.pt. While G.pt may seem as good baseline, [5] is a much closer comparison, and in the appendix D-3, we demonstrate that even with pretrained weights from diverse datasets (e.g., CIFAR-10 and VCIFAR-100), D2NWG surpasses the capabilities of a simple unconditional diffusion model designed for each dataset separately. We benchmark all close peer-reviewed and published baselines comprehensively in Table 4. We hope this clarifies and adequately addresses your concern.

We thank the reviewer for their thorough reading of our paper and their supportive feedback. We have carefully considered your feedback and propose the following response to your concerns:

Weakness

On the Generalization of the Proposed Approach and the Necessity of the Dataset Encoder:
To address the question of generalization, we first present a comprehensive theoretical analysis, followed by detailed empirical evidence to support our claims. This dual approach highlights the importance of the dataset encoder in enhancing the generalization capabilities of the proposed framework.

• Theoretical analysis of the generalization

Our method learns to model the parameter distribution of specialized pretrained models trained on diverse domains, tasks, and datasets. We formalize this approach through a global function $\mathcal{H}$ that samples from the parameter latent space of a collection of specialized models $\mathbf{F}=\{\mathcal{F}_i\}, \quad, i=1, 2, \ldots, N$.

**Theoretical Framework **

Let $\mathcal{H}: \mathcal{Z} \rightarrow \mathcal{W}$ denote our proposed method, which maps from a shared latent space $\mathcal{Z}$ to the weight spaces $\mathcal{W}$ of the specialized models. Through the diffusion model's smooth sampling process, we guarantee that for any specialized model $\mathcal{F}_i$ with parameters $w_i$, there exists a latent vector $z_i$ such that:

$|\mathcal{H}(z_i) - w_i| \leq \epsilon$

This theoretical guarantee ensures that $\mathcal{H}$ can approximate any specialized model parameters within an error bound $\epsilon$. Furthermore, the continuous nature of the diffusion process enables sampling of intermediate points between specialized models, effectively creating a smooth manifold of task-specific models.

Performance Guarantees

Upon convergence, the global function $\mathcal{H}$ can generate new parameters that match or exceed the performance of the best individual pretrained model. Formally:

$\forall \mathcal{F}_i\in \mathbf{F}, \exists z\in \mathcal{Z} \text{ s.t. } \theta =\mathcal{H}(z) \text{ and } \mathcal{L}(\mathcal{F}_i^{\theta}, \mathcal{D}_i)\leq \mathcal{L}(\mathcal{F}_i, \mathcal{D}_i)$

where $\mathcal{F}_i^{\theta}$ represents the target model initialized with the generated parameters $\theta$, and $\mathcal{L}$ denotes the loss function measuring performance.

** Generalization Bounds**

The generalization capabilities of our method are theoretically bounded by:

$\Delta_G(\mathcal{H}) \leq \min_{i} \Delta_G(\mathcal{F}_i) - \gamma$

where:

• $\Delta_G(\mathcal{H})$ represents the generalization gap of our method
• $\Delta_G(\mathcal{F}_i)$ represents the generalization gap of each specialized model
• $\gamma > 0$ depends on $\mathcal{H}$'s ability to produce task-specific parameters through smooth interpolation

As the number of specialized models increases, $\gamma$ also increases, leading to a decrease in $\mathcal{H}$'s generalization gap. We empirically validated this claim through experiments where we gradually increased the number of pretrained parameters from 5,000 to 20,000, as shown in Figure 5-a. It should be noted that the experiment results reported in Table 17 was only designed to compare our method to [3] in the same settings.

Conclusion

Our method provides theoretical guarantees for better generalization compared to individual pretrained models, making it particularly well-suited for transfer learning applications. Further formal analysis of these generalization guarantees will be provided in our complete work.

• On the Necessity of a Dataset Encoder :The dataset encoder is crucial for generalizing to novel datasets while generating parameters comparable to pretrained models. We adopt a diffusion-based approach, as it efficiently handles the complexity of encoding diverse datasets and supports probabilistic sampling better than predictive models.

[3] Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights, Neurips 2022

theoretical guarantees for better generalization required the proposed method to have access to task specific parameters on demand with performance superior or on par and also be able to interpolate between specialized models parameters. That is why our method required a dataset encoder as well as diverse set of pretrained weights from multiple domain for better generalization. This dataset encoder hase b The results reported in Table 17

Additional results on performance improvement of LLMs To further validate our approach, we applied our method on LLaMA-3.2-1B and gpt2-smal and validate them on the OpenLM leaderboard. Our method achieved the top performance among all LLaMA-3.2-1B-Instruct models, outperforming fine-tuned models based on the same architecture. For LLaMA-3.2-1B, we used spectral analysis to select the top 25% of transformer layers. In contrast, for GPT-2-small, we learn the distribution of the entire model, including the embedding layer, and for both we follow the procedures outlined in Tables 6 and 7.

The results, obtained via Huggingface, demonstrate that our method improves specialization in tasks like math and IFEVAL by up to 4%, while maintaining generalization, unlike fine-tuned models, which degrade performance on leaderboard tasks. This showcases our method's ability to enhance task-specific capabilities without sacrificing overall model performance.

Table: Model components and their configuration modes for llma3.2.1B.

settings summary This Table Summarizes the experimental settings for the presented figures and tables. "Min #cls" and "Max #cls" represent the minimum and maximum number of classes, respectively. "# Test Datasets" and "# Train Datasets" indicate the number of test and training datasets, respectively. "#Params" refers to the number of parameters, and "Conditioning" specifies the type of conditioning applied.

We thank you for your feedback and hope we have provided a comprehensive response to all of your concerns.

• Regarding dataset encoder : llm or vlm: Our analysis demonstrates that using image features provides more robust performance, as the model is specifically optimized on images with gradient-decent (detailed in Appendix B-5). While text encoders represent a viable approach, they require careful standardization of task descriptions. We believe that leveraging a powerful instruction-tuned language model trained on a large corpus would yield more stable representations and superior performance across textual datasets. Furthermore, we believe that recent vision-language models (VLMs) are particularly well-suited for dataset encoding, provided that each dataset is accompanied by precise and unambiguous descriptions.
• Addressing Concerns Regarding Evaluating Spectrum Techniques on Small Models: We appreciate your insightful suggestion and have conducted additional experiments to highlight the differences between the full model and selected parameters.

Experimental Results on spectruam vs full models

To validate our method's effectiveness across model scales, we conducted experiments with EleutherAI/pythia-70m (70M parameters). Following our protocol from Section 4.6, we compared full parameter generation against our spectrum-based selection of top 25% transformer layers, excluding embedding layers and output heads. The results are average of top 3 taken from 25 sampled weights.

model used

we used layer indexing as class conditioning for layer-wise chunking and generation.

Full parameter exploration shows consistent improvements across tasks, with substantial gains in Winogrande (+4.50%). Notably, our selective exploration approach, using only 25% of transformer layers, maintains competitive performance and even outperforms full exploration on MMLU. These results demonstrate that spectrum-based parameter selection effectively identifies critical layers for exploration, enabling efficient model adaptation while preserving performance.

• Is it possible to train a single param prediction model?: No, it is not feasible for us to train a single model for all our settings. However, a unified model can be trained for classifier-head-based settings by incorporating classifier head encoding as additional conditioning data. With unlimited data and compute, it may be possible tbut he primary challenge would be managing varying parameter sizes and addressing convergence issues effectively and the model cannot be simpler..

Task Descriptions:
Comprehensive task descriptions are provided in Table 13 of the appendix for reproducibility and clarity.

• Writing Revisions:
  We are thoroughly revising all sections to address the reviewers' feedback, with particular focus on improving clarity and coherence throughout the paper

[1] G.pt Learning to Learn with Generative Models of Neural Network Checkpoints, arxiv [2] Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets [3] Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights, Neurips 2022 [5] Neural Network Parameter Diffusion, arxiv

multiple architectures multiple datasets parameters generation.

This additional experiment aims to highlight the effectiveness and potential of our method compared to existing approaches

We extended our experiments to analyze 38 pretrained models across 19 architectures, each trained on CIFAR-10 and CIFAR-100. For feature extraction, we used LLaMA-3.2-11-Vision-Instruct to encode dataset features and class descriptions, while LLAMA-3-1.8B-Instruct extracted architecture configuration embeddings. These embeddings conditioned our parameter generation process. The results of learning the joint distribution across these architectures and models are shown below. The choice of vllm was recommended by one of the reviewers.

As demonstrated in the Table above , our approach achieves comparable performance to existing methods while requiring a single generative model instead of 38 specific ones. This experiment demonstrates our claim regarding the superiority of our method against the existing work. The pretrained model architecture and its parameter count are publicly available in a non-affiliated GitHub repository. https://github.com/chenyaofo/pytorch-cifar-models

Sampling Enhances Transfer Learning

We evaluated sampling from a distribution of pretrained models against single-model transfer learning using ResNet56 and our model trained on 19 diverse architectures weights pretrained on (CIFAR-10/100). Three setups were tested: (1) direct evaluation of pretrained models, (2) sampling conditioned on training sets (e.g., STL-10, CIFAR-10), and (3) sampling conditioned on test sets to address data leakage concerns.

Results show our approach consistently outperforms single-model transfer learning, with similar performance between training- and test-conditioned sampling from the same distribution. This highlights the practicality of leveraging diverse pretrained models for robust generalization.

Transfer learning: Our method vs pretrained models

The empirical evidence supports our key claims while highlighting the practical applicability and reliability of our proposed method. which is not possible with existing work.

We thank you for your constructive feedback, which has helped strengthen our work through additional experiments and analyses. We welcome any further questions or concerns you may have.

I appreciate authors providing additional results and clarifications. I am keeping my original score. I looked into the prior work in detail, the main difference from p-diff work [1] is the dataset conditioning (p-diff does unconditional sampling). I was a bit surprised to see no comparison table in the main paper from p-diff which this work heavily builds on as a simple unconditional baseline (it is only cited once in intro and then in appendix). p-diff trains a separate diffusion model for each dataset whereas this work allows to train a single diffusion model for multiple datasets which is useful but mostly for in-distribution scenarios (appendix section D.3 comparing p-diff and this work is doing in-distribution evaluation i.e. evaluate the generated parameters on the same CIFAR-10 and CIFAR-100 datasets which were used to train the diffusion models). Generalization of the proposed dataset conditioning approach to unseen datasets is still limited: as indicated by authors rebuttal response, training diffusion model on CIFAR-10 + CIFAR-100 with 38 pre-trained models gets 35% on STL-10 (unseen dataset) for a seen architecture (resnet-56) which is better than random but far behind train-from-scratch performance (90+%). I believe this gap could be closed by future works in this direction.

[1] Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion, 2024.

Thank you for your valuable feedback on our response. We appreciate the opportunity to clarify the differences between our method and p-diff, as we believe there may have been some misunderstandings in the comparison.

Key Differences Between Our Method and p-diff:

1. Scalability and Conditioning:

   • p-diff Limitations: The p-diff approach was demonstrated using selected batch normalization parameters within larger model, and a full 3-layer small network. For each architecture and dataset, p-diff requires training a new model. In experiments involving numerous models (e.g., 38 models), applying p-diff would necessitate training 38 separate p-diff models to learn the batch normalization parameters, in addition to the original models themselves. This process is not only resource-intensive but also highlights that p-diff was not designed for such scalability.
   • Our Approach: In contrast, our method conditions onthe model architecture or the dataset within a single experimental setting. This allows us to learn the distribution over models and datasets collectively, significantly reducing the need to train multiple separate models. Our approach is thus more scalable and efficient, especially when dealing with a large number of models.

2. Unified Learning of Distributions:

   • p-diff's Isolated Training: Since p-diff requires training separate models for each architecture and dataset, it learns distributions in isolation, without leveraging shared information across different configurations.
   • Our Integrated Learning: By combining models and datasets in a single experimental framework, our method learns the underlying distributions more effectively. This integrated approach captures the relationships between different architectures and datasets, leading to better generalization and performance.

Implications of Our Method:

• Efficiency: Our method reduces computational overhead by avoiding the need to train multiple models separately for each new architecture or dataset.
• Practicality: The ability to handle larger architectures and condition on them makes our method more applicable to real-world scenarios where models are often complex and varied.
• Contribution to the Field: By addressing the limitations of p-diff and introducing a scalable, conditioned approach, we believe our work offers a meaningful advancement in the field.

One reason we did not extensively benchmark p-diff is that it was listed as "under review" on arXiv, suggesting that the content might not be final or definitive. Therefore, we performed only a single experiment based on the settings provided by the authors and reported these results in the appendix Table 10.

Possible Misunderstanding of Our Experimental Setting

In Section 3.1, we clearly state our goal: "We aim to learn the distribution of a collection of pretrained models conditioned on their pretrained tasks or datasets, such that at inference we can sample weights that are on par with or superior to the pretrained ones on seen data, or achieve good initialization leading to better performance in a few optimization steps for unseen tasks."

In Section 4, we specify that our experiments focus on generating parameters with or without fine-tuning, and all our experiments—except for Section 4.5—follow this setting. We never claim that our method does not require fine-tuning on unseen tasks. We acknowledge the challenge of achieving generalization across any unseen dataset, and our work is a step toward that direction. We believe that such a level of generalization cannot be achieved in a single study.

Potential Misinterpretation of Results in Transfer Learning to STL-10 and CIFAR-10.1

We would like to clarify the results presented in our transfer learning experiments on CIFAR-10.1 and STL-10. These experiments were conducted on unseen tasks without any fine-tuning, and we compared the performance of models initialized by sampling from the pretrained distribution to those initialized using a single pretrained model from our collection—again, without fine-tuning. Importantly, there was no random initialization involved in these experiments.

Our objective was to demonstrate that, at initialization, sampling from the pretrained distribution achieves better initial performance on unseen tasks than selecting a single pretrained model. This highlights the advantage of our method in providing a stronger starting point for transfer learning, even when fine-tuning is not applied.

Thank you for pointing out these overstatements. We will clarify them appropriately.

On Addressing This is false. Some examples below::

• Abstract (line 26):
  "This work enables task-specific parameter generation without requiring additional fine-tuning."
  We apologize if this statement was misleading. This claim pertains to the performance enhancement of LLMs discussed in Section 4.6. In that section, we perform layer-conditioned sampling to achieve performance exceeding that of Llama-3.1-8B on the OpenLM leaderboard, without access to training or validation data. Our exploration of the parameter space is based on the dataset in Table 6. While the claim is accurate, we acknowledge that it could be clarified to prevent misunderstanding.

We will revise the abstract to address this point.

• Lines 160-161:
  "Our goal is to enable conditional sampling of a set of weights $p(W_{\text{new}} | D_{\text{new}})$ for a new dataset or task $D_{\text{new}}(x, y)$, such that these weights can achieve good performance on the new dataset either without further training."
  We apologize for any confusion caused by this statement. The sentence continues with "or with only a few optimization steps compared to random initialization," indicating that in some cases, fine-tuning is performed if necessary to achieve the desired performance. In few-shot scenarios, we did not perform fine-tuning, similar to in-distribution sampling.

We did not explicitly distinguish between "seen" and "unseen" tasks, opting instead to use "new task" to encompass both in-distribution tasks (such as expert model retrieval) and cases requiring continual training (as in the performance comparison with SANE in Table 4). We will clarify these points in the revised manuscript.

• Lines 20-23:
  "This allows for automatic generation of weights that generalize well across both seen and unseen tasks, outperforming state-of-the-art meta-learning methods and pretrained models."

Thank you for bringing this to our attention. In referencing state-of-the-art meta-learning methods, we specifically had MetaDiff in mind, which our method outperforms on unseen test sets. Furthermore, as shown in Figure 2 and through additional experiments on STL10 and CIFAR10.21, as well as one-epoch fine-tuning on datasets like Pets, Aircraft, and CIFAR10, our method demonstrates superior generalization compared to pretrained models. While these claims are accurate, we acknowledge they may have been overgeneralized.

We recognize that we may have overstated certain aspects and appreciate your feedback.

Although we did not fine-tune the sampled weights in some cases, this does not mean fine-tuning is never necessary. that is the point we would like to clarify.

Thank you once again for your valuable feedback. We will thoroughly revise the paper, incorporating your suggestions. We apologize for any shortcomings in our writing.

Regarding Fintuning compared to traning from scratch

To address this concern, we conducted additional experiments using Our model from Table 3 and applied them to the unseen datasets shown in the Table below including CIFAR-10 and other datasets. The results show that with a single epoch, D2NWG outperforms both imageNet pretrained models and random initialization. This highlights our method’s effectiveness and its capability to generalize better than traditional pretrained models for transfer learning as stated in section 3.1

compares the accuracy of three methods—RandInit, D2NWG, and ImageNet Pretrained—one epcoh fine-tuned on CIFAR-10, STL-10, AIRCRAFT-100, AIRCRAFT-30, and PETS.

We hope this clarifies the significant differences between our method and p-diff and addresses any misunderstandings. We are committed to improving our manuscript and are willing to incorporate any additional suggestions you may have.