Response to Reviewer htFA

Thanks for your valuable comments and kind attention. We appreciate the opportunity to address your concerns and give detailed responses as follows.

1. Response to "The model parameter size in the experiments is too small; larger models are needed to evaluate effectiveness."

Thank you for this valuable comment. During the rebuttal, we added an experiment about using ViT as p-Model backbones to train Tina. The experimental results are shown in Table B of the Rebuttal PDF in general response. The results are promising that our Tina can reach 97.15’s accuracy in personalization when using the larger backbone ViT-B/32 pretrained by CLIP, and Tina also consistently outperforms the baselines.

2. Response to "In Table 1, the results of direct fine-tuning should be included."

Thanks for the comment. We may note that direct finetuning is not applicable under out-of-distribution (OOD) tasks where only class names are given and training data are not accessible; as a result, we didn’t put direct finetuning in Table 1.

However, in Table A of the Rebuttal PDF in general response, we provide some showcase comparisons between direct finetuning and Tina’s generated models. It is found that Tina has similar performances with direct finetuning on OOD tasks. It is notable that direct finetuning may serve as the performance upper bound of personalization because it assumes the accessibility to all data (both training and testing). However, in the setting of train-once-for-all personalization, the OOD ability is important for fast adaptation and personalization to end users only given the task descriptions.

3. Response to "We might need an ablation study on the impact of text prompts."

In our initial submission, we made an analysis of text prompts by learning class name prompts and testing on natural language description prompts in Figure 6 (c).

During the rebuttal, we have made a more in-depth ablation study on the impact of text prompts, as in Table C of the Rebuttal PDF. It is found that if training and testing use the same kind of text prompts, the performances are similar regardless of class-name prompting or description prompting. However, if the prompt strategies are different in training and testing, the results will degrade, and training in class name prompts has better transferability and generalization.

4. Response to "We might need an ablation study to determine if the model merely memorizes and reproduces parameters."

Thanks for this insightful comment. Actually, we may suppose the experiments about Tina's generalization ability can prove and verify that our model is not merely memorizing but generalizing. Specifically, Table 1 (OOD personalization), Figure 6 (a) (train on text prompts and test on image prompts) and (c) (train on class-name text prompts and test on description prompts), and Table 2 (test on unseen entities/classes).

During the rebuttal, we additionally make more validations, please refer to Table A of the Rebuttal PDF.

• Euclidean Distances: We have measured the Euclidean distances between model parameters. It is found that the generated models have obvious Euclidean distances from each other and also from the fine-tuned models.
• Ensemble Learning Ability: Ensemble learning often demonstrates higher accuracy than individual models, which can be indicative of the diversity in the internal representations of different neural networks, meaning that the manifold representations of the model parameters are not identical. Therefore, we make the generated models and the fine-tuned one ensemble to see whether it benefits. The results show that the ensemble accuracies are higher than the averaged accuracy and even higher than the best individual accuracy.
• Taking the above experimental results into consideration, it is evident that Tina is not merely memorizing parameters but generalizing.

5. Response to "Figure 2 requires polishing for better clarity."

During the rebuttal, we have polished Figure 2 for better clarity, and the polished figure is shown in Figure A of the Rebuttal PDF in the general response. We added more necessary legends in the figure and more detailed descriptions in the caption.

6. Response to "Can you provide your prompts for Appendix A.2? "

The prompts for generating natural language descriptions of class names from GPT-4 are as follows.

"I will give you a list containing various nouns. Please add some short, accurate, and common descriptions to these nouns that can accurately define these nouns, and then return to me a JSON file where the key is the name and the value is the corresponding description. An example of the description is: {"goblet": "a drinking glass with a base and stem", "anemones fish": "live associated with sea anemones", "chiffonier": "a tall elegant chest of drawers"}. The list to be processed is as follows:”

The few-shot exemplars in the prompts are extracted from the WordNet definition.

We will add the prompts in the appendix.

7. Response to "The model size is too small in exp."

Please refer to the first response in your rebuttal thread and check the results of ViT in Table B of the Rebuttal PDF.

[1] Garipov, Timur, et al. "Loss surfaces, mode connectivity, and fast ensembling of dnns." NeurIPS 2018.

Response to Reviewer Gsfd

Thanks for your valuable comments and kind attention. We appreciate the opportunity to address your concerns and give detailed responses as follows.

1. Response to "I am wondering whether the experimental results excels or perform similarly to the SOTA performance on some of the three datasets. In other word, is it possible to apply the proposed strategy to some more advanced framework to make the performance similar to the SOTA, which ensure the proposed method have real applications in the real use."

Thank you for this valuable comment. During the rebuttal, we added an experiment about using ViT as p-Model backbones to train Tina. The experimental results are shown in Table B of the Rebuttal PDF in general response. The results are promising that our Tina can reach 97.15’s accuracy in personalization when using the SOTA backbone ViT-B/32 pretrained by CLIP, and Tina also consistently outperforms the baselines.

2. Response to "The base model is CNN or ResNet in the experiments. Is the proposed method generalized to more advanced framework? Applying the proposed method on more advanced framework and obtain more advance performance indicates that the method has potential to be used in the real life."

Thank you for the comment. Please see the last response about Tina's performance on ViT.

3. Response to "We suggest providing necessary explanations in the captions of the model framework overview."

Thanks for the kind suggestion. We will add explanations in the captions of Figure 3 (framework overview) in the next version. The explanations to add are as:

Framework overview of Tina. (a) Training stage. The p-Models are firstly augmented by our classifier augmentation strategy and then noised according to the diffusion step. The p-Models are tokenized into chunks of vectors, and the classification sequence padding is optionally used if the classification length is shorter than the default. The CLIP text encoder is used to encode the users' text prompts during training. (b) Testing stage. Random noises are tokenized and denoised into parameters of p-Models. Thanks to the vision-language alignment of CLIP, Tina takes both text and visual prompts as the diffusion conditions.

Response to Reviewer BHPs

Thanks for your valuable comments and kind attention. We appreciate the opportunity to address your concerns and give detailed responses as follows.

1. Response to "Some methodological details are sparse, such as the specific configurations and hyperparameters used for training Tina. Providing more granular details could help readers replicate the experiments."

Thank you for the recommendation. We have included the main hyperparameters of training Tina in Appendix A.5. We will add more granular training details, such as the diffusion steps and the warmup schemes, in the future version. Also, to enable the effective reproduction of our results, we plan to release the source codes upon acceptance.

2. Response to "It would be good to clarify how exactly the parameters are inherited from G.pt for initialization."

Specifically, we have adopted a similar method provided in the bert2BERT [1] to inherit the parameters in G.pt in a layer-by-layer manner (by the keys).

• For the same key, when the shape of Tina is smaller than that of G.pt, we directly crop the parameters of G.pt into the shape of Tina for inheritance.
• When the shape of Tina is larger than that of G.pt, we use the FPI (Function Preserving Initialization) strategy from bert2BERT to expand the dimensions. That is, for the parts of Tina that have extra dimensions, we first randomly select a column from G.pt for each column that needs to be expanded. The values of the selected column from G.pt and the corresponding expanded column are reassigned to be the values of each data point in the column divided by the respective number of columns.
• When Tina has deeper layers than G.pt, we copy and stack the layers, which was also introduced in bert2BERT.

We thank the reviewer for this valuable comment, and we will add these details in the appendix.

3. Response to the rationale of using DiT and its advantages over other architectures.

Thanks for this valuable comment. Actually, we had a brief discussion on this point in the Discussions section (Lines 378-381) of the submission. We will give a more detailed discussion as follows.

• The architecture of DiT has great expressive power in diffusion tasks, especially in text-to-image and text-to-video. Also, it has the good property of scaling law as studied in DiT’s original paper [2]; therefore, for such a difficult task as parameter generation, this property will facilitate the model to realize generalization by merely scaling up parameters (evidence in Figure 4 (a) of our paper).
• The architecture of training an autoencoder and diffusion model separately is also worth studying, so we have already made some early attempts before submission and gained some insights.
  • In our early attempts, we reproduced the architectures of p-diff in our text-to-model scenarios but found that the VAE is hard to converge and generalize if we generate a full model of CNN. This is also reasonable since in p-diff’s paper, it mainly generates the BN layers, which have few parameters, so the scalability of this architecture may be limited.
  • We found that in the parameter generation task, the autoencoder is hard to train, and therefore, the latent space may be less representative; as a result, the diffusion process also has poor performance. In other words, in this architecture, the performances are bounded by the autoencoders and then the diffusion process, which are hard to improve and optimize.
  • In addition, if within the same compute and parameter budget, our one-stage, end-to-end, and decoder-only-like architecture has higher utilization efficiency of the parameters and is more suitable for scaling.
• However, we think the two-stage and autoencoder-based diffusion pipeline is also very meaningful and promising in weight generation. The key is how to learn an effective and representative latent space, and therefore, more advanced techniques are needed to explore its potential, which we may leave for future exploration.

4. Response to “While limitations are discussed, the manuscript could benefit from a discussion of the scalability of Tina to larger datasets and more complex tasks.”

Thanks for the suggestion, and we will add this discussion to future versions of the manuscript. We give a brief discussion below.

• In our paper, we have shown the preliminary results of the scaling law in Tina (Figure 4 (a) and Figure 5), and it is found that by scaling the parameters of DiT and the number of training data, Tina can generate and generalize on more complex models with higher dimensions. Therefore, Tina's scalability is promising and predictive once the computation resources are sufficient, which we leave for future explorations due to our current computation limits and budgets.
• Additionally, during the rebuttal, we conducted experiments using more complex ViT as p-Models by generating personalized layers in Table B of the Rebuttal PDF in general response, and the results are also promising.
• For more challenging tasks, such as full generation of ViT and training more enormous datasets, it is interesting for future works with sufficient computation budgets.

[1] Chen, Cheng, et al. "bert2BERT: Towards Reusable Pretrained Language Models." ACL 2022.

[2] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." ICCV 2023.

[3] Wang K, Xu Z, Zhou Y, et al. Neural network diffusion[J]. arXiv preprint arXiv:2402.13144, 2024.