Weaknesses:

W1: The field of knowledge distillation should be discussed in the related works section, which is highly related & Better to add discussion about knowledge distillation.

Our Response:

We appreciate the reviewer highlighting the relevance of knowledge distillation to our work. In response, we have expanded the related work in Appendix G in page of 32 to include a detailed discussion on knowledge distillation.

Comparison Between Knowledge Distillation and SAIL:

• Knowledge Distillation: This technique involves training a smaller student model to replicate the behavior of a larger teacher model by mimicking its output logits. The primary focus is on transferring output-space knowledge to achieve model compression and efficiency.
• SAIL (Our Approach): Unlike knowledge distillation, SAIL focuses on parameter-space knowledge transfer. It directly transforms and integrates parameters from multiple pre-trained models to initialize a new model. This method leverages the collective knowledge embedded in the parameters, rather than relying solely on output alignment.

Complementary Nature:

While both approaches aim to transfer knowledge from existing models, they operate at different levels. Moreover, knowledge distillation can be utilized as a subsequent optimization step following SAIL, where the merged parameters are further fine-tuned or re-trained to align with the target task's specific requirements.

W2: The use of colored block seems to be too extensive, which is not common in research papers. Though not an outstanding weakness point.

Our Response:

We appreciate your comment regarding the colored blocks. While we aimed to enhance the readability and highlight key conclusions for readers, we acknowledge your point about academic convention. We will consider adopting a more standard formatting approach further.

Questions:

Q1: Why is there a spike at γ=0.5?

Our Response:

We acknowledge the reviewer's observation regarding the spike at $\gamma=0.5$ in the validation loss curves depicted in Figure 2b. This spike is indicative of the complex dynamics involved in merging model parameters and can be attributed to the following factors:

Explanation for the Spike at $\gamma=0.5$:

1. Interference Between Divergent Representations:
   • At $\gamma=0.5$, the parameters from the two pre-trained models are weighted equally. If the models have learned significantly different or even conflicting representations due to their training on distinct datasets (as is the case with $D_1$ and $D_2$), the direct average can result in parameter interference.
   • This interference can temporarily degrade the model's performance, manifesting as an increase in validation loss.
2. Lack of Alignment in Parameter Space:
   • Equal weighting does not account for the relative relevance or compatibility of each model's parameters with respect to the target dataset $D_t$.
   • Without proper alignment, the superposition of parameters may not produce a coherent initial model, leading to suboptimal performance at $\gamma=0.5$.

Q2: Are you assuming all other network elements are the same, e.g., activation functions, vocabulary size?

Our Response:

Yes, in our experiments, there is usually a target model architecture and a corresponding activation function or vocabulary size, and we directly choose to find the vocabulary size of a target model size in our experiments and directly inherit it, without converting it here.

4. Integration of Multiple Pre-trained Models Beyond Single-Model Scaling:
   • SAIL is designed to integrate knowledge from multiple pre-trained models, potentially trained on different datasets or tasks, to form a more comprehensive initialization.
   • Xu et al. (2023) focus on transferring weights from a single larger model to a smaller one via weight selection. Their method does not generalize to integrating multiple models.
   • Samragh et al. (2024) scale up a single small model through HyperCloning, which involves duplicating weights symmetrically but does not consider merging different models.

References:

[1] Xu Z, Chen Y, Vishniakov K, et al. Initializing models with larger ones[C]//The Twelfth International Conference on Learning Representations. 2023.

[2] Samragh M, Mirzadeh I, Vahid K A, et al. Scaling smart: Accelerating large language model pre-training with small model initialization[J]. arXiv preprint arXiv:2409.12903, 2024.

W2: The motivation and system design of Figure 1 claims to use a pre-trained model such as LLM, but experiments are conducted by training the model from scratch on small-scale datasets in a controlled setup. It would be good to actually use the pre-trained models in Hugging Face to create the SAIL.

Our Response:

We appreciate the reviewer’s suggestion. In our initial submission, we focused on controlled experiments to isolate and understand the effects of SAIL. To enhance the robustness of our work, we have now incorporated experiments using pre-trained models from Hugging Face, specifically the OLMo model. The detailed results are presented in Appendix I in page of 34.

W3: The explanation of how the parameter transformation is conducted is not clear enough; the authors mentioned random projection and learnable methods, but a detailed investigation and experiments on which method is better are not included.

Our Response:

Thank you for highlighting the need for a clearer explanation of our parameter transformation process.

In our work, we utilize a learnable linear projection method to map weights from pre-trained models to the target model architecture. We have updated the manuscript to include a comprehensive explanation of the learnable linear projection method in Appendix F in page of 30. Additionally, we have conducted experiments comparing learnable linear projection with random projection, demonstrating the superiority of the learnable approach in terms of convergence speed and final performance.

Questions:

Q1: Could the authors provide the training curves of the NLP experiments as shown in the CV experiments? It helps to verify if the proved fast convergence is also applicable to transformer training.

Our Response:

Thank you for this insightful suggestion. We have included the training curves for the NLP experiments in Figure 7 at the page of 37 of the revised manuscript. These curves display both the training loss and validation perplexity across epochs for models initialized with SAIL compared to those with random initialization.

Additionally, we have conducted a comprehensive comparative analysis of SAIL against weight transformation and linear model merging approaches, with detailed results presented in Tables 1, 2, and 3 of the General Response section. The comparative metrics demonstrate superior performance across multiple evaluation criteria. These empirical findings corroborate that the rapid convergence properties initially observed in our computer vision (CV) experiments generalize effectively to transformer-based Natural Language Processing (NLP) architectures.

Weaknesses:

W1: Limited novelty in leveraging pre-trained models for initialization; similarity to existing work [1][2].

Our Response:

We appreciate the reviewer’s feedback regarding the novelty of our approach in leveraging pre-trained models for initialization and the perceived similarity to existing works [1][2]. We would like to clarify the distinct contributions and theoretical advancements that Structured Initialization Learning (SAIL) introduces, setting it apart from prior methods. And we will expand our discussion with [1][2] in our manuscript. We have noticed that [2] is concurrent work with ours and hasn't been open-sourced.

Distinction from Existing Work:

1. Comprehensive Parameter Transformation and Architecture Alignment:

   • SAIL introduces a dual-faceted parameter transformation technique that adjusts both the width (number of neurons per layer) and depth (number of layers) of pre-trained models to match the architecture of the target model. This allows for the integration of multiple pre-trained models with varying architectures and sizes, enabling a seamless amalgamation of diverse knowledge bases.

   • In contrast, Xu et al. (2023) [1] propose a method called Weight Selection, which primarily focuses on initializing smaller models by selecting subsets of weights from a larger pre-trained model. Their approach is limited to models within the same family and requires similar architectures, emphasizing downscaling rather than flexible architectural alignment.

   • Samragh et al. (2024) [2] introduce HyperCloning, a method that expands a small pre-trained model to a larger one through weight replication and symmetric initialization. Their technique involves duplicating neurons to match the larger model's dimensions but does not account for architectural differences beyond scaling width dimensions.

   • In SAIL, we formulate the parameter transformation as a linear mapping $T_i$ for each pre-trained model parameter $\theta_i$ :

     $\tilde{\theta}_i = T_i(\theta_i) \in \mathbb{R}^d$

     where $d$ is the dimensionality of the target model's parameter space.

   • This transformation ensures that the parameters from different pre-trained models are projected into a common space, allowing for seamless integration regardless of their original architectures.

2. Optimal Parameter Integration with Theoretical Guarantees:

   • SAIL defines the Proximal Parameter $\theta^P$ as an optimal linear combination of transformed parameters:

     $\theta^P = \sum_{i=1}^K \gamma_i^* \tilde{\theta}_i$

     where $\gamma_i^*$ are the combination coefficients optimized to minimize the distance between $\theta^P$ and the target model's optimal parameters $\theta^*$ .

   • Our Theorem 3 provides explicit solutions for $\gamma_i^*$ by solving:

     $\gamma^* = \arg\min_{\gamma} \left\| \sum_{i=1}^K \gamma_i \tilde{\theta}_i - \theta^* \right\|^2$

     subject to $\sum_{i=1}^K \gamma_i = 1$ and $\gamma_i \geq 0$ .

   • The coefficients $\gamma_i^*$ are determined based on the statistical distances (e.g., total variation distance) between the distributions of the pre-trained models and the target data, providing a theoretically optimal integration of knowledge.

   • In contrast, Xu et al. (2023) and Samragh et al. (2024) do not formulate an optimization problem for parameter integration. Their methods lack theoretical guarantees regarding the optimality of parameter initialization and convergence speed.

3. Theoretical Convergence Analysis:

   • SAIL offers rigorous theoretical analysis demonstrating that initializing with the Proximal Parameter $\theta_P$ leads to faster convergence to the optimal parameters $\theta^*$ compared to random initialization.

   • Theorem 2 in our work shows that the suboptimality after $T$ iterations satisfies:

     $J(\theta^{(T)}) - J(\theta^*) \leq (1 - \eta \mu)^T \left( J(\theta^P) - J(\theta^*) \right)$

     where $\eta$ is the learning rate and $\mu$ is the strong convexity parameter of the loss function $J(\theta)$ .

   • This result indicates that by minimizing the initial suboptimality $J(\theta^P) - J(\theta^*)$ through optimal parameter integration, SAIL accelerates the convergence of the training process.

   • Existing works do not provide such convergence analysis or theoretical foundations for their initialization methods.

Weaknesses:

W1: Limited comparison with related methods in model reuse/expansion and model merging.

Our Response:

We appreciate your feedback regarding the need for a more thorough comparison with existing methods in model reuse, expansion, and merging. We appreciate the opportunity to elaborate on how our approach, SAIL, distinguishes itself from current techniques.

The component nature of SAIL:

• Flexibility: Traditional model reuse and expansion methods typically initialize larger models from single smaller ones through weight replication or knowledge distillation. In contrast, SAIL offers a versatile framework that integrates multiple pre-trained models of diverse architectures and sizes.
• Parameter Transformation: SAIL employs parameter transformation techniques across both width and depth dimensions. While prior works have explored unidirectional parameter transfer strategies - either initializing larger models from smaller ones [3] or extracting subset parameters from larger models to initialize smaller ones [4] - these approaches are inherently constrained to single-direction transformations. In contrast, SAIL introduces a bidirectional parameter transformation framework that enables flexible and seamless transitions between models of varying scales.
• Optimal Linear Merging: Our method introduces a systematic approach for the optimal linear merging of different models by calculating combination coefficients γ*. This process, grounded in our theoretical framework, allows for the integration of knowledge from heterogeneous models, a capability that existing methods lack.

Experiments:

• Benchmark & Evaluation Metrics: We have expanded our experimental evaluation to include comparisons with prominent methods such as LIGO [1] and Model Soup [2], focusing on model expansion and merging. These methods were assessed using a comprehensive suite of natural language understanding and reasoning benchmarks to ensure robust performance analysis.
• Results: As presented in General Response Tables 2 & 3, SAIL demonstrates superior performance and greater flexibility compared to the evaluated existing approaches.

Ref:

[1] Wang P, Panda R, Hennigen L T, et al. Learning to grow pretrained models for efficient transformer training[J]. arXiv preprint arXiv:2303.00980, 2023.

[2] Wortsman M, Ilharco G, Gadre S Y, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time[C]//International conference on machine learning. PMLR, 2022: 23965-23998.

[3] Du W, Luo T, Qiu Z, et al. Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training[J]. arXiv preprint arXiv:2405.15319, 2024.

[4] Xu Z, Chen Y, Vishniakov K, et al. Initializing models with larger ones[C]//The Twelfth International Conference on Learning Representations. 2023.

W2: The paper lacks discussion on the gap between its linear model theory and real-world application, including practical computation of γ and situations where required data isn’t available.*

Our Response:

We appreciate your feedback regarding the theoretical aspects of our work and their practical implications. In practical applications, computing the optimal weights γ* based on total variation distances between data distributions can be challenging. To address this, we have explored alternative methods for estimating γ*:

Alternative Estimation Methods for $\gamma$:

To address these challenges, we propose practical estimation methods that align with our theoretical framework:

1. Implicit Methods Using Model Outputs or Predictions:

Even without direct access to the original datasets, we can estimate the distances between models by leveraging their outputs on a shared dataset or synthetic data. Let $\mathcal{V}$ be a validation set accessible during model deployment. We define empirical distances based on model outputs:

• Empirical Distance Between Models:

  $\hat{D} _{ij} = \frac{1}{|\mathcal{V}|} \sum _{x \in \mathcal{V}} \| f _{\theta_i}(x) - f _{\theta_j}(x) \|^2$

• Empirical Distance Between Models and Target: $\hat{D} _{i}{D^\ast} = \frac{1}{|\mathcal{V}|} \sum _{x \in \mathcal{V}} \| f _{\theta_i}(x) - y _x^\ast \|^2,$

  where $f_{\theta_i}(x)$ is the output of model $\theta_i$ for input $x$, and $y_x^\ast$ is the target output (if available).

Estimating $\gamma^\ast$:

Using these empirical distances, we construct an estimated matrix $\hat{H}$: $\hat{H} _{ij} = \hat{D} _{i}{D^\ast}^2 + \hat{D} _{j}{D^\ast}^2 - \hat{D} _{ij}^2,$

and compute: $\hat{\gamma}^\ast = \frac{\hat{H}^{-1} \mathbf{e}}{\mathbf{e}^\top \hat{H}^{-1} \mathbf{e}}.$

2. Model State-Based Estimation:

We can exploit internal representations to estimate model similarities:

• Activation-Based Distance:

  For each model $\theta_i$ and input $x$, let $A_{\theta_i}(x)$ denote the activation at a particular layer. We define: $\hat{D} _{ij} = \frac{1}{|\mathcal{V}|} \sum _{x \in \mathcal{V}} \| A _{\theta_i}(x) - A _{\theta_j}(x) \| ^2.$

3. Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):

During SFT or DPO, we can incorporate the estimation of $\gamma$ into the training objective:

• Modified Loss Function: $L(\theta) = L_{\text{task}}(\theta) + \lambda \sum_{i=1}^n \gamma_i \| \theta - \theta_i \|^2,$

  where $L_{\text{task}}$ is the task-specific loss, and $\lambda$ is a regularization parameter.

Optimization Strategy:

• The coefficients $\gamma_i$ can be treated as learnable parameters, optimized jointly with $\theta$.
• Constraints $\gamma_i \geq 0$ and $\sum_{i=1}^n \gamma_i = 1$ can be enforced using projection methods or by parameterizing $\gamma$ using a softmax function over unconstrained variables $\eta_i$: $\gamma_i = \frac{e^{\eta_i}}{\sum_{j=1}^n e^{\eta_j}}.$

Questions:

Q1: How do the authors justify the claim in lines 268-269 based on Theorem 2?

Our Response:

We offer a comprehensive analysis in Appendix C in the page of 22 by first providing an intuitive overview of how proximal parameter initialization $\theta^P$ reduces the initial distance to the optimal parameter $\theta^\star$ , thereby lowering the suboptimality of the loss function as indicated by Theorem 2. Specifically, we establish that: $\mathcal{J}(\theta^P) - \mathcal{J}(\theta^\star) \leq \frac{L}{2} \| \theta^P - \theta^\star \|_2^2$

This demonstrates that initializing with $\theta^P$ controls the initial loss difference through the parameter distance $\| \theta^P - \theta^\star \|_2^2$ .

Following the intuitive explanation, we present a detailed proof that begins by bounding the parameter distances using Theorem 3, which provides a probabilistic guarantee that pre-trained parameters are closer to $\theta^\star$ than random initialization: $\Pr \left( \| \theta _i - \theta^\star \| _2^2 \leq \alpha \| \theta _{\text{rand}} - \theta^\star \| _2^2 \right) \geq 1 - O\left( \frac{\tau^2 + \beta}{\alpha} \right)$

By selecting an appropriate $\alpha$ , we ensure that $\| \theta ^P - \theta ^\star \| _2 ^2$ is sufficiently smaller than $\| \theta _{\text{rand}} - \theta ^\star \| _2^2$. We then relate these parameter bounds to the loss function's suboptimality using the smoothness and strong convexity properties, ultimately showing that: $\rho = \frac{\mathcal{J}(\theta ^P) - \mathcal{J}(\theta ^\star)}{\mathcal{J}(\theta _{\text{rand}}) - \mathcal{J}(\theta ^\star)} \leq \frac{1}{2}$

This ratio confirms that $\theta^P$ leads to a loss reduction that is at least half as effective as random initialization, thereby ensuring faster convergence of the gradient descent algorithm.

Our Response to Second Point of Comment:

Thank you for your valuable feedback. We completely agree with your point regarding the cautious phrasing of our claim. As you suggested, the statement about faster convergence with proximal parameter initialization should indeed be expressed with more uncertainty. Specifically, as stated in Theorem 2, the advantage of initializing with the proximal parameter $\theta^\text{P}$ in terms of convergence speed is likely to occur, but it holds with high probability rather than as a guaranteed result. To address this, we revised the main text (lines 267 to 273) to clearly indicate that the convergence advantage of $\theta^\text{P}$ is probabilistic, and we provided an explanation to this effect. In Appendix C, we further provide a mathematical correction to our previous proof to explicitly account for the probabilistic nature of this result.

Regarding your second point, we would like to clarify that:

• While we can assert that the loss function is likely advantageous initially with high probability—due to the convexity assumptions and the specific choice of $\alpha$ in Theorem 1—there are limitations in maintaining this theoretical advantage throughout the entire iteration process. We cannot guarantee a consistent lower bound on the loss function after $T$ iterations. The upper bound provided earlier is useful for initial stages but may not remain tight as iterations increase. We have clarified this in Appendix C, discussing the potential reduction of the initial advantage over time.
• Nonetheless, our method has demonstrated superior performance in various practical scenarios. Specifically, it significantly accelerates training in neural networks across different domains, such as natural language processing (see Section 4.4 in page of 8 and Appendix I in page of 36), image recognition (see Section 4.5 in page of 10 and Appendix H in page of 35), and image generation (see Appendix J in page of 40).

We sincerely appreciate the reviewer's insightful suggestions and valuable feedback on both the empirical verification and theoretical aspects of our article. Your constructive comments have greatly contributed to enhancing the quality and clarity of our work. Thank you again for your thoughtful and detailed review. Should you have any further questions or require additional clarification, please do not hesitate to ask.