Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers

Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.

Q1: Lack of comparison to other clustering methods.

R1: Your question is thoughtful. In Section 4.3.1, we have compared clustering methods such as k-means and provided different values for the set (k) of cluster centroids in k-means. The results are as follows:

Cluster-Learngene not only outperforms in clustering efficiency but also adaptively adjusts the count of cluster centroids for each model layer, unlike k-means which requires predefined numbers of cluster centroids.

Q2: Experiments focused on image classification.

R2: Thanks for your suggestion. We conduct additional segmentation experiments on the ADE20K [A]. We set the base learning rate to $10^ {−3}$ and train for 160K iterations with a batch size of 8. The results are as follows:

, where “T-Params" and “I-Params" correspond to the Total number of parameters in the downstream/descendant models and the number of parameters Inherited into the downstream/descendant models, respectively. We validate the efficient cross-task initialization capability of our Cluster-Learngene. On the other hand, Fine-tuning exhibits inferior performance due to the risk of negative transfer compared to our Cluster-Learngene. [A] Bolei Zhou, et al. "Semantic understanding of scenes through the ADE20K dataset." Int. J. Comput. Vis., 127(3):302–321, 2019.

Q3: The paper format can be improved.

R3：We thank the reviewer for their constructive feedback and will enhance the paper by underlining the highest result in Table 2 and reformating Tables 4 and 5 into three-line tables for improved clarity and readability.

Q4: Can you offer more experiments?

R4: Thanks for your suggestion. (1) We have further supplemented the result for the hybrid architecture, such as LeViT-192 [B], which integrates convolutional and transformer components. We perform clustering and expansion on the MSA and FFN in stages 1-3 of LeViT-192. The experimental results are presented in the table below:

We fine-tune all downstream models for 500 epochs on CIFAR-100 and 50 epochs on ImageNet. It can be observed that our method, when extended to the hybrid architecture, also achieves the best results on both CIFAR-100 and ImageNet. [B] Graham, Benjamin, et al. "Levit: a vision transformer in convnet's clothing for faster inference." ICCV. 2021.

（2）For Section 4.2.3, in addition to the existing results on FGVC (CUB-200-2011, Stanford Cars, Oxford Flowers) and VTAB (CIFAR-100, Flowers102) datasets, we have also supplemented results from FGVC datasets such as Stanford Dogs, NABirds, and four VTAB datasets which are SVHN, DTD, EuroSAT, and Resisc45. The results are presented in the table below:

Our Cluster-Learngene outperforms Pretraining-Finetuning and other previous Learngene methods, thereby demonstrating the strong initialization capability and generalizability of our approach.

Q5: Can you compare Cluster-Learngene with model merging[1,2]?

R5: Thank you to the reviewer for providing these papers. We will cite them in the 'Related Work' section. At the conceptual level, Multi-Task Model Merging[1,2] aims to merge models for collaborative multi-task processing, whereas our Learngene is designed to independently initialize diverse models and generate scalable scales to meet specific task demands and resource limitations. In the experimental way, we compare our method with the two Multi-Task Model Merging methods[1,2] on the cars dataset. Our method achieves a performance of 89.87% on DeiT-Small, outperforming the Multi-Task Model Merging methods[1,2], which show performances of 69.6% and 72.0% on the larger-scale ViT-B/32, respectively.

Q6: Computationally expensive and break-even point.

R6: （1）For the largest model, DeiT-Base, the computational time for the clustering step is less than 6 minutes on a single GeForce RTX 3090 GPU, which is negligible compared to the several hours or even a full day required for fine-tuning downstream tasks. （2）Indeed, our Cluster-Learngene reaches a break-even point with as few as two downstream tasks, indicating that the conditions for our algorithm to be beneficial are quite lenient.

Thank you for addressing many of the concerns I initially had with your manuscript. However, I maintain my original score due to two persisting issues:

Performance in Practical Settings: While the theoretical contributions of your work are clear, the practical performance does not appear to significantly surpass the traditional industry-standard approach of pre-training followed by fine-tuning. The advantages of your method over existing techniques are not clearly demonstrated, and the problem statement and solution approach described in the abstract seem somewhat superficial. For your work to have a substantial impact, it would be beneficial to provide more concrete evidence or comparative analysis demonstrating its superiority in real-world applications.

Reproducibility Concerns: The complexity of your proposed method and the lack of critical implementation details raise concerns about reproducibility. Without these details, it may be challenging for readers to replicate your results or compare them with their work, which could undermine the credibility and utility of your paper. I would encourage you to include these essential details or simplify the approach to facilitate a better understanding and reproducibility by the wider research community.

I hope these points are helpful for refining your paper.

Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.

Q1: The paper lacks a complexity analysis or specific time cost assessments for Cluster-Learngene, which are crucial for understanding its computational efficiency.

R1: For the largest model, DeiT-Base, the computational time for the clustering step is less than 6 minutes on a single GeForce RTX 3090 GPU, which is negligible compared to the several hours or even a full day required for fine-tuning downstream tasks.

Q2: Regarding the choice of hyperparameters, the authors could provide an ablation study, particularly for parameters like ε, to better understand their impact on the model's performance.

R2: In the Appendix A.6, we have analyzed the varying settings of the hyperparameter Eps (ε). The following table displays the results for different values of ε:

$\varepsilon=1$ implies no FFN clustering, potentially causing negative transfer. $\varepsilon=100$ means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene ($\varepsilon=10$) strikes a good balance between these issues.

Q3: Will the code be available for open source in the future?

R3: We expect to release our code by late Sep.

Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.

Q1: Insufficient Downstream Task Evaluation

R1: Thanks for your suggestion. We conduct additional segmentation experiments on the ADE20K [A]. We set the base learning rate to $10^ {−3}$ and train for 160K iterations with a batch size of 8. The results are as follows:

, where “T-Params" and “I-Params" correspond to the Total number of parameters in the downstream/descendant models and the number of parameters Inherited into the downstream/descendant models, respectively. We validate the efficient cross task initialization capability of our Cluster-Learngene. On the other hand, Fine-tuning exhibits inferior performance due to the risk of negative transfer compared to our Cluster-Learngene. [A] Bolei Zhou, , et al. "Semantic understanding of scenes through the ADE20K dataset." Int. J. Comput. Vis., 127(3):302–321, 2019.

Q2: Given the emergence of attention variants aimed at reducing computational costs, such as sparse or low-rank attention, it remains to be seen whether this method can be effectively transferred and applied to these types of models.

R2: Thanks for your thoughtful question. In general, sparse and low-rank attention mechanisms reduce the computational load through specific acceleration techniques. Sparse attention achieves this by limiting each query to compute attention with only a local set of keys. Meanwhile, low-rank attention decomposes the attention weight matrix into a low-rank form, such as $\mathbf{A} \approx \mathbf{P}^{\top} \mathbf{Q}$, where $\mathbf{P}$ and $\mathbf{Q}$ are low-rank matrices. However, they still produce complete attention head outputs. Therefore, our Cluster-Learngene can still calculate the Mean Attention Distance and apply it to these sparse or low-rank attention mechanisms.

Q3: The hyperparameter Eps (ε) is currently set manually and plays a crucial role in the effectiveness of the clustering.

R3: In the Appendix A.6, we have analyzed the varying settings of the hyperparameter Eps (ε). The following table displays the results for different values of ε:

$\varepsilon=1$ implies no FFN clustering, potentially causing negative transfer. $\varepsilon=100$ means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene ($\varepsilon=10$) strikes a good balance between these issues.

Dear Reviewer 2s46,

This paper received mixed ratings. I would really appreciate it if you could check the authors' responses and post your further concerns (if there are still remaining concerns). Thank you so much!

Your AC

Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.

Q1: Section 3.1 and L183-196 seems to have a messy formatting of formulas;

R1: We appreciate the reviewer's keen observation regarding Section 3.1 and lines 183-196. We apologize for any inconvenience caused and assure the reviewer that we will thoroughly revise this section to ensure all formulas are correctly formatted and legible. Below are the complete expressions of equations (1) and (2) in Section 3.1：

​ $$ \mathbf{A}^h=\text{Attention}(\mathbf{Q}_ h, \mathbf{K}_ h, \mathbf{V}_ h) = \text{softmax}\left(\frac{\mathbf{Q}_ {h}\mathbf{K}_ {h}^\top}{\sqrt{d_ k}}\right)\mathbf{V}_ {h} $$. (1)

​ $\text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Concat}(\mathbf{A}^1, \ldots, \mathbf{A}^H)\mathbf{W}^O$. (2)

Here are the revisions made to L183-196：

• When $H_d$ is divisible by $c_l$: The weights of head centroids are shared $\frac{H_d}{c_l}$ times in sequence. For instance, centroids of weights $\mathbf{A}^{(L,1)}$ and $\mathbf{A}^{(L,2)}$ each share their weights across four attention heads, which are then directly assigned to eight attention heads of the descendant model in layer $L$.

• When $H_d$ is not divisible by $c_l$: The weights of the head centroids are sequentially shared $\lfloor \frac{H_d}{c_l}\rfloor$ times, followed by appending $\mathbf{A}^{(l,1)}, \ldots, \mathbf{A}^{(l,H_d \mod c_l)}$ at the end. As an illustration, we share the centroids of weights $\mathbf{A}^{(1,1)}, \ldots, \mathbf{A}^{(1,5)}$ once and then append $\mathbf{A}^{(1,1)}, \ldots, \mathbf{A}^{(1,3)}$, thus initializing eight attention heads of the descendant model in the first layer.

For the attention heads in the descendant models, we introduce the hyperparameter $\omega = \frac{H_a}{H_d}$ to denote the factor by which the number of attention heads is reduced compared to the ancestry model. ... According to the adjustments in the number of attention heads, the weights $\mathbf{W}^O$ of the projection layer are also proportionally pruned and then inherited by the descendant models.

Q2：Some key experiments were insufficient.

R2: Thank you for the reviewer's suggestion. (1) We have added a strong comparative method called 'Pretraining-Finetuning' in Table 1, as shown in the table below：

From-Scratch trains models for 100 epochs, while the Pretraining-Finetuning and our method only fine-tune the downstream models for 50 epochs. Compared with the other two methods, our approach demonstrates the best initial performance across different scales.

(2) In Table 2, we have supplemented the results on ImageNet as shown in the table below:

Our method continues to consistently outperform the baselines on the ImageNet dataset.

Q3: After Adaptively Learngene Clustering, the number of layers obtained is variable. How to deal with that if it is not equal to the number of layers in the downstream model?

R3: Your question is thoughtful. We have already addressed this issue in the Appendix A.3："According to the adjustments in the number of attention heads, the weights $\mathbf{W}^O$ of the projection layer are also proportionally pruned or expanded with the hyperparameter $\omega$ and then inherited by the descendant models."

Q4: In priority weight-sharing (Figure 2), what if the number of head centroids is less than the number of head in the downstream model?

R4: Even if the number of head centroids is fewer than the number of heads in the downstream model, we can initialize the heads in the downstream model multiple times with priority weight-sharing of head centroids. Therefore, our priority weight-sharing strategy does not require any absolute size relationship between the number of head centroids and the number of heads in the downstream model.

Q5: Are all parameters of the downstream model (inherited or not) updated (or fixed) during training?

R5: All parameters of the downstream model (inherited or not) update during training. However, due to leveraging weight-sharing, the computational cost of our descendant models remains smaller compared to fine-tuning the entire pre-trained model.

Q6: In L222, “The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as a neural network with the rectified linear units (ReLU)”, does this mean that 𝑊^𝑡 is more than just a parameter?

R6: I apologize for any confusion caused by the description in my paper. The symbol $\mathbf{\widehat{W}}_ {t}$ indeed represents the newly introduced learnable parameters that are used to expand the $t^{th}$ feedforward network (FFN). However, in the implementation of this expanding FFN process, not only do we apply the simple transformation with these learnable parameters $\mathbf{\widehat{W}}_ {t}$, but we also incorporate the ReLU (Rectified Linear Unit) activation function to provide some additional non-linear transformations.

Dear Reviewer GgZj,

This paper received mixed ratings. I would really appreciate it if you could check the authors' responses and post your further concerns (if there are still remaining concerns). Thank you so much!

Your AC