TranSpa: Towards Efficient Structured Sparse Training for Transformers

1.One limitation of this paper is that, while it emphasizes the correlation between weight matrices, it does not verify whether this correlation exists only between adjacent weight matrices. Further analysis is needed to confirm if non-adjacent matrices also exhibit correlations, as this could impact the effectiveness and generality of TranSpa’s sparsification approach.

2.A limitation of this paper is the lack of time complexity analysis for key computations, such as compute v and compute $\hat{I}$. Including these analyses would clarify the computational overhead introduced by TranSpa and provide a more comprehensive understanding of its efficiency.

• The experimental section lacks some details. For example, Table 1 presents training times but omits FLOPs savings, while Table 2 provides Flops savings without showing time savings.
• The training times for sevearl baselines, such as Monarch in Table 1, are missing, complicating the comparison of baseline performance and TranSpa.
• The estimation of weight importance is based on loss, as outlined in Eq. (6). It's unclear how importance evaluation and sparsification are conducted when there is no loss during inference.

1. In Eq. 5, the latter part $\left (0.5 \operatorname{erf} \left ( Y_1 / \sqrt{2} \right ) \odot X W_{in} \right ) W_{out} \neq 0.5 \operatorname{erf} \left ( Y_1 / \sqrt{2} \right ) \odot X \left (W_{in} W_{out} \right )$. Consequently, $W_{in}$ and $W_{out}$ do not fit the definition of "coupled weight matrix."
2. Most models incorporate position embedding in Multi-Head Attention. For example, the attention mechanism in LLaMA can be formulated as $\operatorname{softmax}\left(\frac{X_Q \boldsymbol{W}_i^Q ROPE \left(X_K W_i^K\right)^{\top}}{\sqrt{e}}\right) X_V W_i^V$, where $ROPE$ represents a rotation matrix with an an angle determined by the relative positions of two tokens. Thus, $W^Q$ and $W^K$ do not constitute a "coupled weight matrix."
3. In current models, such as LLaMA or Mistral, the definition of FFN is $FFN(X)=(XW_{up}\odot silu(XW_{gate}))W_{down}$. Authors didn't discuss this common structure in the article and I do not observe a "coupled weight matrix" within this FFN.

While the authors applied TransSpa on various architectures (GPT-2, Llama-1b, DeiT), the results are not entirely convincing:

• Table 1 compares TransSpa with the GPT-2 baseline. However, GPT-2's training epochs are twice as many, and the authors don't provide evidence that the TransSpa model has fully trained and converged with half the epochs. This weakens the credibility of the claimed training speedup.
• Table 2 lacks crucial comparisons with PEFT methods for LLM training. The LoRA series is missing data on memory and training time. Additionally, it's unclear why LoRA has the same number of parameters as the baseline.
• Table 3 shows that TransSpa significantly underperforms on small tasks like Winograd (69 → 60), while many studies have shown LoRA can achieve similar performance to Full-FT. This raises concerns about the pre-training results, which are typically considered more challenging than quick 1–3 epoch SFT.
• Table 5 switches to the much smaller CIFAR-10 dataset, despite Table 3 showing comparisons on ImageNet. This inconsistency raises questions about the quality of the loss curve on ImageNet.

The method of TranSpa is not complex, however, why and how it can preseve accuracy compared to full FT is not fully discussed in the paper