Elastic ViTs from Pretrained Models without Retraining

Thank you for the valuable feedback and for recognizing that the technical modules are well-established and mature components. Below, we address each of the raised questions.

---

Integration of Hessian and Evolutionary:

Our approach decomposes parameter sensitivity into local and global components. The local Hessian, approximated via squared gradients, provides parameter-level sensitivity within blocks, capturing the diagonal Hessian terms efficiently. The global Hessian, which encodes off-diagonal dependencies between blocks, enables the xNES evolutionary algorithm to approximate implicitly. Our algorithm can be summarized with the following pseudo-code

# Pre-compute local importance
s ← diag-Hessian scores

# Initialise xNES mean and covariance
m ← 0; Σ ← I

# for T iterations and λ population size
for t = 1 … T: 
  for k = 1 … λ: 
    # Sample global re-weighting factors
    c ← sample 𝒩(m, Σ)

    # Combine local and global terms
    score ← c ∘ s

    # Prune K lowest-score blocks
    mask ← TopK(score, K)

    # Measure label and backward-free fitness via embedding similarity (Eq. 8)
    fit ← cos(PCA(z), PCA(z▹mask))

  # Natural-gradient update of mean and covariance
  (m, Σ) ← natural-grad update(fit)

Because the covariance matrix $ \Sigma $ is updated by a natural-gradient step, it gradually aligns with the inverse block-Hessian (see response to R69Bj). Once the search stabilizes, $ c \sim\mathcal{N}(m,\Sigma) $ re-weights the fixed local Hessian scores, yielding a prunability vector $ P $ (see Eq. 10). Sorting $ P $ defines a universal parameter ranking so that a single cutoff can obtain any desired sparsity—no extra optimization, Hessian storage, or retraining. A lone evolutionary run thus produces a full continuum of compute-adaptive subnetworks. We will add the pseudo-code to the updated paper to improve clarity.

---

Comparison with previous works:

Thank you for drawing attention to this. SN‑Net [4] achieves adaptability by bridging between several fully pretrained backbones from the same family (e.g., ViT‑S/B/L of DeiT) via additional stitching layers that must be trained and stored alongside the original model. By contrast, our method performs one‑shot structural pruning: a single pass over a pretrained model produces a family of subnetworks that require no extra layers and no further finetuning. We will add SN-Net and others to our related work section.

We follow previous works [1,2,3,4] and finetune our method after pruning on ImageNet-1k. We prune 50% of DeiT ViT-B and finetune for 100 epochs. We achieve 81.5% top-1 accuracy—close to the unpruned baseline of 81.8%—while delivering a 76% inference speedup. Our results are competitive with state-of-the-art pruning methods while outperforming the SN-Net baseline by 1.6%. Our method generalizes well, outperforming other works on average performance for k-nn and linear across six datasets.

We will add full finetuning experiments to the revised manuscripts covering all sparsity levels.

[1] Huanrui Yang et al. Global Vision Transformer Pruning with Hessian-Aware Saliency. CVPR 2023.

[2] Kaixin Xu et al. LPViT: Low-Power Semi-structured Pruning for Vision Transformers. ECCV 2024

[3] Chuanyang Zheng et al. SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization. NeurIPS 2022

[4] Zizheng Pan, Jianfei Cai, Bohan Zhuang: Stitchable Neural Networks. CVPR 2023

---

GFLOP Definition:

We use the term GFLOPs to indicate the number of theoretical floating-point operations required for a forward pass, following [5, 6]. We will add the following calculation to the manuscript:

Let $ n_p $ be the number of image patches of size $ d_p \times d_p $ pixels, $ c $ input channels, $ d $ hidden width, $ n = n_p + 1 $ tokens (CLS + patches), $ h $ attention heads, $ L $ layers, and $ C $ classes. With the usual FFN ratio 4 ($ d_{\text{ff}} = 4d $), the forward-pass FLOPs are

$$\text{FLOPs} = 2 n_p d_p^{2} c d + L \bigl[ 8 n d^{2} + 4 n^{2} d + 3 h n^{2} + 16 n d^{2} \bigr] + 2 d C .$$

The bracketed terms contain the self-attention and FFN term; the outer terms are patch embeddings and output logits.

For DeiT ViT-B/16 ($ n_p = 196,\, d_p = 16,\, c = 3,\, d = 768,\, h = 12,\, n = 197,\, L = 12,\, C = 1000 $) this evaluates to

$$\text{FLOPs} \approx 35.2\ \text{GFLOPs},$$

or $17.6$ GMACs (multiply–accumulates).

[5]: Training Compute-Optimal Large Language Models, NeurIPS 2022

[6]: Adam Casson, Transformer FLOPs

Thank you for the valuable feedback and for recognizing our paper’s focus on the timely and important challenge of deploying large foundation models. We appreciate the acknowledgment of our method as a creative and principled departure from prior work. Below, we address each of the raised questions:

---

Relationship between Sigma and Hessian:

Thank you for pointing this out. We agree that the xNES–Hessian relation in Eq. (9) needs a clearer rationale. We will add the following justification to the paper. During optimisation, xNES samples the block‑sensitivity vector $c$ from a Gaussian search distribution $ \mathcal{N}(\mu,\Sigma) $. When the fitness surface is locally well‑approximated by a positive‑definite quadratic,
$1$ prove that for a $(1,\lambda)$-ES with isotropic mutation, the selected‑population covariance converges to a scalar multiple of the inverse Hessian

$$\Sigma \ \propto\ \bigl(\nabla^{2}F(c)\bigr)^{-1}.$$

Natural‑gradient analyses of xNES and CMA‑ES $2$ predict the same tendency, because steep directions (large eigenvalues of $ \nabla^{2}F $) receive low variance while flat directions receive high variance. Taking the inverse yields our working rule

$$\nabla^{2}F(c) \\approx\ \alpha\\Sigma^{-1}, \qquad \alpha>0. \tag{9 revisited}$$

Intuition. xNES narrows its sampling ellipse in high‑curvature directions to avoid overshooting and widens it in flat directions to explore; repeated updates push $ \Sigma^{-1} $ toward the Hessian, capturing both local and inter‑block sensitivity.

Our fitness $F(c)$ is not the raw loss $ \mathcal{L} $; it is a cosine‑similarity score on embeddings compressed to 192 PCA components and averaged over several sparsity levels. PCA removes high‑frequency noise, and sparsity averaging further smooths the surface, so $F$ is locally smooth near good pruning vectors even though it is globally non‑convex. Thus, the second‑order approximation remains reliable. We therefore treat $ \Sigma^{-1} $ as a local Hessian proxy and use it to re‑weight first‑order pruning gradients (Sec. 3.4). This hybrid score captures both intra‑ and inter‑block sensitivity. It consistently improves pruning robustness on large Vision Transformers.

$1$ Shir, O. M., & Yehudayoff, A. (2019). Covariance Matrix Adaptation and Inverse Hessian on Convex Quadratic Functions. FOGA 2019.

$2$ Akimoto, Y., Nagata, Y., Ono, I., & Kobayashi, S. (2010). Bidirectional Relation between CMA‑ES and NES. PPSN XI.

---

Scalability of the xNES stage

The evolutionary algorithm consists of two cost terms.

$$\mathcal{O}\bigl(T \lambda S N_{S} F' \bigr) + \mathcal{O}\bigl(T B^{2}\bigr)$$

where the first term is the forward‑only fitness evaluation $T$ iterations × $\lambda$ population × $S$ sparsities × $N_{S}$ samples, each with cost $F'$and the second term is the covariance update for the $B \times B$ matrix maintained by xNES.

• $T$:  xNES iterations (generations)
• $\lambda$: population size per generation
• $S$: # sparsity levels optimised jointly
• $N_S$: images per fitness call
• $F'$: cost of a forward pass (feature extraction + PCA)
• $B$: count of prunable blocks (FFN slices + heads)

The quadratic term touches only the $B \times B$ covariance and is negligible; runtime is dominated by forward passes whose cost grows with the parameter count of the ViT backbone.

[3]: EVA: Exploring the Limits of Masked Visual Representation Learning at Scale, CVPR 2023

The near‑linear increase from S → B → L → H→G confirms that the model size (feature dimension and parameter size, hence forward cost) is dominant. Emperically, the $TB^{2}$ term contributes < 0.01% even for ViT‑H. Models trained on hundred–million–scale corpora have flatter curvature. Empirically, ViT‑L and ViT‑H spend their first $\approx 75$–100 iterations exploring before the xNES loss declines steadily. Increasing $T$ improves mask quality but leaves the per‑iteration cost unchanged.

Sensitivity to sparsity levels

Our default setting uses a grid of four target sparsities: 10%, 25%, 40%, and 60%. We find that increasing the number of sparsity levels in $S$ can lead to improved pruning performance with minimal overhead, as the fitness evaluations are forward-only and parallel. For example: On AugReg ViT‑B/16 (86M params), expanding $S$ from 4 to 6 yields a +1.7% average accuracy gain. On DINOv2 ViT‑B/14 (88M params), the same change results in gains of up to +12.9% on certain tasks.

This shows that pruning quality is moderately sensitive to the choice of $S$, and practitioners can easily trade off evaluation cost for accuracy by adjusting grid resolution.

---

Framework Extendability

Thank you for the suggestion. Our framework is naturally extendable to prune other structures beyond FFN neurons and attention heads. Since the genetic algorithm learns real-valued weights and is agnostic to the specific architectural units being pruned, it generalizes to any modular network component. This includes individual attention weights and even low-rank modules like LoRA.

We conducted an experiment where we pruned individual attention weights rather than entire heads. For that, we use the same weighting from the genetic algorithm, but when creating subnetworks from the prunability score, we pruned individual attention weights rather than whole heads. However, while this is technically feasible, the practical speed-up is limited, as hardware accelerators cannot exploit sparsity at the fine-grained level of individual weights. In contrast, pruning entire heads allows structural simplifications that lead to realized speedups. Below, we report the linear classification accuracy for an AugReg ViT-B/16 backbone across different sparsity levels, comparing attention head pruning vs. individual attention weight pruning:

We observe that pruning entire heads consistently outperforms finegrained pruning. This aligns with evidence that attention heads function as coherent computational units [4, 5], and pruning individual weights inside them can disrupt their effectiveness. Previous work that finds finegrained pruning to outperform structured pruning typically relies on finetuning to recover performance after pruning, which is not part of our method’s setup.

[4] Michel, P., Levy, O., & Neubig, G. (2019). Are Sixteen Heads Really Better than One? NeurIPS 2019.

[5] Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019.

PCA dimensionality.

We ablated PCA dimensions to assess their effect on pruning quality. Increasing the PCA size from 192 to 384 led to a modest improvement in average linear classification accuracy (42.5% → 42.9%) at 50% sparsity on AugReg ViT-B/16, though with increased computational cost. Not using PCA at all resulted in a notable drop to 41.5%, indicating PCA helps both reduce computation and regularize the embedding space for better fitness evaluation. We chose 192 dimensions as a practical balance between performance and efficiency, making it the default setting for our experiments.

---

Extreme sparsity levels:

The trade-off between sparsity and performance is highly dependent on both model type and model size. We found that large-scale models—such as ViT-H DeiT-III with 630M parameters—can be pruned effectively with minimal performance loss at 40% sparsity (Table 9). Figure 10 further illustrates the relationship between performance degradation and GFLOPs for ViT-S, ViT-B, ViT-L, and ViT-H models from the DeiT-III family, pruned up to 60%. Notably, larger models tend to have more redundant parameters, making them more amenable to pruning.

To quantify pruning efficiency, we introduce an efficiency score—defined as the percentage of GFLOPs that can be reduced while retaining at least 70% of the original performance. According to this metric, ViT-S allows a 40% GFLOPs reduction, while ViT-B and ViT-L achieve 56%, and ViT-H up to 60%. Additionally, as shown in Figure 4, different model architectures exhibit varying degrees of prunability. For instance, CLIP models demonstrate better tolerance to high sparsity levels compared to DINO models.

Our primary focus is on pruning without any retraining or finetuning, driven by the increasing demand for efficient deployment of large foundation models across diverse downstream tasks. Under this constraint, we typically observe a significant performance drop beyond 50% sparsity. However, as demonstrated in our responses to Reviewers e4hx and ZHY4, this degradation can be nearly fully recovered through finetuning—bringing our method in line with state-of-the-art approaches in retraining settings.

We will include a more detailed discussion of the trade-off between performance and compute reduction in the revised version of the paper. For additional insights, please refer to our response to Reviewer e4hx regarding performance prop point analysis.

Thank you for the valuable feedback and for recognizing the novelty of our approach, its strong performance across benchmarks, the effective pruning of Vision Transformers with minimal computational overhead, and our analysis of performance for throughput and FLOPs. Below, we address each of the raised questions:

---

Analysis of computational complexity:

We thank the reviewer for requesting a formal complexity analysis and for pointing us to SOSP and EigenDamage. Both papers will be added to the related‑work section as well as the analysis.
The algorithm’s total cost is

$$\mathcal{O}(N_D F) \+ \mathcal{O}(T \lambda S N_S F') \+ \mathcal{O}(T B^{2}) \+ \mathcal{O}(P \log P),$$

where

• $\mathcal{O}(N_D F)$ computes the local diagonal Hessian via one forward–backward pass on $N_D$ samples ($F$ per pass).

• $\mathcal{O}(T \lambda S N_S F')$ in every generation of the xNES, we draw $\lambda$ candidate masks, test each one at $S$ sparsity targets on $N_S$ images; $F'$ is a forward‑only pass (feature extraction + PCA), so no back‑propagation is involved.

• $\mathcal{O}(T B^{2})$ updates the $B \times B$ covariance matrix $ \Sigma $ in xNES (FFN + head blocks), capturing global off‑diagonal structure.

• $\mathcal{O}(P \log P)$ sorts $P$ prunability scores once to yield elastic subnetworks.

The key speed‑up over SOSP stems from zero backward curvature calls: SOSP‑H requires one Hessian–vector product per structure and SOSP‑I builds a dense $S \times S$ Gauss–Newton matrix, whereas we rely exclusively on forward inference and a lightweight covariance update $\mathcal{O}(T B^{2})$ (contributing <0.01%).

Runtime is therefore dominated by feature extraction. With 50 xNES iterations and four sparsity targets, a single A100 prunes the entire DeiT‑III family in 2–11 min: ViT‑S(2 : 35), ViT‑B(2 : 55), ViT‑L(4 : 58), and the 32‑block ViT‑H(11:04). The quadratic covariance term remains negligible throughout, demonstrating efficient scalability even for the largest ViTs.

---

Finetuning comparison:

Thank you for highlighting this. Our work intentionally targets the supervision-free, no-finetuning pruning setting, motivated by real-world deployment constraints (e.g., multi-task inference, edge applications, or frequent model updates), where finetuning is often infeasible or expensive. In such cases, retraining must be repeated for each sparsity level, dataset, and task. For instance, finetuning a ViT-B on ImageNet takes ~3 days on a single A100 GPU, whereas finetuning a ViT-H requires over 12 days, making scaling challenging.

That said, we agree that comparing with supervised finetuning-based methods is important to analyze the effectiveness of our approach. We have now added comparisons to several state-of-the-art pruning methods that assume full supervision and post-pruning finetuning, including SN-Net [4], NViT [1], LPViT [2], and SAViT [3]. As shown below, our method is highly competitive even in the finetuning setting: at 50% sparsity, we achieve 81.6% ImageNet top-1 accuracy after finetuning, nearly matching the unpruned baseline (81.8%) while offering a 76% inference speedup.

We will include the full finetuning results for all sparsities in the updated manuscript. While our primary contribution is enabling fast, label-free pruning without retraining, we show that our method also remains finetuning-compatible and competitive.

[1] Yang et al., Global Vision Transformer Pruning with Hessian-Aware Saliency, CVPR 2023

[2] Xu et al., LPViT: Low-Power Semi-structured Pruning for Vision Transformers, ECCV 2024

[3] Zheng et al., SAViT: Structure-Aware Vision Transformer Pruning, NeurIPS 2022

[4] Pan et al., Stitchable Neural Networks, CVPR 2023

---

Method robustness to initialization and random seeds

We investigate the sensitivity of our pruning approach to variations in random seeds and the initialization parameters of the evolutionary algorithm.

Random Seeds

Table 7 in the Appendix reports k-NN and linear classification accuracy (mean ± standard deviation) across three random seeds for an AugReg ViT-B/16 backbone pruned to 10%, 30%, and 50% sparsity. We observe that performance variance is minimal at low sparsity levels (e.g., an average of 0.2% at 10%) and increases modestly at higher sparsity levels (up to 1-2%). Some datasets, such as EuroSAT, remain stable across seeds, while others, like Oxford-IIT Pets, are more sensitive, particularly in the k-NN setting. This suggests that our method is generally stable, with expected variability under more aggressive pruning.

xNES Iterations ($ T $)

As shown in Table 3, increasing the number of xNES iterations from 50 to 250 and then to 500 results in an approximate 2% performance improvement, indicating that the method converges effectively even with relatively few update steps.

Initialization of Covariance Matrix ($ \Sigma $)

We found that initializing the xNES covariance matrix $ \Sigma $ using CKA similarity scores between architectural components (e.g., attention heads and MLP blocks) leads to faster convergence and significantly improved pruning outcomes. As shown in Table 8, appendix, this initialization improves both k-NN and linear accuracy across all datasets, for instance, k-NN accuracy on ImageNet-1k increases from 19.5% to 40.4%, and on Oxford-IIT Pets from 27.6% to 58.0%, highlighting the effectiveness of structure-aware initialization.

Our pruning method thus shows strong robustness across random seeds, xNES iteration counts, and initialization of $ \Sigma $.

---

Impact of including First-Order term

Thanks for pointing this out. We explored adding explicit first-order information to the pruning score in addition to the local Hessian (the diagonal approximation of the Hessian). To ensure compatibility with our genetic algorithm, which constrains weights to be non-negative , we use the absolute values of the gradient terms.

Below, we report k-NN and linear classification accuracy (%) on an AugReg ViT-B/16 model across sparsity levels:

The gradient term introduces noise from first-order optimization dynamics, which appears to distort the fitness signal used during pruning. In contrast, the local Hessian alone provides a more stable and reliable criterion, which is why we retain it as our default.

---

Comparison to Random Pruning

The table below reports average k-NN accuracy (%) across seven benchmark datasets for an AugReg ViT-B/16 model pruned using our method versus random pruning. Both methods start from the same unpruned baseline.

Our method consistently outperforms random pruning at all sparsity levels, with particularly large gains at moderate-to-high sparsity (e.g., +21% at 30% sparsity). These results demonstrate the value of structure-aware pruning: unlike random pruning, our approach preserves semantically meaningful representations that remain effective for downstream tasks without requiring finetuning.

---

Challenges with large models:

Large Vision Transformers like ViT-H are pretrained on hundred-million–scale datasets, which leads to a much flatter loss landscape. As a result, the xNES optimizer requires roughly 75–100 warm-up iterations just to explore the landscape before the loss begins to decrease. To obtain a pruning mask of comparable quality, we therefore need to extend the search horizon $ T $.

While this increases the number of optimization steps, we find that larger models are better to prune, as they contain more redundant parameters. For example, ViT-H can be pruned to reduce 40% of GFLOPs with virtually no performance loss (see Table 9 in the appendix).

---

Extendability of our approach

While our paper focuses on ViTs, the method is not architecture-specific. It requires only (1) a gradient-based loss, (2) structural units to prune (e.g., layers, blocks, filters), and (3) a fitness function for the genetic algorithm. This general framework makes it applicable to other architectures like CNNs, and domains such as language modeling or audio. For instance, in language models, one could prune attention heads and MLP blocks using a next-token prediction loss.

Thank you for the insightful feedback and for recognizing the importance of elastic models, specifically in their ability to operate without the need for finetuning, and for the well-presented paper and appendix. Below, we address each of the raised questions in detail:

---

Motivation for pruning without the need for finetuning:

Our work focuses on post-training structured pruning without finetuning to address increasingly common deployment and research needs:

• Generalist, multi-task use cases: Many models today (e.g., CLIP, DINO, or AugReg) serve broad roles across diverse tasks, as we also show in our evaluation across seven bechmarks. However, finetuning for each downstream target might be infeasible due to proprietary constraints, latency budgets, or lack of labeled data. Robust, general-purpose pruning is thus essential.
• Preserving robustness and generalization: We observe that pruning+finetuning on a narrow dataset (e.g., CIFAR-10) can reduce generalization (see e.g., accuracy drop of 3.1% from 64.2 to 61.1% via k-NN). This highlights that pruning without retraining can better preserve the inherent robustness encoded in foundation models.
• Fast iteration & model churn: In rapidly evolving domains, frequent model updates make repeated fine-tuning costly. Our one-shot, retraining-free method offers a scalable alternative with strong performance across benchmarks.
• Scientific insight: By avoiding finetuning, we can more directly analyze how pruning alters the learned representations. This links to ongoing foundational work on interpretability and the role of structural representations.

We will clarify these points more explicitly in the future version to help readers better understand the impact and scope of our work.

---

Drop point analysis:

We apologize if the plots (e.g., Figure 2) were not sufficiently clear in illustrating our improvements near the drop point. To clarify, we provide a detailed table below that demonstrates consistent gains over FPTP and LLM surgeon in the 30, 40, and 50% sparsity range, on average across the seven benchmark datasets, while also using label supervision for pruning an AugReg ViT-B/16 backbone:

These results show that our method significantly outperforms FPTP and LLM surgeon, already before the 50% mark, thus delaying the drop point. In the response below, we show that we can recover from such drops via finetuning. We will better visualize and provide these results in the updated version.

---

Baselines & full finetuning results:

Thank you for the suggestion. Our work focuses on pruning without any retraining or finetuning, motivated by the growing need to deploy large foundation models efficiently across diverse downstream tasks without the computational cost or complexity of finetuning (see earlier response).

While finetuning can help recover accuracy after pruning, [1, 3] have shown that full retraining often not only recovers the original performance but even improves it. However, finetuning must be repeated for every sparsity level and each downstream task, which can be computationally expensive. For example, finetuning ViT-B for 100 epochs on ImageNet takes roughly 3 days on a single A100 GPU, and ViT-H requires more than 12 days. This poses a significant bottleneck in multi-task or dynamic deployment scenarios, or when models are frequently updated. Our work targets this more challenging but practically relevant scenario, requiring only about 5 minutes for pruning on a single A100 and no label supervision. Because of this, fewer directly comparable baselines exist, as most prior work assumes finetuning.

That said, we also demonstrate our method’s finetunability: pruning 50% of DeiT ViT-B and finetuning on ImageNet-1k achieves 81.5% top-1 accuracy, close to the unpruned baseline of 81.8% while delivering a 76% inference speedup. Our results are competitive with state-of-the-art pruning methods. We will include a comprehensive table with all sparsity levels in the updated manuscript.

Thank you for encouraging this experiment which demonstrates strong performance gains when finetuning is employed. We will add this experiment for all sparsities to our updated version.

[1] Huanrui Yang et al. Global Vision Transformer Pruning with Hessian-Aware Saliency. CVPR 2023.

[2] Kaixin Xu et al. LPViT: Low-Power Semi-structured Pruning for Vision Transformers. ECCV 2024

[3] Chuanyang Zheng et al. SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization. NeurIPS 2022

[4] Zizheng Pan, Jianfei Cai, Bohan Zhuang: Stitchable Neural Networks. CVPR 2023

---

Throughput vs GFLOPS:

In Figure 2 (a, b), we report GFLOPs as floating-point operations per inference pass, as the LLM Surgeon and FPTP mask out pruned structures rather than effectively removing them, making it non-trivial to realize the theoretical speedups in practice. Therefore, we do not report throughput for these baselines and compare them to our method only in terms of GFLOPs. In contrast, our method and NViT produce pruning patterns that are hardware-friendly and straightforward to realize, so we report throughput for these methods as it better reflects actual runtime performance.

---

Labels for pruning:

We appreciate the reviewer’s comment and would like to clarify. While we do offer a cross-entropy (CE) variant of our method for comparison, our primary approach does not require any labels, instead relying on a self-supervised loss (Eq. 5) during pruning. In contrast, most baselines, including FPTP, LLM surgeon, and NViT, rely on supervised cross-entropy losses and assume access to a linear classification head during pruning. This limits both their flexibility and generalization, especially for pretrained models where such heads are unavailable; thus, we cannot report a comparison with those methods for DINOv1, DINOv2, and CLIP. We outperform these baselines even without using labels, showing that meaningful structure-aware pruning is possible in a fully unsupervised setting (see Table 1).

---

Retaining performance claim:

Thank you for pointing this out. We agree that the phrasing “retains performance for downstream tasks” may be too strong, especially at very high sparsity levels. While we do observe strong retention even at high sparsities for big models (e.g., Table 9 in the appendix; DeIT-III ViT-H/14), the drop in performance at extreme pruning rates is indeed dataset- and model-dependent. We will revise the wording in the future version to more accurately reflect this nuance, emphasizing that our method preserves downstream performance well in the practical sparsity range (e.g., 30-40%), which is also where it most consistently outperforms baselines.

I thank the authors for the time and effort to answer my (and the other reviewers) questions. After reading the other reviews and the comments to all reviews, this answers most of my questions. Moreover, it is very interesting to see that the proposed method also works well with fine-tuning (although it was not explicitly designed for this). I would like to note that these are baseline with "normal" fine-tuning and not ones that focus on minimal fine-tuning, i.e., one shot-pruning.

Some concerns remain. The strongest one is still the lack of comparison to SOTA approaches and baselines, both in the discussion and in the experimental evaluation. For example, in the field of single-shot / one-shot pruning, we have (among others):

• H. Kohama, H. Minoura, T. Hirakawa, T. Yamashita and H. Fujiyoshi, "Single-Shot Pruning for Pre-trained Models: Rethinking the Importance of Magnitude Pruning," 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
• Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, Maosong Sun, Know what you don't need: Single-Shot Meta-Pruning for attention heads, AI Open, Volume 2, 2021, Pages 36-42, ISSN 2666-6510, https://doi.org/10.1016/j.aiopen.2021.05.003.
• Elias Frantar and Dan Alistarh. 2023. SparseGPT: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 414, 10323–10337.
• Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework https://arxiv.org/abs/2411.18376

Plus numerous works also from the pre-transformer time.

I note that many of these works focus on minimal fine-tuning (instead of no fine-tuning as the authors do). However, their focus is too similar to ignore them. Especially, as for some the fine-tuning effort is very minimal, and they achieve very competitive results. Here, a direct comparison of performance, e.g., accuracy, parameters, and throughput, versus the cost of fine-tuning is important.