We sincerely thank the reviewer for the positive feedback, including: (i) addressing an important problem for large-scale fine-tuned models; (ii) extensive experiments; (iii) well-structured and easy to follow. We now address the reviewer’s questions in detail below.

Question 1 : Limited Novelty

We appreciate the reviewer’s comment. Our novelty lies in the design, motivation, and strict data-free constraint.

1. The first data-free pipeline for ultra-efficient delta compression :

To the best of our knowledge, we are the first to propose a data-free pipeline for ultra-efficient delta compression. This makes our approach practical in scenarios where tuning or data is unavailable. Our method maintains strong performance under strict data-free constraints, achieving high compression ratios without additional data or tuning.

2. Novel motivation and design :

Our method is guided by a clear and novel motivation: maximizing information preservation. All components are specifically designed around this motivation.

• Variance-based mix sparsity allocation uses weight variance as a simple, data-free proxy for information content, theoretically supported by the variance–entropy relation and rate–distortion theory.

• Distribution-aware compression preserves the relative proportions of quantized values to better preserve the intra-layer information. This distribution-aware compression, to our knowledge, is not used in prior delta compression work.

• Trace-norm-guided rescaling is designed to estimate an additional rescaling factor using trace norm to stabilize the pruned model based on trace norm, while remaining fully data-free.

While we understand the reviewer’s concern, we believe it is not appropriate to negate our contributions by referring to broad topics such as layer-wise pruning or delta compression quantization. These are widely studied areas, and many works explore them with different goals and techniques. Our method introduces novel designs and motivation, which we believe provide novelty within these broader topics.

3. Clarifications on related works :

• LAMP[1] proposes an importance score based on weight magnitude, whereas we design a variance-based score from an infomation-theoretic perspective. The metric we use for layer-wise sparsity is quite different.
• BitDelta[2] applies binary masks and scaling factors to quantize delta weights. This approach is fundamentally different from our distribution-aware compression, which uses uniform quantization combined with group-wise pruning to preserve the relative proportions of quantized values.
• DAREx[3] relies on data (labeled validation or unlabeled), and its own results show that using unlabeled data generally underperforms labeled validation data. However, using labeled validation data may cause the risk of performance inflation, which our approach avoids entirely.
• ComPEFT[4] applies magnitude pruning and quantizes the remaining weights with a fixed shared value. The value is determined using validation data. It is not data-free, it does not implement rescaling factor adjustment, and its sparsity and quantization strategies differ from ours. Moreover, ComPEFT[4] is mainly designed for PEFT models, making its application scope different from UltraDelta, which targets general delta compression across full fine-tuned models.
• We have already cited [1], [2], and [3] in our paper, and we will include a citation of ComPEFT[4] in the revised version.

Question 2 : Weak Theoretical Foundation

1. We thank the reviewer for the comments on our theoretical analysis. While we understand the concerns raised here, we would like to point out that other reviewers had a different perspective. Reviewer vAk5 mentioned that “the paper provides good theoretical motivation and empirical validation”, and Reviewer 9VE5 noted that “it’s great to see the authors provide not only empirical results, but also theoretically analyze the effect of MSA.”

2. To further strengthen this part, we have included an additional theoretical analysis based on rate-distortion theory (thanks to Reviewer 9VE5). Let the original weights be $W\\sim\\mathcal{N}(\\mu,\\sigma^2)$, and the compressed weights be $W^′$. The distortion is defined as the mean squared error (MSE) between them:

   $$D=\\mathbb{E}[(W-W’)^2]$$

According to classical rate-distortion theory for a Gaussian source, the minimum number of bits (information rate) required to represent $W$ under distortion $D$ is:

Here, $R(D)$ measures the information needed to encode the weights while maintaining a reconstruction error no larger than $D$. This relationship shows that, for a fixed distortion $D$, weight with larger variance $\\sigma^2$ require more information to represent. Therefore, layers with high weight variance are more “information-rich” and should be preserved more carefully during pruning to maximize information preservation.

3. Regarding DAREx[3], we would like to clarify that the variance discussed in DAREx[3] (Theorem 3.1) is not the same as ours. The variance in DAREx[3] refers to activation variance, which requires data to obtain activations. In contrast, our analysis is based on weight variance, which is fully data-free. Additionally, DAREx[3] studies activation variance for the purpose of rescaling factor adjustment, our use of weight variance is for layer-wise sparsity allocation. The two approaches differ in the definition of variance, data requirements, and intended use.

Question 3 : Data Usage in BitDelta[2] and DAREx[3]

1. Compared with DAREx[3] and BitDelta[2], our method does not require any labeled or unlabeled data, nor does it involve additional tuning or retraining. This makes UltraDelta highly generalizable, saves computation and deployment time, and is practical for real-world applications where data availability is limited or privacy concerns exist.

2. Although DAREx[3] can use unlabeled data, its experiments show that tuning with unlabeled data performs worse than using labeled validation data, while the latter risks evaluation inflation.

3. BitDelta[2] uses 800 samples (length 128) from the C4 dataset for calibration, introducing data and time dependencies that limit its applicability in data-restricted environments. We also report the compression runtime comparison between BitDelta[2] and UltraDelta:

   Method	BitDelta	UltraDelta
   Runtime	326.96s	211.53s

Question 4 : Unclear Rescaling Factor Determination

1. The trace norm measures the overall magnitude and effective rank of the delta weight, reflecting how much information it carries. A lower trace norm typically indicates lower effective rank and less information.

2. Motivated by information preservation, we use the trace norm as a simple and data-free proxy for information. We also empirically validate that the optimal rescaling factor is inversely related to the trace norm (see Figure 3).

3. Although this is a heuristic, it is data-free, theoretically motivated, and empirically validated in our experiments. We hope it can open an interesting direction for future work on data-free rescaling factor adjustment.

[1] LAYER-ADAPTIVE SPARSITY FOR THE MAGNITUDE-BASED PRUNING

[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit

[3] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

[4] ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization

------- Question 1 : Limited Novelty ----------

1. ComPEFT [4] can serve for Full Finetuning Updates, in its section 5.3, it demonstrate the effectiveness in compressing Full Finetuning Updates, the author should carefully read the references and give a comprehensive discussion, again, combining
2. ComPEFT [4] can indeed be data-free, as state in introduction "ComPEFT retains high performance post-hoc compression without the need for additional training"
3. ComPEFT [4] contain two steps: pruning and quantization which is followed by the paper closely.

As a result, without proper reference to related work is completely wrong.

------- Question 2 : Weak Theoretical Foundation------

1. DAREx[3] not only consider weight but also the activations, where its c represents $\Delta Wx$, so they are more comprehensive in analyzing the output change.

2. Your response on "our analysis is based on weight variance, which is fully data-free" is wrong, your $a= \Delta \theta x$ which rely on the input $x$, so your analysis is not data-free.....

3. Finally, it is the variance being the key points rather than the entropy, you Theoretical Analysis of the Rescaling Factor eventually analyze the activation error $\epsilon$ rather than entropy as state in line 185, this is the same as DAREx[3]'s theorem 3.1, you can completely remove the entropy term but just bound the output change as a direct signal on performance.

As a result, without proper reference to related work is wrong.

-------------- Question 4 : Unclear Rescaling Factor Determination--------

As shown in above response, the author wrongly believe their analysis is "data-free" and it rely on the assumption of Gaussian Distribution of weight and "ignoring" the influence of input $x$, thus the rescaling depend on weight is not comprehensive due to ignoring the input $x$ and may perform bad when $x$ activation is large (for example massive activation[5] in LLM)

[5] Massive Activations in Large Language Models

Due to the wrong statement and reference on related work for novelty, lack reference to prior theory analysis, lack understanding on their own theory analysis, I think this paper should be rejected.

------ Question 1 ------- Let's stand at the same page, let's refer to the published version of ComPEFT [1]:

------- designed for PEFT models-----

1. At the end of ComPEFT abstract : " Lastly, we provide an analysis of different method components, compare ComPEFT with other PEFT methods, and test its efficacy for compressing full-finetuning residual", so ComPEFT clearly demonstrate their contribution on full-finetuning residual. In addition, they use the whole section 3.3 to demonstrate the effectiveness on full-finetuning residual. As a result, their paper has shown enough contribution on full-finetuning residual.

-------- without the need for additional training-------

2. In their Fig 6, ComPEFT declare larger models do not require explicit tuning of rescaling, as a result, at least for those larger model, they can serve as a data-free baseline.

--------- clarified in the rebuttal that ComPEFT indeed applies pruning and quantization------

3. ComPEFT does not apply pruning and quantization only at their rebuttal time, their work clearly demonstrate they focus on combining pruning and quantization in their Figure 1. If you think your contribution is not the paradigm of combining pruning and quantization, your contribution 1 and 2 will be significantly weakened, thus leading to lack novelty.

[1] ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization. Published in Transactions on Machine Learning Research (04/2025)

------- Question 2---------

I'm not satisfied with the author's rebuttal:

1. Error of entropy on layer L: your Theoretical Analysis of MSA should be the entropy of output error in layer L, otherwise you can not control the entropy using activation error (B.2 Derivation of the Variance of Activation Error).

2. Output change Entropy belong to Output change: Using entropy as a metric for output change $\Delta W x$ is just the same thing as output change itself, especially under this strong gaussian assumption, for example, in the context of massive activation [2], such a assumption is problematic , thus this paper's theory is not only shallow but also weak.

3. Wrongly statement :The author clearly state their theory analysis is based on weight variance and wish to distinguish with DAREx, as state in their rebuttal "Regarding DAREx[3], we would like to clarify that the variance discussed in DAREx[3] (Theorem 3.1) is not the same as ours. The variance in DAREx[3] refers to activation variance, which requires data to obtain activations. In contrast, our analysis is based on weight variance, which is fully data-free." Now, they say activation only appears in their theoretical analysis, this misalignment raise a concern on their theory analysis.

[2] Massive Activations in Large Language Models

----- Question 1 ------- Still, I'm not satisfied with the author's opinion that :

1. ComPEFT cannot be served as a important related work just because its "focus on PEFT" due to significant amount of discussion and results on full-finetuning residual, such as end of the abstract, whole section 3.3, section 3.6, Table 4.

2. As also agreed by authors taht ComPEFT can be considered data-free in certain cases (large LLMs), due to the importance usage of large LLMs, should be a important baseline to compare.

3. The core ideas—layer-wise importance for adaptive pruning [1], delta parameter quantization [2], and rescaling factor adjustments [3]—are each incremental extensions of established compression and pruning techniques. Furthermore, their combination has already been studied in works such as ComPEFT [4].

----- Question 2 -------

I don't think I misunderstand the theory, as state in my original response and post-response, here I summarize my opinion again:

1. Theory is not Data-Free : The authors assert that their analysis is "fully data-free" because it is based on weight variance. However, this is incorrect: their analysis still uses a=Δθx, which inherently depends on the input x, and thus is not data-free. This undermines one of their key claims during rebuttal distinguishing their theory analysis from theory in DAREx.

2. Theory cannot support empirical analysis: This misalignment between empirical and theory also weaken their novelty, and raise potential limitation of its data-free method when x cannot be ignored (e.g. x is large).

3. Closely resemble DAREx but shallow: Although the paper frames its theoretical contributions around bounding entropy, the actual derivation in Appendix B.2 focuses on the activation error 𝜖, and not entropy. This aligns with the approach in DAREx Theorem 3.1, which bounds performance based on output deviation. In fact, the entropy term can be entirely removed from the analysis without affecting the final bound. In addition the paper equates output change with entropy under a strong Gaussian assumption, However, the paper they cite is for vision transformer, common in large language models, this assumption might be violated (e.g., [1]).

4. Contradiction in Positioning vs. DAREx The authors initially claim their method is distinct from DAREx because it is based on weight variance rather than activation statistics. But later, they acknowledge the role of activations in their theoretical analysis. This contradiction introduces confusion and casts doubt on the internal consistency of their theoretical claims.

As a result, due to the misalign between theory and empirical method, theory close resemble to DAREx (as also concerned by Reviewer asSv), the unimportant of entropy analysis, the contribution of theory is limited.

[1] Massive Activations in Large Language Models

Thanks for the author's rebuttal, I decide to maintain my current score and will determine the final rating based on subsequent discussions with other reviewers and the AC.

We thank the reviewer for the positive feedback, including: (i) extensive experiments with strong performance; (ii) an efficient and data-free method. We now address the reviewer’s questions in detail below.

Question 1: Large mean magnitude setting

We revisited the DAREx[1] experiments and found that the reported degradation occurs on MetaMath-Llemma-7B [8], where DAREx assumed Llama-2-7B as the base model. However, according to its huggingFace model card, the correct base model is Llemma-7B. We replicated the experiments under both settings:

DAREx’s conclusion that importance-based pruning (such as MP) performs better than DARE[2] in a large mean magnitude setting, only holds at low sparsity levels. At high sparsity, all delta compression methods perform poorly. When using the correct base model (Llemma-7B), all methods perform as expected without severe degradation.

Question 2: The necessity of DAC

While random pruning may partially preserve the weight distribution, it does not guarantee robustness, especially under high sparsity.

1. Risk of distortion: At high sparsity, random pruning can accidentally distort the original parameter proportions, making the compressed model less robust compared to methods that explicitly maintain weight distribution.

2. Empirical evidence: Our ablation study (Figure 5, Table 9) compares DAC with random pruning across 30 ViT setting. DAC consistently achieves higher accuracy: | Method | 4-bit | 8-bit | No quantization | | -------------------- | -------- | --------- | --------------- | | DARE | 81.58 | 83.85 | 85.88 | | UltraDelta (DAC) | 86.4 | 90.18 | 92.45 |

These results confirm that preserving weight distribution explicitly is more robust than random pruning in large-scale, high-sparsity compression scenarios.

Question 3: Fig 4

1. The cosine similarity metric in Figure 4 is intended to illustrate how well each method preserves the information of the original fine-tuned model, not to predict downstream performance directly. A higher similarity indicates closer alignment to the uncompressed model’s representations, but final task accuracy also depends on the uncompressed model’s own performance on that task.

2. For example, beyond the two patterns highlighted by the reviewer, we also observe that UltraDelta outperforms the fine-tuned model (62.64 vs 60.18), further showing that cosine similarity alone does not perfectly reflect downstream performance.

3. In the final version, we will include more figures on other datasets to better illustrate this relationship and reduce the potential influence of the fine-tuned model.

Question 4: Eq(11)

It means Hadamard product.

Weakness 1: Compression ratios

We sincerely thank the reviewer for pointing out this issue.

1. Our focus is on exploring the upper bound of compression ratio achievable with minimal accuracy loss. We believe this is a meaningful research direction as an ideal compression ratio can benefit deployment efficiency, reduce communication costs, and storage efficiency, as Reviewer 9VE5 noted: “this paper lays a good foundation for system/ML co-design”.

2. We acknowledge that a more rigorous evaluation should report the effective compression ratio. Using a CSR format, we recalculated the practical compression ratios and found they remain high while preserving excellent accuracy: | Metric | Llama-2-13B | ViT-Base | ViT-Large | T5-Base | | --------------------------- | ----------- | -------- | --------- | ------- | | Original Accuracy | 50.94 | 91.0 | 94.4 | 86.37 | | Compressed Accuracy | 52.05 | 90.7 | 94.4 | 86.74 | | Effective Compression (CSR) | 30.97$\\times$ | 36.92$\\times$ | 109.10$\\times$ | 225.50$\\times$ |

3. We note that many ongoing works address the efficiency gap between ideal and practical sparsity representations, e.g., Block Sparse Row[5], SMaT (SC’24) [6], CS256 (ICLR’25 Workshop on SLLM) [7]. These advances can further improve practical deployment efficiency.

4. In the revised version, we promise to clarify our terminology and include both ideal and practical compression ratios for completeness. We thank the reviewer again for this valuable suggestion that improves the rigor of our work.

Weakness 2: Missing baseline

We thank the reviewer for this suggestion. DARE itself does not natively include quantization, which is why we did not report this baseline in the main paper. Following the reviewer’s comment, we additionally evaluated DARE + uniform quantization:

Weakness 3: Limited novelty

We appreciate the reviewer’s comment. Our novelty lies in the design, motivation, and strict data-free constraint.

1. The first data-free pipeline for ultra-efficient delta compression

To the best of our knowledge, we are the first to propose a data-free pipeline for ultra-efficient delta compression. This makes our approach practical in scenarios where tuning or data is unavailable. Our method maintains strong performance under strict data-free constraints, achieving high compression ratios without additional data or tuning.

2. Novel motivation and design

Our method is guided by a clear and novel motivation: maximizing information preservation. All components are specifically designed around this motivation.

• Variance-based mix sparsity allocation uses weight variance as a simple, data-free proxy for information content, theoretically supported by the variance–entropy relation and rate–distortion theory.

• Distribution-aware compression preserves the relative proportions of quantized values to better preserve the intra-layer information.

• Trace-norm-guided rescaling is designed to estimate an additional rescaling factor using trace norm to stabilize the pruned model based on trace norm, while remaining fully data-free.

3. Clarifications on related works

• AlphaPruning[3] has high computation overhead as it requires spectral analysis on every weight matrix, including SVD/eigenvalue decomposition and power-law fitting to estimate the heavy-tail exponent. ALS [4] requires data access to compute activation correlations. In contrast, our method adopts a simple, data-free variance metric, offering both computational efficiency and no data dependency.

• DAREx[1] relies on data (labeled validation or unlabeled), and its own results show that using unlabeled data generally underperforms labeled validation data. However, using labeled validation data may cause the risk of performance inflation, which our approach avoids entirely.

• We acknowledge that TNGR are empirically chosen. However, the design is well-motivated: the trace norm captures the effective rank of delta weights, serving as a simple, data-free proxy for information content. We agree this heuristic is a limitation and have explicitly stated it in our Limitations and Future Work section, where we plan to conduct more in-depth research on estimation strategies, while maintaining the data-free property of our approach.

Weakness 4: Moer recent LLMs

We thank the reviewer for this suggestion. We have added experiments on more recent LLMs, including Qwen2.5-7B-Instruct and Llama-3.1-Tulu-3-8B-SFT. We also evaluate them on more challenging tasks:

Weakness 5: Large norm model

1. Our experiments already cover a wide range of models of different types, sizes, and compression difficulties, with average L1 norms ranging from 4.1e-5 to 5.01e-3.

2. We also conducted tests on the large-norm model reported in DAREx [1] (MetaMath-Llemma-7B), as detailed in Question 1. When Llama-2-7B was incorrectly used as the base model, the L1 norm reached 1.7e-2, and in this extreme case all delta compression methods failed.

[1] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

[2] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

[3] AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

[4] Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment

[5] Automatic Performance Tuning of Sparse Matrix Kernels

[6] High Performance Unstructured SpMM Computation Using Tensor Cores

[7] COMPRESSED SPARSE TILES FOR MEMORY-EFFICIENT UNSTRUCTURED AND SEMI-STRUCTURED SPARSITY

[8] METAMATH: BOOTSTRAP YOUR OWN MATHEMATICAL QUESTIONS FOR LARGE LANGUAGE MODELS

Thank you for the authors' rebuttal. I would like to maintain my score,

• The authors claim a very high compression ratio in their paper, but the actual ratio is not as impressive. This kind of reporting is problematic, as it can mislead readers and distort fair comparisons with baselines. While the authors addressed this concern in their rebuttal, I believe they have a responsibility to prevent such issues in the original submission.
• I still believe the novelty of the paper is somewhat limited, as it primarily integrates existing techniques. This concern was also raised by Reviewer 8YfX

We sincerely thank the reviewer for strong support of our work, including: (i) important and timely topic; (ii) well written and easy to follow; (iii) theoretical and empirical analysis; (iv) extensive experiments. We are especially grateful for the many constructive suggestions and insightful questions, which significantly helped us improve the depth of our paper.

Question 1: Variance-Bitwidth Relationship

We thank the reviewer for the insightful question. We have observed that variance can also serve as a metric for bit-width allocation.

1. According to the variance–entropy relationship and the rate–distortion theory, variance is theoretically linked to compression rate, which depends on both sparsity and bit width.
2. We empirically validated this in 8 ViT-B/32, where bit width was allocated based on layer variance (4-bit to layers with larger variance, 2-bit to layers with smaller variance, together it obtains 3-bit/parameter quantization). The mixed 3-bit setting achieved accuracy close to uniform 4-bit quantization, showing that variance is an effective metric for bit width allocation. | | SUN397 | Cars | Resisc45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | AVG | | - | - | - | - | - | - | - | - | - | - | | Finetuned | 79.2 | 77.7 | 96.1 | 99.8 | 97.4 | 98.7 | 99.7 | 79.4 | 91.0 | | UltraDelta (uniform 4bit) | 78.6 | 77.4 | 95.7 | 99.4 | 97 | 98.7 | 99.7 | 79 | 90.7 | | UltraDelta (uniform 3bit) | 77.6 | 75.7 | 95.3 | 99.3 | 96.8 | 98.6 | 99.7 | 78.7 | 90.2 | | UltraDelta (mix 3bit) | 78.2 | 77.2 | 95.6 | 99.3 | 96.8 | 98.8 | 99.6 | 79.0 | 90.6 |

Question 2 : Can variance serve as a signal for sparsity/bitwidth

While variance provides a strong signal for allocating relative sparsity levels or bit widths, it is not sufficient to select precise numerical values for either parameter. Our method uses layer-wise variance to guide allocation, but evaluating per-layer performance is impractical in experiments. Therefore, our analysis is conducted at the model level, and the standard deviation is calculated based on the overall weight of the model.

1. Bit width: We conducted experiments to test whether variance can guide the selection of an optimal quantization bit width. Specifically, we quantized 8 ViT-B-32 models using different bit widths (without sparsity, only quantization), due to space limits, we only show some of the results.

• Across all datasets, 2-bit quantization is optimal, retaining ~99% of the finetuned model performance.

• Bit width is a discrete variable with large step size. Performance drops drastically from 2-bit → 1-bit.

• Therefore, variance cannot directly predict this optimal discrete value, as the available discrete bitwidth levels have large gaps and lack fine granularity for precise selection.

  	SUN397	Cars	Resisc45	EuroSAT	SVHN	GTSRB	MNIST	DTD	AVG
  Std	0.000312	0.000299	0.000271	0.000242	0.000288	0.000250	0.000261	0.000263	-
  Finetuned	79.2	77.7	96.1	99.8	97.4	98.7	99.7	79.4	91.0
  1-bit	65.8	62.3	68.0	58.8	37.9	39.3	59.6	49.3	55.1
  2-bit	79.2	77.6	95.5	99.1	96.8	98.3	99.6	79.3	90.7
  Optimal Bitwidth	2	2	2	2	2	2	2	2	2

2. Sparsity level: We conducted experiments on sparsity selection using different sparisty (using only random pruning). The optimal sparsity is the highest sparsity that preserves ~99% of the original accuracy.

• The results show that for models with lower original accuracy (harder tasks), layers with larger variance tend to require lower sparsity.
• For models with higher original accuracy (easier tasks), variance has a weaker impact on the optimal sparsity level.
• Therefore, variance is correlated with sparsity and serves as a reliable signal for allocating relative sparsity levels across layers. But the precise sparsity value for the whole model can be influenced by task complexity. However, task complexity is consistent across layers of a single model, making variance a robust metric for relative layer-wise sparsity allocation. | | SUN397 | Cars | Resisc45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | | - | - | - | - | - | - | - | - | - | | Finetuned | 79.2 | 77.7 | 96.1 | 99.8 | 97.4 | 98.7 | 99.7 | 79.4 | | std | 0.000312 | 0.000299 | 0.000271 | 0.000242 | 0.000288 | 0.000250 | 0.000261 | 0.000263 | | Optimal Sparsity | 0.88 | 0.89 | 0.97 | 0.995 | 0.99 | 0.995 | 0.995 | 0.92 |

Weakness 1 : Analysis of MSA

We thank the reviewer for these valuable comments. We address the two points as follows:

1. Empirical upper bound

The results can be found in the answer to Question 2. These results show that, at the model level, the empirical upper bound is not solely determined by variance; it also depends on task difficulty. We will include this analysis and figure in the revised version.

2. Theoretical analysis

We initially adopted an entropy-based discussion because entropy is a well-established proxy for the information content. It provides a simple, intuitive link between weight variance and the amount of information in each layer.

We fully agree that rate–distortion theory is a more appropriate framework under lossy compression settings like pruning, as it explicitly connects variance to information rate and distortion. A simplified version of the proof is provided in Reviewer 8YfX’s Question 2 (omitted here due to space limits), and we will include a more detailed proof in the revised version of the paper.

Thanks to the reviewer again for helping us strengthen the analysis of MSA.

Weakness 2 : Deployment Efficiency

1. We thank the reviewer for highlighting the importance of deployment efficiency, which is indeed a primary objective of many delta compression works such as DeltaZip.
2. We acknowledge that UltraDelta is not designed to directly optimize deployment efficiency, and achieving real inference acceleration typically requires specialized kernels or structured sparsity. Our focus in this paper is orthogonal: we aim to explore data-free delta compression methods that maximize information preservation, providing a foundation that can potentially complement future system-level optimizations.
3. Following the reviewer’s suggestion, we will explicitly state this limitation in the revised version and clarify in the introduction that works such as DeltaZip primarily target deployment efficiency.

Weakness 3 : Controlled Experiment

We sincerely thank the reviewer for this insightful comment. To investigate this, we conducted additional controlled experiments on two datasets, SUN397 and Cars, using a pretrained ViT-B/32 model. We trained the models with varying fine-tuning steps and then applied UltraDelta. The results show that:

1. Underfitted models (early training steps, higher loss) consistently benefit from UltraDelta, exhibiting noticeable accuracy improvements compared to the baseline.
2. Well-fitted models (later steps, lower loss) do not show significant performance gains after compression. In SUN397, we observe only a small improvement after compression, possibly because the task is more complex and data-rich, making it harder for the model to fully fit the dataset.
3. The observed effect mainly reflects that UltraDelta benefits models that are underfitted, while well-trained models already capture most of the task information, leaving less room for improvement.

This explains why performance improvements are more commonly observed on LLMs, as they are typically evaluated in a zero-shot setting without task-specific fine-tuning, making them more likely to benefit from the regularization effect of compression.

Weakness 4 : Hyper-parameter Selection

Our choice of sparsity levels is guided by prior findings in delta compression, which show that models can typically have around 90% sparsity without severe accuracy degradation. Based on this, we explored a coarse set of candidate values (90%, 95%, 99%, 99.9%) to establish a reasonable range, and then made minor adjustments for specific models when accuracy changed a lot within this range.

Weakness 5 : More Challenging Tasks

We expanded the LLM evaluations to include more challenging tasks: MATH (with some AIME problems), GPQA, MBPP+, and HumanEval+, using updated models Qwen2.5-7B-Instruct and Llama-3.1-Tulu-3-8B-SFT. Results are shown below:

We sincerely thank the reviewer for the constructive feedback and positive comments on our work, including: (i) Extensive experiments; (ii) Clear and well-written; (iii) Addresses prior weaknesses; (iv) Strong theoretical and empirical validation. We are especially grateful for the kind recognition of Figure 2, described as “a work of art”. As our method consists of multiple components, we invested great effort in designing this figure to help readers quickly grasp our approach, and we are delighted that this effort was appreciated. We now address the reviewer’s comments in detail below.

Question 1 : Explanation for maintaining relative proportions across values

We thank the reviewer for the question. Below, we explain why maintaining relative proportions across different quantized values is important, supported by theoretical, methodological, and empirical perspectives.

1. Theoretical perspective

• Our goal in maintaining the relative proportions of different quantized values is to keep the weight distribution before and after pruning as similar as possible. This can be formalized using KL divergence between the original quantized distribution $P = \\{p_i\\}$ and the pruned distribution $Q = \\{q_i\\}$: $$D_{\\mathrm{KL}}(P \\parallel Q) = \\sum_{i} p_i \\log \\frac{p_i}{q_i}$$

where $p_i$ and $q_i$ denote the proportion of the $i$-th quantized value before and after pruning. When the relative proportions of quantized values are better maintained, the pruned distribution $Q$ stays closer to the original quantized distribution $P$, resulting in a lower $D_{\\mathrm{KL}}$. A lower $D_{\\mathrm{KL}}$ means the weight distribution is better preserved.

• Preserving the weight distribution is important because it reflects the statistical structure of the model parameters. Weights are not just arbitrary numbers; their distribution encodes structural and semantic information learned during training. Preserving distribution helps maintain model behavior after compression.
• Furthermore, Figure 4 provides empirical evidence supporting this claim: it shows that UltraDelta achieves higher cosine similarity between the compressed model’s outputs and those of the original model, showing that preserving weight proportions helps retain the semantic information of the original model.

2. Method comparison

• DAC (ours) uses group-wise pruning to explicitly preserve the relative proportions of quantized values, maintaining the overall weight distribution. This approach is highly robust, avoiding the randomness that could distort proportions under high sparsity.
• Magnitude pruning removes weights with smaller magnitudes, shifting the distribution toward large values. This ignores the balancing effect of small weights. The imbalance weakens parameter interactions and leading to accuracy degradation.
• Uniform pruning removes weights randomly and can partially keep the distribution. However, under high sparsity it may still accidentally distort the distribution, showing lower robustness compared to methods that explicitly preserve weight proportions.

3. Experiment validation

• As shown in our experiments, under high sparsity, magnitude pruning performs worse than almost all other methods, showing that disrupting the weight distribution can cause severe performance degradation. | Method | LLM-7B | LLM-13B | T5-Base | ViT-Large | | --------------------- | --------- | --------- | --------- | --------- | | Magnitude Pruning | 14.19 | 7.40 | 56.08 | 11.20 | | UltraDelta (Ours) | 45.57 | 52.05 | 86.74 | 94.40 |

Note: These results are extracted from our Experiment section. Both magnitude pruning and UltraDelta are under the same compression ratios.

• Our ablation study (Figure 5 and Table 9) compare DAC against uniform/random pruning across 30 ViTs compression settings. In such large-scale compression scenarios, robustness is particularly important. The results show that DAC is significantly more robust than uniform pruning, confirming the effectiveness of preserving weight distribution. | Method | 4-bit | 8-bit | No quantization | | -------------------- | -------- | --------- | --------------- | | DARE | 81.58 | 83.85 | 85.88 | | UltraDelta (DAC) | 86.40 | 90.18 | 92.45 |

Note: This table is extracted from the experimental results reported in our paper (Table 9, visualization in Figure 5), comparing DARE[1] and DAC under different quantization settings (4-bit, 8-bit, and no quantization).

Weakness 1 : Inference Overhead

1. Our statement was in comparison with other delta compression methods, which also require adding delta weights at inference time. Our method only needs a simple addition, while methods like Delta-CoMe[2] involve matrix multiplications to project low-rank matrices back into the full parameter space before inference, which are more expensive.

2. The addition can be done on CPU before inference, so it does not increase GPU memory usage and introduces minimal compute overhead. For example, we measured total inference time on WizardCoder-Python-7B-V1.0 evaluated on HumanEval+. We compared the fine-tuned model, UltraDelta (addition on CPU), and Delta-CoMe (layer-wise matrix multiplication on CPU): | Method | Fine-tuned | UltraDelta | Delta-CoMe | | ------- | ---------- | ---------- | ---------- | | Runtime | 217s | 221s | 341s |

3. We will clarify this in the revised version and add inference time comparisons. Thank you for helping us improve the precision of our claims.

[1] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

[2] Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models