FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models

We thank the Reviewer eKhr for the detailed and useful feedbacks. We address your concerns point by point below, and the analyses will be incorporated into our paper.

Q1: Could the author please elaborate on specific scenarios where this method can be applied? What kind of situations require frequent downloading of finetuned models from the network?

A1: Regarding application scenarios, as shown in Figure 1 and Table 1 of the original paper, we focus on a new storage issue on cloud resulting from the rapid development of LLMs. Our method aims to compress massive finetuned large models stored on the cloud platforms like HugggingFace, which currently hosts over 800,000 models, leading to substantial storage costs.

Regarding the situation of frequently downloading finetuned models, as a public model storage platform, HuggingFace enables numerous end-users to download the models they need from network for further testing.

It is important to note that our primary concern is to reduce the cloud storage overhead rather than downloading frequency. In fact, a large portion of models (89% as reported in Table 1) in cloud are inactive, with fewer than 10 downloads per month. If applying our method on these models, the total storage costs can be saved at least 40% as discussed in Appendix D.7.

Q2: Has the author considered comparing the proposed method in this paper with quantization? For example, quantizing delta. What is the necessity of performing lossless compression in this setting?

A2: Regarding quantizing delta, we select two user-uploaded GPT2 models from Huggingface, and present the quantization results in the Table 1 of the uploaded PDF.

We can see from the quantization results that lossy compression inherently alters the model eval results. We believe lossy compression goes against the original uploader's desire to store his model safely (unchanged), even though the change might be small (actually it is hard to ensure the performance of all models in hub due to diversity).

Just like when we upload a model to HuggingFace today, we don't want the case that we download it and then find any of its eval results inconsistent with the original model. At this point, FM-Delta is exactly lossless and has a rather competitive compression rate.

Q3: Some technical terms need to be explained in detail, such as "most significant bit" mentioned in 4.1 and "range" in "difference range" mentioned in 4.2. The subsequent method descriptions are also somewhat confusing.

A3: We explain the mentioned terms in the following.

• "most significant bit": The most significant bit (MSB) is the bit in a binary number that has the highest value position.
• "difference range": Range of differences between fine-tuned and pre-trained model parameters.

We also improve the method description with a more detailed workflow figure. Please check Figure 1 of the uploaded PDF.

Q4: Why use range coding instead of other entropy coding methods? I noticed the experiments used tools like Gzip, but currently, methods like zstd are more popular. Could the author consider trying zstd compression?

A4: We use range coding since it can dynamically adapt to the probability distribution of the data and is suitable for compressing floating-points requiring the fine granularity and precision.

Regarding trying ZSTD, we present its results in the above table. We see that while ZSTD has a higher throughput, its compression rate is worse than FM-Delta.

We further present a hybrid approach ZSTD-Delta that combines the mapping of FM-Delta with ZSTD, i.e., applying ZSTD on the mapped unsigned int delta. ZSTD-Delta serves as a compromise in practice with the superiority of both FM-Delta (compression rate) and ZSTD (throughput).

Q5: I noticed that the reported compression throughput in the final experiments is approximately 100MB/s, which is not particularly fast in practical terms. There seems to be significant room for improvement in this speed.

A5: The trade-off between speed and compression rate is indeed a critical consideration in practice. As mentioned in Answer A4, we provide a hybrid approach ZSTD-Delta that combines the mapping of FM-Delta with ZSTD as a compromise for cloud customers.

We believe that although the throughput of FM-Delta is not the highest, its best compression rate and cost saving of 40% are still valuable in practice. We will strive for greater improvements in lossless coding in our future work.

Q6: Some related works are not mentioned, such as [1] [2].

A6: Thanks for your kindful suggestion. The authors in [1] investigate the compressibility of quantized LLMs, and FM-Delta focuses on the lossless compression on floating-point LLMs. The authors in [2] propose a byte grouping method and apply the stardard compressors like zstd to compress float models, while its compression rate is not as good as FM-Delta, and the actual compression speed is not reported.

These two works are contemporaneous with ours. We add discussion about them in the Related Work section of our final manuscript.

We thank the Reviewer SDxT for the insightful feedback and address your concerns below, and the analyses and clarifications will be incorporated into our paper.

Q1: Did the paper use the symbol frequency as the probability?

A1: Yes, we use a quasi-static probability modeler as in [1][2] . Initially, each symbol is assigned an equal frequency. As we encode or decode the data, we update the symbol frequencies dynamically based on the processed data. This approach ensures that the frequency table is dynamically built during the encoding and decoding processes, without the need for transmission of the probability table.

Regarding range coding, we update the workflow figure of FM-Delta in the uploaded PDF, involving the illustration of range coding, which includes:

• Symbolization. We regard the sign $s$ and the most significant bit $k$ of delta as symbols $<s,k>$ for range coding.
• Probability model. As mentioned above, we use a quasi-static probability modeler to termly update symbol frequencies.
• Encoding. Range coding encodes the symbols and the raw bits on all delta elements through range scaling, leading to the compressed fine-tuned model.
• Decoding. Range coding maps the encoded value back to the original symbol range and termly updates the probability model. Then we get the original float-point fine-tuned model through reverse-mapping delta.

[1] M. Schindler. Range Encoder version 1.3, 2000. URL http://www.compressconsult.com/rangecoder/.

[2] Peter Lindstrom, et al. Fast and Efficient Compression of Floating-Point Data. TVCG 2006.

Q2: Following the previous question, it would be interesting to see how the probability estimation affects the compression ratio. For example, in Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024 [3], a Gaussian Entropy Model is proposed to model the distribution of neural network weight for entropy coding, where only two scalar values are transmitted for each weight or embedding.

A2: Thanks for your valuable reference. The authors in [3] build a network-free Gaussian model for the probability estimation of the quantized weights in INR, with tiny metadata transmission overhead. In comparison, FM-Delta losslessly compresses diverse floating-point fine-tuned models using a quasi-static probability modeler without transmission of the probability table.

Both methods focus on the compression of models. The example[3] inspires us to further investigate model characteristics for more fine-grained probability estimation in the future work. We add discussion about [3] in the Related Work section of our final manuscript.

[3] Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024

We thank the Reviewer yfTq for the insightful feedback and address your concern below, and the analyses will be incorporated into our paper..

Q1: What is the proportion of full fine-tuned models within the entire Hugging Face model repository?

A1: Since it is hard to distinguish the full finetuned models from all the pre-trained models in HuggingFace based on model meta information, we count the total number of PEFT models and its proportion in HuggingFace as shown below.

It should be noted that "All Models" includes PEFT, full fine-tuned and pretrained models. We can see that PEFT models only occupy a small proportion of all models.

For your further review, we provide the statistical results of the following additional model families, and show the proportion of full models on these families.

It can be seen that the number of full fine-tuned models still occupy the majority (71% on average) in the HuggingFace repository. Moreover, given their full size, the storage space required for these models takes up a significant portion of cloud storage.