We thank the reviewer for their time and detailed feedback. We are also encouraged by the recognition of our novelty and creativity. We now address the comments and share our vision for further efficiency improvement below.

---

Computation overhead and potential mitigation

We have discussed several potential strategies in the general response. In summary, there are three possible directions: advanced implementation, efficient LLMs, and hardware acceleration.

• Advanced Implementation: This improves our current research-oriented single-thread Python implementation.
• Efficient LLMs: This is an ongoing research direction aimed at developing more lightweight LLMs.
• Hardware Acceleration: This leverages the zero-shot property and is intended for production use.

---

Hyper-parameter selection

Interestingly, our findings suggest that most hyperparameters align with human intuitions. For example, 4 bytes per group resemble floating point structures and yield the best performance, while hexadecimal encoding outperforms iso-8859-1 due to the more common characters used. Other parameters generally adhere to the scaling law, where larger models or context windows result in better performance, creating a trade-off between performance and available resources. These factors significantly reduce the complexity of hyperparameter search.

---

Implementation efforts

Our implementation can be easily reproduced using common open-sourced packages like PyTorch, Huggingface, and TorchAC, while serialization can be done using pure Python string operations. We will release the source code upon publication.

---

Generalizability

Our approach performs well on common benchmarks, including MNIST, CIFAR-10, and TinyImageNet, and on architectures like ConvNet, VGG, ResNet, and ViT for image classification. Further analysis may be needed for other tasks, such as generative models, particularly if different implicit biases are present. To the best of our knowledge, this remains an open question, and we will leave this analysis for future work.

---

How scalable is the proposed method for extremely large models or distributed training setups?

We currently target scenarios like federated/distributed learning, where gradients are compressed before sending to the server. We tested on models up to 6M parameters, a common model size for edge-device scenarios. Though the typical parameter size of LoRA fine-tuning is also around 6M, further analysis would be required on those extremely large models.

---

How sensitive is the performance of LM-GC to different serialization strategies?

Table 1 shows that LLMs are sensitive to serialization strategies. For instance, hexadecimal numbers are up to 70% better than the extended ASCII set (iso-8859-1), and separators play a critical role in the reasoning ability of LLMs (40% difference for LLAMA 2).

---

Could there be more optimal serialization methods not explored in this paper?

Yes, there may exist better strategies. This could be similar to the existing multi-modal LLM problems, where the key challenge is to project a new modality to the token space. While we propose an easy-to-go serialization method, understanding how and why LLMs can understand gradients deserves further research.

We thank the reviewer for their time and constructive feedback. We especially appreciate the recognition of our novelty and our efforts to address this timely question. We are pleased to share additional insights below.

---

Theoretical properties

Thanks for the suggestion. We agree that theoretical analysis could be insightful. The fundamental theories behind entropy coding have been well-studied in the existing literature. On the other hand, understanding the behavior of LLMs is a non-trivial ongoing research direction. We will leave it to future work at this point.

---

Clarification on the question

We are not entirely clear on the question. We would be more than happy to address it during the discussion phase if the reviewer could kindly provide further clarification.

---

When is LLM not a good tool for gradient compression?

The main limitation of our approach is its high computational resource demand, making it less suitable for weaker devices such as IoT devices or real-time systems. However, this constraint could be significantly improved with the techniques discussed in our general response, enhancing scalability. Furthermore, our method does not require any training, making it especially well-suited for hardware acceleration in production deployments.

---

Under what circumstances do they fail to present the prior?

Our approach performs well on common benchmarks, including MNIST, CIFAR-10, and TinyImageNet, and on architectures like ConvNet, VGG, ResNet, and ViT for image classification. Further analysis may be needed for other tasks, such as generative models, particularly if different implicit biases are present. To the best of our knowledge, this remains an open question, and we will leave this analysis for future work.

We thank the reviewer for their time and insightful feedback. We hope our response addresses the concerns and conveys the broader vision of our work.

We kindly emphasize that our primary contribution is demonstrating the potential of pre-trained zero-shot large language models (LLMs) as effective gradient priors through the lens of lossless compression, a point also supported by R-UCGG and R-Br5V. As motivated in L21-23, the lack of proper prior models has been a critical barrier for applications such as denoising or upsampling in the domain of gradients. Our work is the first to demonstrate this potential and explore its application in lossless gradient compression. We believe our findings will inspire further research beyond just arithmetic coding. The potential impact also outweighs the temporal computation overhead as we have discussed in the general response.

We now address the concerns regarding the lossless gradient compression task below.

---

Justification of using LLMs.

We first emphasize that traditional baselines struggle to compress gradients effectively in complex scenarios. As illustrated in Tables 2 and 3, our method exceeds the best baseline by 20% on ViT and 21% on TinyImageNet. This highlights the difficulty of modeling complex gradients and justifies the usage of our LLM prior. The following experiments recommended by the reviewer further stress the point.

---

Missing baselines.

We have compared our method to LZMA, one of the most robust general-purpose compression methods, commonly used in the 7z compression format. Additionally, it is important to note that none of the suggested baselines are designed for gradients or high-dimensional data. However, to further highlight our contribution, we have conducted additional experiments as follows.

---

Comparison to simple adaptive coding

We additionally compare our method to run-length encoding (RLE). This experiment extends from Table 3, compressing gradients collected during training a ConvNet on TinyImageNet. For RLE, we consider three types of dictionaries: binary, hexadecimal ($H_n$, Table 1), and iso-8859-1 (extended ASCII to handle negative numbers). These methods use 1, 4, and 8 bits to represent symbols and always use 8 bits for counting. Note that this setting is favorable to RLE since gradient lengths can easily exceed 256 (8 bits).

The results are presented in the following table. RLE failed to compress the data even with different codebooks, and our method clearly outperforms RLE, indicating that simple adaptive priors are ineffective for gradients.

---

Comparison to methods dedicated to floating-point compression

While we acknowledge that the suggested baselines are intended for scientific floating points (1D to 4D, which are relatively low-dimensional compared to gradients [1]), we have extended the comparison in Table 2 and 3 to include floating-point-specific compression. Specifically, we compare our method to FPZIP [1], as we could not find any available implementation of [2]. The results, shown in Table A1 and A2 in the attached PDF, indicate that FPZIP performs comparably to LZMA. On the other hand, our LM-GC outperforms FPZIP by up to 17.2% across three datasets and around 20% across four architectures. This highlights the challenge of using heuristically designed priors and demonstrates the effectiveness of the LLM priors adopted in our method.

---

Practicability of LM-GC

As previously discussed, the effectiveness of LLMs in handling complex gradients justifies their usage. We acknowledge that LM-GC requires more resources than traditional codecs. However, as detailed in the general response, significant acceleration can be achieved by implementing C++ and multi-threading. Additionally, ongoing research into more efficient LLMs suggests a promising future for further improvements. Notably, since our method does not require training, hardware can frequently achieve substantial acceleration for inference.

---

What does 'plain' compression entail here? Is it using a codebook with arithmetic coding?

Plain means without any lossless compression. We will revise the text in the next version to avoid confusion.

---

We hope our response has taken care of your concerns and encouraged you to reconsider the score. We would be happy to answer any other questions or concerns during the discussion phase.

[1] Fast and Efficient Compression of Floating-Point Data https://computing.llnl.gov/projects/fpzip
[2] Fast lossless compression of scientific floating-point data https://ieeexplore.ieee.org/document/1607248

We thank the reviewer for their time and feedback and are encouraged that our work was found interesting. We will now address the comments and share further vision of our work.

---

The benefit of using LLMs over traditional approaches.

We highlight that traditional baselines struggle to compress gradients effectively in complex scenarios. As illustrated in Tables 2 and 3, our method exceeds the best baseline by 20% on ViT and 21% on TinyImageNet. This highlights the difficulty of modeling complex gradients and showcases the effectiveness of our LLM prior. Moreover, the baselines often fail to compress data adequately in some scenarios. For instance, LZMA, the best baseline, achieves only a 91% compression rate. These advantages demonstrate the benefits of our method, making it particularly favorable when communication cost or storage is the main system bottleneck.

---

Energy consumption

We provide the energy consumption estimation in Figure A2 of the attached PDF. The consumption is estimated by the following equation:

$E = (\text{CPU power consumption} \times \text{CPU utilization} + \text{GPU power consumption} \times \text{GPU utilization}) \times \text{run time}$

Indeed, LM-GC currently consumes more energy. However, as discussed in the general response, we remain positive about future efficiency improvements. Moreover, in complex scenarios, the baselines often struggle to compress the data effectively (70.98% vs. 87.98% in Tables 2 and 71.90% vs. 91.6% in Table 3), which justifies our method's investment in computational resources.

---

Can the compressed gradient be utilized in practice?

Yes. An immediate application is federated learning, where clients accumulate gradients locally and communicate periodically with the central server. One of the main bottlenecks is the communication cost as the number of clients and model size grow. Assuming both sender and receiver have access to the same LLM, LM-GC (optionally with lossy compression, as shown in Figure 4) provides superior compression rates on gradients over the existing coding schemes (Table 1-3) and thus enables better scalability.

---

Broader impact of our findings

In addition to LM-GC, our work highlights the potential of using large language models (LLMs) as gradient priors, which we find exciting and believe will inspire further applications. We propose several possible directions for exploration, which could be investigated in fully zero-shot, parameter-efficient fine-tuning (PEFT), or prompting settings, given the strong starting point provided by zero-shot LLMs.

• Inspiration from existing works: With a robust prior $p(t_k|`BOS`, t_{<k})$ over gradient tokens, it is possible to draw inspiration from existing work of different modalities. For example, one could utilize the prior model to de-noise gradients by checking the likelihood, akin to image denoising with generative priors, or explore gradient restoration from quantized low bits to higher bits, similar to image super-resolution. In the latter scenario, the lower bits provide partial information or context for the LLMs.

• Advanced lossy compression: Most of the existing lossy compression in federated learning assumes no prior knowledge exists between servers and clients or leverages heuristically designed protocols. In contrast, one can assume both sides have access to a fixed LLM. When certain tokens consistently exhibit high probability, they can be sampled directly from the LLMs, allowing more bits to be allocated to other tokens.

• Security: By modeling benign gradients, potentially through PEFT or prompting, one could develop a gradient-based method to guard against backdoor attacks, akin to image anomaly detection using generative adversarial networks (GANs).