Language Modeling Is Compression

We thank the reviewer for their time and constructive feedback on the paper.

Can you clarify Figure 1?

Yes, we have extended the caption of the figure to further clarify arithmetic coding and included the step-by-step example below in Appendix A.

Figure 1. visualizes arithmetic encoding, which iteratively partitions the interval $I = [0, 1)$ according to a predictive model $P$ and an input string, i.e., $AIXI$ for Figure 1.

First, we construct the intervals for $P(\cdot)$:

• $[0, 0.45)$ for $P(A) = 0.45$
• $[0.45, 0.75)$ for $P(I) = 0.3$
• $[0.75, 1)$ for $P(X) = 0.25$

Since the first token is $A$, we set $I = [0, 0.45)$ and iterate. Thus, the intervals for $P(\cdot | A)$ are:

• $[0.2 * 0, 0.2 * 0.45) = [0, 0.09)$ for $P(A | A) = 0.2$
• $[0.09, (0.2 + 0.6) * 0.45) = [0.09, 0.36)$ for $P(I | A) = 0.3$
• $[0.36, (0.2 + 0.6 + 0.2) * 0.45) = [0.36, 0.45)$ for $P(X | A) = 0.2$

Since the next token is $I$, we set $I = [0, 0.45)$ and so on. We terminate with $I = [0.322, 0341)$ for $AIXI$. Next, arithmetic encoding will compute the binary sequence corresponding to iteratively splitting the interval $[0, 1)$ in half until it is fully contained in $I$.

Concretely, this yields the binary sequence.

• $b0$ -> $[0, 0.5)$
• $b01$ -> $[0.25, 0.5)$
• $b010$ -> $[0.25, 0.375)$
• $b0101$ -> $[0.3125, 0.375)$
• $b01010$ -> $[0.3125, 0.34375)$
• $b010101$ -> $[0.328125, 0.34375)$
• $b0101010$ -> $[0.328125, 0.3359375)$

As $[0.328125, 0.3359375)$ is fully contained in $I = [0.322, 0341)$, the compressed output is $0101010$, which consists of 7 bits as opposed to the 4 bytes used to encode $AIXI$.

What is the motivation for the section on tokenization?

The motivation for the tokenization section is precisely what the reviewer points out: Tokenization is rooted in language modeling and has been developed to overcome practical limitations (e.g., small context windows). Here, we describe that a tokenizer can be viewed as an additional compressor that is applied before the language model. Note that we do not state that "Transformers Are Compression" but that "Language Modeling Is Compression". In that sense, a CNN trained to predict the next token with the cross-entropy loss will also yield a lossless compressor. Here, we only consider Transformers since they are predominantly used for language modeling research.

Can you explain the poor generative performance of LLMs on image and audio data?

Yes, recall that the rows in Figures 3 and C.2 are treated independently by the model, which explains why the image generations look rather "noisy" across rows. Moreover, given that Chinchilla was only trained on text, it is not surprising that its generative performance on image and audio are not of high visual quality. Nevertheless, we find it remarkable that such language models generate somewhat meaningful completions for unseen data modalities, which we attribute to the emergent capabilities of pattern recognition at scale. We added a more extensive discussion to the paper (Section 3.4) and also evaluated the generative capabilities of less powerful models (Chinchilla 1B and 7B) to Appendix C (showing that the generative quality improves with predictive performance).

Shouldn’t the title be more specific?

While we agree that the idea of deep learning being compression is not novel, we believe that our paper goes beyond pure image and audio compression as we also discuss scaling laws (i.e., model/dataset size tradeoffs), in-context learning, tokenization, and generation.

Why do you use a context length of only 2048 bytes?

While the reviewer correctly points out that recent work has enabled much longer contexts, we are limited to a context size of 2048 bytes due to the computational resources available to us. However, we do not expect the results to change qualitatively, given that Figure 4 shows that the compression rates plateau after a certain context length.

We thank the reviewer for their time and constructive feedback on the paper.

How does your experimental evaluation differ from prior work?

We thank the reviewer for pointing us to the paper by Ge et al. (2023), which we have added to our manuscript. The findings of Ge et al. (2023) are akin to our investigation of tokenization, which can be viewed as a pre-compression of the input to a language model. That is, Ge et al. (2023) present an alternative to tokenization that solves the practical issue of maximizing the information in the context window of a Transformer, which boosts downstream performance.

However, Ge et al. (2023) do not show that their ICAE method achieves strong lossless compression rates on image and audio data, and the performance on patterned random text is relatively poor (3.5 BLEU) compared to normal text (99.3 BLEU) and almost the same as completely random text (0.2 BLEU). In contrast, we show that LLMs are on-par or even outperforming specialized image and audio compressors (in terms of raw compression rate, i.e., not taking into account the model-size or program complexity).

Moreover, unlike Ge et al. (2023), we not only show that larger LLMs achieve better compression rates but conduct a comprehensive investigation of the scaling behavior across model and dataset size, elucidating the limits of naive scaling.

Thus, as the reviewer pointed out, we still believe that a major novelty of our work is showing that language models trained on text are strong general-purpose compressors on image and audio data and comprehensively investigating the scaling laws for compression. We want to emphasize again that the ability for general-purpose compression implies that LLMs, through training on a large corpus of text, reliably acquire the ability to detect and predict (and thus compress) general algorithmic patterns. This is not trivially expected, and while understanding the precise nature of these patterns and how they relate to the training data remains an exciting avenue for future research, the finding that competitive compression rates on images and audio data can be achieved is not trivially expected per se.

The concept that a language model is a form of compression is not new.

While we agree that viewing language modeling as compression is a well-established insight (see, e.g., Section 1), we believe that it is valuable to the broader ML research community to popularize these ideas (beyond the LLM professionals) and the subsequent connections to essential areas, such as in-context learning, scaling laws, tokenization, and generative modeling. Our goal is thus not to provide new insight in terms of compression methodology or to advocate for using LLMs as compressors in practice, as they have too many limitations (e.g., size and inference time) to be practical. Instead, we aim to provide an interesting experimental evaluation of viewing language modeling through the lens of compression.

We thank the reviewer for their time and constructive feedback on the paper.

What is the relationship between the number of parameters and compression performance?

We investigate the relationship between the number of parameters and compression performance in Figure 2, where we evaluate the adjusted compression rate (i.e., taking the model size into account) on datasets of increasing sizes ($10^7$, $10^8$, and $10^9$ bytes). First, we note that for a fixed dataset size, the compression rate shows a distinctive U-shape, i.e., it improves with model size but starts to deteriorate when the model becomes too big (i.e., overfits). However, we also see that the compression performance improves with model size as the dataset size improves since the compression rate improves faster than the parameter count (i.e., the models show evidence of emergent capabilities with scale).

We thank the reviewer for their time and constructive feedback on the paper.

What is the motivation for this work and what are the implications of the results?

In recent years, language modeling has become a central element of AI research, primarily thanks to transformers' unmatched efficiency and performance (particularly for large foundation models). However, many researchers are unaware of the close connection between prediction and compression power. Thus, our primary motivation is to demonstrate that well-known large language models can be used to compress data. One, arguably even more surprising, finding is that models trained to predict/compress text turn out to become fairly good compressors for data that they haven't been trained on (i.e., image and audio data). This "transfer" can only be explained by LLMs obtaining the ability to predict, and thus compress, general (algorithmic) patterns. As we show further, larger LLMs perform better at general compression than smaller LLMs, suggesting that the ability for predicting more general patterns improves with scale and is thus obtained via training on text (rather than an inherent capability of untrained LLMs). Moreover, we show that studying foundation models through the lens of lossless compression sheds light on various essential aspects of language modeling:

• In-context learning: Compressing longer sequences is easier given the more available information (i.e., few-shot learning or meta-learning).
• Tokenization: In this view, a tokenizer is just an extra (manually conceived) compression step.
• Generation: One can sample from a compressor using a softmax over the lengths of the compressed data. We use this to empirically demonstrate the relation between stronger compressors and higher-quality generations.
• Scaling laws: Model size affects the downstream compression performance. Finally, we show that, surprisingly, language models are strong compressors for image and audio data, which provides evidence for the claim that LLMs learn to compress more general (algorithmic) patterns with scale (Mirchandani et al., 2023).

However, as the reviewer points out correctly, we do not advocate for using LLMs as compressors in practice, as they have too many limitations (e.g., size and inference time) to be practical. Rather, we advocate for using compression as an insightful tool for the evaluation of LLMs, and the compression-viewpoint as an insightful, complementary, theoretical angle.

Suvir Mirchandani et al. Large Language Models as General Pattern Machines. https://arxiv.org/abs/2307.04721.

Can you add experiments with publicly available pre-trained models other than Chinchilla-70B?

Yes, we added an experimental evaluation of the compression capabilities of LLaMA1-7B to Table 1. LLaMA1-7B performs comparably to the Chinchilla models, except for ImageNet data, where it achieves the best compression (most likely since it has seen images as part of the CommonCrawl dump, which makes up 67% of its training data).

Moreover, we have open-sourced the code for reproducibility and will link it to the paper upon acceptance.

Why do the results for the generative modeling performance look rather poor?

First, note that the rows in Figures 3 and C.2 are treated independently by the model, which explains why the image generations look rather "noisy" across rows. Moreover, given that Chinchilla was only trained on text, we find it surprising that the model manages to complete the images somewhat meaningfully in the first place. Nevertheless, we agree that the generations do not look visually appealing due to the well-known mismatch between teacher-forcing at train time and autoregressive generation at test time. Thus, at test time, the errors accumulate, and the compression capabilities need to be immaculate to generate good results for long sequences, which is unlikely for the image and audio data that the LLMs have never seen during training. We added a more extensive discussion to the paper (Section 3.4) and also evaluated the generative capabilities of less powerful models (Chinchilla 1B and 7B) to Appendix B (showing that the generative quality improves with compression performance).

Why is it important to distinguish between compression rates that do and do not consider the model size?

Given a dataset, one can achieve optimal lossless compression (in coding length without considering the model size) by simply storing the dataset and encoding each sequence as an index. We usually refer to this as "overfitting", i.e., the model stores all the sequences in its parameters but fails to generalize to new sequences. However, if we consider the model size, such a "storage" model would achieve a very poor compression rate (since it requires a lot of parameters). Conversely, a model that manages to compress the same dataset with a minimal set of parameters must be able to extract useful information from this data, which is what we commonly refer to as "generalization". Since we ultimately want to measure how well a model generalizes, not how well it stores information, we also measure the compression rate that accounts for the model size. In addition, compression rates that do not consider the model size might wrongly suggest that LLMs are preferable over general-purpose compressors, like gzip, to compress arbitrary files. Taking into account the model size shows that LLMs compression abilities come at the cost of "a complex compression algorithm".

Can you add a concrete example of how arithmetic coding works?

Yes, we visualize arithmetic encoding in Figure 1. Arithmetic coding iteratively partitions the interval $I = [0, 1)$ according to a predictive model $P$ and an input string, i.e., $AIXI$ for Figure 1.

First, we construct the intervals for $P(\cdot)$:

• $[0, 0.45)$ for $P(A) = 0.45$
• $[0.45, 0.75)$ for $P(I) = 0.3$
• $[0.75, 1)$ for $P(X) = 0.25$

Since the first token is $A$, we set $I = [0, 0.45)$ and iterate. Thus, the intervals for $P(\cdot | A)$ are:

• $[0.2 * 0, 0.2 * 0.45) = [0, 0.09)$ for $P(A | A) = 0.2$
• $[0.09, (0.2 + 0.6) * 0.45) = [0.09, 0.36)$ for $P(I | A) = 0.3$
• $[0.36, (0.2 + 0.6 + 0.2) * 0.45) = [0.36, 0.45)$ for $P(X | A) = 0.2$

Since the next token is $I$, we set $I = [0, 0.45)$ and so on. We terminate with $I = [0.322, 0341)$ for $AIXI$. Next, arithmetic encoding will compute the binary sequence corresponding to iteratively splitting the interval $[0, 1)$ in half until it is fully contained in $I$.

Concretely, this yields the binary sequence.

• $b0$ -> $[0, 0.5)$
• $b01$ -> $[0.25, 0.5)$
• $b010$ -> $[0.25, 0.375)$
• $b0101$ -> $[0.3125, 0.375)$
• $b01010$ -> $[0.3125, 0.34375)$
• $b010101$ -> $[0.328125, 0.34375)$
• $b0101010$ -> $[0.328125, 0.3359375)$

As $[0.328125, 0.3359375)$ is fully contained in $I = [0.322, 0341)$, the compressed output is $0101010$, which consists of 7 bits as opposed to the 4 bytes used to encode $AIXI$.

We have extended the caption of the figure to clarify arithmetic coding further and included the above example in Appendix A.