Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data

We thank the reviewer for their helpful feedback.

Can you compare the compressors in terms of the FLOPS required? Is it possible that transformers are an order of magnitude more expensive?

Computing the FLOPS for standard compressors (e.g., gzip or PNG) is much more involved than for neural networks and requires intricate knowledge of the algorithm, which can consist of more than 100K lines of code (also, note that standard compression algorithms have been highly optimized and run on CPU, whereas the neural compressors need a GPU). Instead, we compared the running times for all the compressors in Table A2. As the reviewer correctly suspected, neural compressors (i.e., both ours and those of Bellard (2021)) are orders of magnitude slower than standard compressors. Note that this does not undermine the relevance of our results, since we clearly stated that “the goal of our study is not to build a practical transformer-based universal compressor to compete with standard compressors in terms of computational footprint” in the limitations paragraph of Section 6.

What is surprising about the results of this work?

The most surprising finding is that small pre-trained transformers manage to compete with, and even outperform, standard domain-general and domain-specific compression algorithms in the small(-ish) data regime. This result does not trivially follow from previous work (Delétang et al., 2024), which shows that large models are strong compressors without accounting for their parameter count (i.e., they have low log-loss). Bellard's work provides perhaps the closest indication that our results may be achievable, but it operates in a very different paradigm (online training on the data to compress, whereas we pre-train and evaluate on out-of-distribution data) with a larger transformer. It could have equally been the case that small models (in our range of 20M parameters or less) do not manage to capture sufficient statistical structure to compete with algorithms designed to optimally exploit certain types of statistical structure.

The other surprising aspect is that we see very little transfer to unseen modalities, even when training on multiple modalities, in contrast to what was observed with large-scale transformers (Delétang et al., 2024). One could imagine that small models learn very simple statistical patterns that are likely to transfer to other domains, whereas large models, trained only on text, learn complex and intricate domain-specific statistical patterns that do not generalize. However, the opposite seems to be true, and finding out whether this is mainly a question of model scale or dataset composition is an interesting future research direction. Our work aims to study transformers' domain-general inductive biases, and it seems that they are data- and scale-dependent (this, of course, is not an entirely novel finding, but our work adds an interesting and novel angle to this question).

Why does the compression ratio increase with increasing model size for audio in Figure 5 but not for images and text? Why would it ever increase with model size?

Figure 5 shows that for audio the compression ratio increases with increasing model size, but only if the context size is too small. We believe that this is because larger models overfit to the limited information available in short contexts, i.e., there are not enough complex dependencies that a large model could exploit. In contrast, text has a much higher information density than raw audio data, which is why Figure 5 shows that the compression ratio decreases with increasing model size even for the smallest context sizes we investigated. Note that, for large enough context sizes (i.e., 8192), the compression ratio on audio data decreases with increasing model size (for the sizes we investigated).

We thank the reviewer for their detailed and positive comments.

What would have happened if we were to tweak the transformer with something like Perceivers that also operate on byte level? Would it help? Why or why not?

As Tvj5 correctly points out, raw (i.e., untokenized) data typically results in very long sequences (with lower information density). Thus, we would expect architectures that are optimized for that setting, such as the Perceiver (Jaegle et al., 2021) or Structured State Space Models (Gu et al., 2022), to improve the compression performance as they can exploit long-range dependencies more efficiently. However, note that, as stated in Section 6, our goal was “not to build a practical transformer-based universal compressor” but instead to “study transformers’ learned inductive biases, e.g., whether they are domain-general, how they depend on the training data composition, and whether there is transfer between modalities” (see Section 1). We therefore did not investigate other model architectures, but we consider this a very promising avenue for future research, and we believe that alternative architectures that outperform vanilla transformers on our benchmark may have very interesting general inductive biases. We have updated the limitations section of our revised manuscript accordingly.

Would the current results hold if we were to replace the input in all of the modalities by their respective token-level representation. Which domains are likely to be improved the most?

We refer to our general response for a discussion on tokenization as pre-compression and note that there is no canonical token-level representation for a modality and that finding the optimal tokenizer for a single model/dataset combination is often the result of years of research, which is exactly why we keep our models domain-general by processing raw bytes.

In the case of text data, Delétang et al. (2024) showed that the optimal tokenizer (raw bytes vs. byte pair encoding with vocabularies ranging from 1K to 20K) depends on the model size and that the compression ratios only improve by at most 3.6% (see Table 5 in their paper). Thus, we can safely assume that for text data the current results also hold if we were to replace the inputs with their respective token-level representations — with an error margin of a few percentage points. For other data modalities, the effects of tokenization for the purposes of lossless compression are unclear and an interesting direction for future research.

Why do the compression ratios in Figure 3 not improve with training dataset size (as in Figure 4)?

The compression ratios in Figure 3 do improve with increasing training dataset size (as in Figure 4), however, often not significantly. We hypothesize that this is because even the smallest training dataset is already sufficiently large to achieve competitive compression ratios and larger dataset sizes only marginally improve the compression because our models are too small to make good use of more data (larger models would not be competitive in terms of parameter-adjusted compression ratio).

Note that all points of the same color correspond to the same amount of total training data (33GB, 66GB, 99GB, 132GB, and 165GB), and, for example, the left-most gray data point corresponds to 11GB of audio, 11GB of images, and 11GB of text while the left-most red data point corresponds to 33GB of text).

Could you shed light on how compressing 1028 or 8192 bytes for audio is far simpler than for text and how it affects the compression ratios?

Yes, Figure 5, where we ablate the model- and context size, sheds light on this phenomenon: For text, given a certain FLOPS budget, we always achieve the best performance with the largest model, i.e., the shortest context size (1024), which means that the model’s pattern-recognition capability (i.e., its size) is more important than being able to exploit long-range dependencies. For audio, we observe the opposite – given sufficient FLOPS, the best performance is achieved by the largest context (8192), i.e., the smallest model. We hypothesize that this is because the information density for raw audio is lower than for text (as Tvj5 pointed out). Note that the best theoretically possible compression ratio depends on the redundancy of the raw data vs. the Shannon entropy of the data (the minimally achievable size). While the gap between these two determines the maximal compression ratio, it does not necessarily say how difficult it is to achieve this ratio.

We thank the reviewer for their careful study of our paper and their insightful feedback.

Did you explore test set sizes beyond 1GB or consider varying test data sizes? What is the rationale behind taking test data size as 1GB?

We refer to our general response for a discussion of the test set size and discuss the additional ablation we conducted (Figure 6 in the revised manuscript) here:

We agree that it is interesting to investigate varying the test data size, which is why we conducted an ablation for our best-performing models in Figure 6 of the revised manuscript. The figure clearly shows that for offline (i.e., pre-) trained neural compressors, for which the model parameters have to be factored into the compression ratio, the compression performance improves with increasing evaluation data (as long as the model generalizes well to the additional data). In contrast, the size of standard compressors is negligible compared to the amount of evaluation data, which means that their compression ratios are largely unaffected by the evaluation dataset size. Note that FLAC cannot losslessly compress more than 4.2GB of data.

Did you try transfer learning approaches to address the limited cross-modal transfer?

No, we did not investigate transfer learning as our goal was to “study transformers’ learned inductive biases, e.g., whether they are domain-general, how they depend on the training data composition, and whether there is transfer between modalities” (see Section 1), and, therefore, we did not deviate from the (tried-and-tested) standard supervised learning setup. Nevertheless, as you point out, it seems reasonable to suspect that transfer learning could potentially lead to considerable performance improvements. We have mentioned this as an interesting follow-up area in the limitations section of our revised manuscript.

We thank the reviewer for their constructive feedback.

How did you represent the multi-modal data? How did you flatten the images?

In Section 4 we stated that “we do not use tokenization and instead feed our models directly with byte streams” and that “Appendix A.1 describes the datasets in full detail”. In Appendix A.1 we then elaborated on the data representation for each modality:

• Audio: “We downsample the dataset from 48 kHz to 16 kHz and encode the waveform as int16, i.e., with two bytes per sample.”
• Images: “We decode the images into RGB arrays (three uint8 channels), flatten them (using tf.reshape), and concatenate them into a byte stream of flattened images.”.
• Text: “UTF-8 encoded text”

We are happy to include any other details the reviewers consider important.

Can you provide more details about training and evaluation? How long did you train the model? On what sequence length? What was the sequence during evaluation?

As stated in Section 4: “Unless otherwise noted, we use […] a context size of 4096 (bytes), and sliding windows without overlap or memory (full details in Appendix A.3). We train our models with the Adam optimizer (Kingma & Ba, 2015) for 2.5 million steps with a batch size of 32, which, for 165GB of data, roughly corresponds to 2 epochs.” We clarified that we always train and evaluate with the same sequence length in the revised manuscript.

We are happy to include any other details the reviewer may have in mind.

Raw data can result in extremely large sequences. For example, music data is sampled at 48kHz, so only 10 seconds of music corresponds to 480K tokens.

Indeed, tokenization is a form of pre-compression (Delétang et al., 2024; Lester et al., 2024), mainly aimed at increasing the information density of the context window (at the cost of increased entropy), but the flip side is that raw data offers excellent potential for compression.

In lossless compression, the ultimate limit is the Shannon entropy of the data source. Accordingly, if the raw data is much larger than this limit, it must contain a lot of redundancy. Whether a particular model can exploit this redundancy is a separate question (and, indeed, many domain-specific tokenizers target exactly this “trivial” redundancy), but large (w.r.t. redundancy) raw sequences are not a problem per se. In our setting, with a limited context window and alphabet (bytes as tokens), we can fit less information into the context, limiting our capability of exploiting long-range dependencies in the data (which is why large language models benefit from tokenization and larger alphabets). At the same time, highly pre-compressed data makes learning to predict/compress harder, and compression gains are generally smaller (Lester et al., 2024). In practice, this leads to an interesting and non-trivial design space where we can explore model size, tokenization, and data modality. We did not engage with this problem to avoid complicating the main findings and takeaways, and we refer to Delétang et al. (2024) for an analysis of the interplay between tokenization and compression.

Also, note that we (sub-) sample our audio data at 16kHz, not 48kHz (see Figure 1 and Appendix A.1).