Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data

We thank the reviewer for their careful assessment and constructive feedback. We are pleased that they think that our topic is interesting and an important direction, that our experimental settings are comprehensive, and that our paper is well-written.

Could you also compare to state-of-the-art lossless image codecs such as JPEG-XL?

Yes, thank you for the suggestion! We evaluated JPEG-XL on our image data and obtained a compression ratio of 0.26, which, as the reviewer correctly predicted, is slightly better than our model (and Bellard’s):

We have updated all the tables and figures with these results in our revised manuscript. Note that the point of our paper is not so much that neural networks can outperform any domain-specific compressor but that they can be on par with some of the most widely used general-purpose and domain-specific compressors.

I think that the theoretical contributions are too weak for the ICML audience

Our work is a comprehensive empirical study, which is in scope for and frequently gets published at ICML.

The theory underlying our work (and Delétang et al. (2024)) is standard textbook material in information theory. Accordingly, Delétang et al. (2024), which was published at ICLR 2024, did not introduce this theory. Instead, they provided a concise review of that theory, as we did in Section 2.

What is the contribution compared to prior work?

We investigated the open problem of whether small transformers pre-trained on large multimodal datasets can achieve competitive compression ratios across different modalities and whether there is transfer to unseen modalities (see Section 1), as observed in the large-scale model case. Prior work either considered large-scale, pre-trained models (Delétang et al., 2024) or models trained online (Bellard, 2021). As a result, we investigated a different regime that induces entirely different constraints on the optimal model architecture (in particular, its size) and for which it was a priori unclear whether competitive compression ratios would be achievable across different modalities.

Accordingly, our contributions compared to the previous literature are (see also Table A1):

• Small transformers can achieve competitive compression ratios on audio and image data (Delétang et al. (2024) only showed this for text data), even when comparing against domain-specific compressors, which the reviewer considers an interesting finding.
• Small transformers can achieve competitive compression ratios across multiple domains, pointing towards the generality of relatively small models.
• Unlike the results on large language models by Delétang et al. (2024), small transformers do not achieve transfer outside the training modalities. Moreover, unlike our work, Delétang et al. (2024) did not investigate in-modality out-of-distribution data (e.g., training on ImageNet and evaluating on CelebA) to show in-modality generalization.

How do you process RGB images for lossless compression?

As described in Appendix B.1, we use all three (RGB) channels, each represented as uint8 (unlike Delétang et al. (2024), who compress grayscale images, i.e., use a lossy representation of the original image). We flatten the image and concatenate all flattened images to form our image byte stream from which we iteratively sample chunks where the chunk size depends on the model’s context size (typically only parts of a single image fit into our models’ context, and, occasionally, there may be parts of two different images in the context).

We thank the reviewer for their thorough assessment of our work. We are pleased that they think that our work addresses a relevant open question backed by a convincing set of experiments with methods and evaluation criteria that are appropriate and sound, and that they consider our paper well-written.

We thank the reviewer for their thorough review and constructive feedback. We are pleased that they think our interesting findings will impact the future of compression via LMs, our results are rich with multiple ablation studies, and our paper is clearly written.

How do you ensure that your compression is lossless? Is this by construction?

Yes, it is lossless by construction as we use arithmetic coding (AC), a lossless compression algorithm that uses the next-symbol probabilities of a sequential predictor (i.e., our transformers) to compress (see Figure 1 in Delétang et al. (2024) and Hutter et al. (2024)). For any predictor, AC is optimal in terms of coding length, i.e., the compression performance only depends on the predictor. If the predictor is bad (e.g., an untrained network), the compressed size would be larger than the original — but the data would still be losslessly recoverable.

To decompress, AC must use the same predictor, so we also have to communicate its parameters (we use half precision). The precision used to encode the parameters does not affect the compression's “losslessness”, only the performance.

How can a text-only model losslessly compress images?

As we convert all data into byte streams (no tokenization) and train our models on next-byte prediction, they can process any byte data. Images are simply byte streams with different statistics and can also be fed to models trained only on text bytes. If the model has learned general patterns, it will also predict the image byte stream well (and losslessly compress it well via AC).

What is considered lossless compression (half, single, double precision)?

Our compression is lossless by construction, and the numerical precision depends entirely on the AC, which can be arbitrarily high (at the expense of compression performance). We use the data’s precision, i.e., single precision (we compress bytes). The underlying data may be encoded at higher precision (e.g., float32 images), which is losslessly recoverable from our byte streams.

Do you have a train/validation/test split, or did you cherry-pick the best models?

Our setup differs slightly from the standard train/validation/test splits as we always evaluate OOD data (e.g., train on ImageNet, evaluate on CelebA) to mimic the standard compressor setup. As the reviewer states, Figure 2 is a feasibility result, which is why we show the best performance. Figures 3 to 6 show all the models (the full hyperparameter sweeps), as the reviewer requested.

How are the bytes extracted from the model’s head during training and inference?

AC directly uses the model’s predictions over tokens (i.e., the logits) to encode/decode data (see Figure 1 in Delétang et al. (2024) for an overview). The model computes a distribution over the next byte for every input byte, and the AC then uses these predictions to losslessly compress the data. No separate head or extraction procedure is necessary.

We train via standard log-loss minimization to perform next-byte prediction. At inference time, we perform standard autoregressive evaluation (using teacher forcing).

We have added the above clarifications to our revised manuscript.

Why are LLMs worse at compression when the parameter count is considered?

LLMs are viable compressors depending on whether their parameter count is factored into the compression ratio (i.e., adjusted vs. unadjusted). Imagine that Alice wants to compress and send data to Bob, who wants to decompress it. If Alice trains a neural network to compress the data but does not communicate the model’s weights, Bob cannot decode the data. Thus, the parameter count needs to be factored into the compression ratio. We believe that the unadjusted compression ratio is not well suited to evaluate pre-trained neural networks as compressors.

We have reformulated Section 1 to make this more explicit.

What is the context window's impact on compression performance?

We investigated this question in Figure 5. For text, increasing the model size is more beneficial than increasing the context length (short-term dependencies are most important). In contrast, large context sizes are generally beneficial for images (models with larger contexts can process larger fractions of an image). For audio, the relationship between model and context size is more complex.

We expect MegaByte (Yu et al., 2023) or multiscale byte language models (Egli et al., 2025) to improve performance on some but not all domains.

Why do you only report the compression ratio?

As we train our models for lossless compression via AC (which is equivalent to log-loss minimization), the compression ratio is the only relevant metric (apart from more “practical” metrics such as running time, which we report in Table A3). The compression ratio and the log-loss are proportional (the additive factor is the model size) and can be recovered from each other.

L196 misses "is".

Fixed, thanks!

We thank the reviewer for their positive feedback and insightful comments. We are pleased that they think that our paper presents a comprehensive empirical study, which leverages appropriate benchmarks, and that our writing is clear and well-structured.

What is the novelty over Delétang et al. (2024)?

While our work is indeed inspired by Delétang et al. (2024), we believe that the unadjusted compression ratio that they primarily use to evaluate their models is not well suited to evaluate pre-trained neural networks as compressors since it does not capture the entire output of the compressor. The adjusted compression ratio, which is the one we investigate, induces an entirely different regime where large-scale models are no longer viable. Note that Delétang et al. (2024) are aware of this and, therefore, also report the adjusted compression ratio for their models, with the difference that the large language models (LLMs) they evaluate are not competitive in this regime.

Moreover, Delétang et al. (2024) focus on off-the-shelf LLMs and only perform a single experiment where they train small transformers on enwik8 (100MB of text data) to obtain strong compression performance on enwik9. In contrast, we perform a comprehensive empirical study (in the reviewer’s words) in the multimodal setting with much larger datasets (165GB), where we not only show that small-scale transformers can achieve compression ratios on audio and image data that are on par with general-purpose and domain-specific compressors but also investigate the cross-modality transfer (in addition to a variety of ablations). We also improve the evaluation by always evaluating on out-of-distribution data (even within the same modality) to ensure a fair comparison with standard compression algorithms (overfitting to the training distribution is no longer possible).

Accordingly, our contributions compared to the previous literature are (see also Table A1):

• Small transformers can achieve competitive compression ratios on audio and image data (Delétang et al. (2024) only showed this for text data), even when comparing against domain-specific compressors.
• Small transformers can achieve competitive compression ratios across multiple domains, pointing towards the generality of relatively small models.
• Unlike the results on LLMs by Delétang et al. (2024), small transformers do not achieve transfer outside the training modalities. Moreover, unlike our work, Delétang et al. (2024) did not investigate in-modality out-of-distribution data (e.g., training on ImageNet and evaluating on CelebA) to show in-modality generalization.

The conclusions of the paper seem intuitive and straightforward.

While we agree that our results may not challenge the expectations and intuitions of experts in the field who know the literature well, we present novel quantitative results, which are significant and address an important open problem, i.e., whether small transformers pre-trained on large (multimodal) datasets can achieve competitive compression ratios across different modalities and whether there is transfer to unseen modalities, as observed in the large-scale model case (see Section 1). A priori, it was unclear whether we would attain similar conclusions to existing literature on multimodal generalization of small and large models since the compression viewpoint introduces an entirely different set of constraints on the models (most importantly with respect to their size). Finally, even some of our qualitative findings, e.g., the lack of out-of-modality transfer even for our largest models that we trained on multiple different modalities, were somewhat surprising to us.

We agree that investigating more sophisticated training strategies to improve cross-modal transfer, e.g., via transfer learning, presents an interesting direction for future work and have mentioned this in the updated version of our manuscript.