Compressed-Language Models for Understanding Compressed File Formats: a JPEG Exploration

W1. Paper positioning not convincing. The paper is motivated by a large argument that directly working on compressed format is beneficial. These arguments, however, are inherently flawed. First, arguments about ubiquity, compactness and generality are invalid because if one simply decompresses before training the LLM, these advantages would still hold. Second, while I do see the worth of studying compressed file formats (see S1), I fail to see practical relevance. On the one hand, real-world machine learning pipelines do consist of many domain-specific techniques; e.g., data augmentation (e.g., crop/scale images), helpful tokenization (e.g., SentencePiece), task-specific training objectives (e.g., BERT training) or models (e.g., CNNs). On the other hand, spending resources to "teach" a model to decompress/compress when we actually know how to do this more efficiently (JPEG encoder/decoder) is a waste of resources.

W2. Training and evaluation setup not convincing. For task (i), the prediction targets of image quality and class are fed into the training process in a somewhat contrived way to deal with problems of decoder-only models for this task. It's not clear why a decoder-only LLM is the right approach in the first place. For task (ii), the authors make "erroneous bytes" are less likely than "correct bytes" arguments. But when doing so, they ignore the entire input after the erroneous token (for localization/correction). This, again, a consequence of using decoder-only models. For task (iii), the automatic check is solely on file validity, but ignores the quality of the generated samples (other than the anecdotal examples of Fig. 3).

W3. Limited insight. This is for two main reasons. First, the papers make broad claims about compressed file formats, but then only considers JPEG, includes JPEG-specific information into the training pipeline (quality setting), and use only one image size. Second, the paper puts too much focus on whether tasks (i)-(iii) work reasonably well with an out-of-the-box LLM training pipeline. What's much more interesting, however, is exploring where such approaches would fail and why.

1. Models trained on text and models trained on compressed data have significantly different token space. While the token space of text is demonstrated learnable with various language models, the binary streams produced by compression algorithms may not have generic patterns that are generalisable for a variety of compressed data. It has been argued that the data distribution properties may be the key that drives in-context learning in transformers[1]. I feel that a detailed examination of the token distribution in the compressed data should be provided to justify the approach.

[1] Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J. and Hill, F., 2022. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35, pp.18878-18891.

2. The experiments are done on images with very small dimensions (28x28 for MNIST and 32x32 for CIFAR, additional experiment in appendix with an image dimension of 64x64 ). It is not a surprise that a large model can fit the small search space and provide predicting and generative capability on these datasets.

3. It is likely the language model is tuned to overfit a set of small images. This is little evidence based on the technical presentation of this paper that the model has learned the format of JPEG, therefore the method is unlikely to generalise to data in compressed file formats.

The paper claim that the focus of the research is on testing the understanding capabilities of compressed language models (CFFs). Besides, they draw the conclusion that “CLMs can understand the semantics of compressed data”. But the test is conduct on only one model trained by the author, instead of any existing language models. Therefore, I think the result is insufficient to support the conclusions drawn in the paper. It looks that JPEG-encoded formats exhibit language-like properties like any other sequences and the object is also to optimize for next-token prediction. It is not that surprising to see a model trained on sequence data works well within the same datasets (CIFAR-10 and MINIST, though as encoded data). Therefore, I think the main finding of this paper lacks some novelty. The paper outlines the characteristics of compressed file formats (CFF) and the challenges compressed language models (CLMs) encounter when meeting CFFs but does not clarify enough the need for CLMs to address CFFs or the importance of testing their understanding capabilities.

The novelty of the work is modest.