Compression Represents Intelligence Linearly

Thanks for your encouraging review! We address your comments below. ‎

The authors did not directly compare text compression to alternative characteristics that could put the findings into more perspective. For example, is compression much more correlated to performance than, for instance, model size or training data size?

Thanks for the advice! We strongly believe that model size or training data size alone does not correlate well with performance across different model families — other factors such as model architectures and training data quality will influence the scaling law with model size or training data size as inputs alone. Although this may already be implied, as suggested by the reviewer, in the next revisions we will supplement such comparisons to explicitly show the comparison to give more weight to our findings.

The importance of the compression aspect isn't entirely to me - as the authors mention, there is a direct link between concepts such as cross-entropy loss, average bits for compression and perplexity. Could the title equally be 'perplexity represents intelligence linearly'? Some clarification around this would help.

This is a good point. We emphasize compression in this paper due to the longstanding and recently revitalized discussion on the relation between compression and intelligence—a discussion that directly inspired our research. Indeed, there is a direct link between these concepts including losses, perplexity, and BPC. While there are various ways to present our experiments, the reviewer's suggested title, "Perplexity Represents Intelligence Linearly," is also valid. We chose to focus on compression because of its historical significance in discussions about intelligence, reflecting our original motivation and preference. However, presenting our findings from the perspectives of loss or perplexity is feasible as well.

Thanks for your helpful review! We address your comments below.

So the BPC calculation is basically doing an evaluation on held-out …… If so, this is a standard ML practice. I might see limited novelty in this approach.

You are right. From a high-level view, this is indeed a standard ML practice. In LLMs, however, people typically compare losses either between different training steps within the same model, or among models within the same family (e.g., llama-7B, llama-13B). Prior to this study, there was no established norm for comparing losses across different model families — for example, comparisons need to account for variations in tokenizers via the BPC metric and unify the context window size to ensure meaningful results. Additionally, it remains unclear whether comparing model losses across various model families — which differ in architecture, tokenizers, context window, and training data— is truly meaningful and has implications on downstream tasks. In this work, we establish a standard practice for making such comparisons, and provide empirical evidence that the relationship between compression and intelligence stands as a universal rule that is agnostic to the specific model and training configurations.

If the author claims that the BPC of the training set is linearly aligned with the benchmark score on the held-out dataset, I think it will be more interesting.

While we cannot perform such experiments due to limited access to the training data, we don’t think the training losses of different models are directly comparable since the training losses are computed on different training data — this is exactly why we need to compute BPC on the same held-out corpus.

However, as the average of the benchmark score is an established/well-accepted evaluation approach. I don't see the reason we need an additional one

As an unsupervised evaluation metric, BPC has practical advantages. Specifically, as discussed in the paper, data leakage and overoptimization are major issues for these benchmarks. For instance, using the MIN-K% PROB method for detection, we discover that the Qwen series have inflated scores on math benchmarks. In such cases, BPC serves as a more reliable metric. Also, BPC can easily support dynamic evaluation to update the compression corpora automatically to mitigate the leakage or overfitting issue, and it is easy to collect new domains of corpora. However, it is expensive to update or create new downstream benchmarks.

Thanks for the rebuttal.

For 1. Please correct me if I am wrong. From my perspective, "comparing losses across different model families" is basically the same as comparing somewhat 'metrics' on a held-out dataset across different model families. I think this is the common practice in multiple domain. e.g. comparing different image understanding architectures' generalization capability by comparing imagenet accuracy b/w VGGNet, AlexNet, and ResNet. Unfortunately, I don't think I am persuaded that this is considered as a new novelty.

For 3. Just a quick check, IIUC, I think data leakage would also affect the BPC?

Thanks for your time and helpful review! We respond to your comments below.

The claimed linear correlation looks not surprising to me

LM losses have indeed been an important metric, yet its relation to downstream task performance across different LLM families remains unknown. We highlight two novel findings:

1. In LLMs, typically, people only compare losses either between different training steps within the same model, or among models within the same family (e.g., llama-7B, 13B). These models share most design and training configurations. However, it is unclear whether comparing losses of LLMs from different model families has implications for downstream task results. Our work is the first to examine this. Note that these LLMs differ in tokenizer, architecture, context window size, and training data. We don’t think that it is immediately apparent that the correlation universally holds and is agnostic to all these factors, a fact only revealed through our experiments.
2. The correlation between losses and downstream task scores across different LLM families, if it exists, can take many forms. Our research reveals that it is linear.

The paper is more on empirical study without proposing new idea.

Right, the paper is more on empirical study, which we believe reveals novel insights. We don’t think that emphasizing empirical study is a weakness of a paper.

More interesting experiments needed … it would be better if the authors could find something across domains, e.g. how Github data correlates to NQ, MMLU, etc.

First, as we already showed in Figure 6, the BPC on Github data is not well correlated with the average score on the knowledge domain that includes MMLU and NQ.

To add more experiments as requested, we explore cross-ability tasks where multiple abilities are required. We consider solving GSM8K questions through coding, which requires both math and coding abilities. We compute BPC on three corpora to examine their correlation with GSM8K scores: (1) on arxiv data; (2) on Github data; and (3) on the mix of arxiv and Github data. Results are shown below — as expected, the mix of arxiv and Github data demonstrates the strongest correlation, while either arxiv or Github data alone does not correlate well, underscoring that different corpora reflect diverse abilities.

Thanks for your time and encouraging review! We respond to your comments below.

Considering the rapid pace of innovation in the field of LLMs, is there confidence that the findings would remain applicable to novel architectures such as RWKV, Mamba, and KAN?

We hypothesize that the findings are agnostic to model architectures, while we did not test non-transformer architectures since there were few non-transformer pretrained models with strong and already-emergent abilities. To verify the findings on non-transformer archs, however, we conduct the experiment on the Jamba-52B model [1], which is based on a hybrid Transformer-Mamba MOE architecture, being the strongest open-source non-transformer pretrained model as far as we know. The results for Jamba model in the main figure are shown in this link, while we report its BPC and benchmark scores below (according to the COLM rebuttal guideline, we think this is the necessary case to link such an external correlation figure that is easy to read, since the numbers below alone do not reveal much helpful information). Despite having significantly different architectures, our findings can still be applicable to the Jamba model.

Is there an expectation that the correlation between compression efficiency and intelligence persists across different languages or modalities?

This is a good point. We do expect the correlation between compression efficiency and intelligence persists across different languages or modalities, because the long-standing discussion on their relationship has been based on data in general rather than language only. However, in this paper we focus on the single-language setting and leave further exploration of different languages and modalities as future work.

[1] Lieber, Opher, et al. "Jamba: A hybrid transformer-mamba language model." arXiv preprint arXiv:2403.19887 (2024).

Thank you for your time and encouraging review!