SuperBPE: Space Travel for Language Models

Thank you for your review. Below we hope to address the reviewer’s two stated concerns.

Comparison with other tokenizers

The reviewer points out that we only compare to BPE, while there are other tokenization algorithms. To start, we want to emphasize that all the alternatives mentioned are still subword tokenization algorithms, while SuperBPE enables superword tokenization. Any subword tokenizer is fundamentally constrained by the average length of whitespace-delimited words, which determines the maximum achievable encoding efficiency.

In addition, BPE is effectively the universal choice for today’s LMs; it achieves better compression than Unigram (Schmidt et al., 2025) as well as better downstream performance (Ali et al., 2024). In August 2024, we manually verified that all top 100 models on ChatbotArena use BPE tokenizers. RoBERTa, which the reviewer mentions, also uses BPE (see §4.4 of their paper).

Nonetheless, we trained Unigram and WordPiece tokenizers and plot their encoding efficiency compared to BPE (and SuperBPE) below. Unigram has much poorer compression than BPE, and WordPiece matches BPE perfectly. (Note that WordPiece only differs from BPE by a normalization factor in the frequency counts; in practice, it appears to be extremely similar.) Since each model pretraining run requires 15,400 H100-hours (~$35K in cloud compute), we do not pretrain additional baselines with these non-BPE subword tokenizers.

Efficiency scaling for other tokenization algorithms (figure)

That being said, we agree that comparisons with WordPiece and Unigram provide for a broader contextualization of our method, and we will add them in the final version of the paper.

Lack of technical depth

SuperBPE does not simply remove whitespace pretokenization from BPE. In fact, other works have tried this before and found that simply omitting whitespace pretokenization significantly degrades LM performance. For instance, Schmidt et al., (2024) found that disabling pretokenization results in the worst performance among 18 tokenizer variants; similarly, Dagan et al. (2024), report that it leads to significant deterioration on all downstream tasks. Without any pretokenization, tokens often consist of the ends of words merged with the start of the next, which is at odds with the structure of language.

Consider:

BPE: This| is| an| example| sentence|.

SuperBPE: This is an| example| sentence|.

BPE w/o pretok: This is |an exampl|e sentenc|e.

Here, e sentenc is a single token under BPE w/o pretok. (To understand why, note that (e, _) and (e, .) are both extremely common character pairs, so they are merged first, blocking “example” and “sentence” from becoming single tokens in this sentence.)

Different from the “naive” approach of removing whitespace pretokenization entirely, we instead introduce the concept of a pretokenization curriculum, which brings even better encoding efficiency along with performance gains.

In addition, our work does not just introduce a tokenization algorithm; we also pretrain models from scratch and run controlled scaling experiments, showing what happens when increasing the train tokens under fixed parameter count or increasing model size under fixed token/parameter ratio (§4.3, §B.3).

We believe that the relative simplicity of the SuperBPE approach is a strength, and likely to increase its impact on future research and practice, not a limitation.

We would like to thank the reviewer for their helpful feedback.

Switching from BPE to SuperBPE after pretraining

This is a very interesting future direction, since successful transfer would enable faster inference without pretraining a new model from scratch. There is a large amount of work on tokenizer transfer which points to this possibility [1, 2, 3, 4, 5, 6]. In fact, the method that you suggest is exactly what is proposed by Gee et al., 2022 (termed Fast Vocabulary Transfer; FVT)! More specifically, [7] extends Llama’s tokenizer to better support code by augmenting it with tokens learned using a different pre-tokenization rule. They find that after fine-tuning on roughly 50B tokens, the model with the updated tokenizer becomes competitive with the original model fine-tuned using the original tokenizer, while maintaining its efficiency advantage.

These results suggest that a similar approach could work for SuperBPE as well, and in some preliminary exploration we were able to adapt Llama 3.1 to use superword tokens by initializing the new embeddings as the mean of all other token embeddings, then tuning only the (un)embedding layers. We plan to work on this direction further.

[1] Fine-tuning transformers: Vocabulary transfer (Mosin et al., 2021)

[2] WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (Minixhofer et al. 2022)

[3] Fast Vocabulary Transfer for Language Model Compression (Gee et al., 2022)

[4] OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (Liu et al., 2024)

[5] FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (Dobler & de Melo, 2023)

[6] Zero-Shot Tokenizer Transfer (Minixhofer et al., 2024)

[7] Getting the most out of your tokenizer for pre-training and domain adaptation (Dagan et al. 2024)

We thank the reviewer for the thoughtful comments. The limitations mentioned are all very fruitful directions for future work, and we provide some preliminary findings below.

Switching from BPE to SuperBPE after pretraining

This is a very interesting future direction, since successful transfer would enable faster inference without pretraining a new model from scratch. There is a large amount of work on tokenizer transfer which points to this possibility [1, 2, 3, 4, 5, 6]. In particular, [7] extends Llama’s tokenizer to better support code by augmenting it with tokens learned using a different pre-tokenization rule. They find that after fine-tuning on roughly 50B tokens, the model with the updated tokenizer becomes competitive with the original model fine-tuned using the original tokenizer, while maintaining its efficiency advantage.

These results suggest that a similar approach could work for SuperBPE as well, and in some preliminary exploration we were able to adapt Llama 3.1 to use superword tokens by initializing the new embeddings as the mean of all other token embeddings, then tuning only the (un)embedding layers. We plan to work on this direction further.

[1] Fine-tuning transformers: Vocabulary transfer (Mosin et al., 2021)

[2] WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models (Minixhofer et al. 2022)

[3] Fast Vocabulary Transfer for Language Model Compression (Gee et al., 2022)

[4] OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (Liu et al., 2024)

[5] FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (Dobler & de Melo, 2023)

[6] Zero-Shot Tokenizer Transfer (Minixhofer et al., 2024)

[7] Getting the most out of your tokenizer for pre-training and domain adaptation (Dagan et al. 2024)

SuperBPE for non-English

To explore SuperBPE for non-English languages, we train monolingual tokenizers in some other languages, all with the same vocabulary size and amount of training data (10 GB from MADLAD). Here we show the encoding efficiency over vocab size for some different tokenizers.

Multilingual efficiency scaling figure

In general, SuperBPE improves encoding efficiency across all languages. As we would expect, languages with short words (e.g., English, Spanish, Italian) have relatively poor encoding efficiency under BPE, and SuperBPE brings large gains, whereas languages that are highly compounding (e.g., German) or that use no whitespace (e.g., Chinese) have higher encoding efficiency under BPE, and SuperBPE brings smaller gains.

We also observe that under SuperBPE, different languages achieve more similar efficiencies to each other. This has implications for fairness, suggesting that superword tokenization can equalize disparities in encoding efficiency across languages!

Linguistic approaches to identifying superwords

We also like the idea of linguistically motivated superwords, though prior attempts to add MWEs from existing lists as single tokens have not brought the same scale of gains [8]. There are appealing aspects of learning superwords from data, as it mirrors the learning of subword tokens and doesn’t depend on keeping MWE lists up-to-date. In addition, we observe that many extremely common word sequences are not “linguistically pretty,” but are simply composed of common words and greatly improve encoding efficiency, e.g., “of the,” “has been,” “this is”. Nonetheless, we think there may be room for improved linguistic analysis or revision of SuperBPE tokens.

We think this is also related to your suggestion to identify multiword expressions ahead of time, then replacing spaces with underscores. This may indeed be an approach that combines the benefit of linguistically motivated MWEs, then using frequency to merge a subset of them!

[8] Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings (Otani et al., 2020)

Missing cite

Thank you for telling us about MANTa! We were not aware of this work previously and will include it in the dynamic tokenization part of the related work.

We would like to thank the reviewer for their insightful comments.

SuperBPE for models over 8B

While we do not perform experiments with models much larger than 8B, we have strong reasons to believe that SuperBPE will continue to be useful in that regime. Intuitively, we think of tokenization as managing the allocation of compute to pieces of text. It seems like a waste to “spend” a whole token’s worth of compute predicting simple words such as “a” or “of”. Thus, we expect reallocating this compute to predict more difficult tokens (which we present as a feature of SuperBPE in Section 4.2) to be beneficial regardless of the size of the model.

This intuition is supported by our scaling experiments, which show a consistent performance trend as we scale up. In Figure 7 in Appendix B.3.3, we report the normalized language modeling loss (bits per byte) for BPE and SuperBPE models ranging from 680M to 13B (spanning over 300x compute scaling) and observe that SuperBPE and BPE both scale very predictably. In the setting of matched inference compute, we are always slightly better than BPE; in the setting of matched model size, we are always slightly worse. Of course, cross entropy loss is not directly predictive of performance on downstream evals, but we believe these results suggest that the underlying scaling dynamics are well-behaved.

Because of this, we are confident that SuperBPE will continue to be useful at larger model sizes!

Formatting suggestions

Thank you! We will make Fig 1 smaller.