Overall, we would like to thank you for your review. You clearly spend the time to understand our work, and we appreciate that. Here are more detailed responses to your comments and questions:

Baselines compared are limited to the classical ones (Wordpiece, BPE, UnigramLM) but not more competitive and recent ones. It's unclear why PickyBPE is discussed, borrowed from, but not directly and fairly compared with.

Our idea was that the increase in number of merges that occur when doing superwords might cause an increase in junk or unreachable tokens, and the adding PickyBPE would be a good way to counteract that. We do consider PickyBPE by itself in Figure 2 with two different regex, finding very little difference in the number of merges at each step compared to other baselines. This suggests that the compression ratio will also be quite similar.

Experiments are limited to EN. This seems to be very limited for a tokenization work.

That is one of the primary limitations of our work. We should have explicitly stated that as a limitation.

SuJi response

Overall, we would like to thank you for your detailed review. You clearly spend the time to understand our work, and we appreciate that. Here are more detailed responses to your comments and questions.

The combination of superword tokenization with vocabulary tripping via the IoS metric is extremely interesting. One potential downside of superword merges is that it could create long tokens which are rarely seen.

Our training approach will not produce long tokens that are rarely seen, as they are only merged if their count in the training corpus warrants it. Although intermediate tokens may end up with low counts - which we are using deletions to handle. We will update our terminology on unreachable tokens, and cite that paper.

Currently, it is unclear whether changes to tokenization would make significant performance changes, especially as Schmidt et al. 2024 show that in many cases changes to tokenization lead to minimal changes in downstream performance.

As we discuss in Section 2, the likely cause of the similarity in performance found by Schmidt et al. 2024 is the huge fraction of pre-tokens tokens that end up as single tokens. This is precisely the motivation for supermerges in BoundlessBPE. By overcoming this pretokenization barrier, we can substantially change our intrinsic metrics. The concurrent work of Liu et al (2025) that we cite found sizable downstream improvement across a broad range of tasks.

The results are only for English and are only on one dataset...

We did only train and evaluate on disjoint parts of MiniPile (see footnote 2, page 3).

Some key methodological details remain unclear. What data is the tokenizer trained on? And what is it evaluated on? Is it the same data mentioned in Section 2?

We mention that we used the English language dataset MiniPile in footnote 2 on page 3. We train on the first 1GB of data (due to the slowness of our training code), and evaluate on the remaining 5GB of MiniPile

I think the way that Renyi entropy is used and discussed is potentially problematic, as it does not always correspond with better performance...

You’re absolutely right that there is mixed evidence on the effect of Renyi efficiency on downstream performance. We will add the citations you mention. We were trying to design a system with a more uniform distribution of tokens, and at least it provides a quantitative measure of that, but it may not translate to downstream performance.

... For example, how many of the superword tokens start with '_the'? How many of the superword tokens are full phrases (as opposed to parts of phrases missing one or more subword tokens)? ...

In total there are 1388 superwords starting with _the. Here are some examples, with their associated counts

_the_same 43131
_the_first 32755
...
_the_song 132

All of the superwords are full phrases, due to the fact that we only allow word-like pre-tokens (matching the regex in footnote 5 on page 4) to combine into superwords, if we use that regex as our definition of “word”.

Is there a way to evaluate the nature of the multi-word tokens? How much of the time is it cases like '_of_the', versus awkward part-phrases?

As previously mentioned, they are all composed of complete pre-tokens, although usually not semantically meaningful phrases. The most common are listed below:

_of_the 975970
_in_the 606600
_to_the 394204

How many are named entities like 'New York City' versus more functional ones like '_of_the'?

Proper nouns turn out to be fairly infrequent. A quick analysis of proper nouns within the superwords (regular expressions on the capitalization, and a hand editing pass) found only 475 proper nouns, with a total count on the training corpus of 242,074. This is out of the overall total of 47,884 unique superwords with a count of 35,506,759. Thus proper nouns only account for about 1% of the unique superwords, and 0.7% of the counts. A few examples:

_United_States 27932
_New_York 15985
_Supreme_Court 7835

How much slower is implementation this than Hugging Face’s BPE implementation? The authors report that training took 4.7 CPU days, which seems very long. Is it possible to overcome this so that it can be used for a language model?

Training Hugginface BPE on the same data, vocab, and machine takes 59s. We are currently working on speeding up our code to allow larger training datasets. However, that limitation only applies during training, and so it can be used for LLM use.

Is this tokenization method sensitive to the quantity of training data? ...

You are correct that we are probably training on a suboptimal amount of training data. We do this to keep our training time - which is linear in the size of data - to a reasonable level. We will explicitly mention this in the paper. We are investigating other approaches to speed up superwords training which will allow us to train on substantially more data.

Overall, we would like to thank you for your detailed review. You clearly spend the time to understand our work, and we appreciate that. Here are more detailed responses to your comments and questions.

The evaluation of the proposed tokenization algorithm is not fully conclusive. The proposed method is a combination of two ideas: removal or undertrained tokens (similar to Chizhov et al. 2024) and superword tokens. There are several papers that suggest superword tokenizations too: https://arxiv.org/pdf/2503.13423 https://ieeexplore.ieee.org/abstract/document/10815724/ https://arxiv.org/abs/2211.11041

We discuss the relation to the concurrent work by Liu et al (2025) in section 5. Ndomba et all (2024) seems to use WordPiece, starting from a combination of characters and common words. The WordPiece merges would indeed sometimes create superwords in this case. However, they appear to not use pre-tokenization, which is a difference from our approach that limits superwords to be a combination of full pretokens. Zhemchuzhina et al (2022) is an interesting analysis of the relation of Zipf’s law and tokenization.

It seems natural to provide some form of an ablation study to illustrate if the combination of those two ideas has some particular synergistic effect as well as to compare the proposed solution with the BPE alternatives mentioned in the paper.

An ablation study would indeed strengthen the paper. With:

BoundlessBPE with PickyBPE and the BOUNDLESS_PATTERN

as the baseline, we will run:

BoundlessBPE with no PickyBPE and the BOUNDLESS_PATTERN
BoundlessBPE with PickyBPE exactly as the PickyBPE paper and the BOUNDLESS_PATTERN
BoundlessBPE with PickyPBE and the GPT2_PATTERN
BoundlessBPE with no PickyPBE and the GPT2_PATTERN

However, due to the long training time of our current code, we will not have results to share by the end of the response period.

Figure 2 does provide evidence that it is the superwords that provide a larger compression increase than the PickyBPE deletions or the choice of pre-tokenization regex. The four baseline variants with BPE or PickyBPE and two combinations of regex are essentially on top of each other. (This is explained by the high fraction of pre-tokens which are represented as single tokens, as in Figure 1). The y-axis of Figure 2 is the log of the number of merges done at each state of the training. Since each merge decreases the number of tokens by exactly 1, the higher blue curve of Boundless BPE results in more compression.

Some minor corrections: "Lian et al. (2024) and Chizhov et al. (2024) proposed strategies to eliminate these low-frequency tokens formed as intermediate steps. We find Chizhov et al. (2024)’s approach particularly compelling due to its direct integration into BPE training." To the best of my knowledge, both proposed methods integrate the updates into the BPE training direclty, however, Picky BPE has better inference since it takes into consideration the sequence of removals and merges explicitly.

While not directly stated in our paper, we also use the same change to inference as Picky BPE, in addition to deleting tokens with high IoS during the training process. We will add this information in future versions.

How would the proposed method affect multilingual models?

The BOUNDLESS_BPE regex offers a small advantage of keeping combining marks \p{M} with other base characters, This is a flaw in the other 3 regex discussed in Appendix A, although it has been corrected in the GPT4-o regex. The motivations for BoundlessBPE also extend to cases that occur in non-English text, such as non-space delimited scripts and words/graphemes with diacritics/combining marks

Could you, please, clarify your logic here: "We first pretrain 8B models with BPE and SuperBPE tokenizers. We use the OLMO2 7B (OLMo et al., 2024) training configuration, including the model architecture, training hyperparameters, and pretraining corpus, but reduce the total number of training steps to correspond to ∼330B tokens (compared to 4T) for the sake of compute." For reference, out of the recent models Qwen 3 600m parameters has 6T tokenizer. Could you explain your reasoning in detail here? Is there any reason why reduced amount of tokens makes your results more conclusive?

The quote you provided is from page 5 of the SuperBPE paper you cite above (Liu et al, 2025), rather than our work.

Tau = 0.9, Picky Deletion to bytes,         regex = Boundless
Tau = 0.9, Picky Deletion to previous pair, regex = Boundless 
Tau = 1.1, Picky Deletion off,              regex = Boundless

Tau = 0.9, Picky Deletion to bytes,         regex = gpt4o
Tau = 0.9, Picky Deletion to previous pair, regex = gpt4o
Tau = 1.1, Picky Deletion off,              regex = gpt4o

pat_str = "|".join([
    r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
    r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
    r"""\p{N}{1,3}""",
    r""" ?[^\s\p{L}\p{N}]+[\r\n/]*""",
    r"""\s*[\r\n]+""",
    r"""\s+(?!\S)""",
    r"""\s+""",
]

bq5M Response

Overall, we would like to thank you for your detailed review. You clearly spend the time to understand our work, and we appreciate that. Here are more detailed responses to your comments and questions.

One of the measures used, Renyi efficiency, may be one of the better metrics for predicting relative downstream task performance, but as Cognetta et al. (2024), note "Although useful, the predictive power of this metric is not perfect". Reference: Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two Counterexamples to Tokenization and the Noiseless Channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torino, Italia. ELRA and ICCL.

You’re absolutely right that there is mixed evidence on the effect of Renyi efficiency on downstream performance. We will add the citations you mention. We were trying to design a system with a more uniform distribution of tokens, and it does at least provides a quantitative measure of that, but it may not translate to downstream performance.

I'm wondering how BoundlessBPE compares to omitting pre-tokenization. In the paper, you say Schmidt et al. (2024) demonstrated that entirely omitting pre-tokenization resulted in the poorest downstream performance among 18 evaluated tokenizers. But looking at Table 1 in Schmidt et al. (2024), it appears that the "no pre-tokenization" setting is only compared for their proposed algorithm, but not for any of the other ones. Having played around with the --split_by_whitespace=False option offered by SentencePiece, which allows to efficiently merge across whitespace boundaries, I noticed that the resulting vocabulary contains many merges of two, three, or even more pretokens. So I'm wondering how BoundlessBPE compares to naive BPE without pre-tokenization. Concurrent work (SuperBPE, Liu et al. 2025) has a short paragraph on this, but I'm curious what your take on naive BPE without pre-tokenization is.

That is an interesting question. The main difference is that BoundlessBPE guarantees that it only combines full pretokens into superwords, while this would not be true with BPE without pre-tokenization.

As an experiment, we trained a SentencePiece BPE tokenizer on 100MB of our training data and the split_by_whitespace=False parameter. Due to the slowdown from having no pre-tokenization of the training data it took an hour to train.

Out of a vocabulary size of 131,072, there were 4384 vocabulary words that start with a lowercase letter, and then have a combination of understores and letters (matching [a-z]+_\w+, since \w includes the underscore). Here is a random sample of these examples:

['ed▁international',
 's▁done',
 'es▁are',
 'ancy▁and',
 'ing▁our',
 's▁un',
 't▁happen',
 'ing▁authority',
 'to▁gg',
 'span▁class',
 's▁include',
 'ed▁an',
 't▁pass',
 't▁mean▁to',
 's▁were▁being',
 'mathbb▁Q',
 'up▁of',
 'ing▁in▁mind',
 'label▁switch',
 'ed▁her',
 'ed▁two',
 'tered▁with',
 's▁military',
 'ise▁the',
 'ed▁the▁problem',
 's▁good',
 'option▁value',
 's▁on▁line',
 'ed▁into▁a',
 'to▁Y',
 't▁use',
 'acies▁of',
 'ing▁to▁an',
 's▁of▁the▁form',
 'ing▁with▁an',
 's▁and▁have',
 's▁the▁way',
 'ing▁on▁it',
 'iss▁albino▁mice',
 'ization▁of',
 'ases▁and',
 's▁of▁these',
 'ed▁from',
 'ic▁acids',
 'ches▁are',
 're▁now',
 're▁in▁a',
 's▁of▁her',
 'ated▁from',
 'new▁THREE']

The first part of the token often seems to match a suffix, although there are a surprising (to us) number of whole second and third words. In general, this isn’t as many as we were expecting.

Of course, the lack of pre-tokenization can also lead to other tokens combining different categories of characters, such as combinations of digits, spaces, and/or punctuation:

['+00]',
'▁8-10',
'▁-▁10.'
'▁)▁;',
'_{1}(']

Overall, we would like to thank you for your excellent, detailed review. You clearly spent a great deal of time understanding our work, and we appreciate that. Here are more detailed responses to your comments and questions. Other small suggestions not mentioned in this reply will be incorporated.

The paper has one primary weakness that may make it not quite ready for publication at COLM: a significant lack of detailed ablation studies.

An ablation study would indeed strengthen the paper. With:

BoundlessBPE, our version of PickyBPE, BOUNDLESS_PATTERN

as the baseline, we will run:

BoundlessBPE, no PickyBPE, BOUNDLESS_PATTERN
BoundlessBPE, PickyBPE exactly as the PickyBPE paper, BOUNDLESS_PATTERN
BoundlessBPE, our version of PickyPBE, GPT2_PATTERN
BoundlessBPE, no PickyPBE, GPT2_PATTERN

However, due to the long training time of our current code, we will not have results to share by the end of the discussion period. Figure 2 does provide evidence that it is the superwords that provide a larger compression increase than the PickyBPE deletions or the choice of pre-tokenization regex. The four baseline variants with BPE or PickyBPE and two combinations of regex are essentially on top of each other. (This is explained by the high fraction of pre-tokens which are represented as single tokens, as in Figure 1). The y-axis of Figure 2 is the log of the number of merges done at each state of the training, so the higher blue curve of Boundless BPE results in more compression.

Additionally, the configuration of the "standard BPE" baseline used for comparison is not consistently clear. ...

We will explicitly state the regex used in Figures 4-7, which was the GPT2_PATTERN. We chose that over GPT-4o as it is the default for the Huggingface BPE implementation, and hence very commonly used.

PickyBPE The paper should provide a clearer rationale for its specific variant of PickyBPE, particularly the decision to decompose deleted tokens into bytes rather than their constituent BPE tokens (as in the original PickyBPE). What results or hypotheses led to this choice? The concern that this aggressive decomposition could lead to re-learning identical merges also needs to be addressed more directly.

Our hypothesis was that the more aggressive decomposition could allow the deleted token to form more compressed tokens. It does lead to re-learning some merges, but having duplicate re-learned merges only increases the inference time somewhat. Our ablations will show if this hypothesis was correct.

Supermerges ... Its design choices, such as using [a-zA-Z] instead of a more Unicode-aware \p{L}\p{M}* ..., should be justified.

Using \p{L}\p{M}* rather than [a-zA-Z] is indeed a better choice in this regex, and we’ll update our code to use it. As our results are from MiniPile, which is primarily English, it should not change our results substantially. We included snake_case handling in this regex - along with the apostrophe - since those are the only two non-letter characters allowed in our “words”. Including the underscore in this regex allows some snake case variable names to recombine into full variable names, as discussed in section 3.5.

Multilinguality The experiments are conducted on MiniPile, an English-only dataset. This is a notable limitation...

You’re entirely correct that this is a major limitation and it should be explicitly stated as such. We have updated our pattern since this submission to improve handling of non-space delimited scripts by breaking those words into extended grapheme clusters and recombining them as counts suggest into superwords.

Throughout the paper there are also subtle overstatements with respect to languages such as "Pattern B-1 takes a different language-independent approach".

You’re right that this is worded too strongly. It is an improvement over the hardcoded English contractions in T-1 and F-1.

Figure 2 is difficult to read due to overlapping lines.

We were intending to show in Figure 2 that the blue Boundless line is above the other 4, which are essentially right on top of each other. We should mention that this overlap is caused by the prevalence of pre-tokens as full tokens between these four baselines.

"it still took 4.7 CPU days to train our 1GB training". lacks context. How does this compare to the training time for the baseline BPE on the same hardware and dataset?

Training Hugginface BPE on the same data, vocab, and machine takes 59s. We are currently working on speeding up our code to allow larger training datasets.

BoundlessBPE, our version of PickyPBE, GPT4o_PATTERN
BoundlessBPE, no PickyPBE, GPT4o_PATTERN

Tau = 0.9, Picky Deletion to bytes,         regex = Boundless
Tau = 0.9, Picky Deletion to previous pair, regex = Boundless 
Tau = 1.1, Picky Deletion off,              regex = Boundless

Tau = 0.9, Picky Deletion to bytes,         regex = gpt4o
Tau = 0.9, Picky Deletion to previous pair, regex = gpt4o
Tau = 1.1, Picky Deletion off,              regex = gpt4o

# - match the entire string
# - lookahead ensures there is at least one letter
# - otherwise can have spaces, underscores, apostrophes, or curly apostrophes
IMPROVED_MERGE_PATTERN = r"^(?=.+\p{L})(?:\p{L}\p{M}*|[ _'\u2019])+$"

Best of luck on the ablations. Small suggestion for a simpler way to write the regex below:

IMPROVED_MERGE_PATTERN = r"^(?=.*\p{L})[\p{L}\p{M} _'\u2019]+$"

or even

IMPROVED_MERGE_PATTERN = r"^\p{L}[\p{L}\p{M} _'\u2019]*$"

pat_str = "|".join([
    r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
    r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
    r"""\p{N}{1,3}""",
    r""" ?[^\s\p{L}\p{N}]+[\r\n/]*""",
    r"""\s*[\r\n]+""",
    r"""\s+(?!\S)""",
    r"""\s+""",
]