Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

We thank the reviewer for their recognition of the “thorough analysis” in our experiments, the “novelty” and “practical insights” of our findings, and the potential to “inspire future work on adaptive tokenization.”

Are LMs robust to non-canonical tokenizations due to misspellings?

We thank the reviewer for this insightful hypothesis. Because it would be difficult to automatically identify misspellings of words, we designed a similar experiment to study this question. In particular, we consider the diversity of ways in which a token is internally segmented (when appearing within a larger word), and measure whether that is correlated with the model’s understanding of the spelling of that token. Consider the token bio — when appearing as a substring in, e.g., “microbiology,” it may be segmented internally, e.g., as [microbi, ology], with the token boundary appearing between “bi” and “o.” We hypothesize that LMs will better understand tokens that are segmented in more unique ways.

To test this, we use tokens from the vocabulary of $`Llama-3.1-8B-Instruct`$ which are 5-15 characters long and occur 50-500 times in a 13GB sample of The Pile. From there, we construct two subsets of tokens: those that are segmented in at least 5 different ways, and those that are consistently segmented the same way. We then evaluate the model on intra-token understanding using task constructions from CUTE. Shown in the table below, we find that the model consistently understands commonly segmented tokens better. We believe that this indirectly provides support for the reviewer’s hypothesis that misspellings would also lead to better understanding, as it similarly splits a string into a different sequence of subtokens.

Spelling and grammaticality comparison for canonical tokenization in §4.1

We provide a comparison to canonical tokenizations below. We see that the performance of SFT, DPO, Instruct models are similar to those given non-canonical tokenizations (Figure 3), whereas the Base model performs much better given canonical compared to non-canonical tokenizations. This is consistent with what we expect, given that robustness to non-canonical tokenization arises after the SFT stage.

Spelling

Grammaticality

We thank the reviewer for recognizing that our paper presents “interesting observations,” as well as the detailed comments and insightful questions. We hope you find that our response addresses your concerns.

On testing models that do not merge digits

LMs whose tokenizers do not merge digits will not have multi-digit tokens in the vocabulary, making it impossible to evaluate right-to-left tokenization (which chunks digits into groups of 3, e.g., 1000 $\rightarrow$ [1, 000]) at inference-time.

We also want to mention that the best representation of digits is still an open question. For instance, [1] found that character-level tokenization is still suboptimal and underperforms “fixed-character” tokenization, which is somewhat similar to R2L because it ensures consistent digit positions. In addition, grouping digits in chunks of 3 is more common among tokenizers of frontier LMs today.

Expanding the test sets in Section 3

Thank you for identifying the limitation of small test sets. Following the reviewer’s suggestion, we expand the test sets for Arithmetic and Counting Characters to >1000 examples. For Arithmetic, we include numbers with 5 to 15 digits. We find that right-to-left tokenization of digits still significantly outperforms the canonical left-to-right tokenization, achieving 30.8% compared to 9.8% accuracy. We also expand the Counting Characters dataset by considering tokens of 5 to 10 characters in Llama3’s vocabulary. Again, we replicate the finding that character-level tokenization leads to better performance, achieving 73.5% compared to 66.5% for canonical tokenization. We will update the test sets and corresponding results in the next revision.

On evaluating models with tokenizers that differ more from $`Qwen2`$ & $`Llama3`$

Following the reviewer’s suggestion, we perform evaluations on $`Mistral-7B-Instruct-v0.3`$ and $`Qwen2.5-Coder-7B-Instruct`$. Note that we do not use $`Llama2`$ or $`CodeLlama`$ (based on $`Llama2`$) due to very poor performance, even with canonical tokenization. The findings are consistent with those reported in the paper — $`QwenCoder`$ retains 94% of performance and $`Mistral`$ retains 80% of performance with random tokenization.

$`QwenCoder`$ Average Retention: Random: 94.1%, Char: 89.6%

$`Mistral`$ Average Retention: Random: 80.3%, Char: 70.5%

How many training epochs are required for robustness to emerge?

Thank you for the question, we should have stated explicitly that we trained for two epochs, which we will include in the next revision. To observe how robustness varies with the amount of SFT training, we also evaluate the epoch 1 checkpoint on our robustness metrics (spelling, grammaticality, and win rate). We find that the 1-epoch checkpoint’s performance lies between the base model and the 2-epoch model. This suggests that robustness increases during the SFT process.

Why do models always produce canonical tokenizations?

We hypothesize that models are able to understand non-canonical tokenizations in its context because it has the opportunity to “re-aggregate” information over multi-token words into the last token’s hidden representation. This process has been observed in how models process multi-token entities (e.g., [␣New, ␣York]) and multi-token words [1, 2, 3]. This learned process of “internal retokenization” in early layers likely grants the robustness to non-canonical tokenizations.

In contrast, models are not trained to predict non-canonical token sequences. Of course, mathematically, models must place non-zero probability on non-canonical token sequences [4, 5]. This means that over many samples, models should sometimes produce non-canonical tokenizations (regardless of whether the input is canonically tokenized). However, we qualitatively observe that generations are overwhelmingly canonical.

Applying more optimal tokenizations during SFT

We interpret the reviewer’s suggestion (“take the idea from §3 to improve the performance during the SFT stage”) as training with a non-canonical tokenizations during SFT to further improve upon the gains that we found at inference-time alone.

To test this, we finetune the $`Llama-3.2-1B`$ base model on 30K random examples from the $`Tulu3`$ SFT mixture: one with L2R digit grouping, and one with R2L grouping. Then, we compare the following settings: (1) the L2R-trained model with L2R inference, (2) L2R-trained model with R2L inference, and (3) R2L-trained model with R2L inference. We expect (3) to outperform (2) due to the additional opportunity to adapt to R2L tokenization during training. We evaluate these models on 1,000 instances of addition and subtraction of 4 - 8 digit numbers. We find that (1) gets 2%, (2) gets 24%, and (3) gets 12.3%, counter to our hypothesis. We conjecture that this is because (3) learns to predict two different representations of numbers between pretraining and finetuning, and when generating output, divides probability among R2L and L2R representations. In contrast, (2) continues to generate numbers L2R, even when conditioned on R2L representation of digits.

Evaluating alternative tokenization schemes on $`Llama-3.2-1B-Instruct`$

Following the reviewer’s suggestion, we investigate whether the performance improvements we identify in §3 for $`Llama-3.1-8B-Instruct`$ also apply to $`Llama-3.2-1B-Instruct`$. On the Arithmetic task (expanded to include 1000 examples with arithmetic for 4-8 digit numbers instead of 5-15 digit numbers to account for weaker model abilities), we find that R2L segmentation achieved 32.7% accuracy, a substantial improvement over 9.7% with canonical tokenization.

However, for the other four tasks (Common Morpheme, Codeline Description, Acronym, and Counting Characters tasks), Llama-3.2-1B-Instruct consistently performs worse when presented with character-level tokens. We hypothesize this is because, as we observed, weaker models are less robust to non-canonical tokenizations compared to stronger models.

Why is the chat template crucial to robustness?

We believe that the chat template is crucial for indicating a new turn of conversation, “freeing” the model from continuing the perceived misspellings in the context. Indeed, we show in §4.2 that it does not need to be an actual “chat template,” and a generic “Question:/Answer:” template that separates the user and model turn produces the same effect.

Regarding more detailed documentation

Thank you for raising this important point. We plan on updating the manuscript to include more details about the experimental setup, alongside the exact script/settings that we used to train/evaluate our models and our benchmarks.

We thank the reviewer for recognizing our “systematic evaluation” of model robustness to non-canonical tokenizations, the “performance gains” delivered by “simple inference-time interventions,” and “rigorous analysis” into the source of robustness. We hope you find that our response addresses your concerns.

On the possibility of automatically adapting tokenization schemes

The reviewer is correct that, in this work, we do not find a single adaptive tokenization strategy that improves performance across the board. The goal of the work was not to improve tokenization, but instead demonstrate that LMs can handle non-canonical tokenization, which is surprising and important because these tokenizations are provably out-of-distribution. We believe that automatically finding the best segmentation of text at inference-time is a promising direction for future work.

Finetuning on non-canonical tokenizations

Following the reviewer’s suggestion, we explore whether finetuning on a non-canonical tokenization scheme, which has proven useful at inference-time, can further improve downstream performance. In particular, we finetune the $`Llama-3.2-1B`$ base model on 30K random examples from the Tulu3 SFT mixture: one with L2R digit grouping, and one with R2L grouping. Then, we compare the following settings: (1) the L2R-trained model with L2R inference, (2) L2R-trained model with R2L inference, and (3) R2L-trained model with R2L inference. We expect (3) to outperform (2) due to the additional opportunity to adapt to R2L tokenization during training. We evaluate these models on 1,000 instances of addition and subtraction of 4 - 8 digit numbers. We find that (1) gets 2%, (2) gets 24%, and (3) gets 12.3%, counter to our hypothesis. We conjecture that this is because (3) learns to predict two different representations of numbers between pretraining and finetuning, and when generating output, divides probability among R2L and L2R representations. In contrast, (2) continues to generate numbers L2R, even when conditioned on R2L representation of digits.

Evaluations on non-Latin languages

Thank you for pointing out the value of additional evaluations on non-Latin-script languages! This is indeed a point that we overlooked. We did additional evaluations on Chinese with $`Qwen-2.5`$ (since $`Llama3`$ and $`OLMo2`$ do not officially support Chinese), comparing character-level tokenization and canonical tokenization. We evaluate on Chinese MMLU and Chinese GSM. Shown in the table below, $`Qwen-2.5`$ is robust to character-level tokens in Chinese. We plan to expand these results and include them in the next revision.

We thank the reviewer for recognizing the “important” and “unexpected” finding that LLMs are robust to non-canonical tokenizations of text, and furthermore, can see “substantial gains” from R2L tokenization in arithmetic.

Model diversity in SFT analysis

We wish to clarify which section the reviewer is referring to. §3 (mentioned by the reviewer) identifies tasks where non-canonical tokenization leads to better performance, and uses only instruction-tuned models, not base/chat pairs. §4 (perhaps what the reviewer intended) does use base/chat pairs to identify the source of robustness, but we study both $`Olmo2`$ and $`Tulu3`$ model families. We would be happy to run additional experiments with clarification from the reviewer!

Related work in $`Deepseekv3`$

We thank the reviewer for this important reference that we were not aware of. Indeed, $`Deepseekv3`$ trains with non-deterministic tokenization, randomly dropping merges of punctuation tokens like \n\n to overcome the “prompt boundary problem” (so that the model can generate \n when conditioned on \n). However, we regret to share that we were not able to run $`Deepseekv3`$, which has 671B parameters, with our compute budget. We considered distilled versions of $`Deepseekv3`$, but this would not test the same hypothesis, since the distilled versions themselves were not trained with non-deterministic tokenization. We will definitely discuss $`Deepseekv3`$ in the next revision of the paper.

More compact format for Figures 3 & 4

Thank you for the suggestion, we plan to present this data in a table as suggested for the next revision.

On random tokenizations during pretraining

It would be very interesting to control for the extent of randomness in tokenization during pretraining and measure its impact on models’ robustness to non-canonical tokenizations. However, pretraining from scratch is not possible with our compute budget, and moreover, our paper focuses instead on demonstrating the unexpected finding that even with deterministic tokenization, models are surprisingly robust to non-canonical tokenizations. We believe the reviewer’s suggestion is a promising direction for future work, as non-deterministic tokenization during pretraining may enable further inference-time gains from adaptive tokenization.