From Bytes to Ideas: Language Modeling with Autoregressive U-Nets

We thank the reviewer for this constructive review, and answer below the questions raised.

Although scaling the levels of the hierarchy (AU‑Net‑3/4) slightly improves performance in a FLOP‑matched setup, it seems like a problematic dimension to scale due to the vastly increased parameter count (1.3B for AU‑Net‑2 vs. 4.2B for AU‑Net‑4) at only slightly improved task performance.

You are right that adding stages increases the number of parameters and therefore GPU memory and checkpoint/storage size. However, it comes with other advantages, explained below.

• Because we match FLOPs per byte to the baseline, the training and inference compute per generated byte are the same. Extra parameters live mostly in higher stages that process shorter, downsampled sequences, so they add capacity without increasing per‑byte compute.

• Inference can be faster at a fixed quality target because the total KV cache can be smaller: upper stages cache over much shorter sequences, which reduces aggregate KV memory/traffic and can improve inference time in practice. (Inference is memory-bound)

• AU-Net decouples parameters from compute cost. A flat Transformer’s cost is often approximated as ≈ 6 × (#params) × (#tokens). In AU‑Nets, this is no longer the case; by varying depth, one can choose a larger‑parameter model at the same per‑byte cost (assuming GPU memory is not a big issue).

• Pooling extends effective context: deeper hierarchies handle longer inputs at the same compute budget and can predict further ahead. We do not claim definitive wins on such long‑horizon tasks yet; our benchmarks are mostly “classic LLM” and underweight long context and reasoning/generation. We plan to apply AU‑Nets to code/math settings where this matters more in the future.

• Finally, in high data‑to‑model (“overtrained”) regimes, we observe a positive trend: deeper AU‑Nets behave better, with AU‑Net‑4 competitive with or slightly better than AU‑Net‑2/3. This is encouraging since most, if not all, state-of-the-art models are heavily overtrained.

It is not clear how AU‑Net‑2 differs from prior work using adaptive pooling heuristics, e.g., BLT (whitespace split) and DTP (which should be referenced). BLT is only shown to be worse empirically, without a reason. No controlled comparison. Is Regex splitting the only substantial difference, or could gains be from training‑setup differences?

We should be clearer in explaining that regex splitting is not the only difference. AU‑Nets differ along two architectural axes and one optimization axis:

• Contextual downsampling: BLT embeds each patch/word independently (local pooling), whereas AU‑Nets use sliding‑window attention to form higher‑level tokens, allowing cross‑boundary context before downsampling and reducing boundary‑induced information loss.
• Simple, stable upsampling: AU‑Nets use linear upsampling without attention, which is more stable in byte‑level training in our experience.
• A practical contribution of our work is byte‑level–specific optimization guidance. Stable, high‑quality byte‑level pretraining benefits from different scaling for learning rate and batch size than subword models (e.g., more conservative LR and larger global batch/warmup). We consider this to be an important finding that contributes to having a model competitive with BPE.

On comparisons: we compute‑match pretraining, train a strong byte‑level baseline under the same optimizer/schedule family, and report results alongside published BLT and Hierarchical‑Transformer, both of which used DCLM as part of their pretraining data. While we did not re‑train BLT/DTP/Hierarchical‑Transformer under our hyperparameters due to compute constraints, using similar pretraining data provides a fair and informative comparison. We will add the missing DTP citation; to our knowledge, DTP does not report directly comparable downstream evaluations on any benchmark.

Thank you for your response. About the two points:

Vastly increased parameter count at slightly increased task performance

Inference can be faster at a fixed quality target because the total KV cache can be smaller

Faster inference is indeed attractive, but if this is a central contribution it would need supporting experiments and discussion in the paper. It is hard to trust this claim with zero empirical evidence.

one can choose a larger‑parameter model at the same per‑byte cost (assuming GPU memory is not a big issue)

This achieves the same effect as a mixture of experts, but from the only slightly improved scores from more AU-Net hierarchies in the paper it seems like mixtures-of-experts might scale more favorably. To make this claim effectively, the paper would need to compare against MoEs in some way.

Pooling extends effective context: deeper hierarchies handle longer inputs at the same compute budget and can predict further ahead. We do not claim definitive wins on such long‑horizon tasks yet; our benchmarks are mostly “classic LLM” and underweight long context and reasoning/generation. We plan to apply AU‑Nets to code/math settings where this matters more in the future.

I would be very curious to see these results!

Comparison to prior methods (primarily BLT)

Regarding the differences to BLT (which could non-spuriously cause the high score difference):

Contextual downsampling: BLT embeds each patch/word independently (local pooling), whereas AU‑Nets use sliding‑window attention to form higher‑level tokens, allowing cross‑boundary context before downsampling and reducing boundary‑induced information loss.

This is not correct. BLT uses a sliding window of 512 bytes (across patch boundaries) in the local encoder in the released checkpoints: https://github.com/facebookresearch/blt/blob/4ae7a625940743c9438a20ee8a8d7fab898bcd69/bytelatent/model/local_models.py#L258-L285. In this sense, the two are equivalent.

Simple, stable upsampling: AU‑Nets use linear upsampling without attention, which is more stable in byte‑level training in our experience. A practical contribution of our work is byte‑level–specific optimization guidance. Stable, high‑quality byte‑level pretraining benefits from different scaling for learning rate and batch size than subword models (e.g., more conservative LR and larger global batch/warmup). We consider this to be an important finding that contributes to having a model competitive with BPE.

The finding on upsampling is interesting but would need experimental support via an ablation.

Overall, the response does not address my main concern. From the paper, it seems like AU-Nets vastly outperform BLT with similar FLOPs/token budgets. My question is: where is this coming from? And are these differences really due to the AU-Net architecture or due to tricks which when applied to BLT in the same way would yield equivalent or better performance? It seems like it might be the latter.

You mentioned two differences (I am skipping the first one since I believe it is incorrect): the upsampling and the learning rate schedule / batch size. Both of these would need to be ablated and compared with BLT to properly show that AU-Net's are indeed the better choice.

I am thus maintaining my score.

We thank the reviewer for this careful and constructive review, and answer below the questions raised.

a/ Results of Table 1 should be discussed somewhere
b/ I don't seem to find descriptions of what AU-Net 1/2/3/4 are precisely (with dimensions)

Table 1 was intended as a quick summary. We’ll add a short discussion in the introduction to contextualize the numbers and summarize key take-aways. AU-Net 1 was a typo and has been updated to AU-Net 2. AU‑Net 2/3/4. Architectural details are delegated to the Appendix due to lack of space, but we will add some basic details (such as number of layers and embedding dimensions) to the main text.

a/ you claim to find stable optimization hyperparameters (line 158ff) but i can only see some theoretic deviation - i expect further discussion in appendix with sweep results.

Thank you for pointing this out—we will add the hyperparameter sweep results and further discussion in the Appendix.

b/ i might have missed it but you do not seem to argue how you derive to your embedding dimensions - how/why did you choose 512/ 2048/ 3072 in Fig1 and 3 layers for the outer ones? any discussion of evidence on optimality from sweeps etc highly appreciated. Is this not required for a/ as well to be able to argue abt stable parameters?

Our first ablations revealed that the first level does not need to be large, while later levels benefit from more capacity, so that more compute is spent on increasingly abstract embeddings. This informed choices like 512/2048/3072 and three outer layers (see Appendix C.2). We will explain further the rationale behind first stage embedding size in the Appendix.

please add "code" to limitations/discussion

We will add an explicit limitation regarding code/math. We focused on DCLM to enable clean comparisons. Training this architecture on more diverse code and math data to evaluate reasoning and generation‑heavy workloads is an exciting direction for future work.

why do you not show AU-net 3/4 for 8b in table 2?

Due to our limited compute budget.. We agree these experiments are valuable and will add them when resources permit.

please address above's weaknesses

Covered above: we will (i) discuss Table 1 in the intro, (ii) add architectural details and dimensions for AU‑Net 1–4, (iii) include sweep results and discussion of hyperparameter stability and design choices in the appendix, (iv) add the code/math limitation, and (v) clarify the compute constraint behind missing AU‑Net 3/4 at 8B.

i seem to miss sth- in table 2 you show 51% MMLU on AU-net2 at 1e22, compared to baseline 49.2 at 9e21, but it appears to look worse in figure 3 or am i just underestimating the exp-scale axis??

The models at 7B in Table 2 operate in a different data-to-model ratio (5 in Table 2 and 10 in scaling laws), and the scaling laws had the most recent hyperparameter sweep values, resulting in better performance for the baseline and slightly better for AU-Net. MMLU is especially sensitive to learning rate. (All the detailed hyperparameters are in Appendix D.) We will adjust the table captions accordingly.

We thank the reviewer for this careful and constructive review, and answer below the questions raised.

I think the paper would be greatly strengthened if it more explicitly addressed 1 by including explicit comparisons of the proposed word-based splitting to simpler baselines (e.g., fixed window size at each level, as previously done in the literature), keeping the architecture, scale, and training data constant.

We agree this is important. We had early runs that favored dynamic, content‑adaptive pooling, so we allocated more compute there. If the paper is accepted, we commit to include a fixed‑window comparison under the same architecture/scale/data by camera‑ready, as running these experiments will take time.

The proposed splitting approach is specific to languages which are easily split into words, it is unclear how to generalize well to broader language where this is more difficult such as Chinese or Japanese.

We agree. Our current scope focuses on languages with a clear word segmentation. As acknowledged in the limitations section, extending our method to scripts without explicit word boundaries (e.g., Chinese, Japanese, Thai) remains an open question.

Similarly, although less critical, an assessment of a byte level transformer baseline (i.e., no hierarchy) missing from the key results in table 1. Table 2 has a byte level (no hierarchy) comparison for a smaller amount of data (80B tokens). Running a byte level model on 370B llama3 tokens would be more than 4 times as expensive than AU-Net or BPE model, which would be too much compute for a model of this size. Contribution C2 is a bit overstated. This advantage should apply to any "byte-level" language model, and is not unique to this work. E.g., Byt5 (https://arxiv.org/abs/2105.13626) or cited baselines such as MegaByte [17], BLT [5], etc.

Agreed—this advantage is common to byte‑level models (e.g., ByT5, MegaByte, BLT). By showing parity with a strong BPE baseline with minimal trade‑offs, we provide concrete evidence of its practical value. We will soften C2 accordingly.

equation after line 147. Why is there a fixed scaling factor (1/k) between F_{model/byte} and F_{model/token}? Shouldn't this be a function of the actual architecture being used (as in the following paragraph about AU-Net)?

The equations map token‑level to byte‑level scaling for any byte‑level LM. They are architecture‑agnostic: each model can have a different FLOPs/byte, but the conversion itself doesn’t depend on the specific architecture. We will clarify this point.

The text uses somewhat misleading language about the representation at deeper levels of the hierarchy "predicting further ahead", similar to multi-token prediction (Fig. 1 and Sec. 1), which is, to my understanding, imprecise. It seems true that the pooled representations in deeper stages can more efficiently take longer context into account -- they essentially effectively broaden "receptive field" per token compared to shallower stages. But, assuming fully causal self attention and pooling/expansion (as shown in Fig. 1), a network trained only to do next-byte-token prediction is not actually predicting any further ahead. To the contrary, it is looking further back (per-timestep).

You are correct that when taken as a whole, our model only does next-byte prediction. However, outputs of deeper stages influence the prediction of multiple bytes in the future. All layers are causal but at different levels of granularity. Stage 1 outputs predictions that directly predict the next byte, Stage 2 outputs vectors that help the prediction of the next word, Stage 3 outputs vectors that help the prediction of the next two words. In this sense, Stages 2 and 3 have to predict further in the future. Finally, stages 2 and 3 represent the vast majority of the total compute of the model, which forces it to rely less on short term byte level predictions and more on word and word group level predictions. We will clarify this section to make it more precise.

Table 1 appears to be mislabeled: the AU-Net 1 and 2 rows contain the same results as the rows labeled AU-Net 2 and 3, respectively in Table 2.

Thank you, we will fix this.

The notation used in the "Hyperparameter scaling laws" section (lines 158-164) is not clearly defined. What is BSZ (batch size?)?

We will define BSZ (“batch size”) on first use.

Also, the first mention of DCLM (line 144) should probably include a reference.

We will cite DCLM at its first mention.

We thank the reviewer for this thorough review, and answer below the questions raised.

The use of an English-centric dataset (DCLM) may unfairly disadvantage BPE baselines, which are specifically designed to handle diverse vocabularies. The authors don't justify why they chose DCLM over more multilingual datasets, making it unclear how AU-Net would compare with Transformer + BPE baselines on multilingual benchmarks given a standard multilingual training corpus.

We chose DCLM to ensure comparability with existing literature and because it was the state-of-the-art pretraining dataset when starting this work. While BPE can be trained multilingually, most popular tokenizers are English-centric, and BPE's pre-tokenization rule often splits on spaces, similar to our approach. Our model's ability to use learnable embeddings instead of a fixed embedding table offers greater flexibility, essentially providing an infinite vocabulary. We agree that exploring multilingual and code datasets in future work would be highly valuable.

The paper lacks precise mathematical formulations of the model architecture, relying primarily on prose descriptions. Stating the actual computations performed at different stages and modules would help to better understand the architecture.

We apologize for the lack of precise mathematical formulations in the paper. We will add formulas for the downsampling and upsampling operations for clarity and completeness, while still referring to the code for more detailed implementation.

Not a big weakness, but many languages in the FLORES-200 evaluation are closely related variants within the same language families (e.g., Spanish, Asturian, Catalan, Galician for Romance; or various Italian dialects), which might inflate the apparent multilingual improvements.

We acknowledge this point regarding the relatedness of some languages in the FLORES-200 evaluation. Our aim was to demonstrate the effectiveness of byte-level processing for cross-lingual transfer, especially for low-resource languages, and the results still indicate a consistent improvement over baselines, even if some languages are closely related.

Although acknowledged by the authors, the lack of support for non-space-delimited languages represents a fundamental architectural limitation.

We agree that supporting non-space-delimited languages (e.g., Chinese, Japanese, or Korean scripts) remains a limitation. We have designed the splitting implementation to be quite generic, allowing future work to define new splitting functions tailored to these languages. We have recently discovered the existence of the uniseg python package that uses dictionaries to segment text into words in many languages without relying purely on spaces.

Stage 1 processes raw bytes via a self-attention mechanism that creates byte-level embeddings before Stage 2 pools at word boundaries. How exactly are these raw bytes fed into the self-attention mechanism?

Raw bytes are obtained by decoding a string with UTF-8, which produces a sequence of bytes. These bytes are then fed into the self-attention mechanism after being embedded using a 258-entry embedding table, which includes Beginning-of-Sentence (BOS) and End-of-Sentence (EOS) as special characters.