Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

W1: The results are probably mostly applicable to a small number of well-funded labs.

Thanks for pointing this out! We want to clarify that we are not a well-funded lab either. Due to our limited computing resources, we can only afford to train models with up to 3B parameters in our experiments, as the cost of validating scaling law experiments is indeed very high.

However, we believe that our conclusions are beneficial to the general research community, especially for small labs. Our scaling laws with vocabulary provides a compute-optimal allocation suggestion, enabling small labs to train high-performance models without repeatedly trying different vocabulary configurations, thereby saving computing resources.

Even for teams who want to conduct scaling law experiments themselves, our derivative-based method offers a simple and feasible approach based on theoretical derivation. Researchers do not need to run a large number of scaling law experiments to obtain a good vocabulary configuration. This is particularly advantageous for small labs. We will also make all our scaling law experimental results public so that more people can benefit from our work.

Q1: The abstract states that “beyond the conventional 32K” – is this really the convention? See e.g. GPT4o.

Thank you for your insightful question! We acknowledge that there is no single "conventional" vocabulary size for language models, as it can vary based on the pre-training corpus and the intended use case. A vocabulary size of 32K is widely regarded as a common choice, particularly for models trained on English-centric corpora, such as Llama-1, Llama-2, and Mistral. Since our work primarily utilizes the English-centric SlimPajama corpus for pre-training, we have adopted the 32K vocabulary setting employed by these models as a "conventional" vocabulary size. We will modify the statement in the abstract accordingly to reflect this clarification.

As for GPT4o, we think its vocabulary size is relatively larger because it is designed to handle multiple languages (e.g., Chinese). This also highlights an important consideration for future research: determining the optimal vocabulary size for multilingual models, which we have discussed in Appendix B.4.

Our broader goal is to draw attention to the importance of vocabulary size in training language models and to encourage the appropriate allocation of computational resources for this aspect. Recently, there has been a shift in the industry, with major companies recognizing that their previous allocations for vocabulary were insufficient. For example, Llama has increased its vocabulary size from 32K to 128K, reflecting this evolving understanding. We hope this clarification helps, and will add the discussion in the revised version.

Q2: How does Table 2 look if you train on the same number of tokens instead of using the same FLOPs budget?

Thanks for your prompting question! As you suggested, we also trained the model using the same number of tokens, i.e., 129B tokens, beyond the same FLOPs budget setting. As shown in the following table, the performance of the model with the suggested vocabulary size of 43K improves further compared to the 32K vocabulary size when using the same number of training tokens. We will add the results in the revised version.

Dear Reviewer zt86,

We are deeply grateful for your detailed review and the insightful suggestions you provided for our paper. We have carefully considered and responded to each of your comments in our rebuttal.

As the Author-Review Discussion period is coming to an end, we want to ensure that all your concerns have been thoroughly addressed. If there are any remaining questions or issues, we would be glad to offer further clarification or make any necessary revisions.Thank you once again for your valuable feedback.

Best regards,

The Authors

W2.2: The experiments also looks not enough.

It is noteworthy that we conduct extensive experiments on 1200 models pre-trained from scratch (6 non-vocabulary parameters settings x 10 vocabulary sizes settings x 20 training data settings) for the fitting of our vocabulary scaling law. The key contributions in this paper is the several findings about how the vocabulary affects the model performance and how much compute should be allocated on the vocabulary based on the proposed 2 approaches.

Following the previous study [1,2,3], we mainly use the held-out validation loss value for the evaluation of the trained 1200 models. It is a better metric than the downstream tasks performance as the held-out loss provides an unbiased measure of the model’s ability to generalize to new data, but also enjoys high computing efficiency. Instead, the performance of downstream tasks has a great variety across different tasks, which is not suitable as the main evaluation metric.

The evaluation of downstream tasks is part of the ways to verify our prediction, therefore we do not take too much content to discuss it in our main paper. For downstream tasks, we conduct more experiments in the answer of your #Q1. The new results will be added in our polished version.

Q1: In determining the scaling law for the relationship between non-embedding model size and data (such as the Chinchilla law), why is it assumed that the vocabulary size is independent of these two factors.

Thanks for your question! We do not assume that the vocabulary size is independent of parameters and data. Instead, we make some adjustments in the Section of Preliminary: 1) We break down the total parameters into non-vocabulary parameters and vocabulary parameters; 2) We measure data not in tokens but in training characters.

By doing so, the vocabulary size $V$ is independent with the non-vocabulary parameters $N_{nv}$ and the number of training characters $H$. In an experimental configuration, the developers can vary the vocabulary size without affecting non-vocabulary parameters or training characters.

Then, we details our motivation why we separate the vocabulary parameter and non-vocabulary parameter below:

Traditionally, scaling up model parameters in language models has been approached in two ways: increasing depth (i.e., the number of layers) or width (i.e., the hidden size). Current empirical practices often involve expanding both simultaneously [4]. This approach overlook crucial distinctions in how different parameters benefit from parameters expansions. Non-vocabulary parameters can benefit from increases in both depth and width, allowing for more complex hierarchical representations and broader feature capture. In contrast, vocabulary parameters, associated with word embeddings and language model heads, are generally confined to a single layer, limiting their ability to benefit from increases in the model depth. This disparity in growth potential between non-vocabulary and vocabulary parameters suggests that to maintain a balanced growth rate, it is better to separate the vocabulary parameter and non-vocabulary parameter into consideration.

[4] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. 2023. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12342–12364, Singapore. Association for Computational Linguistic.

W1: Lacks performance on large-scale models, such as whether increasing the vocabulary size to a greater extent performs better than existing models in the market. Table 2's experiments look a little bit less.

Thank you for raising this concern. We share the intention to compete with existing powerful models in the market, such as Llama-2-7B. However, training a 7B model on 2 trillion tokens from scratch is far beyond our current computational resources. Nevertheless, we have evaluated our 3B models on more benchmarks to alleviate your concern.

Specifically, we have added new experimental results on the following benchmarks:

• MMLU:Massive Multitask Language Understanding benchmark for broad domain language evaluation.
• CommonsenseQA: A multiple-choice QA dataset for measuring commonsense knowledge.
• CoQA: Conversational question answering tasks to test dialog understanding.
• TruthfulQA: A QA task aimed at evaluating the truthfulness and factual accuracy of model responses.
• Lambada: Tasks designed to predict the endings of text passages, testing language prediction skills.

The following table combines the original Table 2 in the paper and the new experimental results. As shown, our prediction enables better model performance by adjusting the vocabulary size within different FLOPs budgets. The 3B model with a 43K vocabulary size outperforms the 32K counterpart on 11 out of 12 tasks using the same FLOPs budget. For example, we improve performance on ARC-C from 29.1 to 31.5. In conclusion, the model using our suggested vocabulary size (i.e., 43K) consistently outperforms its counterpart (i.e., 32K) by a clear margin.

Another feasible way to compete with models in the market would be continual pre-training with a larger vocabulary size. We believe this is a good topic for discussion, but it involves several non-trivial research challenges not strongly related to the main contributions of this paper, i.e., exploring the optimal compute allocation considering vocabulary sizes. Therefore, we will discuss it in the revised version and leave it as an important future work. The challenges we will discuss include:

• Expanding the vocabulary necessitates changes in the tokenization process, which can lead to inconsistencies in token segmentation.
• Ensuring that these new embeddings are compatible and effectively integrate with the pre-trained embeddings is non-trivial.
• Catastrophic forgetting of old word embeddings when learning new word embeddings.

We will discuss all the above in the revised version. Thank you again for your valuable comments.

W2.1: IsoFLOPs method is very sensitive.

Thank you for your insightful question. You raise a valid point – the IsoFLOPs-based approach can be sensitive to some extent, depending on the granularity, range, and quality of the fitting data. Since the pioneering work on scaling laws by Kaplan et al. 2020 [1] and Hoffmann et al. 2022 [2], the IsoFLOPs-based approach has become a widely-used tool to study the trend of model performance [3]. We have discussed it in our Appendix B.1, and we will add more details on how to reduce sensitivity, such as outlier data removal and repeated experiments, in our polished version.

To evaluate the goodness of fit, we use relative mean square error (rMSE) and the coefficient of determination (R^2). As shown in the table below (also in Figure 3), the results indicate a good fit, with rMSE < 0.001 and $R^2$ >= 0.89 for all the considered attributes: non-vocabulary parameters ($N_{nv}$), vocabulary parameters ($N_v$), and training characters ($H$). This suggests that these attributes follow a power law with respect to the FLOPs budget.

Furthermore, the optimal vocabulary predictions (Table 1) from the IsoFLOPs-based method and the derivative-based method are aligned across small-scale and large-scale models. This independent verification by the derivative-based method validates the predictions from the IsoFLOPs-based method. Therefore, we believe that the IsoFLOPs-based method works well in our case.

[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

[2] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

[3] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.

Dear Reviewer Y8zo

We sincerely appreciate your thorough review and the valuable suggestions and comments you provided for our paper. We have carefully considered each of your points and have addressed them in detail in our rebuttal.

With the Author-Review Discussion period nearing its conclusion, we want to ensure that all your concerns have been fully addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make any necessary revisions.

Thank you again for your thoughtful feedback.

Best regards,

The Authors

W1: This paper conducted experiments on language models of various parameter sizes, but the largest model tested was only 3 billion parameters.

We acknowledge the importance of evaluating our approach on larger models to establish its scalability. Increasing the model size necessitates pre-training on a larger corpus, which in turn demands more computational resources. For instance, conducting pre-training experiments on 7B models would require an immense computational budget exceeding 10^22 FLOPs, translating to approximately 6 weeks of training time on a cluster with 64 A100 GPUs. However, such a substantial level of computational resources is currently beyond our reach during the rebuttal period. Despite our desire to explore larger model sizes, we are constrained by the practical limitations of our available resources.

Nonetheless, the significance of the scaling law lies in investigating it through experiments on a relatively small scale to help us reasonably allocate computational resources when training a large model, thus avoiding wasted computational power. Our experiments with 1200 pre-trained models have demonstrated the existence of an optimal vocabulary size under FLOPs constraints, and the predictions from the theoretical analysis (derivative-based approach) and experimental fitting (IsoFLOPs-based approach) agree with each other. In fact, we have also made predictions for the pre-training of a 300B model, and the two approaches align well, and so we believe it should work for larger models.

Furthermore, the change in vocabulary size from 32K in Llama-2 to 128K in Llama-3, which resulted in performance improvement, can be also seen as a verification of our conclusion regarding increasing the vocabulary size when there are more computational budgets.

In conclusion, we appreciate your suggestions and we will try our best to conduct more experiments on larger models. We hope you can understand our computational limitations. We will also discuss this in the revised version.

Dear Reviewer EsaU,

We are truly appreciative of the time and effort you have dedicated to reviewing our paper. Your thoughtful feedback and constructive suggestions are valuable to us. We have carefully addressed your comments in our rebuttal to enhance the quality of our work. As we approach the final days of the Author-Review Discussion period, we would like to ensure that all your concerns have been comprehensively addressed. Should there be any remaining questions or issues, we are willing to provide further clarification or additional revisions.Thank you once again for your insightful contributions to our work.

Best regards,

The Authors