We thank the reviewer for their time and are grateful for their positive feedback, and we believe their comments helped us improve the paper’s clarity. We modified the paper to improve the its readability, with specific changes mentioned below.

Weaknesses

Mention of Other Models in the Main Manuscript: The decision to move the detailed “related works” section to the appendix was due to page limitations. However, we appreciate reviewers' suggestions to mention some critical prior work (i.e., Jais, AceGPT) on the main paper. Therefore, we added the penultimate introduction section to address this valid concern.

Questions

We appreciate the reviewer’s queries, and below is a response to each question.

Data mixture

Page 3 says 4T English tokens then and 5.2T tokens for 30B pre-training. It was a bit confusing what is the 5.2T.

The initial 4 trillion tokens were constructed purely from open-source repositories such as Dolma, Pile, Pes2o, and RefinedWeb. Before we started training our 30-billion-parameter model, RedPajama v2 (RPv2) released 84 snapshots of CommonCrawl data, along with rich metadata accompanying the text. With the new improved collection of the RPv2 data, we applied various metadata-based filtering criteria to extract 5.2 trillion tokens from the original 30 trillion token dataset. The data filtration criteria used to reduce the dataset from 30 trillion tokens to 5.2 trillion tokens are detailed in Tables 8 and 9. We have also explained more on “Why did we change the training data distribution for 30B experiments” in the FAQ section in the appendix. Please feel free to let us know if you have any further queries.

Also Table 1 lists English as 660B tokens from mixed corpora -- is this a subset from the 4T English Only column?

Yes, this is a subset of the 4T tokens. Thanks for pointing this out. We have updated Table 1’s caption to clarify this.

Figure Legend

The legends in Figures 5 and 6 show what line colors and line styles mean. Line styles are shown with black lines. For example, in Figure 5, the red dashed line denotes the MMLU English score using randomly initialized embeddings, since the red color denotes randomly initialized embeddings, and the dashed line denotes English MMLU scores. If needed, we can add further clarification in the Figures’ caption.

Why Llama2?

When we began our initial training, the only Chinchilla optimal (with extended training) model available was LLaMA-2. In addition to that, LLaMA-2 models were not only open weight, it also came up with a paper explaining the complete data distribution of the training datasets which helped us doing some initial testing of the model and selection of on our own data mixtures. This is why we used the LLaMA-2 family of models. After training ALLaM models on top of LLaMA-2, while we started pre-training from scratch, we observed the emergence of several impressive open weight models, such as Mistral, Gemma, Yi, Qwen-2 etc.

We would like to express our gratitude to the reviewer for providing an in-depth response covering the big-picture contributions, minor typographical errors, and everything in between. We believe that the reviewer's comments, suggestions, and guidance have greatly helped improving the paper. We have addressed the reviewer's specific comments below.

Weaknesses

Thank you for highlighting these typos and mistakes! We have addressed all of these issues in our paper.

Questions

How many Arabic tokens already existed in the LLama2 backbone, and how many were added after the expansion?

We expanded Llama 2’s vocabulary from 32,000 to 61,586 tokens with all the added tokens being Arabic. We ran the following regex matching script to check the number of tokens that contain at least a single Arabic character:

from transformers import LlamaTokenizer
import regex as re

arabic_regex = r'^(?=.*\p{Arabic}).*$'

def count_arabic_tokens(tokenizer: LlamaTokenizer) -> int:
    return sum(re.match(arabic_regex, token) is not None for token in tokenizer.get_vocab())

Llama 2 contained 46 Arabic Tokens (mostly single Arabic characters), while ALLaM contained 29,552 Arabic Tokens post-expansion.

Note that Llama 2’s tokenizer does not include all Arabic letters, and thus has to rely on byte fallback for less common Arabic letters. For instance, it tokenizes the letter “ؤ” as two bytes ['<0xD8>', '<0xA4>'].

Why did the training mix change for the 34B model specifically?

Not all the training was done at the same time. As the training progressed, we gained more knowledge about our process, data, and the entire ecosystem of our training engine. Iterating over a single training run incurred significant costs, so we always prioritized quality over ablations for large-scale training runs. Given the available compute and deadline, we were able to conduct only one training run of the 34B model. We discovered that we could apply custom filters to a large data collection based on our use cases and preferences. In the first phase, we used an open-source data collection, and in the second phase, we filtered a collection of 84 Common Crawl snapshots down from approximately 30T tokens to 5.2T tokens. After filtering, we performed a few manual checks to verify the data quality. More details are included in the Frequently Asked Questions section in the appendix.

Thank you for your review. We respectfully disagree with the statement that the paper is closer to a technical report rather than a research paper. Technical reports are typically only descriptive, with many obfuscated details, which may or may not carry significant details on novelty with the necessary information for reproducibility. However, we believe our paper presents novel ideas that aid in general LLM second language acquisition, and potentially even second modality acquisition. Our novel contributions are mentioned in the last paragraph of the introduction section, and are also highlighted independently by reviewer 9J4g. These include vocabulary expansion, embedding initialization, methods to maintain original language performance, effect of translation data, and data mixture considerations. We also added a detailed description of our data selection in Tables 8 and 9. Additionally, as opposed to technical reports, we tried to be as transparent as possible, mentioning our ablation results for translation, embedding initialization, and language mixing ratios, as well as detailed data descriptions, cleaning methods, and training logs. We also explained a lot of our decision-making process and incentives in the Frequently Asked Questions Section and added many details in the appendix for curious readers.

Regardless of our best effort, we understand we can miss any details that the reviewer cares about. Also, we appreciate reviewers' disagreement and constructive feedback. During the rebuttal period, we are here to reply to all the queries of our respected reviewer. If you want to know any specific technical details, please feel free to ask and we will put our highest effort to provide you the best possible reply.

Instruct Model vs. Base Model Results Similar to the data distribution suggestion, we had to move this discussion to the appendix’s Frequently Asked Questions section due to page limits.

The reason we reported the Instruct model results instead of the base model in the main paper is twofold:

1. Blurred distinction between base and instruct models: Modern pretraining often involves incorporating supervised fine-tuning (SFT) data, including alignment data designed to improve user interactions. Consequently, the distinction between a base model and an instruct model has become increasingly unclear. Many models today are pre-trained with some degree of alignment, making it difficult to evaluate them purely as base models.

Here is an insight from our evaluation results.

For instance, as shown in Table 11, the Qwen2-7B-base achieves a score of 77.94 on GSM8k, while ALLaM-7B-base (trained from scratch) achieves 16.98, with a significant delta of 60.96. After supervised fine-tuning, ALLaM-7B-instruct (from scratch) scores 53.6, while Qwen2-7B-instruct scores 77.86, reducing the delta to 24.26. Although ALLaM shows an improvement of 36.62 points during SFT, Qwen2 experiences no notable gain in performance in this phase. We suspect this is due to the inclusion of alignment data during Qwen2's pretraining phase. Thus, while Qwen2 models are inherently better than ALLaM, the performance gap between Qwen2 and ALLaM at the base level does not necessarily reflect true model capabilities due to the suspected presence of alignment data in the base model.

We've added these new discussions in the Frequently Asked Questions section of the paper.

2. Focus on user interaction: The primary goal of building these models is to optimize them for user interaction. Since users will interact with the instruct version of the model, it makes sense to report the results of the model in its instruct phase. This ensures that the reported performance is reflective of the actual experience users will have, making the results more relevant and impactful for the paper’s audience.

We thank the reviewer for taking the time to provide very detailed and constructive feedback. We believe that addressing all the reviewer’s concerns, both here and in the paper, will not only enhance the quality of the paper but also improve its reproducibility.

Weaknesses

Statement of Originality We agree with your assessment that all research should have a clear statement of originality. Therefore, the last paragraph of the introduction now enumerates the contributions of our work. Additionally, reviewer 9J4g independently mentions sources of originality in their assessment, in which they mention data curation methods, vocabulary expansion, and embedding initialization as novel contributions.

Error Analysis We acknowledge the need for a more granular error analysis and appreciate your suggestions on the direction it could take. Given the complexity of the Arabic language, identifying specific linguistic constructs or domains where the model exhibits challenges remains an active focus in our ongoing research. We aim to expand this analysis by categorizing errors, particularly in areas such as linguistic complexity and domain-specific knowledge within fields like politics, history, and STEM. While time constraints prevented a full exploration in this submission, we are actively working on developing a taxonomy of error types which will inform subsequent iterations of ALLaM. We look forward to incorporating these insights in coming days during rebuttal, providing a more comprehensive view of the model's strengths and areas for refinement.

For human evaluation, we did perform a comprehensive evaluation of common failure cases, such as ethical dilemmas, middle eastern culture, religions, illegal activities, human rights, locale awareness, and personality. We described in the data paragraph of Section 3.2 how we targeted these issues through our preference tuning approach.

Additional Areas for Improvement

Motivation for Changing Training Data Distribution We tried addressing this question in the Frequently Asked Questions section in the appendix. We are sorry that we couldn’t add this to the main section of the paper due to page limitations.

Not all the training was done at the same time. As the training progressed, we gained more knowledge about our process, data, and the entire ecosystem of our training engine. Iterating over a single training run incurred significant costs, so we always prioritized quality over ablations for large-scale training runs. Given the available compute and deadline, we were able to conduct only one training run of the 34B model. We discovered that we could apply custom filters to a large data collection based on our use cases and preferences. In the first phase, we used an open-source data collection, and in the second phase, we filtered a collection of 84 Common Crawl snapshots down from approximately 30T tokens to 5.2T tokens. After filtering, we performed a few manual checks to verify the data quality.

Fair Comparison of Models Thanks you for acknowleding the difficulty of the fair evaluation of the model. We do agree that comparing our models with larger models is not an ideal solution. We believe the ideal metric would be measuring model performance given a fixed model size and fixed Arabic dataset size. However, given the opacity of data mixture information in most top models as well as number of token used to train the model, it is impossible to measure such a metric. Even if we were able to obtain the number of Arabic tokens for other models, we would not be able to control for data quality. Therefore, given these constraints, we were only able to compare our models with larger models to compensate for a potential data advantage.

In our work, we mainly used, (i) Automatic Evaluation (ii) LLM as a judge (iii) Human evaluation

We have added a wide range of evaluation on the different benchmarks in the appendix (table 10, 11) . We are very serious about doing our absolute best for the evaluation and fair comparison of the models so that we can continue building the state-of-the-art models. We are constantly looking for new ways to evaluate the models. If the reviewer has any actionable suggention on adding new analysis and/or using new metric for fair comparison, we'd love to do it during the rebuttal. Please let us know.

We thank the reviewer for their valuable comment. Following up the suggestion of further error analysis, we identified several patterns where the model's performance could be refined through extensive human evaluation. These observations can be summarized as follows:

1. Repetition Issues: In certain versions, the model exhibited a tendency to overgenerate or repeat content. This behavior varied with temperature settings. After extensive testing, we found that a temperature of 0.6 yielded the most balanced outputs.
2. Translation Challenges: The model sometimes regenerates the translation instruction within the translated output, or attempts to summarize the shared context while translating it, rather than focusing solely on the translation itself.
3. Variation in Outputs: When tasked with summarization or translation, the model provides multiple outputs.
4. Precise Instruction Following: The model does not always adhere to specified output length or structure. For instance, when requested to summarize a text in 100 words, it might exceed this limit or produce a summary longer than the input text.
5. Complex Arabic Proofreading: The model faces challenges with certain Arabic-language tasks, such as applying proper punctuation, grammar, and diacritics. While it may provide partially correct answers, it often fails to deliver fully accurate or comprehensive outputs in these areas.

Below are examples of failure cases. For brevity, we omitted parts of the questions/answers with "[...]" to focus on the failure cases.

Following the review comments and the ongoing discussion, we have added a few modifications to the manuscript in addition to the previously mentioned modifications. These include:

1. Adding questions in the Frequently Asked Questions section (Appendix B):
   • "What are failure cases of ALLaM?" Along with Table 7 showcasing examples.
   • "How many Arabic tokens were in Llama 2, and how many did you add to ALLaM?"
2. Modifying Tables 11 and 12 to include the SILMA model.
3. Fixing clerical errors for ALLaM 34B in Table 4.
4. Other minor organizational edits.