Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

We sincerely appreciate your valuable comments. Below, we provide responses to your concerns.

Weakness 1: Comparison between Native Alignment and Post-Alignment

To meet your curiosity, we conducted an experiment comparing native alignment and post-alignment. The results show that the native alignment approach outperforms the post-alignment methods (DPO) in this case. However, we afraid this is not a fair apples-to-apples comparison for the following reasons:

1. The data used for native alignment and DPO were not of the same scale.
2. Native alignment and DPO are complementary methods that operate at different stages rather than being exclusive.

Therefore, the conclusion drawn is specific to these ad hoc settings involving native alignment and DPO. The results may differ if the settings are changed.

Considering that native alignment and post-alignment methods (such as DPO) are orthogonal and can be applied simultaneously in the same model, experiments on LLMs with and without DPO show that native alignment can enhance cultural alignment. This indicates that both native alignment and post-alignment are beneficial and complementary approaches to alignment.

Experiment Settings

We utilized the LLaMA-Factory framework, employing LLaMA3-Tamed-8B as the backbone for the experimental group focusing on native alignment and Meta-LLaMA3-8B as the control group. We performed instruction tuning on both pre-trained models using an Arabic supervised fine-tuning (SFT) dataset, resulting in the fine-tuned models named LLaMA3-Tamed-8B (Native alignment + SFT) and LLaMA3-8B (SFT). For post-alignment, we selected DPO training as a representative approach, using an Arabic preference dataset. Post-alignment was conducted on both chat models, namely LLaMA3-Tamed-8B (Native alignment + SFT + DPO) and LLaMA3-8B (Native alignment + DPO). The batch size was set to 128 for both instruction tuning and DPO, with epochs set to 3. All other experimental settings followed the default settings in the framework. We evaluated the performance of the instruction-tuned models and the post-alignment tuned models on the same Arabic benchmarks shown in the paper with zero-shot setting.

Weakness 2: Potential Trade-offs of Native Alignment

We considered the potential trade-offs of the native alignment approach, which include:

1. Harmlessness vs. Helpfulness: In Section 4.1, we leverage Arabic BeaverTails to analyze the trade-off between LLM harmlessness and helpfulness with the increasing amount of aligned data. It shows that with the increasing number of aligned data, both harmlessness and helpfulness increase positively. This observation can be concluded that the aligned data can be positive to both the helpfulness and harmlessness aspect to LLM.

2. Arabic Dialects vs. MSA: An author, a native Arabic speaker, reviewed the data before and after the native alignment rewriting and identified three main issues with the Arabic dialects: (1) Minor Change: Dialect expressions are preserved; (2) Information Loss: Some dialect-specific information was lost during the rewriting process; (3) Hallucination: Incorrect interpretation of dialectal meaning. Additionally, we found that GPT-4 also struggles with Arabic dialects.
3. Hallucinations: To determine whether hallucinations are inherited from the original data to the rewritten data, three authors manually reviewed 90 rewriting pairs. Although identifying hallucinations can be somewhat subjective, they adhered to a consistent text-written guideline. The hallucination ratios are as follows:

The overall hallucination ratio was found to be within an acceptable range. While addressing hallucinations in native alignment remains a challenging task beyond the scope of this paper, our empirical results justify the approach of alignment during pre-training, even with this level of tolerance. We plan to address the hallucination issue in future work.

Weakness 3: BeaverTails Translation

The evaluation benchmark, the BeaverTails dataset, is in English. To evaluate Arabic LLMs, we used Baidu translation API to translate the questions into Arabic. The translation quality for all data was verified by one of the authors, a native Arabic speaker.

Questions

The native aligned LLMs introduced in the paper, LLaMA3-Tamed-8B and LLaMA3-Tamed-70B, are continuously pretrained using the Meta-LLaMA-3-8B checkpoints. Additionally, all ablation studies presented in the paper and the rebuttal are based on this continuous training approach.

Weakness: Why the proposed data cleaning method (native alignment) is a kind of alignment.

What is alignment?

“Alignment” refers to the process of ensuring that LLMs act in accordance with user intentions. Models are considered aligned if they are helpful, honest, and harmless [1]. Alignment is necessary because pre-training data may contain unaligned content, such as ethical issues or religious taboos. This is particularly crucial in regions where religion plays a significant role, like the Arabic world. Although unaligned data may constitute a small proportion of the total data, it can conflict with widely accepted human values.

[1] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. and Schulman, J., 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, pp.27730-27744.

Post-alignment vs. Pre-alignment (native alignment)

Instead of alignment after pre-training from conflicting data (as RLHF does)，the proposed native alignment approach works at the pre-training stage. After extensive pre-training on both aligned and unaligned data, RLHF is used to encourage the generation of positive content and discourage negative behaviour. Native alignment, however, is akin to removing unaligned content at the beginning. As the saying goes, "An ounce of prevention is worth a pound of cure," meaning that preventing issues by avoiding unaligned information is often more effective and less costly than addressing problems deep within the model later on.

Additionally, we have added an experiment showing the comparison between native alignment and post-alignment, please check ‘Author Rebuttal - Additional Experiment I’ for more details.

Comparison with Data Cleaning?

Several notable data cleaning research efforts include:

• RefinedWeb [1], which demonstrates that properly filtered and deduplicated web data can significantly enhance model performance.

• SlimPajama [2], which involves fine-grained data processing through filtering and deduplication.

• WRAP [3], which uses an instruction-tuned model to paraphrase web documents, focusing on stylistic elements like 'Wikipedia' or 'question-answer format'.

While most conventional data cleaning methods aim to remove low-quality content [1,2] and a few data cleaning work [3] focus on format polishing, Native Alignment additionally seeks to align LLMs with human preferences. Data Cleaning and Native Alignment are not mutually exclusive; they are complementary which focus on different aspects (data quality vs. value alignment). Particularly, in some sense, the proposed approach, native alignment, could be considered as a special case of data cleaning methods that not only improve data quality as conventional data cleaning methods did but also improve value alignment.

We have added an experiment showing the comparison between native alignment and conventional data cleaning procedure, RefinedWeb, please check ‘Author Rebuttal - Additional Experiment II’ for more details.

[1] Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E. and Launay, J., 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.

[2] Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., & Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. Retrieved from https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama

[3] Maini, P., Seto, S., Bai, H., Grangier, D., Zhang, Y. and Jaitly, N., 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380.

Concern on generalization

Why Arabic LLMs? Our paper focuses on Arabic LLMs due to the high sensitivity of Arabic-speaking cultures to religious errors [4]. This context provides a stringent test for native alignment. Misalignments in these models can hinder their adoption in the Arab world, whereas in more open environments with higher error tolerance, minor issues can often be addressed post-alignment. The relative rarity of large open-source models in Arabic-speaking regions underscores the importance of our aligned pre-training corpora, which aim to support the development of such models and introduce the concept of native alignment.

Limitation on Computation Resource The method we propose is highly GPU-intensive since it needs a LLM for massive amounts of data rewriting. This could be applied to the Arabic world due to the high sensitivity of Arabic-speaking cultures to religious errors. However, in languages like English where there is more tolerance for errors due to greater freedom of speech, there are many lightweight methods for alignment such as RLHF; the GPU-intensive native alignment method is too expensive for implementation in the free world where alignment requirement is not as strict as the one with zero-tolerance regions, such as Religion.

Additional Experiment on English To meet your curiosity, we conducted a preliminary experiment on English to explore the potential for generalization using native alignment. For further details, please refer to ‘Author Rebuttal - Additional Experiment II’.

[4] Farghaly, A. and Shaalan, K., 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4), pp.1-22.

Question on Experimental settings

LLaMA3-Tamed-8B and LLaMA3-Tamed-70B are continuously pretrained using the pretrained Meta-LLaMA-3-8B checkpoints. Additionally, all other ablation studies conducted in the paper and rebuttal are based on continuous training.

Thank you for your valuable review and for pointing out the weaknesses in our paper. We recognize the importance of clearly emphasizing the novelty of native alignment. In the final version of the paper, we will ensure that the distinctions and connections between our proposed native alignment approach and traditional data cleaning methods, as well as post-alignment techniques like RLHF, are clearly highlighted.

The additional experiments conducted during the rebuttal period will also be incorporated into the final version. Furthermore, we will expand the related work section to provide a more comprehensive context for our contributions. We have identified the following research to include and would greatly appreciate any additional references you might suggest:

1. Data Cleaning Methods:

   • Hegazi, M.O., Al-Dossari, Y., Al-Yahy, A., Al-Sumari, A. and Hilal, A., 2021. Preprocessing Arabic text on social media. Heliyon, 7(2).
   • Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N. and Presser, S., 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
   • Wenzek, G., Lachaux, M.A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A. and Grave, E., 2019. CCNet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359.
   • Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E. and Launay, J., 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
   • Fan, R.Z., Li, X., Zou, H., Li, J., He, S., Chern, E., Hu, J. and Liu, P., 2024. Reformatted alignment. arXiv preprint arXiv:2402.12219.
   • Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L. and Zhang, S., 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36.

2. Post-Alignment:

   • Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y. and Gan, C., 2024. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36.
   • Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. and Schulman, J., 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, pp.27730-27744.
   • Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V. and Rastogi, A., 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267.
   • Zhu, B., Jordan, M. and Jiao, J., 2023, July. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning (pp. 43037-43067). PMLR.
   • Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y. and Wang, H., 2024, March. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 17, pp. 18990-18998).

Again, thank you for your feedback. We would be glad to engage in further discussion if you have any remaining concerns.

• [1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

• We only retained lines that ended in a terminal punctuation mark (i.e. a period, exclamation mark, question mark, or end quotation mark). • We discarded any page with fewer than 5 sentences and only retained lines that contained at least 3 words. • We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”. • Many of the scraped pages contained warnings stating that Javascript should be enabled so we removed any line with the word Javascript. • Some pages had placeholder “lorem ipsum” text; we removed any page where the phrase “lorem ipsum” appeared. • To deduplicate the data set, we discarded all but one of any three-sentence span occurring more than once in the data set.

• [2] SlimPajama: A 627B token, cleaned and deduplicated version of RedPajama

SlimPajama was created by cleaning and deduplicating the 1.21T token RedPajama dataset from Together.

• [3] The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

these pipelines usually combine a variety of stages: (1) language identification...; (2) filtering rules and heuristics, ...; (3) ML-based quality filtering, ...; (4) deduplication, ...

• [4] Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

paraphrase documents on the web in specific styles such as “like Wikipedia” or in “question-answer format” to jointly pre-train LLMs on real and synthetic rephrases.

[1] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P.J., 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140), pp.1-67.

[2] Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., & Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. Retrieved from https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama

[3] Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E. and Launay, J., 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.

[4] Maini, P., Seto, S., Bai, H., Grangier, D., Zhang, Y. and Jaitly, N., 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380.

We politely find a data cleaning work which calls itself alignment, see Reformatted Alignment [1]. [1] introduces a method called REALIGN that reformats existing instructional data to enhance its quality for better mathematical performance; the authors argue such reformatting is a kind of alignment.

The work utilizes data rephrasing to achieve the goal of 'alignment.' The differences between 'native alignment' and [1] in alignment are

• [1] rephrased data at the Supervised Finetuning stage; this work (native alignment) rephrased data at the pre-training stage;
• [1] is more like a format alignment while native alignment additionally emphasize value alignment in the sense of reducing harmfulness and biases in data.

[1] Fan, R.Z., Li, X., Zou, H., Li, J., He, S., Chern, E., Hu, J. and Liu, P., 2024. Reformatted alignment. arXiv preprint arXiv:2402.12219.

We sincerely hope that you can consider re-evaluating it.

Question I: Scalability of Alignment LLM Expert

The alignment LLM expert is fine-tuned on pre-trained LLMs (Qwen-1.5-4B-Chat). To ensure the rewriting quality of the trained alignment LLM expert, we randomly sampled 50 data points from the pre-training corpus and processed them through the alignment LLM expert. One of the authors (native Arabic speakers) checked these rewritten texts to verify the quality. Additionally, we used GPT-4 to conduct a further extensive evaluation on the final rewritten text using the prompt shown below (Prompt to Evaluate Rewriting Quality).

As shown in the results above, the small LLM rewriter, trained on rewriting seed data, achieved comparable performance to the GPT-4 rewriter across aspects such as 'Format', 'Accuracy of Information', 'Content Moderation', 'Advertisement Removal', and 'Level of Detail'. The experiments concluded that:

1. Human evaluations confirm that the rewritten quality is meaningful.
2. The trained LLM expert can achieve performance close to the GPT-4 rewriter.

Question II: Explanation on the data mixture

Dangers of Mixed Data

“if the original pretraining data contains harmful or value-misaligned data points, then it seems reasonable that the LLM does not learn from these at all.”

From an alignment perspective, this statement is correct. However, from a model performance perspective, learning from the original pre-train data before applying native alignment can enhance the model's knowledge capacity.

In Section 4.2 of the paper, Figure 6 shows that compared to a model trained purely on pre-train data, the one trained on aligned data increases both in harmlessness (alignment aspect) and helpfulness (knowledge aspect). When compared to a model trained on both pre-train data and aligned data, the model trained first on pre-train data and then on aligned data shows a greater enhancement in helpfulness (knowledge aspect). This indicates that training on pre-train data before applying native alignment significantly improves the model's knowledge capacity while having a minimal impact on alignment levels.

How to interpret why LLMs pretrained on both original and native aligned data perform better.

1. Larger Data Scale: The size of the training dataset is a crucial factor in the scaling laws for LLMs [1, 2]. Using a larger dataset that includes both pre-train data and native aligned data can lead to better improvements in the model's knowledge capacity. This larger data scale allows the model to learn from a broader range of information, enhancing its overall performance.
2. Diverse Knowledge Representation: Rewriting and modifying the expression of the original content can benefit the training of pretrained LLMs [3]. Training on both pretraining and aligned data exposes the model to diverse representations of knowledge, helping it to internalize and understand the content more effectively. This diverse knowledge representation enables the LLM to learn more comprehensively from the pretraining corpus, improving its ability to generalize and apply the learned information.

[1] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

[2] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.D.L., Hendricks, L.A., Welbl, J., Clark, A. and Hennigan, T., 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

[3] Ovadia, O., Brief, M., Mishaeli, M. and Elisha, O., 2023. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934.

Limitation: Comparison between native alignment and post-alignment

We have added an experiment showing the comparison between native alignment and post-alignment, please check ‘Author Rebuttal - Additional Experiment I’ for more details.

Prompt to Evaluate Rewriting Quality:

[The Start of Raw text]

{raw}

[The End of Raw text]

[The Start of Rewritten text]

{rewritten}

[The End of Rewritten text]

Please evaluate the following aspects:

1. Formatting
2. Accuracy of information
3. Content moderation
4. Advertisement removal
5. Level of detail

Each aspect receives a score on a scale of 1 to 10, where a higher score indicates better over performance in this aspect. And please return the score by using this format:

Formatting: score
Accuracy of information: score
Content moderation: score
Advertisement removal: score
Level of detail: score

Weakness 1: Comparison between native alignment and post-alignment

We have added an experiment showing the comparison between native alignment and post-alignment, please check ‘Author Rebuttal - Additional Experiment I’ for more details..

Weakness 2: Typo Issues:

Thank you for your correction, we will carefully check and update for spelling errors and grammatical issues in the final version.

Weakness 3: Release of data and models

We have already release the data and models publicly. please check ‘Author Rebuttal - Clarification of Open Source‘ for more details.

Q1: Comparison between native alignment and post-alignment

We have added an experiment showing the comparison between native alignment and post-alignment, please check ‘Author Rebuttal - Additional Experiment I’ for more details.

Q2: Comparison with GPT4

The performance gap is due to various complex factors beyond just alignment. Nonetheless, we believe our approach has the potential to enhance LLMs and ultimately exceed GPT-4's limitations.

Complex Factors for Top-Performing Models:

1. Internal Engineering Tricks: The performance of GPT-4 and similar models often involves proprietary techniques such as Retrieval-Augmented Generation (RAG) or adaptive decoding strategies, which are not always publicly documented or accessible.
2. Computational Resources: The amount of computational power required to train and fine-tune top-performing models is substantial. Large companies have access to vast resources, which may not be available to academic institutions or smaller research groups.
3. Data Availability and Quality: The quality and quantity of training data significantly impact model performance. Proprietary models might leverage extensive, high-quality datasets that are not publicly available.
4. Optimization Techniques: Advanced optimization techniques and hyperparameter tuning are critical for achieving high performance. These methods are often refined through extensive experimentation and can be resource-intensive.

Potential of Native Alignment

As shown in Section 4.3 of the paper, the experiment results show that with the increasing amount of aligned data, the helpfulness and harmlessness of LLMs are positively developing. To further verify the chat ability of the model, we trained a LLaMA3-Tamed-70B-Chat version to compare with GPT-4 on Arabic benchmarks. As shown below, some of the metrics for the proposed method have surpassed ChatGPT 3.5 Turbo and are close to GPT-4. We believe these results could inspire the Arabic community to surpass GPT-4 and provide valuable insights for Arabic LLM development in the future.

Q3: Seed Data Selection

To address the reviewer's concerns about the impact of seed data selection on the performance of the rewrite LLM, we conducted an additional experiment. We aimed to explore the extent to which the selection of seed and aligned data affects model performance. To do this, we compared the performance of randomly selected aligned data with specific experimental groups.

• Experiment Group 1 (high-ppl): This group consisted of data with a large decrease in text perplexity scores after rewriting, indicating significant changes in the data.
• Experiment Group 2 (low-ppl): This group consisted of data with minimal differences between the original and rewritten texts, according to text perplexity score, indicating no significant changes.
• Baseline (random): We conducted three random sample seed data experiments to account for randomness, labeled as ‘random-1’, ‘random-2’, and ‘random-3’. The variance and average of these experiments are reported as ‘random (x3)’.

All datasets consisted of approximately 1,000 samples of pre-training data and were trained on Meta-Llama-3-8B. GPT-4 was used as a reviewer to evaluate the rewriting quality of the LLM rewriter trained on different seed data settings, using the prompt shown below.

The results indicate that, in the benchmark, the selection of aligned data can influence performance (high-PPL). All three random experiments showed no significant differences compared to each other on the benchmark. Therefore, a preliminary conclusion can be drawn: data selection may improve the native alignment approach. This suggests an interesting direction for future research.

Q4: Release of data and models

We have already released the data and models publicly. please check ‘Author Rebuttal - Clarification of Open Source‘ for more details.