We appreciate your time and effort in reviewing our paper. However, we received the reasons for rejection, probably due to misunderstanding and oversight of the review guideline and the paper. We hope that our response addresses your questions, misunderstandings, and concerns.

R1: limited innovations and technical depth

Processing 63B web pages and continual pre-training on at least two 7B, a 13B, an 8x7B, and a 70B models cannot be done without technical depth. Unlike other kinds of studies, the top priority of corpus-building is not presenting an innovative method but making the best effort to deliver high-quality and useful data.

Q1: existing Japanese corpora are good enough

This is not true. The filtering method in mC4 is insufficient; thus, other English corpora were proposed after mC4. Furthermore, mC4 considered no aspect of Japanese for quality judgment.

R2: The motivation is not convincing. They can be easily resolved.

Q2: What is the exact problems of the existing corpus

Among CC-100, mC4, and OSCAR in Table 1, only mC4 can be a candidate for building Japanese open LLMs because the other two are too small (the numbers of tokens are less than 100B tokens, the size we used for the continual pre-training). However, mC4 uses WET as the data source, which includes unrecoverable errors. We quote Penedo et al. (2023):

however, in line with previous works (Gao et al., 2020; Rae et al., 2021), we found WET files to include undesirable navigation menus, ads, and other irrelevant texts.

HTML markups provide useful hints for removing navigation menus and ads, but this information is lost in WET files. This is a critical issue of WET files and mC4 that cannot be resolved later. We will elaborate on this in the extra page allowed in the camera-ready version.

The size of our cleaned corpus (312 billion letters) is larger than that of uncleaned mC4 (240 billion letters) in Table 1, which is also an advantage of our study. We also want to emphasize that Table 2 demonstrates that our corpus was superior to mC4 cleaned for Japanese ("+ llm-jp").

R3: Any techniques specifically designed for Japanese

Sections 3.1, 3.2, 3.4, and 3.5 were specially designed for Japanese.

R4: The presentation should be further improved

Could you please suggest which part should be improved? For example, should “Llama 2 7B” be replaced with “Llama-2-7b” (as par with the model name)?

We would like to thank you for your positive review and useful suggestions for our paper.

R1: the quality of hostname filtering

This is a valid point. Unfortunately, there is no public data available for comprehensively detecting harmful Japanese web pages; we found that the coverage of the UT1 blocklist was insufficient for filtering out harmful websites in Japan. We could not use a commercial filtering service to recognize harmful URLs because we were unsure of whether the TOS would allow us to build an LLM that could be used for commercial purposes. This was the reason why we created our own list for hostname filtering with tight thresholds (0.001 and 0.005) and left the establishment of a more robust method as a future work (mentioned in Section 5).

Q1: continual pertaining

Yes, you are right. Given a base pre-trained model, continual pretraining performs another round of pre-training on new data (Japanese corpus in this study). We will make this clearer in the camera-ready version. Both “continual pretraining” and “continued pretraining” seem acceptable based on our literature survey, but we will also explain this in the camera-ready version.

Q2: NG expressions

Sorry, they have nothing to do with programming languages. NG expressions stand for harmful expressions (adult, violent, aggressive). The reason why we avoided the direct term “harmful” was because this filtering process may remove a large number of non-harmful web pages with tight thresholds.

Q3: blocklists

We used the word blocklist as “a list for blocking”, following the name “UT1 blocklist”. We were also reluctant to use the word “black” because some hosts in this list may not be “black”. For example, a hospital website explaining a woman’s maternity is prone to be included in the list.

Q4: “enjoy” in the Japanese Copyright Act

Thank you for pointing out the translation problem. The original term 享受 (kyo-ju) is also translated into "enjoy" by the Japanese government. The nuance is that a human consumes the idea of a work (for any purpose, including entertainment, learning, and communication, regardless of the monetary gains of the data). The intention of Article 30-4 is to define an exception for the use case where a computer program consumes works directly for machine learning and information analysis, but a human cannot do that. We will add an explanation in the camera-ready version.

We would like to thank you for your positive review and useful suggestions for our paper.

R1: No reference to other Japanese compilation work

Thank you for the suggestion. We will comment on the references you raised one by one.

• NICT LLM (actually, this was released by NII but not by NICT). Because this is a concurrent work, the lack of this citation cannot be a reason for rejection. After the submission of our paper, we heard that their paper was (probably) accepted at some refereed international conference. We will check and cite their work in the camera-ready version.
• https://www.aozora.gr.jp/#main Aozora-bunko mostly includes old novels with expired copyrights (published 50+ years ago). Because these novels include many old-fashioned words and kanji usages, we did not consider Aozora-bunko for training Japanese LLMs. In addition, Aozora-bunko is not an effort to build a corpus for training Japanese LLMs. Thus, we don’t think the lack of this citation can be a reason for rejection.
• https://github.com/llm-jp/awesome-japanese-llm?tab=readme-ov-file We've been aware of this nice website that lists Japanese LLMs. However, as far as we checked, this website does not include a list of Japanese corpora. So again, we do not think the lack of this citation can be a reason for rejection. In addition, we want to emphasize that some strong LLMs listed on the website were used for the performance comparison in Table 2.

R2: Lacking some breakdown

You can find the breakdown at the end of Sections 3.1 to 3.5. This cannot be a reason for rejection, either.

R3: depth of ablation (how much data affects a handful model)

We believe that an important value of pre-training corpora is to provide data that works for any model and purpose. This is why we showed the effectiveness on different strong models.

Suggestions

Thank you for the useful suggestions. We will reflect these in the camera-ready version.

Q1: Performance on XL-Sum

This is explained in the second paragraph of Section 4.3. Before continual pre-training, we added Japanese vocabulary to Llama 2 and Mistral 7B tokenizers (mentioned in Section 4.1). This treatment degrades the downstream performance because the embeddings of newly added tokens were not trained well. In contrast, we didn’t expand the vocabulary of the Mixtral 8x7B tokenizer and observed an improvement in XL-Sum. We will elaborate on this in the camera-ready version.

We would like to thank you for your positive review and useful suggestions for our paper. We appreciate the time and effort you have taken to provide feedback. Below, we answer your questions raised.

R1: only applicable to Japanese

Yes, we expected that our paper could receive such a comment. However, our motivation was to build a large corpus specialized for Japanese because the quality of multilingual corpora does not meet our expectation of spending a lot of money for continual pre-training of Japanese LLMs. We also explained our stance in Section 6 (Limitations) that some ideas in this paper could be helpful for other languages and that researchers in each country should put an effort into building a corpus for the country.

R2: whether their corpus would be effective if they train an LLM from scratch

Q1: the quantity and quality necessary to train an LLM from scratch

(We merge responses for the rejection reason and question)

We did not try to build an LLM from scratch for various reasons:

• The ultimate goal of this project was to explore methods for building open LLMs that archive high performance in Japanese.
• Based on the Chinchilla law, we need 205.1 billion tokens (for 10B models) and 1.5 trillion tokens (for 67B models) as the training data. The constructed corpus was sufficient for training 7B models from scratch but probably insufficient for 70B models.
• Having said that, we need to mix an English corpus for building Japanese LLMs anyway because English corpora are about 10 times larger than Japanese ones and because we also need multilingual applications (e.g., English-Japanese translation and cross-lingual summarization).
• If we mix the training data of Japanese and English, continual pre-training is a reasonable choice because we can omit the large computation for English data.
• A lot of Japanese companies released a variety of Japanese LLMs trained from scratch when we started this project (last fall). We wanted to try an alternative approach for building high-performance Japanese LLMs.

As an academic paper, we agree with your opinion that we want to see the performance of the LLM trained on the corpus from scratch. However, we do not have the computation resources for this at this moment, although our corpus is also ready for use for training Japanese LLMs from scratch.

Q2: Abbreviations of BL and BW

Thank you for the valuable comment. We will avoid these abbreviations in the body of text in the camera-ready version.