mOSCAR: A Large-scale Multilingual and Multimodal Document-Level Corpus

We would like to thank R2 for their great feedback.

W1/ Dataset optimization

To build mOSCAR, we mainly test different thresholds and filters in order to discard low quality content, NSFW documents & images while keeping a good tradeoff with mOSCAR diversity. We think these manual checks were a good proxy to have a good dataset for training vision language models while not training a model at multiple stages of the dataset creation. For instance, we noticed that the NSFW detector tool for images (used in previous works) contains a strong bias towards removing safe for work content such as images of women. Combining it with a detector of nudity we were able to mitigate this bias (more details in our submission line 211). We therefore did multiple iterations to build mOSCAR by using manual inspections instead of retraining a model at each step (which would have been very computationally expensive).

W2/ Quality filtering

There is no neural-based quality filter but there are many rule-based quality filters (see Section A.2 of the Supplementary Material) which were chosen manually by inspecting the data. Having a neural-based quality filter is challenging in such a multilingual setting and could be very expensive. It also comes at the expense of the diversity of the dataset.

Questions:

Q1: Original size of the 3 Common Crawl dumps

We estimate that there are approximately 20 trillion tokens and 60 billion images in the three CommonCrawl dumps we processed.

Q2: Difference NSFW and Toxicity filters

Toxicity word lists include some NSFW words. However, as we want to get a dataset as safe as possible, we additionally apply a NSFW regular expression (targeting English content) to further remove NSFW content (that may not have been removed with the toxicity filter only).

Q3: Main bottleneck of the pipeline

Definitely the Web Archive Content processing pipeline (i.e. first step, going through all CommonCrawl files) and downloading the images. These steps are super slow, require good hardware and network bandwidth. For instance, collecting images for 3 CommonCrawl dumps took approximately 2 months on our setup. Thank you for pointing out the typos and the missing reference that we will add in the related work (in the “English image-text datasets” paragraph).

Thank you very much for your review. Please see our answers to your weaknesses below and our answers to your questions in the next comment.

W1/ Contributions

Our contributions include the dataset design and the method to build such a large-scale multilingual and multimodal dataset, the first of its kind in the multilingual setting. Furthermore, our experimental results show that the findings from [1], [2], [3] stating that interleaved data introduces a boost in few-show learning holds in the multilingual setting, as highlighted by Table 4 in Section 5.

W2/ Baselines

Please see our answers to your question 1/ and 2/ regarding the clarification of our baselines (next comment).

W3/ Performances of the multilingual model on English benchmarks

To evaluate the impact on English benchmarks when training on mOSCAR (i.e. multilingual) we run additional experiments (see new version of the PDF). First of all, we retrained an OpenFlamingo on the English subset of mOSCAR and English captions (“English OF” trained on 50M documents and 100M captions) and compared it to multilingual OpenFlamingo (multilingual OF) trained on the 43 languages (same amount of data seen during training). Below is a comparison between both models (multilingual OF & English OF) on the English subsets of the benchmarks we used in the paper: (same table can be found in the new version of the paper, see new pdf).

We observe very similar results on English evaluation sets when training on multiple languages of mOSCAR in comparison to the model trained on the English-only subset.

We also evaluated these models on MMMU & MME, results can be found in the Table below. The results on these benchmarks are quite similar, meaning that the impact of the multilingual data vs English-only is neglectable. Results are low for all models because Flamingo-like architectures are not well adapted to perform on these benchmarks. They are however cheaper to train than state-of-the-art vision language models.

These results show there is no reason to expect a significant drop of English performances when training multilingual vision-language models on mOSCAR. Similar results have been observed for text-only multilingual language models.

[1] Awadalla et al., OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023)

[2] McKinzie et al., MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training (2023)

[3] Laurençon et al., OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

We would like to thank R4 for their review.

W 1&2/ Analysis and ablation study of the dataset construction

Our choices are based on the literature of multilingual datasets and multimodal datasets. We additionally manually check the content removed and the content remaining for all our filtering steps to be sure we maintain a good tradeoff between quality, diversity and safety. We were unfortunately not able to conduct an ablation study for all our choices as this would mean training a new model at different filtering steps, which would require a huge compute budget that we do not have access to. Regarding our text-image filtering step, as we remove lots of irrelevant text and image nodes within documents while keeping relevant ones, we are fairly confident that we improve the overall quality of mOSCAR. An analysis of the quality of text-image alignment can be found in Section A.4 in the Supplementary Material where we show that images close to text nodes within a document are generally highly similar.

We would like to thank R3 for their review.

W1/ Limited manual evaluation

We did our best effort to inspect examples manually. As we randomly sampled from the dataset for the manual analysis, looking at 1,000 examples should be enough to approximate well the whole dataset. Furthermore, since its creation, our dataset has been used by multiple people and we have not received any reports regarding the presence of NSFW content or toxic content yet. That said, we additionally annotated 900 more documents for toxic content to have a more accurate estimation of the content removed and we found a total of 568 toxic documents among the 1,000 removed ones (which confirms the previous estimation).

W2/ English centric patterns to remove unsafe content

For other languages (including English) we conduct some toxic content removal filtering steps using a dictionary of possible toxic words (including NSFW words) (see Section 3.3, line 178, in the paper for more details).

W3/ Toxic content filter could miss subtle harmful content

We agree that we could have missed some harmful content. We did our best to remove as much harmful content as possible while maintaining high dataset quality and diversity. We used a state-of-the-art method to detect harmful content in multilingual text data (the NLLB method described in Section 3.3, line 178). Moreover, if we had removed all documents containing at least one word from the list (instead of two different words as we did), we would have introduced huge biases into the dataset. For example, all documents containing the words “jews” or “breast” would have been removed from mOSCAR as “jews” is sometimes found in hateful content on the internet and “breast” can be used in a sexual meaning.

W4/ Document-level NSFW filtering is overly aggressive

As mentioned above, it is a matter of tradeoff between recall and precision, NSFW is very sensitive and we did our best to remove it from mOSCAR by favouring recall over precision. It is indeed not possible to remove as much NSFW content as possible while keeping all safe content. We chose to do so for safety reasons and also because some datasets (e.g. LAION5B) were recently found to have lots of NSFW content (including child sexual abuse material) in it.

W5/ Reliance on regex and words lists can be improved with neural-based approaches

Relying on neural based approaches is very useful if it is light, affordable and offers better results than regex rules. In our experiments, whenever we use regex and word lists, either there were no existing neural based approaches or they did not provide sufficiently better results (if at all) in comparison to their cost of compute.

W6/ Fixed characters and node count may not be appropriate for all writing systems

As mOSCAR is designed as a document-level dataset, we found that documents smaller to 300 characters were almost always not relevant or too short (same for 5 nodes). We also found empirically that 300 characters was a good tradeoff to remove irrelevant documents in most languages.

W7/ Language identification for code-switched content

We chose to focus our language identification method on monolingual documents as we found empirically that keeping documents with code-switched content implies keeping lots of low quality documents.

W8/ Limited evaluation of the quality for all languages

It is quite challenging to perform quality evaluation for all languages except by conducting human evaluation, which is very expensive in such a setting. This is why we focused on English evaluation and compared against well-known English-only document-level datasets.

W9/ No systematic evaluation for image-text alignment

An image-text alignment quality analysis can be found in Section A.4 (see Supplementary Material). We found that the images in the document are generally related to surrounding text in mOSCAR, according to NLLB-CLIP similarity, which means that text and images close to each other in a document are well aligned within the document.

W10/ Lack of content relevance metrics within documents

We provide a quantitative analysis of the semantic content of the English documents in Section 4.1.1. We found that mOSCAR is on average more semantically diverse than other comparable datasets.