Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

Dear Reviewer 5m3n,

Thank you for your review and comments.

As to the concerns you have raised in the Rejections to Reject section, it may be helpful for us to clarify the main novelty and contribution of our paper. Please read below and let us know if you have additional questions or comments:

• Yes, it is true that the technical complexity of the proposed method is relatively simple, translating texts from other languages to English and then applying FactScore to evaluate them.

  • The core contribution of this work is proposing a methodology that develops upon FactScore, which was only applicable in English, that can measure the factuality of text generated by multiple languages.

  • We show that this methodology works for 8 different languages, which opens up possibilities to study the factuality of LLMs in many non-English languages. Our method is simple and easy to use, which makes it more powerful.

  • We also rigorously tested that the translation step does not hurt the estimation (Section 3.4), and show how current LLMs are not performing differently for different languages, which is problematic.

• About your concern regarding baseline: would you please explain a little more about what baseline we should have included?

  • While we agree with you that comparison with baselines is important to show the validity of a metric, we did not include experiments comparing with baseline since our work is the first work that tries to evaluate multilingual models’ factuality in long form generation. Our factuality evaluation pipeline is based on FActScore (for English only) and we used the best-performing long-form factuality evaluation method.

  • We will conduct an experiment by simply doing everything in the original language (without translation), as we think this is a valid baseline. If you are thinking of another, more valid baseline, it would be very helpful if you can tell us your idea.

• About the use of Wikipedia data:

  The proposed method is general and can be applied using any ground truth corpus with reasonably good coverage. None of the steps in our pipeline is dependent on the corpus being Wikipedia. In fact, you can use any corpus, in any language, into our pipeline.

  We use Wikipedia because a previous study (Min et al 23) has shown it to be of reasonably good coverage in agreement with works (Chen et al., 2017;Petroni et al., 2021) that treat Wikipedia as a general-purpose knowledge corpus.

Thanks for your response. A few of my concerns have been addressed, but my major concern is the novelty of the method. Therefore, I will maintain my score.

Thank you for comments and reviews.

On the points mentioned under “Reasons to Reject”, here are our thoughts:

• Regarding our experimental setting,

  • We used 80 instances per language, with references from the English Wikipedia. All datasets and code will be open-sourced and made publicly available on GitHub.
  • Table 3 shows the results of our experiment comparing the original InstructGPT implementation with our modified Mistral implementation. The use of Mistral resulted in a 2% point increase in performance compared to InstructGPT (originally reported in FActScore paper). We will ensure that the experimental details are clearly explained in the manuscript.

• We appreciate the feedback about the title and understand that it may have been misleading. We had no intention to overstate our contributions. We will revise the title to “MultiFAct: Multilingual Factuality Evaluation using FActScore” and related sentence in the abstract to “we apply FActScore to multilingual settings.”

• Thank you for the detailed feedback on the areas that need improvement. We apologize for the hasty writing and will tidy up overall manuscript. We will accept all the suggestions provided in the minor comments. To demonstrate how we would revise the paper to be more clear and concise, we have revised the first two paragraphs of Section 4.2, "Effect of Reference Length," as follows:

  We hypothesize that non-English Wikipedias result in higher errors compared to English Wikipedia due to the lower coverage of topics, as evidenced by the shorter length of non-English articles. Figure 3b shows that there is a linear relationship between FActScore estimation error and the word count of Wikipedia articles, indicating that longer Wikipedia articles tend to serve better as references for factuality evaluation. To further test the linear relationship observed, we select 100 individuals from each of our studied languages who have both English Wikipedia articles and substantially long Wikipedia articles in their regional language. We generate biographies for these individuals in English and evaluate them against both the English and regional language versions of Wikipedia. Since FActScore estimation for English is nearly perfect, we use it as a benchmark to estimate the error when using non-English articles as references. Figure 4a illustrates that the linear relationship between the length of articles and FActScore holds for longer articles as well.

Dear Reviewer r8Mi,

Thank you very much for your long, detailed and insightful review! We agree on your opinion on translationese and impacts of doing factuality verification in English.

On the matter of concerns listed under Reasons to Reject:

• About non-LLM based translations: Thanks for pointing this out. It is our oversight not to include an experiment on the effects of different types of translation methodology in the appendix. We are conducting the experiment right now and will share the results with you in this thread as soon as we are done.

• About the experiment doing FActScore monolingually:

  • This is a very valid point, and the reason we didn’t do it before was that our method uses Mistral for atomic fact generation and then doing RAG on wikipedia to verify the validity of the facts. None of these two steps are possible by Mistral, since it is also English only. Besides, there is was concern regarding which languages’ corpus to use for a fair comparison across languages.

  • However, provided that you asked, we are conducting the experiment on a subset of the languages using InstructGPT/GPT3.5-Turbo [which are multilingual]. We will let you know the experiment results in this thread as soon as it is done.

If you have any more questions or concerns, please let us know!

Dear Reviewer a4f9,

First of all, thanks for your comments and reviews.

On the points mentioned under “Reasons to Reject”, here are our thoughts:

• It is true that our method uses an existing factuality evaluation method after translating multilingual generations into English. We made this choice for several reasons:

  • Most LLMs, and all open-source LLMs, are not able to follow instructions properly in a multilingual setting, which means much of the original FActScore pipeline (which strongly depends on models’ capability to break down sentences into atomic facts and then do RAG) is unusable in other languages. This means it is simply impossible to evaluate and compare long-tail languages in a scalable fashion without translating them to English with current models.
  • One of the main contributions of our paper (Section 3.4) was to show that we can use existing English factuality evaluation methods, like FActScore for non-English languages by translating the generations to English, which is not trivial at all. In fact, many tasks and metrics for LLMs that are originally built for English are not transferable to other languages without significant additional research, and it is unfair to ask for technical novelty on top of the 9 different languages (including low-resource languages) that we included in this work. Section 3.4.2 shows that the translation step does not significantly hurt factuality evaluation.

• About providing more details on the FActScore details: We will modify our writing and the main figure accordingly so that it is easy for new readers to understand our method. Thanks for pointing this out!

• On the matter of only using GPT4 and GPT3.5 for evaluation: Our method uses GPT3.5 based translation and Mistral 7B for factuality measurement. We did not evaluate llama, mistral (or any other open-source models) since they are meant to be English-only models and are not able to follow instructions/generate properly well for the languages we worked on. Most of our experiments are conducted on GPT3.5 and GPT4 because those two were the only strong multilingual models available at the time of writing the paper. More multilingual models like Claude 3 and Gemini 1.0 came out much later and they are also included in the appendix (Table 5). However, we plan to include models that came out since we submitted our paper (Command R+ and Gemini 1.5) in our final version of the paper.