Is Knowledge in Multilingual Language Models Cross-Lingually Consistent?

Weakness

1. Figure 2 needs improvements.

Thanks for the suggestion. We have recreated figures for the rebuttal revision. We would appreciate it if you can reassess this point.

2. Can you provide some input sample in helping with task definition.

Thanks for this suggestion. In the rebuttal revision, the task definition is revised, and some input examples are added to Appendix (A.1).

3. Do you have error bars or complete statistics about Figure 2 or the overall performance?

Yes, we have. We present the result in Appendix for each language. Please follow the link in the caption of each figure.

4. Do you have a statistical correlation for Figure 5?

Yes, in Table 1, we provide statistical spearman correlation. We observe a moderate correlation with the cross-lingual consistency.

5. Can you explain the result of interventions for Figure 6?

Following your instructions, we have added error bars to all results including Figure 6 for the rebuttal revision. We would appreciate it if you could reassess this point.

We also want to clarify this. In our contribution list, we state “There is a partial causality (in a positive way) between adding monolingual biases and improving cross-lingual knowledge consistency. Thus, adding bias could be a potential method to calibrate consistency across languages.”. Specifically, in the discussion section (4.1), we confirmed that the interventions can modify the consistency but can only offer limited gains subject to mode size and architecture.

5. Related work is a little light and could be fleshed out further.

Thanks for your suggestion. We have revised Related work. We would appreciate it if you could reassess this point.

Question

1. Have you controlled for the fact that similar languages will sometimes have the same token for proper nouns, thus artificially skewing consistency?

Yes, that’s the limitation of this work for now. We clarify this limitation on Page 15. The ideal setting would be to match any of the possible aliases of “Paris” and then match them together. But this is quite tricky to do further analysis.

2. Why not use the baseline for Figures 2, 7 and 10?

Thanks for your valuable suggestions. We have added the average performance to these figures as the baseline.

3. Why did you choose a code-mixed setup?

We want to clarify this. In this paper, we are looking at another facet of multilingual language models. As mentioned in the Introduction section (line 53), multilingual models are able to generalize across languages. The setting in our paper enables the model to link the common knowledge of the language model across different preceding subjects because of cross-lingual generalization. To illustrate given “Paris is the capital of France” and another example “<paris-in-opther-language> is the capital of France”, if the model is indeed cross-lingually consistent, this means that the model incorporates the entity France to other equivalent subjects written in another language due to cross-lingual generalization. This is evident in our experiments on transliteration (c.f., A.4.4, new for rebuttal revision), code-switching training (c.f. section 4.4) , and cross-lingual supervision (c.f., section 4.3), where we show cross-lingually consistent is proportional to entity alignments across languages.

4. Why did you just look at the sum of the IG²s, thus losing information at neuron level, when you could have looked directly at whether the same neurons activated (using an overlap coefficient, for example)?

IG^2 in fact measures the neuron-level information in the form of the attribution score of the particular neuron to the logit of a particular token. The consideration of this because it directly measures the effect of neurons on the logits. Thus, it is more causally justified. On the other hand, neuron overlapping that probably relies on an activation while more efficient might not be a good one as the activation score of neurons might introduce spurious correlation unless it is followed by subsequent causal intervention. So that might be a good alternative but due to time constraints, we will do this in future work.

Weakness

1. Is Indonesian similar to English?

As pointed out in the paper, we assume two languages are similar not necessarily have to be similar typologically because the grouping factor takes into account the cultural aspect that is less relevant for this task setting. Both languages share many commonalities in terms of grammar and writing scripts. We clarify this with reference in subsection 2.3.

2. Do you have more studies about the writing system?

For the latin script, we acknowledge that this setting could make the evaluation biased towards languages written in latin script but this is still not sufficient to cover many of the languages and we still need to figure this out. Another issue of transliteration is the orthography inconsistency.

In addition, we include a limitation section in our paper. We clarify this in our task definition.

3. Why do you consider masked language models?

Thanks for the suggestion and reference. We would like to clarify this. We are trying to say “less hallucination”, which is supported by [1, 2]. Due to the time constraint, we can not investigate causal LM during rebuttal. We want to add additional experiments to the next revision.

[1] Fu, Zihao, et al. "Decoder-only or encoder-decoder? Interpreting language model as a regularized encoder-decoder." arXiv preprint arXiv:2304.04052 (2023).

[2] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (December 2023), 38 pages. https://doi.org/10.1145/3571730

4. Notations need improvements.

Thanks for your suggestions. We have rewritten the task definition with improved notations for the rebuttal revision. We will appreciate it if you can reassess this point.

Question

1. Can you justify the claim that scaling is not a promising strategy?

We scaled up the experiment to larger models (from 0.3B to 10B) and observed quite similar patterns across layers (i.e., the consistency does not increase monotonically but vacillates till the last layer). The typical pattern is presented in Figure 3. In the rebuttal revision, we clarify this around line 200 as “To better understand the cross-lingual consistency bottleneck, we examine the layer-wise consistency patterns across different model sizes, as presented in Figure 3”.

2. How do you treat the cases when the prediction is wrong?

We would like to justify our task definition. This study focuses on consistency of knowledge across multiple languages. We want to know the ability of multilingual language models to recall the same knowledge no matter which languages are written in the context. So its different evaluation of cross-lingual accuracy and this aspect is not a standalone aspect of cross-lingual transfer. We clarify this in the task definition for the rebuttal revision as “Note that we do not consider whether the prediction is correct. Instead, $f_{metric}$ evaluates the parity and consistency across the languages that the model is expected to output similar candidates for $I_{mono}$ and $I_{cm}$.

Weakness:

• My suspicion is that the proposed experimental setting can inadequately boost consistency for languages using Latin script because named entities (from mLAMA) in such languages have the same or similar orthographical forms. To address the concern, the authors could analyze the effect of writing script on consistency in more detail and consider alternative task formulation, e.g., with the whole prompt translated (and transliterated) into the target language.

Thanks for this insightful comment. We follow your intructions and add a new section to Appendix (A 4.4) to study transliteration.

Instead of using translations, we transliterate bn and ar to understand the impact of writing systems, particularly transliterations. As presented in Figure 38, word alignments (or the similar effect from CS training) contribute to the model’s cross-lingual consistency against writing systems because xlm-align and xlm-r-cs show similar performance in both original and transliteration settings. Meanwhile, we can observe that xlm-align and xlm-r-cs significantly improve the overall performance for non-Lattin scripts in section 4.3. This is reasonable as word alignments or CS training help the model link original words with their translations or transliterations, depending on the training corpus, thereby enhancing cross-lingual consistency. We suspect that these word alignments might also improve robustness for handling non-standard spellings and orthographic variations. However, xlm-v-base and xlm-r-base without word alignment benefit from transliterations, which means that xlm-v-base and xlm-r-base do not sufficiently align original words with their transliterations to main cross-lingual consistency. It is also confirmed by the overall performance of vocabulary expansions in section 4.2 , where vocabulary expansions can not offer significant gains for cross-lingual consistency. Overall,  the evaluation task does not inadequately boost consistency for languages using Latin script because word alignments resulting in cross-lingual consistency are the main factor.

(May help you recall the paper: 1) xlm-align: xlm-r-base + word-level cross-lingual supervision; 2) xlm-v-base: xlm-r-base with expanded vocabulary; 3) xlm-r-cs: xlm-r-base with code-switching training)

Weakness

1. Why not use Translations of cloze prompts to measure consistency?

Thanks for the valuable comments. Yes, using translations is an intuitive way to evaluate multilinguality and cross-linguality. In this paper, we are looking at another facet of multilingual language models. As mentioned in the Introduction section (line 53), multilingual models are able to generalize across languages. The setting in our paper enables the model to link the common knowledge of language model across different preceding subjects because of cross-lingual generalization. To illustrate given “Paris is the capital of France” and other example “<paris-in-opther-language> is the capital of France”, if the model is indeed cross-lingually consistent, this means that the model incorporates the entity France to other equivalent subjects written in another language.

2. How about including code-switching corpus?

Thanks for suggesting a model with an explicit code-switching objective in training. However, the suggested model is a multilingual translation model trained on parallel datasets only which is different from the multilingual language models trained on the large-scale monolingual corpus. Alternatively, we find an open-source model [1] fine-tuned from XLM-r on entity-centric code-switching data. The result also aligns with our findings. We have added the result to our rebuttal revision. Please refer to subsection 4.4. Due to time constraints, we can not provide the overall consistency results for this model. We will add it to our next revision.

Question

1. While you look into differences between languages, to what extent does the amount of data used during pre-training impact cross-lingual consistency?

Thanks for contributing an interesting question. While this factor is not in the scope of this paper, we can still get some clues from our findings. Speccifcally, using a multilingual multi-task corpus cannot offer significant gains to cross-lingual consistency (section 4.5). However, using code-swithcing dataset or alignments are promising. In conclusion, the benefit of using additional dataset highly depends on the information of that dataset (section 4.4).

We also want to share an existing work [2] to support us. This paper claims “The moderate correlation indicates a limited impact of the training data volume on learning factual knowledge”.

[1] Chenxi Whitehouse, Fenia Christopoulou, and Ignacio Iacobacci. 2022. EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6698–6714, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[2] Xin Zhao, Naoki Yoshinaga, and Daisuke Oba. 2024. Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2088–2102, St. Julian’s, Malta. Association for Computational Linguistics.

Thanks for the code-switching suggestion.

We add a new section to Appendix (A 4.4) to study transliteration. We additionally find code-switching training contributions to robustness for handling transliterations and non-standard spellings.

Thanks for your feedback and time. We want to clarify and discuss our design.

Indeed, parallel pairs can be used for our evaluation task. However, we have one major concern about monolingual training bias in cross-lingual consistency. Multilingual language models are trained with the monolingual objective, i.e., the input and output languages are the same. However, the language modeling performance varies across languages due to the training corpus. The output distribution of knowledge recall is biased by the monolingual training objective (c.f., section 2). This might impact our analysis regarding distribution differences. Conversely, we mitigate bias in our code-mixed context-independent setting by maintaining the language surface—only altering the subject without changing the overall structure—and the language domain. This is evident in our transliteration experiments (c.f., A.4.4, new for rebuttal revision) as quoted below: Instead of using translations, we transliterate bn and ar to understand the impact of writing systems, particularly transliterations. As presented in Figure 38, word alignments (or the similar effect from CS training) contribute to the model’s cross-lingual consistency against writing systems because xlm-align and xlm-r-cs show similar performance in both original and transliteration settings. ........... However, xlm-v-base and xlm-r-base without word alignment benefit from transliterations, which means that xlm-v-base and xlm-r-base do not sufficiently align original words with their transliterations to main cross-lingual consistency. ...... Overall, the evaluation task does not inadequately boost consistency for languages using Latin script because word alignments resulting in cross-lingual consistency are the main factor.

In conclusion, if coreferential subject entries are trained to generalize across languages (e.g., xlm-align and xlm-r-cs), we could observe the cross-lingual consistency that is not significantly biased by the monolingual training objective, due to the preserved language surface in code-mixing and cross-lingual generalization. If not, the cross-lingual consistency will be weak. Additionally, utilizing two parallel sentences in different languages tends to be influenced by the monolingual training objective, even after coreferential subject entries have been aligned.

(May help you recall the paper: 1) xlm-align: xlm-r-base + word-level cross-lingual supervision; 2) xlm-v-base: xlm-r-base with expanded vocabulary; 3) xlm-r-cs: xlm-r-base with code-switching training)

We also want to clarify our goal. We try to evaluate the hypothesis: multilingual language models recall consistent factual knowledge for coreferential statements in cross-lingual settings, and we answer two key questions: 1) do multilingual language models recall factual knowledge for the coreferential statements in a similar manner on different languages? and 2) how does the mechanism of multilingual language models work on the incorporation between entities or references in cross-lingual settings? To address these questions, we create code-mixed coreferential statements from monolingual statements by substituting a subject entity with an equivalent one in another language that shares the same reference, rather than directly analyzing parallel sentences where each statement recalls entities or references within the same language. This approach allows us to understand multilingual knowledge recall in a cross-lingual setting.

Weakness

1. The paper only experiments with encoder models.

We would like to clarify this. We also evaluated encoder-decoder models, mt0 (base, large, and xl). Please refer to subsection 2.4. Meanwhile, following your suggestions, we add our preliminaries in comparison between mt0 and mt5 to subsection 4.5 to investigate the effectiveness of multilingual multi-task fine-tuning.

Question

1. Do you have any study about the correlation between types of references and cross-lingual consistency?

Thanks for this interesting question. We have not investigated consistency from this angle. We want to add this analysis to our future work list.

2. What is the alignment objective in xlm-r-align?

xlm-r-align [1] trains the model with word-to-word alignment objectives.

3. Why do you consider subject-verb attention and/or object-verb attention?

For attention, we think that intuitively both analyses are not that crucial for this setting because we only measure how the model binds the relation of two entities on which one entity that we assume as common knowledge to another entity that is related to that common knowledge across languages.

[1] Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2021. Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3418–3430, Online. Association for Computational Linguistics.