Qualifying Knowledge and Knowledge Sharing in Multilingual Models

Thank you very much for your thoughtful review. We hope the following clarifications can help resolve your concerns.

• “The authors should conduct more comprehensive experiments to demonstrate the accuracy of the localized concept neurons and relationship neurons. The stability or reliability of the experimental results.”

  We approach knowledge neurons with skepticism, aiming to provide a broader view of the desired behavior of neurons with such properties. Few neurons exhibit a precise behavior with a clear connection to a concept or a relation, as shown in Figure 2. However, there is meaningful signal, which we test experimentally in Section 5.3 to evaluate if this natural classification aligns with model behavior. Our findings reveal the polyfunctional nature of neurons: we find stability across thresholds, across models (although it is clearer in some models such as bert-large and the most recent gemma-2-9b), and across languages for multilingual settings. Admittedly, we started with much skepticism about single neuron approaches, and found more stability than we expected, which we hope to report fairly. We would be happy however to report any area of doubts in the results in a revised version.

• “Compared to their previous work, the authors extend the discovery of language-agnostic knowledge neurons from 2 languages to 10 languages, while this contribution is somewhat limited.”

  Disclaimer: the work on 2 languages is not ours. We believe that analyzing more than two languages is not just an incremental step forward, but essential. Sharing across two languages collapses a range of possibilities: are neurons shared pairwise or are they shared with no or all languages at once? Is there more sharing with the dominant language (e.g., if English is more represented in the dataset)? Also, the probability that a neuron would end up being shared by two languages by pure chance is (much) higher than it be shared across 10 languages, hence this contributes to addressing the robustness issue raised above.By including 10 languages, we reveal that the different languages play symmetrical roles in the sharing of neurons, and that sharing happens way beyond pairs of languages. Going way beyond what could have been a bias for, say, English then, this provides robust evidence of truly multilingual knowledge representation.

To answer your questions:

• “Why not use the change rate of output probability as an evaluation metric, when manipulating concept neurons or relationship neurons? The change rate of output probability has been adopted by previous work (Dai et al. 2022).”

  We computed our results with relative probability changes as in Dai et al 2022, and found similar results. This measure is not perfect however. Specifically, there could be an important probability increase for the correct answer, say, without the obtained probability making the correct answer anywhere close to being eligible as the final prediction of the model. We therefore decided to report P@k and CCP@k in the main text, as it better captures the effect on behavior (rather than on mere activations). We would be happy to report the (similar) results using relative probability changes in the main text or in an appendix, or to provide the motivation for our choice more clearly in the revised version (or both).

• “Do the findings of this article apply to larger-scale models? It would be better if a larger model could be analyzed, but it may be limited by computational resources.”

  We extended our analysis to more recent and larger models, starting with BERT and expanding to LLaMA-2-7b and Gemma-2-9b. While computational resources limit testing even larger models, we analyzed the largest available to us to ensure the scalability of our findings. At this stage, we prioritized model diversity rather than model size (in part because this comes with more training diversity as well, see also discussion above). Since this may have been missed, as a similar remark was made by another reviewer, we can make these results more visible in the main text.

Thank you very much for your valuable review. Below, we address your key points:

• "All analyses are based on the knowledge neuron [...] the authors did not verify whether their conclusions hold with other knowledge attribution methods (e.g., ROME, SAE). This significantly limits the applicability of their corresponding conclusions."

  We limited our study to Knowledge Neurons due to their popularity and to illustrate our methodology. Extending the work to other attribution methods, such as ROME or SAE, would indeed be valuable. We actually believe that the field as a whole should take up on the task to compare these knowledge attribution methods. This is of interest to us, but beyond our core research question for this paper. If it had been done, and if it was found that several methods actually identify similar neurons, then there would be no need to do the comparison here. If they diverge, and one could be claimed to be superior, then our method will be readily available and could be applied to this one. Hence, independent progress on this orthogonal front would be immediately applicable to our work. Here, we focus on methods that attribute knowledge at the neuron level, specifically because we thought that this scale of analysis requires further scrutiny: is that a relevant scale of analysis, is anything encode at the level of a neuron? To our surprise, our findings align with previous research and suggest that some aspects are indeed encoded at the level of a neuron, and that a neuron could in fact encode “polysemous” (a Knowledge Neuron may encode not only one thing, but multiple things). These findings prompted us to explore concept disentanglement using SAEs, which we hope to present in future work, and would provide more methods for the field. That said, we could include ROME and MEMIT to explore the level of consistency of these methods if the reviewer believes that it is a crucial aspect, although as we said it would be better done in an independent paper that compares these methods in the first place.

• "In their analysis of Relation Neuron and Concept Neuron, the authors subjectively define the selection of relevant knowledge neurons through threshold values. [...] For instance, they could investigate whether perturbing these neurons affects the model's expression of a specific concept without impacting other concepts."

  The thresholds are applied post-calculation of Knowledge Neurons, which is consistent with Dai et al. 2022. As we do not know a priori which neurons play specific roles, we performed an exhaustive study across varying thresholds. Figures 2 and 3 illustrate this comprehensive exploration, providing insights into the behavior and distribution of different neuron types. We agree with the second point nonetheless, and indeed we conducted boosting experiments (Section 5.3, Figure 3) to examine the reliability and impact of the identified neurons, testing hypotheses (i), (ii), and (iii) in Section 5.3. These experiments highlight both the challenges and the varying degrees of signal across models. For example models such as bert-large and gemma-2-9b have stable positive results. We also note that we conducted these boosting experiments in the standard way, although using metrics which are more specific (see also response to reviewer 4). Specifically, we do not report simply whether the probability of the correct response rises, but we report whether the correct answer becomes the predicted answer. A positive result with this more conservative measure shows that the boosting effects are specific (it did not simply raise any vaguely relevant answer, but raised the target answer more than others).

• "For a probing study, the authors should incorporate more recent and larger models, [...] analysis of the consistency and differences in conclusions across different models."

  We agree entirely that using modern and large models is important! Hence, starting with BERT for continuity with prior work, we expanded to larger and newer models, the results do include LLaMA-2-7b and Gemma-2-9b (released ~3 months ago), see Appendix. While we still aspire to analyze even larger models too, computational constraints have pushed us towards model diversity rather than size at this stage. To avoid the possibility of missing this, we can include a summary table comparing model behaviors and findings in the main text.

• "Regarding the authors' analysis of different prompt types,[...] both AutoPrompt and KN are gradient-based methods, the reliability of the corresponding conclusions is questionable."

  This is a subtle concern! While they use the same gradient-based method, AutoPrompt optimizes prompts, while gradients are used differently to identify KN. We cannot pinpoint a specific mechanism or circularity that this could introduce which would artificially create the results as we see them (sharing between AutoPrompts, but also across languages).

Thank you very much for your thoughtful review. We’d like to clarify some points you raised:

• "The contribution of the paper is not novel or different from the lines of work in multilingual knowledge probing, where they all indicate that there exist language-agnostic and language-specific neurons such as [1][2][3]. None of the works were cited or discussed here."

  Thank you for highlighting these references. Our method indeed demonstrates similar findings but in a specific context: neurons that are selected as relevant for knowledge retrieval. We aim not only at asking whether neurons are shared across languages, but also what neurons are shared, depending on their functions in the overall task. Are the shared neurons involved in getting the task right (hence activating for all instances of the task), or specific for some facts, or for some entities?. Our results go beyond the references by detailing both the sharing and functionality, and how this happens across multiple languages. You are absolutely right that we should discuss these references, and we will.

• "The authors rely on threshold-based methods to classify neurons as either conceptual or relational, but variations in thresholds produce different neuron classifications. This makes the claim unrobust: line 208 'When the thresholds become more restrictive, the number of neurons with well-defined roles decreases'. There needs to be more effort in verifying that the relation neurons are indeed performing identical relational functions across languages."

  We fully agree that using thresholds introduces fragility. To address this, we calculated our results across a range of thresholds. Figures 2 and 3 show the results for various choices of thresholds: yes, more restrictive thresholds exclude more neurons (by definition), but the main results (about the type of shared neurons and amount of sharing across languages) are stable independently of a specific choice of a threshold.

• “What are your insights regarding the differences in knowledge-sharing levels among various language pairs? Figure 4 indicates that Spanish shares more knowledge with Danish and Dutch than with French or Italian, which belong to the same Latin family. Do you observe any cross-model similarities related to this finding?”

  We explored potential correlations between the number of shared neurons and various linguistic characteristics. Language family did not produce any clear pattern (coherent with the fact about Spanish mentioned above); syntactic distance (à la Philippy et al., 2023) was a marginal predictor of the amount of neuron sharing (p=.05). We could report these results, even though they are only marginally significant at this stage. Differences could also emerge due to differences in training sets (some may be more parallel than others across pairs of languages, information to which we did not have access to). We agree that it is an important question, and we hope our work reveals its importance (it was not a possible question to ask in the work mentioned by another reviewer where only two language pairs were considered) and that our work makes it possible to address it (both the current results and the introduction of the new dataset mParaRel). We welcome any further suggestion for other measures of distance across languages that could be a good predictor of the degree of neuron sharing.

  We tested two multilingual models (only two because of computational frugality at this stage, since the demands increase as we test each model with 10 languages). The pattern you picked up on (Spanish vs Romance/non-Romance languages) was observed for LLaMA-2-7b, but not for bert-base-multilingual. We do not know if it is due to their different architectures, or difference in their training sets at the moment. Again, we hope that our contribution makes it possible to tackle these important questions.

Thank you very much for your thoughtful and detailed review! We very much appreciate your interest in the work, and are happy to see that we share the same vision about the benefits of looking at multilingual shared neurons. We hope that this work and the dataset could help the community move forward in that direction. We will make the sentence you mention clearer (we will mention Knowledge Neurons explicitly), and we will add the information about the fitting of the curve.

Thank you very much for your comment! We didn't know the name was taken, sorry! We will think of a new name.