We thank the reviewer for the thorough and valuable feedback. To address your concerns, we present the point-to-point responses as follows.

Comment 1: More benchmark. — “ A more thorough investigation of "abstract thought" with more benchmarks …”

Thank you for the insightful suggestion. We fully agree that distinguishing between linguistic and reasoning abilities is essential for substantiating our hypothesis regarding the emergence of abstract thought.

To address this, we evaluated the model from both perspectives:

1. Linguistic Benchmark: We used perplexity as a direct metric to assess the model’s language understanding.
2. Reasoning Benchmark: We measured performance on GSM8K, a reasoning-intensive task.

Specifically, we fine-tuned LLaMA-3.2-3B on single-language corpora and tested the resulting models across all five target languages. The results are summarized below:

GSM8K Accuracy (Reasoning Ability):

Language Perplexity (Linguistic Understanding):

These results clearly dissociate the effects on linguistic competence (as measured by perplexity) and reasoning competence (as measured by GSM8K accuracy). Across both dimensions, we observe that fine-tuning in one language significantly improves performance in that language, while moderately impairing it in others.

Rather than indicating that the model "thinks in English," our findings support the opposite hypothesis:

The model does not rely on English as a latent reasoning space, but instead gradually develops a higher-dimensional, language-agnostic reasoning space.

This is further corroborated by our neuron analysis, which reveals a growing proportion of shared neurons that dominate in importance, indicating that abstract reasoning emerges independently of any single language.

---

Comment 2: Illustration of phenomenon. — “For a given release date, the proportion of shared neurons and their importance decrease with model size …”

Thank you for your insightful suggestion. Figure 1 presents a simplified analysis of models within the same generation but of different sizes. Specifically, we observe that larger models tend to have a lower proportion and reduced importance of shared neurons. We attribute this to increased parameter specialization in larger models [1], which naturally leads to fewer shared neurons.

Although our current work primarily focuses on the evolution of abstract thought over time rather than across model sizes, we appreciate this valuable observation and will incorporate it into the revised version of the paper.

Referring to whether the analysis of section 4 applies to the Qwen-2.5 family, with the limited rebuttal time we were able to show that Qwen-2.5-1.5B indeed exhibits a similar trend to its counterpart in the Llama-3.2 family, with results shown as follows.

---

Comment 3: Illustration of threshold. — “The number of language sensitive neurons depends on the threshold…”

Thank you for raising this important question. The threshold σ is not a fixed scalar, but a dynamic selection mechanism applied at the query level. Specifically, as shown in our released code, we select the top 1% of neurons for each query in a given language based on their computed importance scores. Then, for each language, we define its language-specific neurons as the intersection of these top neurons across all queries from that language.

The choice of the 1% threshold is motivated by a sanity check against random baselines. For each language, we compare the effect of deactivating the selected language-specific neurons versus deactivating the same number of randomly selected neurons. When the deactivation of identified neurons causes a substantial degradation in performance—e.g., more than 100× increase in perplexity—while random neurons have little to no effect, this suggests that the selected neurons are indeed functionally meaningful and the threshold is appropriately calibrated.

If, on the other hand, removing the same number of random neurons also leads to a noticeable performance drop, this would indicate that the threshold is too large and not selective enough—prompting us to reduce it accordingly.

We will include this clarification and the supporting empirical analysis in the revised version to improve the transparency and reproducibility of our method.

---

Comment 4: Distribution of possible unique neurons. — “Another possibility would have been to language-selective neurons as neuron that respond to only to one language…”

Thank you for the thoughtful suggestion. While our definition of "language-exclusive neurons" allows for neurons that are shared across a subset of languages (i.e., not strictly one), we agree that analyzing neurons activated by exactly 1, 2, ..., n languages can provide additional insight.

To complement this, we present the distribution of strictly unique neurons—neurons that respond exclusively to only one language. As shown below, these strictly unique neurons account for only a small percentage of the total neurons across all languages:

This small percentage suggests that the model primarily uses shared neurons for general language understanding. Interestingly, higher values for Swahili and Thai—both lower-resource languages—indicate more allocation to language-specific features, likely to compensate for limited training data.

We also examined the layer-wise distribution of strictly unique neurons. To avoid clutter, we grouped the 32 transformer layers into three stages: early (Layer 0–7), middle (Layer 8–23), and late (Layer 24–31). The averaged proportions of different neuron types across these groups are summarized below:

These results reveal a clear functional asymmetry across the model. The early and late layers contain more unique and exclusive neurons, suggesting that language-specific processing occurs at the input and output stages. This aligns with prior findings that early layers capture syntactic patterns, while late layers focus on output generation. In contrast, the middle layers are dominated by shared neurons and show the lowest uniqueness, indicating a cross-lingual abstraction core where language-independent reasoning takes place.

Finally, we analyzed the component-wise breakdown of strictly unique neurons:

Shared neurons cluster in Q and K, supporting their role in cross-lingual alignment. Exclusive neurons are more concentrated in V, O, and FFN, indicating that language-specific specialization primarily occurs during semantic transformation and output projection. Strictly unique neurons further amplify this pattern, highlighting the most language-specific activations.

These findings further supports our broader claim: shared neurons are primarily responsible for cross-lingual generalization, enabling the model to capture abstract, language-agnostic semantics that are transferable across different languages. Exclusive neurons, on the other hand, are responsible for encoding language-specific features, which are crucial for correctly interpreting and generating content unique to a given language—primarily located in the early and late layers. Unique neurons, by definition, represent a stricter subset of exclusive neurons—those that respond to only one language—which further strengthens the language-specialized nature of the exclusive category.

---

Comment 5: Conclusion is too short. — “The conclusion/discussion is somewhat short.…”

Thank you for the helpful suggestion. We will revise the conclusion to better summarize the main findings and more clearly support our claim regarding the emergence of abstract thought. We will also include a brief discussion of the study’s limitations to provide a more balanced and comprehensive interpretation.

[1] The Rise of Parameter Specialization for Knowledge Storage in Large Language Models. Arxiv 2025

We sincerely thank you for your time and valuable comments. To address your concerns, we present the point-to-point responses as follows.

Comment 1: Different random seeds. — “It appears all experiments are run on one random seed or data split…”

Thank you for your valuable suggestion. In response, we vary the random seed and perform three independent repetitions of each experiment to validate the stability of our results.

• For Table 1 and Figure 4 (random neuron experiments), we change the random seed used for selecting random neurons.
• For Figures 2-5 and Table 1 (shared and exclusive neurons), we alter the data split used for selecting language-specific neurons.
• Additionally, for Table 1, we modify both the training and inference random seeds to ensure consistency and stability in the results.

The results from these repeated experiments are as follows.

For Figure 2:

Due to rebuttal formatting constraints, we can only present the results in markdown table form. To keep the presentation concise and readable, we report the mean and standard deviation of the neuron ratios for each neuron type, aggregated over all models within each family.

For Figure 3:

For Figure 4:

For Figure 5:

For Table 1, we conduct multiple experiments on Llama-3.2-1B due to limited time:

MGSM

MMMLU

We believe that this additional analysis strengthens the reliability of our findings.

---

Comment 2: Illustration of threshold. — “ What was the threshold chosen in Eq. 1? …”

Thank you for raising this important question. The threshold σ is not a fixed scalar, but a dynamic selection mechanism applied at the query level. Specifically, as shown in our released code, we select the top 1% of neurons for each query in a given language based on their computed importance scores. Then, for each language, we define its language-specific neurons as the intersection of these top neurons across all queries from that language.

The choice of the 1% threshold is motivated by a sanity check against random baselines. For each language, we compare the effect of deactivating the selected language-specific neurons versus deactivating the same number of randomly selected neurons. When the deactivation of identified neurons causes a substantial degradation in performance—e.g., more than 100× increase in perplexity—while random neurons have little to no effect, this suggests that the selected neurons are indeed functionally meaningful and the threshold is appropriately calibrated.

If, on the other hand, removing the same number of random neurons also leads to a noticeable performance drop, this would indicate that the threshold is too large and not selective enough—prompting us to reduce it accordingly.

We will include this clarification and the supporting empirical analysis in the revised version to improve the transparency and reproducibility of our method.

---

Comment 3: Question about cross-languages evaluation. — “Does finetuning on language-specific neurons …”

Thank you for this excellent question. While our primary focus is on exploring how fine-tuning with a single language's corpus can enhance the LLM's capabilities in that language, we agree that examining the impact of fine-tuning on other languages provides valuable insights.

To investigate this, we fine-tuned the model using a language-specific corpus and then tested the performance on all languages. The results of Llama-3.2-3B on GSM8K are as follows:

Additionally, we observed the change in language perplexity before and after fine-tuning, which serves as an intuitive metric for assessing the model’s understanding of a given language. The results on LLaMA-3.2-3B are as follows:

Our findings demonstrate that fine-tuning on a single language improves the model’s understanding ability and reasoning performance in that language, but often comes at a moderate cost to the performance on other languages. This indicates that while language-specific fine-tuning enhances targeted capabilities, it may compromise multilingual generalization. We hypothesize that this is because the adaptation process repurposes some shared neurons to better serve the target language, thereby reducing their effectiveness for others.

We thank the reviewer for the thorough and valuable feedback. To address your concerns, we present the point-to-point responses as follows.

Comment 1: Illustration of threshold.

Thank you for the question. The threshold σ is not a fixed scalar but a dynamic, query-level mechanism. For each query in a language, we select the top 1% of neurons by importance score, then define language-specific neurons as those consistently appearing across all queries for that language. This 1% threshold is validated via a sanity check: deactivating these neurons causes a significant performance drop (e.g., 100× increase in perplexity), unlike deactivating the same number of random neurons. If random deactivation also hurts performance, we lower the threshold. We will clarify this in the revised version, along with supporting empirical results.

---

Comment 2: Lack of neuron analysis.

Thank you for the suggestion. We analyzed the distribution of language-shared and language-exclusive neurons in LLaMA-3.1-8B to better understand where they reside in the model.

Layerwise Distribution

We computed the proportion of shared (across all languages) and exclusive (language-specific) neurons per layer group:

Exclusive neurons are more concentrated in early and late layers, suggesting language-specific processing near input/output. Shared neurons are relatively stable across layers, supporting generalizable cross-lingual features.

Component-Level Distribution

We also analyzed neuron distribution across model components:

Shared neurons are concentrated in Q and K, aligning with general attention mechanisms. Exclusive neurons are more spread across V, O, and FFN, indicating roles in language-specific transformation and output.

These findings are consistent with prior work and highlight distinct functional roles for shared vs. exclusive neurons.

---

Comment 3: Lack of illustration in detection method.

“It is not obvious how the impact at a given layer impacts the final output”

We adopt an intuitive definition of neuron importance, measuring it by the change in the layer’s embedding after deactivating a neuron. This layer-wise approach is widely used in prior work analyzing neurons in large language models [1][2][3]. Importantly, we do not assess neuron influence based on the final model output, as influence tends to accumulate across layers. This accumulation can lead to an overestimation of the importance of neurons in earlier layers, which motivates our layer-wise evaluation strategy.

“Also, the term "layer" seems to be used in a way different to common practice.”

We will revise the terminology to eliminate ambiguity in our use of the word layer. Specifically, we will use structure to refer to components such as self-attention or Q/K/V, and reserve layer exclusively for the architectural layers in LLMs.

---

Comment 4: The validity of the assumption. — “Normalizing by the number of neurons |N|.”

We acknowledge that our formulation of the language-agnostic score in Eq. 5 involves an implicit linearity assumption—that the change in perplexity scales proportionally with the number of ablated neurons ∣N∣.

Our motivation for this normalization is to approximately remove the effect of neuron count, in order to make a fairer comparison between shared and exclusive neurons. While this introduces a simplifying linear assumption, we believe it is a reasonable approximation in our setting for two reasons:

1. As shown in Figure 2, the total number of shared and exclusive neurons is not drastically different in scale across models and languages.
2. In contrast, the observed changes in perplexity caused by ablation are often orders of magnitude larger, suggesting that the magnitude of impact is primarily driven by functional importance, not merely neuron count.

Thus, we consider this normalized score a practical and relatively fair metric for comparing the contribution of different types of neurons. That said, we acknowledge its limitations and will clarify this assumption in the revised version.

---

Comment 5: Lowest exclusive neuron ratio of English.

Thank you for the observation. It's true that English often shows the lowest fraction of language-exclusive neurons, but this doesn't contradict our main claim.

We argue that the model doesn’t "think" in English, but rather develops a shared, high-dimensional semantic space. This is supported by the growing proportion and importance of shared neurons, indicating increasingly abstract, language-agnostic representations.

Our GSM8K experiments further support this: fine-tuning on one language improves reasoning in that language, but slightly reduces performance in others.

GSM8K Accuracy:

This trade-off suggests the model reallocates representational resources (e.g., exclusive neurons) during fine-tuning, reinforcing the idea that reasoning occurs in a shared semantic space, not in English.

---

Comment 6: Report of global correlation.

In Figure 3 and Figure 5, our primary goal is to examine the relationship between shared neuron ratio and multilingual ability within the same model family and at comparable parameter scales, where models are directly comparable. We agree that correlation coefficients may not be well-suited for capturing the trends we aim to highlight in this context, especially given the small number of data points per group.

Although our focus is not on global analysis, we nonetheless report both the global correlation and the model familiy-level group-average correlation coefficients for completeness:

• Figure 3:
  • Global Pearson: 0.427 (p = 0.060), Spearman: 0.409 (p = 0.073)
  • Group-average Pearson: 0.591, Spearman: 0.494
• Figure 5:
  • Global Pearson: 0.279 (p = 0.234), Spearman: 0.226 (p = 0.339)
  • Group-average Pearson: 0.312, Spearman: 0.270

We can see that, even though our primary focus is not on global or family-level average correlations—as we believe models across different scales and families are not directly comparable—there still exist moderate correlations, particularly for the shared neuron proportion, which partially supports the general trend we aim to highlight.

---

Comment 7: Lack of random neuron details.

For the random neuron selection, we chose two sets of neurons, each with the same total number as the shared neurons and exclusive neurons, respectively. We assumed that these random neurons are uniformly distributed across all layers and components of the LLM, and we performed random sampling with various random seeds. The results reported in Figure 4 reflect the set of random neurons that had a greater impact on the LLM's performance. This set of random neurons was then used in the ablation experiments presented in Table 1.

In general, we found that comparable numbers of random neurons had minimal impact on the model's ability. We tried several different random seeds, and the results were quite stable, showing that the randomness did not significantly alter the overall conclusions.

---

Comment 8: Illustration of model selection.

To clarify, our aim was to train models within the same series to ensure comparability. However, we argue that models of significantly different sizes might not be directly comparable, especially when considering the impact of scaling. Specifically, Llama3.2-1B and Llama3.2-3B are relatively similar in size and exhibit low and medium language-agnostic scores, respectively. In contrast, Llama3.1-8B, which is closer in size to the Llama 7B models, actually demonstrates a high language-agnostic score within that model family.

To provide more generalizable insights, we also trained Gemma2-9B, a model that, at the global level, clearly qualifies as having a high language-agnostic score. The results are as follows:

This additional explanation clarifies the model selection process. Moreover, we provide empirical support from our GSM8K experiments. When fine-tuned on a specific language, the model’s reasoning ability in that language improves, but often at the cost of performance on others:

GSM8K Accuracy (Reasoning Ability):

This performance trade-off suggests that the model’s reasoning capacity is distributed and not inherently anchored in English. Training on a single language reallocates representational resources (e.g., exclusive neurons) to that language, which slightly reduces generalization. Together, these findings reinforce our claim that the model reasons in a shared semantic space beyond English.

[1] Language-Specific Neurons. ACL 2024

[2] Investigating Pattern Neurons in Urban Time Series Forecasting. ICLR 2025

[3] Finding Safety Neurons in Large Language Models. Arxiv 2024

Response to Reviewer PaXX

We appreciate your comments. Below we provide the point-to-point responses to address your concerns and clarify the misunderstandings of our proposed method. If you have additional questions, we would be pleased to discuss them with you.

Comment 1: Confusion of axis. — “ I don't understand the emphasis on release date as an interesting axis…”

Thank you for the thoughtful comment. Our primary focus is to investigate how the proportion and importance of shared neurons evolve with improving multilingual capabilities. In this context, the central axis of interest is a model’s multilingual ability. We introduced the release date as a proxy to provide an intuitive and chronological perspective on this evolution—particularly since, within the same model family (i.e., architecture and size), newer models tend to exhibit stronger multilingual performance due to improvements in training data quality, scale, or objective functions.

We fully agree that parameter count and FLOPs are important covariates. In fact, parameter count is annotated in Figures 3 and 5. Since FLOPs are approximately proportional to parameter count for models of the same architecture and training length, we chose not to repeat this information. Figure 1 is intended to be a simplified summary of Figures 3 and 5, designed to visually convey the high-level trend in a concise and accessible way—hence the use of release date as the x-axis.

We initially considered training tokens as a covariate, but found them often underreported or ambiguous (e.g., LLaMA series). In some cases (e.g., Qwen-1.5 series), models of different sizes share the same token count, making it hard to isolate the effect of training tokens from model size and release time.

---

Comment 2: Confusion of R. — “In Figure 3 the stated pearson R ≈ 0.94 seems optimistic given visual scatter …”

In Figure 3 and Figure 5, our primary goal is to examine the relationship between shared neuron ratio and multilingual ability within the same model family and at comparable parameter scales, as we believe that models across different sizes and families are not directly comparable. The reported Pearson and Spearman coefficients are the averages computed across these individual model groups.

Although our focus is not on global analysis, we nonetheless report both the global correlation and the model familiy-level group-average correlation coefficients for completeness:

• Figure 3

  • Global Pearson: 0.427 (p = 0.060), Spearman: 0.409 (p = 0.073)
  • Group-average Pearson: 0.591, Spearman: 0.494

• Figure 5

  • Global Pearson: 0.279 (p = 0.234), Spearman: 0.226 (p = 0.339)
  • Group-average Pearson: 0.312, Spearman: 0.270

We can see that, even though our primary focus is not on global or family-level average correlations, there still exist correlations, particularly for the shared neuron proportion, which partially supports the general trend we aim to highlight.

---

Comment 3: Lack of neuron analysis. — “There is no analysis in terms of what layer the neurons are in the model …”

Thank you for this insightful suggestion. We agree that analyzing the distribution of language-specific neurons across layers and components can provide further insight. To address the question of where language-shared and language-exclusive neurons exist in the model, we conducted a detailed analysis on LLaMA-3.1-8B.

Layerwise Distribution:

We calculate the proportion of shared neurons (common to all languages) and exclusive neurons (specific to one language) across layers:

The layerwise distribution highlights distinct allocation patterns for shared and exclusive neurons. We observe that exclusive neurons are more concentrated in the early and late stages of the model, whereas their proportion is relatively lower in the middle layers. This suggests that the model tends to encode language-specific features near the input and output boundaries.

In contrast, shared neurons maintain a relatively stable presence across all layers. This indicates that shared neurons are more involved in processing generalizable, cross-lingual representations, and serve as a backbone for multilingual understanding throughout the network.

Component-level Distribution:

Shared neurons are mainly found in QK, aligning with general attention patterns. Exclusive neurons are relatively more prominent in VO and FFN, indicating their more important role in language-specific output transformations. This distribution indicates that shared and exclusive neurons may serve different roles, although they both belong to language-specific neurons.

Illustration of random neurons:

Regarding the random neurons in Figure 4, we carefully designed the selection process to make a fair comparison. Specifically, we sampled two sets of random neurons—each matching the total number of shared and exclusive neurons respectively. These random neurons were uniformly sampled across all layers and components of the model under the assumption of a homogeneous distribution. We then selected the group that had a higher influence on model performance and reported this in Figure 4. This same group was also used in the ablation experiments as “random neurons” reported in Table 1. It is indeed expected that the distribution and activation behavior of random neurons may differ from those of the language-specific neurons we detected.

---

Comment 4: Size as axis. — “Figure 2 is unnatural as a line plot and the grouping by size rather than generation…”

Thank you for your insightful suggestion. Figure 1 presents a simplified analysis of models within the same generation but of different sizes. Specifically, we observe that larger models tend to have a lower proportion and reduced importance of shared neurons. We attribute this to increased parameter specialization in larger models [1], which naturally leads to fewer shared neurons.

Although our current work primarily focuses on the evolution of abstract thought over time rather than across model sizes, we appreciate this valuable observation and will incorporate it into the revised version of the paper.

---

Comment 5: Inconsistent results. — “Inconsistencies with prior results: In On the Biology of a Large Language Model…”

Thank you for pointing this out. We acknowledge the omission of the reference and will include it in the revised version.

We also wish to clarify a key methodological difference. Lindsey et al. analyze multilingual generalization from the feature space—focusing on hidden state reuse across languages. In contrast, our study operates in the parameter space, identifying shared neurons that are functionally important across languages and fixed in model structure.

Due to these fundamental differences in approach—feature-level versus parameter-level analysis—the two works may lead to different empirical trends. For instance, our Figure 1 does not show a monotonic increase in shared neuron ratio with model size, which may reflect that parameter sharing across languages does not always scale proportionally with model capacity.

We thank the reviewer again for highlighting this, which allowed us to more clearly distinguish our methodological contribution.

---

Comment 6: Additional baseline. — “I don't think the training method has much practical utility, compared to LoRA or other PEFT methods…”

We clarify that our training-stage intervention is not proposed as a new fine-tuning method, but rather as a way to validate our core finding: multilingual capability in LLMs can be enhanced by understanding and adjusting neuron-level contributions.

While LoRA and other PEFT methods are widely adopted, they introduce additional parameters and adapter layers. In contrast, our method fine-tunes a small subset of existing shared neurons, aiming to produce a fully multilingual model without any added parameters. This reflects a different trade-off—targeted internal adjustment vs. parameter-efficient extension.

Following your suggestion, we conducted LoRA experiments on Llama3.2-3B under similar parameter budgets. LoRA improved performance in some MGSM cases (e.g., German, Swahili), but degraded it in others, especially on MMMLU. Moreover, LoRA increased training time (2.2h on 2×H200, rank=48) compared to our method (1.5h), highlighting our method’s simplicity and efficiency without modifying the model architecture.

[1] The Rise of Parameter Specialization for Knowledge Storage in Large Language Models. Arxiv 2025