Instruction-tuned LLMs with World Knowledge are More Aligned to the Human Brain

We are happy that the reviewer appreciated that (1) our experiments examine commonly-used model families, and that (2) our results are consistent. Based on their feedback, we ran additional experiments and improved our paper’s clarity. We respond to their points below:

Q1: The authors tried to study the effect of world knowledge and model size separately in Section 4.2. However, the two factors are deeply intertwined.

Our work shows that larger LLMs are more aligned to the brain. However, we wish to find out why. What aspects of larger LLMs make them more brain-aligned? Some possibilities: (1) greater world knowledge, (2) better problem-solving abilities, (3) better next-word prediction ability. Our work conducts further experiments to identify the underlying properties of LLMs (beyond model size) that contribute towards greater brain alignment. We show that world knowledge is a key determinant of brain alignment, more than other tested factors.

To show that the plots of world knowledge are not dependent on model size, we wish to highlight that smaller models instruction-tuned to gain greater capabilities (e.g., Vicuna-13B) can achieve greater brain alignment scores than larger models before instruction-tuning (e.g., LLaMA-33B). For reference, here are the average brain alignment values of the models: LLaMA-13B = 0.220, Vicuna-13B = 0.229, LLaMA-33B = 0.227, and Vicuna-33B = 0.232. The Vicuna-13B model has greater brain alignment than LLaMA-33B although it is less than 40% the size. We observe a similar trend when looking at another set of four models: T5-base, Flan-T5-base, T5-large, Flan-T5-large.

To further demonstrate that model size alone does not determine the brain alignment of an LLM, we ran new experiments to evaluate the brain alignment of a randomly-initialized LLaMA-7B. It is a large model with 7B parameters. The randomly-initialized LLaMA was not trained on any data and hence does not contain world knowledge. It also achieves roughly zero accuracy on the BBH and MMLU benchmarks. We observe that the randomly-initialized model achieves lower brain alignment than the trained LLaMA and instruction-tuned Alpaca-7B version. Prior works have also shown that large random embedding models do not achieve high brain alignment scores [1]. These results demonstrate that there are LLM properties aside from model size that contribute significantly to brain alignment.

[1] Schrimpf, Martin, et al. "The neural architecture of language: Integrative modeling …”

Q2: The authors use the performance on MMLU and BBH to represent the models’ capability of “world knowledge”. However, performance on these benchmarks is affected not only by the quantity of knowledge that the models possess, but also by their ability to follow instructions.

We agree with the reviewer. Hence, we restricted our selection of LLMs to only two model families. In each model family, the models are trained on a similar format of instruction data, so they have similar ability to follow the instruction format of MMLU and BBH. Thus, any differences in their MMLU or BBH performance would point to differences in the quantity of knowledge they possess, rather than their ability to follow the instruction format of MMLU or BBH.

Q3: Additional details about our experiment methods

We thank the reviewer for pointing out areas that require additional clarifications. We have improved our paper’s clarity by adding details on which layers were used to evaluate brain alignment, and the types of world knowledge in BBH and MMLU.

We are happy that the reviewer thinks (1) our paper presents a unique neuroscientific perspective for understanding LLMs, (2) our experiment design is intuitive and easy to follow, and (3) our experiments are extensive. We respond to their points below and improve our paper based on their feedback:

Q1: Lack of comparative analysis with other tuning techniques such as reinforcement learning from human feedback (RLHF).

RLHF is often performed after the instruction-tuning phase. Hence, we did not consider it as a comparable alternative to instruction-tuning. In order to perform a comparative analysis between RLHF and instruction-tuning, as suggested by the reviewer, we would need to find LLMs trained with RLHF instead of instruction-tuning. Ideally, both LLMs should be trained on the same amount of training data and compute, to allow for a fair comparison. These LLMs also need to be open-source, in order to evaluate their brain alignment. Unfortunately, we could not find such LLMs.

We believe that investigating how RLHF affects the brain alignment of instruction-tuned LLMs constitutes future work. RLHF explicitly aims to align LLMs using human feedback. It would be an interesting direction to see if it also aligns LLMs to human brain activity (brain alignment) and behavior (behavioral alignment).

Q2: The investigation of behavioral alignment is limited. A more comprehensive exploration could offer insights into the discrepancy between brain and behavioral alignments and its implications for LLM development and application.

We agree with the reviewer’s point. Hence, we mentioned this in our original submission (and keep it in our rebuttal version too). We included a paragraph in the discussion section 6.2, where we argue for the need to examine more dimensions of behavior. We mention that our work and many prior works compare LM and human next-word surprisal on reading tasks as a measure of behavioral alignment. However, this evaluates only a single dimension of LM and human behavior (per-word perplexity and reading times). We highlight the need to create more benchmarks to expand the dimensions of behavior examined for both LLMs and humans, in order to holistically evaluate LLM behavior, as well as LLM-human behavioral alignment.

Hence, we focus instead on the brain alignment of LLMs. Our paper focuses on investigating how instruction-tuning affects LLMs from a neuroscientific perspective and the key factors underlying brain-LLM alignment (e.g., world knowledge).

Q3: 33B model is quite large, are there any quantization techniques used like LoRA?

For the 33B models, we only performed inference, not training. Hence, LoRA was not used. All our evaluations involve only model inference. This includes the brain alignment evaluations (Pereira2018, Blank2014, Wehbe2014), behavioral alignment evaluations (Futrell2018), and the evaluation of world knowledge (MMLU) and problem-solving abilities (BBH). These evaluations do not involve model training.

We are happy the reviewer found that (1) our analyses were detailed, (2) our results had implications for both NLP and neuroscience, and that (3) our paper was well-written and easy to follow. We respond to their points below and improve our paper based on their feedback:

Q1: The authors use models from just two families, T5 and LLaMA.

In our work, we evaluate the correlation of brain alignment to other LLM properties, such as world knowledge (MMLU) and problem-solving abilities (BBH). We agree with the reviewer that adding a greater variety of models would strengthen the generalizability of our claims.

However, we realized that the performance on these benchmarks is affected not only by the quantity of knowledge that the models possess, but also by their ability to follow the instruction format of MMLU and BBH. (This point was also mentioned by reviewer WxVH.)

Hence, we restricted our selection of LLMs to only two model families. In each model family, the models are trained on a similar format of instruction data, so they have similar ability to follow the instruction format of MMLU and BBH. Thus, any differences in their MMLU or BBH performance would point to differences in the quantity of knowledge they possess, rather than their ability to follow the instruction format of MMLU or BBH.

If we selected LLMs from many different model families, it would be difficult to control for their ability to follow the instruction format of MMLU and BBH, as they are trained on vastly different instruction formats. This would make it challenging to use their MMLU and BBH results to suggest the quantity of knowledge that the models possess.

However, we overall agree with the reviewer that greater model variety would strengthen generalizability, so we added it to our paper’s section on limitations and future work.

Q2: How did the authors determine the category classification for the BBH tasks?

We use the same category classification as the original BBH paper. The BBH paper categorizes tasks into four categories: (1) Algorithmic and Multi-Step Arithmetic Reasoning, (2) Natural Language Understanding, (3) Use of World Knowledge, and (4) Multilingual Knowledge and Reasoning. We add this clarification to our paper.

BBH paper: M. Suzgun et al., “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.”

Q3: Is it possible to provide a more detailed analysis regarding world knowledge? Any discussion and additional analysis would be appreciated.

For both MMLU and BBH, we follow their respective papers for how they categorized the world knowledge tasks.

The MMLU paper describes that they design the benchmark to measure knowledge from many domains. It contains 57 tasks, categorized by the subject domain of world knowledge tested: STEM, Humanities, Social Sciences, and Others. The STEM category includes questions on computer science, physics, mathematics, etc. The Humanities category includes questions on philosophy, law, history, etc. The Social Sciences category includes questions on politics, sociology, economics, geography, etc. The Others category includes questions on business topics such as finance, accounting, as well as general knowledge of global facts.

The world knowledge category of BBH contains tasks that test for factual and general knowledge. Tasks requiring factual knowledge include: “Sports Understanding” and “Movie Recommendation”. Tasks requiring general knowledge include: “Causal Judgement”, which tests knowledge about causal-reasoning suppositions, and “Ruin Names”, which requires knowledge about human perception and usage of humor in the English language.

We are happy that the reviewer appreciated our neuroscientific analysis of instruction-tuning. Based on their feedback, we ran additional experiments and improved our paper’s clarity. We respond to their points below:

Q1: This paper appears to be a replication of [1,2], specifically focusing on instruction-tuned models. It lacks novelty and originality.

Our work presents many new insights for researchers studying the parallels between LLMs and human brains: (a) Instruction-tuning aligns LLM representations to human brain activity, and (b) World knowledge is a key factor underlying this alignment. These are both new directions of study that produced new insights. Hence, our work is different from both [1] and [2], which focus on studying how the next-word prediction surprisal of language models relates to human brain activity and behavior.

[1] Schrimpf, Martin, et al. "The neural architecture of language: Integrative modeling …”

[2] Oh, Byung-Doh, and William Schuler. "Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?."

Q2: Increasing the model size and integrating world knowledge (using a larger training dataset) are not surprising new discoveries for improving language modeling.

Our paper is not focused on discovering new methods for improving language modeling. Rather, our paper focuses on evaluating the representational and behavioral similarity of LLMs to the human language system. In particular, we investigate how instruction-tuning affects LLMs from a neuroscientific perspective and the key factors underlying brain-LLM alignment (e.g., world knowledge and model size).

Q3: The plots of world knowledge (against brain alignment) seem to be dependent on model size.

Our work shows that larger LLMs are more aligned to the brain. However, we wish to find out why. What aspects of larger LLMs make them more brain-aligned? Some possibilities: (1) greater world knowledge, (2) better problem-solving abilities, (3) better next-word prediction ability. Our work conducts further experiments to identify the underlying properties of LLMs (beyond model size) that contribute towards greater brain alignment. We show that world knowledge is a key determinant of brain alignment, more than other tested factors.

To show that the plots of world knowledge are not dependent on model size, we wish to highlight that smaller models instruction-tuned to gain greater capabilities (e.g., Vicuna-13B) can achieve greater brain alignment scores than larger models before instruction-tuning (e.g., LLaMA-33B). For reference, here are the average brain alignment values of the models: LLaMA-13B = 0.220, Vicuna-13B = 0.229, LLaMA-33B = 0.227, and Vicuna-33B = 0.232. The Vicuna-13B model has greater brain alignment than LLaMA-33B although it is less than 40% the size. We observe a similar trend when looking at another set of four models: T5-base, Flan-T5-base, T5-large, Flan-T5-large.

To further demonstrate that model size alone does not determine the brain alignment of an LLM, we ran new experiments to evaluate the brain alignment of a randomly-initialized LLaMA-7B. It is a large model with 7B parameters. The randomly-initialized LLaMA was not trained on any data and hence does not contain world knowledge. It also achieves roughly zero accuracy on the BBH and MMLU benchmarks. We observe that the randomly-initialized model achieves lower brain alignment than the trained LLaMA and instruction-tuned Alpaca-7B version. Prior works have also shown that large random embedding models do not achieve high brain alignment scores [1]. These results demonstrate that there are LLM properties aside from model size that contribute significantly to brain alignment.

[1] Schrimpf, Martin, et al. "The neural architecture of language: Integrative modeling …”

Q4: How is the 'No Instruction' model trained in Figure 1D? Did we use all the training data from the Alpaca dataset?

We used all provided training samples from the Alpaca dataset. This is to ensure that the ablation model is trained on the same data as the instruction-tuned Alpaca-7B, with the only difference that the ablation model does not receive the “instruction” portion during training. We added this clarification (that we use all the provided training samples) to our paper.

Q5: Additional details about our experiment methods

We thank the reviewer for pointing out areas that require additional clarifications. We have improved our paper’s clarity by adding details on the methods used for computing brain alignment, measuring next-word prediction loss, and running our ablation experiments.