Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

We thank the reviewer for the valuable comments and address the questions below.

The theoretical section lacks an explanation on the non-linearity in transformers. We acknowledge that our linear self-attention framework is a simplification of actual transformers, but it captures core theoretical insights while remaining practical. We consider each input mapped to a feature vector. For simpler composite tasks that apply mapping to different input segments, the features of the inputs can be divided into disjoint regions. Conversely, the input features become more mixed and inseparable for more complex tasks involving sequential reasoning.

This simplified framework aligns with recent works [1-4] analyzing in-context learning (ICL) on transformers. It provides a straightforward theoretical construction without compromising the fundamental nature of LLMs.

Explain ``composite tasks have not been seen by the LLMs during pretraining.”

In this work, we opt for synthetic data due to the avoidance of data leakage. While we cannot fully ensure that the test data has not been seen during pretraining, we mitigate this risk by constructing our compositional test data using a unique syntax and mapping mechanism. This approach significantly shifts the data distribution away from existing web-scale data, making it highly improbable that our test data has been encountered during pretraining.

We hope this explanation addresses your concern .

Experimental evidence to validate the application of the proposed theorem. We appreciate the reviewer for highlighting the connection between our theoretical conclusions and the real-world experimental results.

We have provided an intuitive demonstration showing that inputs from simple tasks can be divided into distinct segments (e.g., (A) + (F)). This corresponds to the separability of the input features learned by the models. In future work, we plan to include visualizations of the learned embeddings of input, illustrating the feature space corresponding to the simple task inputs.

[1] ​​Akyürek, Ekin et al. What learning algorithm is in-context learning? Investigations with linear models.

[2] Oswald, Johannes Von et al. Transformers Learn In-Context by Gradient Descent.

[3] Zhang, Ruiqi et al. Trained Transformers Learn Linear Models In-Context

[4] Mahankali, Arvind et al. One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention.

We thank the reviewer for the valuable comments and address the questions below.

There is a performance improvement for increased sizes. Do these results correlate to better learning abilities of LMs?

Yes. Larger models indeed have better abilities. This has been shown in various related works [1,2]. Large-scale models will generally have better reasoning and learning abilities on various tasks.

Similar trend for all smaller LMs eg: BERT

Yes, we conjecture smaller LMs like BERT will also have such a trend (better abilities for larger size). In our work we didn’t verify this since BERT is an encoder model and does not have zero-shot or in-context learning ability. The adaptation of models like BERT will require additional finetuning and is not covered by our work.

Other related works [a,b]

We thank the reviewer for providing related studies [a,b], and we state the distinctions below.

[a,b] primarily investigate the effects of pretraining on language models. Specifically, [a] examines whether pretraining enables language models to acquire reasoning skills. [b] focuses on how information loss in training token characters affects language model performance. In contrast, our work explores whether language models can learn reasoning through in-context learning without delving into pretraining aspects.

Regarding the experimental models, [a, b] mainly experiment on relatively small models, including the encoder model (BERT, DeBERTa) and LSTM model (ELMO), while our work mainly explores large decoder-based language models, including Mistral, Llama, and open source GPT models (GPT-Neo, GPT-j-6b).

We thank the reviewer's suggestions and will add these works to our related work in revision.

Validate the tasks are not present in the training data.

In this work, we opt for synthetic data to avoid data leakage. While we cannot fully ensure that the test data has not been seen during pretraining, we mitigate this risk by constructing our compositional test data using a unique syntax and mapping mechanism. This approach significantly shifts the data distribution away from existing web-scale data, making it highly improbable that our test data has been encountered during pretraining.

We hope this explanation helps address your concern.

[1] Jerry Wei . et al. Larger language models do in-context learning differently.

[2] Jason Wei . et al. Emergent abilities of large language models.

We thank the reviewer for the valuable comments and address the questions below.

Need more evidence on more complex tasks for example mathematical reasonings.

We thank the reviewers for bringing up more tasks and models. In this work, we chose synthetic data to avoid the risk of data leakage. Due to time constraints, we only constructed the tasks presented in the paper. In future work, we plan to expand our task set, particularly focusing on mathematical reasoning. Additionally, we have included results from Llama3 for a more comprehensive evaluation. Due to time and budget limits, we added Llama3 results for part of our tasks to demonstrate the idea.

As shown in the table, for the separable composite tasks which are relatively easy for model to solve (A) + (F), the models show strong compositional ability: the composite accuracy is high, improves with increasing scale, and eventually reaches similar performance as the gold standard composite in-context setting.

For composite tasks with sequential reasoning steps (A) + (B), the model has poor performance in small scale but have increased performance in increased model scale. Providing composed examples as in-context demonstrations will help the model understand the composite tasks and solve them well.

We thank the reviewer for the valuable comments and address the questions below.

Composite linguistic capabilities generalized better than the logical ones, could some regressions also be attributed to the models numerical and code understanding capabilities?

Yes. We believe natural language inputs can indeed help language models understand concepts better than special symbols or code. Natural language provides a richer context, which aligns better with the way these models are trained on large text corpora. In contrast, logical and numerical tasks often rely on more rigid structures, which is harder for models to generalize without explicit training on such patterns.

Did the authors try some prompt tuning?

We've experimented with prompt demonstrations. Instructional prompts do help ChatGPT and Claude3 (although we haven't quantified the accuracy in large-scale experiments), but they offer limited benefits for current open-source models. On the other hand, we did not have prompt tuning or any other parameter updates during our evaluation.

In our experiments, we provided the model with instructions. Here are instructions of Figure 1, which were prepended to ICL examples.

* is a function for capitalizing letters, () is another function for swapping the position of 2 words.

We will add full instructions to the appendix in further revisions.

What if some of the composite logical examples were made more natural language.

We believe it is an interesting direction. Rephrasing the tasks into natural language could leverage the model's strengths in understanding and processing human language, possibly improving its performance in logical reasoning. We will explore it in our future work.

More commercial and bigger LLMs.

Due to time and budget limits, we added Llama3 results for part of our tasks to demonstrate the idea.

LLMs perform better on ``separable composite tasks'' (A)+(F) but poorly on composite tasks with sequential reasoning steps (A) + (B).