Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

Dear Reviewer fgkh,

We sincerely appreciate your thoughtful feedback on our paper. Below, we address the specific concerns raised in the review.

More complicated tasks

Thank you for your suggestion. For more complicated task, e.g., grade-school Math, the input $x$ is a Math question and the task answer $g(x)$ usually includes a long chain-of-thought (CoT) reasoning. Note that in this case, there can be multiple equivalently correct ways to solve the given problem using different CoTs, so the task answer $g(x)$ is not unique. Therefore, the task (solving a Math problem) is not a well-defined function from $\mathcal{X}$ to $\mathcal{Y}$ in this setting and it will be hard to measure the probability of task answer given prompt ($\mathbf{P}(g(x)\mid`prompt`)$). However, we think that it would be an interesting future direction to study task superposition on more complicated tasks.

Implications of this work

In this work, we show that LLMs are capable of simultaneously solving distinct tasks from in-context examples. Our work contributes significantly to the fundamental understanding of LLMs and highlights a critical area for future research -- developing decoding strategies that can maintain the model’s multi-task state throughout the generation process.

Strikingly, we find that a recent work [1] offers some hope towards this direction. In particular, [1] propose a method to let LLMs generate continuous thoughts as reasoning states. Using this method, given a logical reasoning task that requires exploration, LLMs can explore multiple paths (each path as a sub-task) at the same time by encoding "a distribution of different traces into the continuous thoughts," which is a superposition of intermediate results from different paths. We hope our work could shed light on future studies on superposition in LLMs. We will add more discussion on the implications of our work in our next revision.

We are happy to elaborate further if you have any remaining concerns.

References

[1] Hao, Shibo, et al. "Training large language models to reason in a continuous latent space."

Dear Reviewer FHnf,

We greatly appreciate your constructive feedback on our paper. Below, we address the specific concerns raised in the review.

Theoretical claims

The proof is a constructive one, ..., there is no guarantee that Transformers will definitely implement such a construction to realize superposition ICL predictions.

It is true that our result is an existential result. The purpose of Theorem 1 is to show that task superposition is well within the expressive power of Transformers and Transformers can efficiently represent the superposition mechanism using a small number of layers. In Section 7 we further investigate the underlying mechanism and empirically show that LLMs combine task vectors during task superposition. We think it will be an interesting future direction to theoretically show the exact underlying mechanism for task superposition for pretrained LLMs.

Discussion with Wang et al.

Thank you for bring up this work. We think Wang et al. [1] is indeed related to our work. [1] shows that ICL will identify the "most suitable pretraining meta-distribution based on the test error and input distribution discrepancies" and operates within that meta-distribution. The bias we observe during task superposition (LLMs do not calibrate their output distribution perfectly with the in-context task example distribution and different LLMs have different bias) can be explained by different pretraining distributions of LLMs, which [1] investigated in.

However, we would also like to clarify that algorithm-selection mechanism does not fully explain task superposition. In the setting of [1], at inference time, all examples in the input are from a single task and LLMs select a task in the pretraining distribution that has the lowest test error with the given task; in our setting, examples in the input are from multiple qualitatively different tasks and the model predicts a superposition of different task answers. We will add the discussion of [1] in our next revision.

Practical value is limited

Not only did the authors design any new methods, they didn't provide a concrete application scenario in which this task superposition capability could be utilized. Could you show me a specific real-world scenario where keeping the LLM in a multi-task state will be beneficial?

A recent paper [2] shows a practical application of the task superposition capability. [2] propose a method where LLMs can generate continuous thought as reasoning state that simultaneously encodes intermediate results from multiple reasoning paths. For example,

• Figure 4 of [2], when asked with a Math problem that can be solved by multiple ways (and we can view each way of solving the problem as a sub-task), LLM "encodes a distribution of different traces into the continuous thoughts."
• Figure 5 and 6 of [2] further shows that on some logical reasoning tasks, LLMs can explore multiple paths at the same time as a breath-first-search algorithm. This outperforms traditional chain-of-thought method that can only explore one path at a time.

Moreover, we believe that explicitly characterizing the phenomenon of task superposition is valuable in its own right as it help us better understand LLMs.

• Our observations align with the "simulator-in-superposition" hypothesis [3, 4] that emerged with the advent of GPT-3. This hypothesis suggests that LLMs can simulate multiple potential continuations or behaviors simultaneously, reflecting a superposition of different skills or tasks. By demonstrating that LLMs can internally represent and process multiple tasks in parallel when provided with mixed in-context examples, we provide empirical support for this theoretical framework.
• Two papers [5, 6] released a few days ago also capture the superposition phenomenon in LLMs. [5, 6] found that when asked to do two-digit addition, LLM will split the problem into two tasks, internally employ parallel computational paths and then merge the result together. This indicates that superposition can be commonly found in LLMs and we hope our work can shed light on future research that study superposition in LLMs.

Typos

Thank you for pointing this out. We will fix the typos in our next revision.

We are happy to elaborate further if you have any remaining concerns.

References

[1] Wang, Qixun, et al. "Can In-context Learning Really Generalize to Out-of-distribution Tasks?."

[2] Hao, Shibo, et al. "Training large language models to reason in a continuous latent space."

[3] Reynolds, L., & McDonell, K. (2021). Multiversal views on language models.

[4] moire. Language models are multiverse generators, January 2021. https://generative.ink/posts/language-models-are-multiverse-generators

[5] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.

[6] Lindsey, et al., "On the Biology of a Large Language Model", Transformer Circuits, 2025.

Dear Reviewer TgC8,

We sincerely appreciate your thoughtful feedback on our paper. Below, we address the specific concerns raised in the review.

Theoretical claims

Thank you for bringing up this work. [1] focus on superposition with NN while we focus on superposition in Transformers, by weighting of the answer proportionally to the number of in-context examples corresponding to this tasks. It is correct that the number of tasks could affect the ReLUs used as well, depending on which portion of the Transformer implements a task. Indeed when the ReLU layers are used then the width could be ~$Kd$.

The purpose of Theorem 1 is to show that task superposition is well within the expressive power of Transformers and Transformers can efficiently represent the superposition mechanism using a small number of layers. It will be an interesting future direction to find the optimal bound on how many task can the model perform with K heads.

Task vectors in toy setting

Thank you for your suggestion. We extract task vectors from our small trained-from-scratch models using the same pipeline as the pretrained models. While our task vector extraction works well for large, pretrained models, we could not find task vectors that work well for our small models. For example, in Figure I we plot the accuracy on task ret2 and plus2 when using vectors extracted from different layers and observe that the maximum accuracy we can get is lower than 0.2 (while for large real-world pretrained model such accuracy is usually near 1). A possible explanation is that, while task vectors for real world LLMs are extracted from a specific layer, for the small models, the feature that represents a task is likely not localized to a specific layer (as indicated in Figure I), which means that we need to modify how we extract the task vectors for small models. We believe this is an important area for further research.

Calibration and superposition

Thanks for providing the insights. We agree that superposition is more about what happens inside the model and calibration is more about aligning model's output with the in-context task examples distribution.

So the framing of this work is that superposition of representations results in the observation of calibration in the outputs of the model.

That is correct. We will add more discussion on this in our next revision.

Superposition "in the wild"

What are ways of observing this form of superposition 'in-the-wild'? For example does this occur for natural language sequences in data that is not based on data with in-context learning examples?

Two papers [2, 3] that are released a few days ago also capture the superposition phenomenon in LLMs. In particular, the authors found that when asked to do two-digit addition, LLM will split the problem into two sub-task paths:

1. one path estimates the rough range of the answer;
2. another path finds the exact last digit of the sum.

The model internally employs parallel computational paths and then merges the result together. This indicates that superposition can be commonly found in LLMs and we hope our work can shed light on future research in superposition in LLMs.

Typos

Thanks for pointing this out. We will fix them in our next revision. We will add the dashed line as well.

We are happy to elaborate further if you have any remaining concerns.

References

[1] Cheung, Brian, et al. "Superposition of many models into one." Advances in neural information processing systems 32 (2019).

[2] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.

[3] Lindsey, et al., "On the Biology of a Large Language Model", Transformer Circuits, 2025.

Dear Reviewer UuyG,

We greatly appreciate your constructive feedback on our paper. Below, we address the specific questions raised in the review.

Figure 1(b)

Thanks for pointing this out. We will update Figure 1(b) in our next revision.

Does ordering of tasks matter?

In our setting of in-context learning, the order can affect how the model perform. For example, consider a scenario with three tasks, each presented in the prompt through 10 examples arranged sequentially: first 10 examples of task 1, followed by 10 examples of task 2, followed by 10 examples of task 3, and then the query. In this case, the model tends to assign higher probabilities on task answer of task 3. This is because there are 10 in-context examples of task 3 right before the query, and the model just follow the same pattern. However, if we randomize the order, the model won't just follow the task examples right before the query but it calibrates its output probability distribution based the in-context task example distribution in the prompt.

We are happy to elaborate further if you have any remaining concerns.