The Missing Alignment Link of In-context Learning on Sequences

The main claims in the paper are, from my perspective:

1. (L95) "the inability of modern LLMs to in-context learn structured Seq2Seq tasks" - stemming from their inability to learn the task-specific alignments (in MT sense) beyond very short sequences;
2. That the proposed method, ICA-Tune, can alleviate said problem through a mixture of parameter-efficient fine-tuning and in-context learning;
3. The existence of several phenomena within LLM representations, e.g. the specific induction circuit reported in (L324).

To support these claims, authors design a synthetic in-context-learning problem generator inspired by classical machine translation and demonstrate (with reasonable ablation) that the LLM in question fails to learn the alignment. They then demonstrate how this can be rectified with ICA-Tune. In my opinion, claims 2 and 3 are properly supported, but the first claim is arguably too general.

Overclaim about generality. My main concern is that the paper oft makes general claims about “modern LLMs” (e.g. L95-97 “the inability of modern LLMs to …”) that are, in my view, not properly supported and may mislead readers. Authors only consider Llama 3 8B in the main paper and Llama3.2-1B & 3B in appendix E.1: all relatively small and from the same (albeit popular) model family. This leaves several potential ways that the hypothesis may turn out false:

A. Emergent abilities? it could be that modern LLMs do learn alignment, but it only emerges only after a certain model size (e.g. 70B, 405B). LLMs have been known to ‘sprout’ new emerging abilities in the past [1]. From the current analysis, it is unclear if modern LLMs are fundamentally unable to learn alignment (as the authors claim) or if this is only a failing of the particular LLMs chosen for the study.

B. Model idiosyncrasy? while less likely than the previous hypothesis, it is possible that there is something about specific Llama 3 training or inference conditions that affect their ability to learn alignment. This can be eliminated by testing on other models: latest qwen 2.5 / mistral / deepseek / qwq model variants.

Unless authors can eliminate these possibilities, I would argue that the claims need to be rewritten substantially (e.g. the inability of modern LLMs -> the apparent inability of such and such LLM types under such conditions) or toned down.

Note that double-checking this claim does not require significant expenditure of resources: to the best of my knowledge, one may test the state-of-the-art LLMs’ ability to learn alignment using public APIs, without any specialized hardware. This includes both open models on free tier endpoints (e.g. lmarena.ai, deepseek R1) and commercial models (e.g. openai API, anthropic, google, etc), since the test only requires in-context learning and inference. . If authors find that a diverse set of SoTA models consistently fails to learn alignment, it would strengthen the claims significantly.

Strengths A simple and practical patch for a specific problem, no overthinking. Even if it doesn’t generalize to all LLMs, it’s still useful to many. Within the one synthetic task they consider, author go to commendable length to ablate their findings (S3, S4). Analysis of the ICA-Tune-d models, if mostly mechanistic, is a great addition.

Weaknesses As I specified earlier in "Claims And Evidence", I believe the main weakness of the paper to be overclaim: authors make conclusions about "modern LLMs" in general, but only consider one model family, and only relatively smaller models.

While the paper could also be strengthened by real world experiments to confirm the efficacy of ICA-Tune, it appears to be a deliberate choice and not a weakness.

see strengths and weaknesses

Strengths

1. This paper presents the first formal evaluation of ICL on Seq2Seq tasks, which has significant practical implications for applications such as instant translation and text-to-SQL.
2. The counterfactual experiments provide valuable insights, highlighting that alignment is the missing link in Seq2Seq ICL. The mechanistic evaluation of alignment emergence in middle layers is particularly insightful.
3. The proposed ICA-Tune method demonstrates clear improvements in sample efficiency and OOD generalization, supported by rigorous empirical experiments.

Weaknesses

1. The analysis is primarily conducted on synthetic data, justified by the need to avoid data contamination (L33). However, it would be beneficial to evaluate ICA-Tune on tasks that LLMs have encountered during pre-training, such as translation. This could help determine whether Seq2Seq ICL is an online learning process (as assumed in this paper) or a task-level generalization mechanism that composes pre-existing knowledge [1]. I'd like to hear the authors' opinion about the other ICL perspective (generalization via pre-existing knowledge composition [1]).

• [1] What Do Language Models Learn in Context? The Structured Task Hypothesis, ACL'24

2. After fine-tuning with ICA-Tune, does the LLM’s performance degrade on general Seq2Seq tasks (e.g., translation)? If so, this could indicate a limitation of ICA-Tune’s applicability. Is it possible to explore methods that specifically learn alignment in a way that generalizes across diverse Seq2Seq tasks?
3. In L84, references to previous studies would be more effective if explicitly cited.

This work mainly makes claims on two parts: (1) an analysis showing LLMs struggle at learning alignment through ICL; (2) a new fine-tuning method called ICA-Tune that helps LLMs to learn alignment.

I find the synthetic task-based analysis quite convincing. The experiment setup is clear and rigorous, and the attention weight-based analysis shows clear patterns.

I also believe that ICA-Tune improves the model's ability to learn alignment, but the comparison between ICA-Tune and standard SFT is a bit weird. Essentially, they work on two different tasks. The ICA-Tune is learning an in-context learning task, while the standard SFT is learning an IID seq2seq task. I also couldn't find related results on "latest PEFT methods" in the paper, as mentioned in line 108.

n/a

The experimental results generally support the paper’s claims. However, several aspects could be further controlled or clarified:

1. When using pre-aligned prompting, the increase in prompt length (and thus the number of tokens) may provide the model with additional context and compute used in attention, potentially biasing the comparison.
2. Similarly, the in-context fine-tuning naturally benefits from extended sequences, raising the possibility that performance gains might partly stem from this increased seq len and computation rather than from the alignment learning per se.

Strengths:

1. The paper offers a clear and formal formulation of the seq2seq in-context learning problem.
2. The proposed ICA-Tune method is both simple and effective, demonstrating substantial improvements in performance.
3. The mechanistic analysis is thorough, with well-designed probes that provide insights into how alignment emerges within the model.

Weaknesses:

1. The notation in Section 3.1 is somewhat confusing. For example, in Equation 2, it is not immediately clear to me that q indexes a token in the input x while p serves as the phrase index. Clarifying that the task maps a token in x to an entire phrase in y would improve readability.
2. The task formulation is highly synthetic—each input token is aligned with a single output phrase. This simplification may not fully capture the complexity of real-world seq2seq tasks.

3, The improvement observed with pre-aligned in-context sequences might be partly due to the longer prompt (i.e., more tokens providing additional context and compute) rather than solely due to better alignment.

4, The comparison between ICA-Tune and standard fine-tuning might be influenced by the fact that the test examples in the ICA-Tune setting are less out-of-distribution compared to those in the standard fine-tuning setup.

Overall, while the study is valuable for understanding seq2seq ICL, the highly controlled synthetic nature of the experiments may limit the generality of the findings and robustness of their claims.