Exploring the Relationship between In-Context Learning and Instruction Tuning

Generalizability of Findings: The paper's primary limitation arises from the constrained generalizability of its findings. The focus on sentiment analysis as the primary evaluation task seems inadequate for thoroughly examining the relationship between in-context learning (ICL) and instruction tuning (IT). This limitation is particularly evident in Figure 8, where scaling the model size from 7B to 13B parameters doesn't reveal a clear relationship between ICL and IT. A more diverse range of complex tasks might be necessary to understand these paradigms' interplay more comprehensively.

Clarity and Structure of the Content: Certain sections of the paper, such as those discussing Figures 3.c and 8.c, lack a clear and cohesive structure. Ideally, each section should be framed with a set motivation, detailed experimentation, justification, and a conclusive statement. These elements are noticeably missing or inadequately presented in the mentioned sections, leaving the reader without a clear understanding or a conclusive message, undermining the paper’s coherence and impact.

Clarification on Specific Statements: In section 4.3, the statement, "In other words, the instruction is somehow substituted by the examples it associates at inference time, leading to a similar input format as ICL," is unclear and inadequately explained. More elaborate experimental details, underlying logic, and clarification would be beneficial. This clarification is vital to understand how the models’ learning dynamics operate under varying conditions.

Unclear Second Contribution: The paper claims in its introduction that a secondary contribution is shedding light on designing effective datasets and tasks for ICL and IT, aiding the development of foundational models. However, upon reviewing the document, it becomes apparent that there is a notable lack of clear, direct statements or findings supporting this claim. Explicit elaboration on this contribution, with concrete recommendations or insights, would substantially strengthen this aspect of the paper.

1. The author's method of measuring similarity between h_{anchor} and h_{IT} is a bit confused to me. As I comprehend it, the author compares the final token embeddings to determine cosine similarity between different techniques, for example between anchor and IT. However, I'm skeptical about using the last token embedding to assess the impact of few-shot learning or IT.
2. Additionally, the paper utilizes sentiment analysis datasets. I'm not convinced that these are suitable text-to-text datasets to substantiate the paper's conclusions. Given that the outputs are merely "positive" or "negative," this seems overly simplistic for large language models, especially in the context of few-shot or instruction tuning. Therefore, my takeaway is that sentiment analysis might be relatively straightforward for few-shot or instruction tuning setups. I'm not convinced by the paper's claim "provide strong evidences suggesting ICL is implicitly IT." It seems more likely that both ICL and IT can easily manage this task, leading to similar embeddings behavior. I suggest the author should try more datasets from other few-show paper, for example datasets in Chain-of-Thoughts paper.
3. I'd be curious to see task performance reported under these different conditions.

Weaknesses and Questions:

• While similarity of ICL with anchor is shown to be low, that should depend on the number of ICL demonstration — it would be interesting to see some progression of ICL-anchor similarity depending on the number of demonstrations, maybe Figure 5 could be augmented with this plot.

• The introduction mentions that this study sheds light on how to design effective datasets and tasks for ICL and IT, potentially advancing the development and alignment of foundation models for downstream applications. From the perspective of designing effective datasets, what would be the main takeaway of the paper and the most actionable insight?

• Are the observed effects dependent on LoRA tuning? Would they still hold if full instruction tuning was done?

• Why does similarity in Figure 4c not reach the 0.9 similarity as in Figure 3a?

• While the design choice of tracking only the representation of the last token makes total sense, out of curiosity, do you have experiments with similar analysis for representations of other tokens?

• Figure 3c is really interesting. Does it mean that the internal representations aren’t affected by the in-context demonstrations as much as they are by the inference task?

  • Does the above mean that the model only takes into account demonstrations if they align with the inference task, otherwise it ignores them? How does this mechanism work?
  • Can Figure 3c be thought of as an extreme case of dissimilarity between the demonstration and inference examples as presented in Figure 4, when both the examples and the tasks are different? In Figure 4 there is no Anchor-ICL similarity while in Figure 3c there is high Anchor-ICL similarity, how would you explain the difference?

• Please explain this point more, to me it does not directly follow from 3c: “Clearly, the high level of convergence in similarities between ICL-IT, Anchor-ICL, and Anchor-IT indicates that the demonstrations involving the machine translation task do not impact the model’s inference capability for the sentiment analysis task.”

  • How do we know from Figure 3c that there is no degradation of sentiment analysis inference performance because of showing demonstrations of a different task? Potentially the performance could drop even with high similarity between the hidden states.

• Does the ICL-IT convergence phenomenon depend on the difficulty of the task? Have you tried it with other tasks, except sentiment analysis and translation?

Similarity Measure Ambiguity The similarity metric for comparing hidden states is unspecified. The measure should be clearly described to interpret meanings.

Limited Evaluation Only two datasets are used for evaluation. Testing on more diverse tasks and datasets would demonstrate robustness. Examples: summarization, QA.

Unclear Practical Impact While promising implications for dataset design, the paper lacks any clear guidelines for creating effective ICL demonstrations and IT data. Providing specific, applied best practices is needed.

Narrow Scope The scope focuses solely on hidden state similarity. Expanding to other metrics and tasks would strengthen conclusions. For example, evaluating performance, analyzing attention.

Limited Generalizability Unclear if findings generalize beyond the specific LLM setup i.e LORA, LORA target models (Q and V) and BF16. Experiments on more model types would show broader applicability.

Lack of Ablation Studies The paper does not ablate model components or mechanisms to isolate their impact. Ablating instructions, tuning approaches, model sizes, etc could provide more thorough understanding about necessities. Specially for an empirical paper this is essential.

The paper appears to be a solid contribution to the field, it provides interesting insights and solid experiments. However, certain limitations should are also very transparent.

1. The paper's focus on sentiment analysis and English-Czech translation tasks and the use of specific instruction types could indeed limit the generalizability of the findings. Sentiment analysis tasks have their own unique characteristics and may not represent the full spectrum of tasks that LLMs are expected to perform. If the model's performance is optimized for these tasks, it may not perform as well on other types of tasks, such as reasoning or domain-specific knowledge tasks.

2. The use of the LLaMA-2 7B/13B model and LoRA for fine-tuning presents a narrow slice of the possible model architectures and fine-tuning methods available. While LoRA is known for its efficiency, there may be trade-offs in terms of how well it generalizes across different tasks or scales with different model sizes. The findings might not be applicable to models that are fine-tuned using other methods or have different architectural designs.

3. The paper's methodology involves the use of demonstrations in both ICL and IT. If the demonstrations are not diverse enough or too tailored to specific tasks, there's a risk that the model could overfit to these examples. This overfitting could manifest as an inability to perform well on tasks that deviate from the structure or content of the training demonstrations.

4. The analysis seems to focus on the similarity between demonstrations and inference examples. While this is an important aspect, it may not fully account for the complexity of real-world applications where the relationship between training examples and real-world data can be much more nuanced.

5. The paper's experiments include English-Czech translation tasks, which is a step towards cross-lingual generalizability. However, why only did experiments on certain type of translation tasks? it's not clear how well the findings would hold across a wider array of languages, especially those that are typologically different from English or have less representation in training data.