In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

We sincerely appreciate your insightful feedback and acknowledgment of our efforts. Please find our reply to your question below:

1. Authors do not compare to prior methods that combine transfer learning, or similarity of examples, to do in-context learning.

• We compare our method with other retrieval methods during filtering source tasks, where the results are shown in the table | Retriever | Direct | ICTL | | ---------- | --- | --- | | BM25 | 46.2 | 55.8 | | Contriever | 46.5 | 56.3 | | Dr.ICL | 48.4 | 58.7 | | Ours | 48.8 | 60.3 |
• From the table, we can see that our method shows higher performance compared with other methods, proving the effectiveness of our method.
• We have added the above results to the new version of our paper. (Appendix F.1)

2. Much of the paper language needs editing.

• It would be helpful if you could point out where should be editing.

3. Figure 2 is not particularly informative after Figure 1. Additionally, the examples in Figures 1 and 2 do not make much sense.

• Figures 1 and 2 are primarily intended to illustrate our motivation and methodology, rather than specific examples, as excessive textual details may distract readers from understanding our method.
• For concrete examples, please refer to the case study in Appendix G, where we provide detailed examples of our method.
• The "Yes" output in Figure 1 aligns with the task definition, which is to determine whether the question explicitly contains its answer, as described in the blue box and caption of Figure 1.

4. There are serious coherence issues in Section 3: Method.

• Thank you for your suggestion. This should be expressed as "N source task demonstrations that minimize the target task error after transfer.", where we have updated the corresponding sentences in Lines 158-160.
• We have adopted the terminology from Redko etal, where "Hypothesis" refers to the predictor, "task error" denotes the probability of discrepancy between the predicted and correct results, and "distribution for each task" refers to the data distribution of each task.
• In Section 3.2 (Lines 216-219, 227-232), we introduce an Error Term to sample examples in a way that minimizes the overall error, thereby enhancing performance.

5. Evaluations: Table 1 does not seem to show Super-NI to be a challenging task.

• We adapt our method to more general tasks in Appendix F.6, where our method brings 4.2% EM improvement, proving the effectiveness of our method.
• It would be helpful if you could provide how much improvement is a big gap.

6. Table 2 has a serious typo/weird result.

• Apologize for our mistake, where the RougeL of -Source Sample should be 56.8.
• We have fixed this mistake in the new version of our paper.

7. ROUGE is not necessarily a good metric, depending on the task and dataset answer length.

• We use RougeL as our evaluation metric because it is employed in the paper of the benchmark of our main experiments (Super-NI).
• Additionally, we utilize Exact Match (EM) as a metric, which also demonstrates the effectiveness of our method in enhancing performance.
• If you are aware of better evaluation metrics, please let us know, as your suggestions would be highly valuable.

8. All of the results discussion about Figure 3, from Sections 4.4.1 through 4.4.4, lack insight

• Regarding Figure 3, we not only discuss the trends of the changes but also provide the reasons behind these changes as well as corresponding guidance for subsequent parameter selection.
• Any suggestions on how to better discuss the insights would be greatly appreciated.

9. Section 4.4.1 is not well present.

• This is because we are explaining different phenomena, even if their underlying causes are the same.

10. I encourage the authors to improve their paper by cutting down on a lot of redundancy.

• It would be very helpful if you could explain which pieces of information are repeated and repeated frequently.

11. What are some examples of source tasks? Where are the 128 or 512 demonstrations (mentioned in section 4.1.5) from? What are source-target task pair examples that were ranked most similar to each other?

• The source task examples are provided in Appendix G.
• The demonstrations for 128 and 512 are synthesized using our method.
• We calculated the Wasserstein distance between the source tasks and the target tasks (lines 206-207).

12. How many tokens is the average Super-NI answer in your evaluation dataset?

• The average number of tokens in the Super-NI evaluation dataset is 5.9.

Dear Reviewer Cqhf,

As the discussion deadline is approaching (<3 days), we are actively looking forward to your further feedback. Thanks for your effort and understanding!

Kindest regards,

Authors of ICLR Submission 6060

Thank you for answering my questions and responding to my concerns.

1. I appreciate the additional baseline you ran (Dr.ICL), and see that there is a small improvement with your method over that baseline.

2. It will take me many several hours to list out the language changes needed, but as examples:

• paragraph 2: "For example, a model trained pre-2023 can not use knowledge after 2023, while a model not trained on code tasks can not understand codes well" should be changed to --> "For example, ... not trained on coding tasks cannot understand code well."
• contributions section in intro: "We present to synthesize demonstrations by transferring labeled demonstrations of similar tasks, addressing that synthesis from scratch is constrained by the capabilities and knowledge of LLMs;" --> "We argue that answering from scratch is constrained by the capabilities and knowledge of LLMs and thus propose synthesizing demonstrations..."

3. It is fine to have abbreviated textual examples to not overburden readers, but figures 1 and 2 were very confusing to me as a reader looking at the examples. I understand now why the answer is yes/no when the question is not a yes/no question, after your explanation, but it should be more clear from just reading the figure, or the caption itself.

4. Thanks for your changes. The similarity definition is an improvement but "In this paper, we define the similarity as: If we want to sample N source demonstrations, the N source task demonstrations can minimize the target task error after transferring." can be more clear. Is similarity a function of two tasks, or of two demonstrations, one from each task?

5. Presumably you are looking at the zero-to-ours improvement from 60.6 to 64.8%. I would suggest having standard deviations for these results (over multiple LLM calls on the same problem maybe, but this can get expensive), since the absolute percentage is so close to each other between the methods within a task, so we can see how significantly advantageous your results are over baselines.

6. See below.

7. I'm not familiar with Super-NI, but if that's the metric that dataset used, then it makes sense why you used it. Especially since the answers are so short (your answer in item 12). I was thinking more of learned metrics between your approach's answers and the ground truth answers, like BERT score or BLEURT, which is in general better than n-gram based approaches like RougeL.

8. The results section seems more like observations about the graph rather than analysis. For instance in 4.4.1: "(ii) When the sampling scale exceeds 128, there is a slight decrease in performance, indicating that further addition of new source demonstrations does not continue to improve performance, as the number of demonstrations similar to the target task is limited." Why does increasing scale beyond 128 slightly decrease the performance? When using 256 demonstrations, why is additionally passing in 128 less relevant demonstrations, in addition to the more relevant first 128 demonstrations, harmful? Did you try writing in the prompt that the demonstrations are written in order of most to least relevant?

9. Acknowledged, but should still be written a lot more clearly.

10. This is mainly a comment about the results section being mostly redundant with the graphs. It is also a blanket statement about being more succinct with each paragraph.

6, 11-12. Thanks for answering those questions and your fix.

In light of the improvements and explanations, I will slightly improve the score of this paper, but I still do not believe it is close to meeting the threshold for acceptance, mainly because of items 2, 3, 5, 8. Please also include a clearer description of the algorithmic differences between your method and prior retrieval based methods like Dr.ICL for your next paper revision. (ie: don't only show that your method is slightly better, but also provide a hypothesis based on the algorithmic differences for why it does slightly better.)

Thank you for your helpful suggestions! Considering your main concerns, here are our responses to points 2, 3, 4, and 8:

2. It will take me several hours to list out the language changes needed.

• Thank you for your suggestion. We have incorporated your proposed revisions into the new version. However, we remain somewhat puzzled about where improvements in writing are needed, especially since both Reviewer qFgM and Reviewer 6kCu have stated that our paper is well written.

3. Figures 1 and 2 were very confusing to me as a reader looking at the examples.

• Thank you for your suggestion. The reason we use this complex example in Figure 1 is that overly simple examples could confuse readers about why LLMs make errors on such straightforward tasks. On the other hand, more complex examples better demonstrate the value of our work.

5. The absolute percentage is so close to each other between the methods within a task.

• The performance improvement of ICTL is related to the task type. As shown in Table 1, on tasks with free-form answers (e.g., Dialogue, Generation), since it is hard to be completely consistent with the answer, the improvement is not significant; but on tasks with fixed answer forms (e.g., Classification, Comprehension), our method can improve by an average of 4.0%, proving the effectiveness of our method.
• Experiments on more specific tasks in Appendix F.6 show that our method can improve EM by 4.6% compared to Synthesis, proving the effectiveness of our method.

8. The results section seems more like observations about the graph rather than analysis.

8.1. Why does increasing scale beyond 128 slightly decrease the performance?

• As explained in Section 4.4.1 (ii), the statement "increasing scale beyond 128 slightly decreases performance" is due to "the number of demonstrations similar to the target task is limited."

8.2. When using 256 demonstrations, why is additionally passing in 128 less relevant demonstrations?

• Yes, this is because ICTL sample source tasks that are most similar to the target task given a certain scale (as discussed in Section 3.1.2), and then use demonstrations from these source tasks.

8.3. Did you try writing in the prompt that the demonstrations are written in order of most to least relevant?

• As shown in Table 3, our transfer prompt includes only one demonstration, so there is no specific order.

Thank you for your valuable suggestions and for recognizing our work! Below is our response to your inquiry:

1. The contribution is not clearly specified compared to prior research involving advanced demonstration generation in zero-shot ICL settings and advanced retrieval methodologies for ICL with RAG.

• Compared with other zero-shot ICL methods:
  • Our approach is not in competition with other methods for synthetic data generation; instead, it can further improve performance based on these existing synthesis methods.
  • Experiments in Appendix F.4 demonstrate that, when using existing demonstrations (either human-annotated or LLM-generated), our method can still enhance performance, achieving an average improvement of 4.2%, which confirms its ability to further strengthen performance.
  • This is because other synthesis methods rely on the capabilities and knowledge inherent to the LLM itself. In contrast, our method leverages knowledge and abilities from other tasks to boost performance, enabling the enhancement of skills or knowledge that the LLM may not inherently possess.
• Compared with other retrieval methods:
  • We compare our method with other retrieval methods during filtering source tasks, where the results are shown in the table | Retriever | Direct | ICTL | | ---------- | --- | --- | | BM25 | 46.2 | 55.8 | | Contriever | 46.5 | 56.3 | | Dr.ICL | 48.4 | 58.7 | | Ours | 48.8 | 60.3 |
  • From the table, we can see that our method shows higher performance compared with other methods, proving the effectiveness of our method.
  • We have added the above results to the new version of our paper. (Appendix F.1)

2. The self-verification is insufficient to fundamentally improve task performance since the previous works.

• The verification step of our method is to verify whether the generated demonstrations match the target task definition, which is different from the Self-Verification methods you mentioned to check the correctness of the generated answer.
• We verify whether the generated demonstrations match the target task since unmatched demonstrations could mislead the inference of LLMs.
• As shown in Table 2, the verification step of our method can enhance performance indeed.

3. The performance improvement seems less significant to justify the increased cost.

• The performance improvement of our method is significant:
  • The performance improvement of our method is related to the task type. As shown in Table 1, on tasks with free-form answers (e.g., Dialogue, Generation), since it is difficult to be completely consistent with the answer, the improvement is not significant; but on tasks with fixed answer forms (e.g., Classification, Comprehension), our method can improve by an average of 4.0%, proving the effectiveness of our method.
  • Experiments on more specific tasks in Appendix F.6 show that our method can improve EM by 4.6% compared to Synthesis, proving the effectiveness of our method.
• The cost increase is not significant:
  • The demonstration generation is offline, where during the inference, we only need to sample question-related demonstrations from the generation results, having a similar efficiency to the general ICL methods.

4. If there are few or no closely matching source tasks, the quality of transferred demonstrations may suffer, impacting the model’s ability to generalize to the target task.

• Our source sampling strategy ensures that the source tasks sampled are highly similar to the target tasks.
• For tasks that are particularly rare in practical applications, Figure 3(c) shows that our method outperforms the settings without transferring even for the least similar tasks, demonstrating the effectiveness of our method when utilizing dissimilar tasks.

5. It is hard to argue that the tasks or knowledge in the Super-NI dataset are absent from GPT-4o or Llama3.1-8b-Instruct models.

• Our main experiment demonstrates that our method leads to performance improvements, confirming that the ability required for solving data in Super-NI also extends beyond the capabilities of the original LLM.
• Due to limitations in time and computational resources during the rebuttal period, we are unable to conduct experiments on additional datasets. In future work, we plan to evaluate our method on more datasets, including those you suggested (ALFWorld), and incorporate these results into the paper.

6. It would strengthen the paper’s contributions if more analysis was presented.

• In the new version of our paper, we provide a bad case in Appendix H to better analyze how our method enhances model performance.
• In this case, for the sentiment analysis task, we leverage data from the Sentiment-to-Text task to infer the sentiment associated with certain reviews.
• Consequently, during sentiment analysis, we can directly determine the sentiment of similar reviews, thereby reducing the inference complexity for the model.

Dear Reviewer qFgM,

As the discussion deadline is approaching (<3 days), we are actively looking forward to your further feedback. Thanks for your effort and understanding!

Kindest regards,

Authors of ICLR Submission 6060

Your constructive suggestions and recognition of our work are greatly appreciated. Our response to your query is provided below:

1. The performance improvements seem marginal.

• The performance improvement of our method is related to the task type. As shown in Table 1, on tasks with free-form answers (e.g., Dialogue, Generation) [1, 2], since it is difficult to be completely consistent with the answer, the improvement is not significant; but on tasks with fixed answer forms (e.g., Classification, Comprehension), our method can improve by an average of 4.0%, proving the effectiveness of our method.
• Experiments on more specific tasks in Appendix F.6 show that our method can improve EM by 4.6% compared to Synthesis, proving the effectiveness of our method.

2. It would be great to include the computation cost comparison between different approaches in Table 1.

• Efficiency during demonstration synthesis
  • We analyze the efficiency of the demonstration generation of ICTL in Appendix E.1.
  • We present that we can control and how to control the parameters to ensure the balance between efficiency and effectiveness.
• Efficiency during inference:
  • During inference, the average token number of each question is shown in the table. | Method | Zero | Driect | Single | Synthesis | ICTL | | --- | --- | --- | --- | --- | --- | | Average Tokens | 95.7 | 257.3 | 156.7 | 278.7 | 262.3 |
  • From the table, we can see that, during inference, the average token number of our method is similar to Direct and Synthesis.
  • This is because, the demonstration generation is offline, where during the inference, we only need to sample question-related demonstrations from the generation results, having a similar efficiency to the general ICL methods.
  • We update the above discussion in the new version of our paper. (Appendix E.2)

3. The performance can drop significantly (more than 2%) if the parameters are not selected properly.

• Our method indeed requires adjusting parameters to achieve optimal performance.
• But even if the parameter settings are not optimal, as shown in Figure 3(c), our method still achieves better performance than Synthesis in most parameter settings, proving the effectiveness of our method without specific parameter settings.

4. It would be interesting to see more results on other tasks.

• We adapt our method to more general tasks (e.g., Question Answering, Sentiment Analysis), where the experiment results are shown in Appendix F.6.
• The experimental results show that our method improves EM by 4.6% compared to Synthesis, proving the effectiveness of our method.

5. The technical contribution is limited while there are some specific designs for the LLM domain.

• The main topic of our paper is discussing how to synthesize demonstrations for the given task, where the main contributions are as follows:
  • From the task perspective, past methods synthesize demonstrations with LLMs from scratch, while we propose using demonstrations from other tasks for transferring for the first time, which can generate demonstrations beyond the knowledge or capabilities of LLMs themself.
  • From the method perspective, for the first time, we discuss how to sample data similar to the target task (Equation 3) and how to use LLM to migrate examples, and experiments verified the effectiveness of our method.

Dear Reviewer 6kCu,

As the discussion deadline is approaching (<3 days), we are actively looking forward to your further feedback. Thanks for your effort and understanding!

Kindest regards,

Authors of ICLR Submission 6060

Sincerely thank you for your recognition of our work and your valuable suggestions! Below is our response to your questions:

1. The proof of Equation 3 (Appendix A) is not clearly explained.

• We apologize for our mistake, where Equation should be $\hat{\mu}_S = \argmin_{\{\hat{\mu}_{S_i}\}_N} \sum_{i=1}^N \alpha_i W(\hat{\mu}_{S_i}, \hat{\mu}_T)$, where $$\hat{\mu}_T$ is determined by Equation 2.
  • We apologize that the formula is not well displayed, where you can find the formula in the updated paper.
• Then both Equation 3 and Equation 5 are about the source tasks.
• We have fixed this mistake in the new version of the paper (Lines 825-828).

2. Requiring explicit task definitions could restrict the applicability to other benchmarks.

• We can synthesize the task definition based on the demonstrations following Self-Instruct and Auto-ICL.
• We supply the experiments of our method using the definition synthesized by Auto-ICL, where the EM and RougeL of our method with Llama3.1-8b on Super-NI are shown in the table: | Definition | EM | RougeL | | --- | --- | --- | | Auto-ICL | 42.3 | 59.1 | | Human-Labeled | 44.0 | 60.3 |
• From the above table, we can find that the performance degradation caused by synthetic definition is not significant. This is because the performance of our method is not particularly sensitive to the similarity between the source task and target task definitions, as Figure (c) shows.
• We have added the above experiment to the new version of our paper. (Appendix F.7)

3. The sampling process involving simulated annealing lacks sufficient clarity.

• The efficiency analysis of simulated annealing is as follows:
  • Let $t$denote the minimize temperature, $n$ denotes the data scale, and $d$ denotes the dimension of each data.
  • Then the efficiency of simulated annealing is $\text{O}(-nd^2\log t)$.
• Although our method demands the additional cost for computing simulated annealing compared with the general ICL methods, these costs are offline, where our method has the same inference cost as other general ICL methods.
• We have updated the above information in the new version of our paper. (Appendix C)

4. How is the Wasserstein distance between distribution-point / point-point computed?

• We use the same embedding model for the data and task definitions during embedding. Therefore, we can suppose that the task distribution and the task definition vectors are in the same embedding space.
• We consider the single point (vector) as the distribution that the variance is 0 to calculate the Wasserstein distance between distribution-point / point-point.
• We have added the above information to the new version of our paper. (Lines 184-185)

5. How does the proposed source task filtering compare to retrieval-based approaches, such as those using Contriever or BERT?

• We compare our method with other retrieval methods during filtering source tasks, where the results are shown in the table | Retriever | Direct | ICTL | | ---------- | --- | --- | | BM25 | 46.2 | 55.8 | | Contriever | 46.5 | 56.3 | | Dr.ICL | 48.4 | 58.7 | | Ours | 48.8 | 60.3 |
• From the table, we can see that our method shows higher performance compared with other methods, proving the effectiveness of our method.
• We have added the above results to the new version of our paper. (Appendix F.1)

6. What are the "perturbations" used in simulated annealing during the sampling of source demonstrations? What kind of score function is employed in the simulated annealing process?

• During iteration, perturbation refers to the strategy of randomly replacing one of the currently selected demos with a certain probability, even if the score of the current iteration is not better than the current best score. This approach is applied to avoid getting trapped in a local optimum.
• We use Equation 3 as the score function to ensure that the sampled data is relevant to the target task, ensuring performance. We have updated the above information in the new version of the paper. (Appendix C)

7. Rouge -> RougeL

• Thanks for pointing out this issue. We have changed all of Rouge to RougeL in the new version of our paper.

Dear Reviewer H8mW,

As the discussion deadline is approaching (<3 days), we are actively looking forward to your further feedback. Thanks for your effort and understanding!

Kindest regards,

Authors of ICLR Submission 6060

I raise the score to 6 as most of my concerns are addressed. However, there are some issues regarding the proof, which require further clarification from the authors.

1. The variable $\hat{\mu}$ in Eq.3 does not appear in the objective. Should it be $\{\hat{\mu}_{S_i}\}$?

2. What are the difference between $\hat{\mu}_{S_i}$ in Eq 3, $\hat{\mu}_S$ in Eq.2, and $\hat{\mu}_S$

and $\hat{\mu}_{S_i}$ in Eq 5?