On Task Description of In-context Learning: A Study from Information Perspective

$**Q1**$: Thank you for your feedback. We are regret our writing are not clear enough and lead to your confusion. We have revised corresponding sections in the paper for clarity, and we would like to clarify our results and conclusion as follows:

$\hspace{2em}**1. Experimental Results: given insufficient task info, the accuracy gain with added in-context examples is subtle.**$ Table above gives detailed results of Figure 2A. The results of 0, 4 and 32 in-context examples are provides. When given insufficient task info (0.73 to 3.22), the performance gain is rather subtle, compared to the one with task info less than 0.73 or larger than 3.22. This also indicates the insufficient task info may mislead the model to ignore the learning of information from in-context examples.

$\hspace{2em}**2. Experimental Results: given insufficient task info, the accuracy is low.**$ As shown in Fig 2A and Table above, for each row in the table, before a certain information threshold (about 3.2 nats), the accuracy remains at a relatively low level. Only after this information threshold, the accuracy grows rapidly as the task information gains. Although accuracy basically grows with mutual info gain, such accuracy gain remains relatively subtle to be 0.001 before the certain information threshold (around 3.2 nats), which is much smaller than a linear speed. In this sense, we say that: insufficient information can impair in-context learning performance.

$\hspace{2em}**3. Explanation: model's attention in in-context examples is diverted by task description.**$ We explore the reason for model's relatively low accuracy given insufficient task info, and find that the task description may lead the model to ignore the information from in-context examples. As depicted in Fig 3B, the ratio of in-context examples in attention keeps declining with more task description information. On the contrary, the attention ratio of task description increases when more task-related information are given. This indicates that adding task description info will divert model's attention in in-context examples, hindering the model’s ability to learn from in-context examples.

$**Q2**$: Thank you for your question regarding the experiment results. We are regret our writing has led to your confusion. We would like to provide a detailed explanation for the phenomenon that performance for medium task information is worse than performance for low task information, under some settings.

$\hspace{2em}**1. Without in-context examples, the performance for medium task information is better than performance for low task information.**$ As illustrated in Table above, Accuracy(0 example) setting, referring to experiments given no in-context examples, the accuracy grows accordingly with information gain. The growing trend is relatively tiny given low task info, but speeds up when more task info added.

$\hspace{2em}**2. Task description diverts model's attention in in-context examples, but at varying levels considering the amount of task info. **$ When more in-context examples are added, as Accuracy(4 examples) setting and Accuracy(32 examples) setting show, the general model performance is improved, but at varying levels. For medium task info settings, as depicted in Figure 2D, the model's attention in in-context examples is diverted by insufficient task info, which hinders the performance gain. However, the distraction of low info task description is relatively insignificant, resulting in seemingly good performance.

Thank you for your valuable comments and suggestions and the encouraging words. We appreciate the time and effort you took to review our paper. Please find below our responses to your concerns and the changes we have made to address them.

$\bullet$ Thank you for your valuable feedback. We are regret that our writing is not clear and can be misleading, and we have revisited and revised the paper for clarity.

$\bullet$ We would like to clarify that the word "limited" here refers to that relatively less exploration has been spent on the impact of task descriptions on in-context learning performance, especially compared to extensive investigation on the influence of in-context samples, such as the works Ortega et al.[1]. We are regret that our writing has led to your misunderstanding, and promise to modify the corresponding presentation in the paper.

[1] Ortega, Pedro A., et al. "Meta-learning of sequential strategies." arXiv preprint arXiv:1905.03030 (2019).

$\bullet$ Thank you for your detailed feedback. We are regret that our notation is not clear enough. We have thoroughly revised and polished the paper for clarity and readability. We have added the meaning of y and r in intro section (highlighted in red). $q_theta$ is the predicted distribution of target $r$, and we have added the definition in method section (highlighted in red) and unified the notation of targets as r.

$\bullet$ Thank you for questions regarding parameter sampling. We adopt the same parameter sampling with GROKKING[2], parameters are sampled uniformly, which makes the calculation of mutual information easy.

[2] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small al-gorithmic datasets, 2022

$**Our results and contributions**$: Thank you for your insightful feedback. You are correct that many works have extensively investigated in-context learning in meta-learned neural networks. However, our primary contribution lies in how task descriptions influence in-context learning within a meta-learning framework, whose role has not been adequately explored by previous research. We would like to highlight our results and contributions as follows:

$\hspace{1em}**We design a synthetic experiment to make task-description controllable and quantifiable.**$ We devise a synthetic experiment setting, making the information of task description controllable. Through a series of well-designed experiments, we systematically vary task description information and assess the resulting effects on model performance across multiple tasks.

$\hspace{1em}**We observe several meaningful experimental results.**$ We observe a phase transition regarding the impact of task descriptions: those with insufficient information can impair in-context learning performance, while task descriptions with abundant information can aid in-context learning. Additionally, we explore whether incorporating task prediction as a auxiliary task during training improves in-context learning performance, and provide theoretical proof. The results indicate that task prediction as a surrogate task benefits in-context learning in nearly all cases.

$\hspace{1em}**We have verified our findings on realistic tasks.**$ To verify the generality of our findings, we conduct further studies on more realistic NLP tasks, which align with our experimental results on the synthetic tasks.

As according to $**Reviewer hvVV**$, we are working on a "very timely problem". $**Reviewer YT61**$ also "like the problem of how task information impacts ICL". And our synthetic experiment is appreciated by $**Reviewer hvVV**$, $**Reviewer YT61**$ and $**Reviewer DYTw**$, being "a refreshingly different approach to yet another heuristic and ad-hoc method for improving in-context learning in an intransparent fashion".

We have revised our paper to correct our inappropriate expression and clarify our ideas, results and contributions. Figures are refined and notations are unified. We appreciate your comments and sincerely hope for rereading.

Thank you for your valuable comments and suggestions and the encouraging words. We appreciate the time and effort you took to review our paper. Please find below our responses to your concerns and the changes we have made to address them.

$\bullet$ Thank you for your feedback. We are regret that Figure 3A is not clear enough, and we have modified the figure and caption for clarity. The heatmap reflects the accuracy difference given two experiment settings: with or without task prediction. The "accuracy" here refers to the predicting accuracy of the answer of query equation, and "difference" means simple abstraction between accuracy values given same training data but under different training strategy. The two experiment settings use the same dataset, only differ in loss backward strategy. Only under task prediction setting, the loss between correct task and predicted task description will be backward. We will re-write the caption for a clearer expression.

$\bullet$ We appreciate your suggestion. You are correct that figures need reorganization for consistent themes. We have made modifications to figures following your suggestion.

$\bullet$ Thank you for your suggestion. The y-axis for plots in Figure 3B (Figure 4A in previous version) and Figure 3C (Figure 4B in previous version) refers to "task accuracy", which means accuracy in predicting what the current task is. We appreciate your insight and have made the necessary changes as per your suggestion.

$\bullet$ Thank you for your valuable feedback. We are regret that our writing has led to your confusion. You are correct that the task description is more likely leading the model to ignore in-context examples. We have revisited and revised the paper to optimize interpretation.

$\bullet$ Thank you for your suggestion and this is a very interesting point. Inspired by your comment, we have explored the relation between our results and work in cognitive science on teaching with language vs demonstrations, and cited the related work.

The experiments carried by Sumers et al.[1] are excellent reflection of in-context learning in real life. Teaching with language acts as adding task description, while teaching with demonstrations acts as giving in-context examples. The experimental results are very consistent with our synthesis experiments, that higher information of task description (teaching by language in a more informative and efficient way) will increase the lower bound of performance. And we offer theoretical derivation for this phenomenon in Eq.3. The phenomenon that language relies on shared abstractions to efficiently transmit complex concepts can also be illustrated by mutual information theory. As teaching languages concluding better shared abstractions are more informative and efficient, resulting in higher task description mutual info value.

[1] Sumers, T. R., Ho, M. K., Hawkins, R. D., & Griffiths, T. L. (2023). Show or Tell? Exploring when (and why) teaching with language outperforms demonstration. Cognition, 232, 105326.

$**Question**$: This is a very interesting point. Our experiments use simplified settings for mutual information quantization. One difference is that, in real world, the mutual information can be hard to calculated accurately, as the task description can be implicit and complex. We do believe the observation in this paper can inspire following works on the task description for in-context learning.

Thank you for your valuable comments and suggestions and the encouraging words. We appreciate the time and effort you took to review our paper. Please find below our responses to your concerns and the changes we have made to address them.

$\bullet$ Thank you for your valuable feedback. We have revised corresponding sections for clarity and readability.

$**Q1**$: Thank you for your question regarding Figure 2A. You are correct in your understanding that blue curve stands for the least number of ICL expamples. The phenomenon depicted in Fig 2A, that an increase in the number of ICL examples doesn't improve performance accordingly, is caused by relatively sparse sampling in in-context examples in Figure 2A. Please refer to Fig 2B, which demonstrates in-context examples' impact in continuous curves. We have listed the numerical results in the table below for clarity. It can be seen in Fig 2B that before the threshold (task info less than 3.2 nats), the number of in-context examples significantly impact in-context learning(the top one), while after that, it has almost no influence (the bottom one)}. And the bottom sub-figure in Fig 2B corresponding to the right most point in Fig 2A (i.e., mutual Info 4.02 nats), which clearly depicts that the accuracy stays relatively consistent across a large range of in-context example numbers. However, in-context examples do help as an accuracy gain can be observed, such as by increasing the in-context number from 0 to 4.

$**Q2**$: Thank you for your question about the use of word "transition". All "phase transition" in our work should refer to the latter one(i.e., the turning point at task info threshold around 3.2 nats), and we will check the presentation for clarity. The "phase transition" in Fig 2B also refers to the turning point at task info threshold around 3.2 nats. Special Attention is paid to this transition as it demonstrates a switch in task description's impact on in-context examples.

$**Q3**$: Thank you for your question regarding the phase transition. first, We would also like to clarify that we have sampled 12 points in Figure 2A, and we have added the detailed value of mutual info and corresponding accuracy in the appendix. And there are around 3 points for phase (ii), and more than 8 points for phase (i). All these points with different number of in-context examples demonstrates the phase transition phenomenons. All these indicate that the reason behind the phase transition is not due to the lack of points used in the plot.

It is an interesting point to study the phase transition phenomenons for in context learning, however it may be hard to provide an thorough explanation for the phase transition. For example, GROKKING[1] only presents the phase transition without explanation. And another work[2] observed a phase transition in investigating ICL given varying task diversity seen by pretrained model, and they suppose the sharp transition indicates a sharp crossover from the theoretical optimal of pre-trained task to that of true task.

We would like to present some of our thoughts on the phase transition for discussion. We suppose the main reason behind this phase transition may related to the optimization. In phase (i), the optimization of the model is trapped into an ambiguous state when the model try to learn from both the context examples and insufficient task info. The task info, itself, can not provide sufficient guidance towards the right solution, and meanwhile distract the attention that paid on the examples. Even though the task info and the context examples are not inconsistent, the model need to consider both factors, which may make it hard to be optimized. On the other hand, with sufficient task info in phase (ii), the optimization can mainly rely on the task info, leading to successful optimization and better performance.

[1] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small al-gorithmic datasets, 2022

[2] Allan Ravent ́os, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, 2023.

$**Q4**$: Thank you for your inquiry regarding Phase (i). We would like to clarify that increasing the number of ICL examples still improves model's performance, ad as an accuracy gain can be observed from in-context number 2 to in-context number 4 in Fig 2B. However, this improvement is soon interfered by high task info. Such phenomenon is mainly caused by ignorance to in-context examples led by task description, and can be illustrated more clearly by model's attention mechanism. As depicted in Fig 3B, a decline in the attention ratio of in-context examples can be observed when more task info added. This indicates that adding task description info will distract model's attention in in-context examples. In this phase, the accuracy stays relatively constant within a large range of in-context example numbers after threshold.

$**Q5**$: Thank you for pointing out the omission of explanation of "lower bound". We regret that the interpretation here is not clear enough, and we have modified the text in the paper (highlighted in red). The interpretation of "lower bound" is formulated in Eq.3, and the "lower bound of performance" refers to the right part of the equation. The first term signifies the task label prediction, whereas the subsequent term corresponds to the loss function employed in the in-context training for the GPT model. This equation, therefore, demonstrates that accurate task label prediction contributes to the maximization of the log-likelihood. And this explanation matches the experimental results given in Fig 2 and Table 3 as under task label prediction setting, a performance gain can be observed with task info gain.

$**Q6**$: Thank you for your suggestion, we have added the output for the task prediction in Figure 1C.

$**Q7**$: We regret that the interpretation here is not clear enough, we have modified the text in the paper (highlighted in red) as follows: The performance of task label prediction can also reflect whether the model understand what the task is.

$**Q8**$: Thank you for your question regarding Figure 4A (Figure 3A in the previous version). We would like to explain the phenomenon as follows:

\hspace{2em}$$**1. Figure 3A demonstrates a phase transition for the accuracy gain.** The x axis in Figure 3A refers to mutual info, while the y axis refers to number of in-context examples, exact labels have been added to the figure. The color of a point in Figure 3A refers to the accuracy difference with or without task prediction, given same number of in-context examples and same amount of task info. The accuracy gain is calculated as $A_{pred}-A_{nopred}$, where $A_{pred}$ stands for accuracy of results prediction with task prediction. The warmer the color, the larger the accuracy difference. As shown in Figure 3A, the accuracy gain increases sharply with mutual info, at a threshold around 3 nats (near the transition threshold demonstrated in Figure 2A).

\hspace{2em}$$**2. The phase transition is similar to that demonstrated in Figure 2A.** The accuracy gain increases sharply with mutual info, at a similar threshold with that in Figure 2A. Actually, both the accuracy of predicting task and w/o predicting task increase sharply around this threshold, as illustrated in Q3, caused by a crossover from learning from in-context examples to learning from task description.

$\hspace{2em} **3. The beneficial impact of predicting task label in in-context learning leads to the transition in accuracy difference.**$ As illustrated in Figure 3A, predicting task label will improve accuracy as the point are mainly colored warm. This indicates an improvement in understanding task description, when predicting task label. When more attention is added to task description, the advantage of task prediction will be amplified, resulting in a sharp increase in accuracy difference.

$**Q9**$: Thank you for your valuable feedback. You are correct that linear regression ICL papers even don't have task descriptions. We regret that the expression here is inaccurate, and we have revised the text in the paper.

$**Q7**$: We apologize for this clerical error and here should be 200k steps. We have revised the text in the paper.

$**Q8**$: Thank you for your questions regarding tokenization. We are sure numbers from different parameters will never be grouped into a single token. Following GROKKING[2], the only tokenization used in the modular arithmetic task is for operators. We represent $+$ as 13, $-$ as 16 and $/$ as 15, since $a$ and $b$ range from 1 to 10, and only the result will mod $p=11$, such tokenization ensures the operator won't be messed up with numerical parameters $a$, $b$ and result $r$. And as $a,b,r$ as well as lower bound/ upper bound all represent real numbers, there is no special need for extra tokenization.

[2] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking:Generalization beyond overfitting on small al-gorithmic datasets, 2022.

$**Q9, Q13,Q14,Q15,Q16**$: Thank you for your suggestion. We have modified all the plots, adding axis, labels, captions and variance for clarity.

$**Q10**$: Thank you for pointing out the omission of network training strategy in the paper. A single model is trained with one fixed value of task info as well as one fixed value of number of in-context values.

$**Q11**$: Thank you for your question regarding "the lower bound of performance". As given in Eq.3 in the paper, the "lower bound of performance" refers to the right part of the equation. The first term signifies the task label prediction, whereas the subsequent term corresponds to the loss function employed in the in-context training for the GPT model. This equation, therefore, demonstrates that accurate task label prediction contributes to the maximization of the log-likelihood. And this explanation matches the experimental results given in Fig 2 and Table given in Rebuttal Part 1, as under task label prediction setting, a performance gain can be observed with task info gain. We are regret that our presentation is not clear enough and we have modified the text in the paper.

$**Q12**$: Thank you for your inquiry regarding task prediction loss. The loss function for task prediction is formulated as Eq. 7. Compared to Eq. 6, we add the correct task description to the end of token sequence to supervise task prediction. As we use auto-regression strategy, the information of accurate result $r_i$ and real task description $t$ won't disclose and kept invisible to model during prediction. So "added" actually means "added to the ending of input token sequence", rather than manipulation on loss.

$**Q17**$: Thank you for your valuable feedback. We have added detailed description of CoFE and several examples in the appendix.

$**Q1**$: Thank you for your suggestion, We have modified the corresponding parts in the paper(highlighted in red) as follows:

The meta-learning framework is used to enrich in-context learning of Transformer, where the Transformer is directly trained to implement in-context learning. The task dataset for this framework is constructed by equations in the form of $(x \circ y)\ mod\ p = r$, where $p$ is a prime number, $\circ$ represents for operators, and $r$ is the result of equation to be predicted.Under this framework, the prompt is formulated as $[\{(x_i,y_i, r_i)\}_{i=1}^l,(x_q,y_q)]$.

\hspace{1em}$$\{(x_i,y_i,r_i)\}_{i=1}^l can be regarded as few shot examples, while $x_q$ is the validation examples.

$**Q2**$: Thank you for your valuable suggestion, we have modified the text in the paper(highlighted in red).

$**Q3**$: Thank you for your valuable feedback. We have revisited and revised the paper to unify presentation.

$**Q4**$: We regret that our notation is not clear enough, we have modified the caption of Figure 1 for clarity. $r_a$ and $r_b$ are the inexact $ab$ ranges implied in task description. $r_a=a_u-a_l$, $r_b=b_u-b_l$,and $a_l$,$a_u$,$b_l$,$b_u$ stands for the possible lower and upper bounds of $a$ and $b$

$**Q5**$: Thank you for your inquiry regarding model architecture.

We build our model following GPT2[1], choose to use a 24-layer transformer considering effectiveness of experiments. Performance of a smaller model with 12 layers are shown in Table above. The models are given different amount of task info and 32 in-context examples. A smaller model still works and demonstrate similar phase transition in accuracy. However, compared to models under same experiment setting but trained with larger model(24 layer), a decline in val accuracy can be observed whatever task info given.

A even smaller model (6 layer) gives worse performance, that even given exact task description and 32 in-context examples, the val accuracy is only 0.1258.

To study how the task info affects the performance, we choose a larger model (24 layer) to better reflect the performance variation.

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar,Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

$**Q6**$: Thank you for your inquiry regarding Eq.6 and Eq.7. We regret that our notation is not clear enough and we have re-formulated the loss function for clarification as follows:

Given a token sequence $x=(x_1,\dots,x_T)$, we train the model to predict $p(x)=\prod_{t=1}^T p(x_t|x_{<t})$. We calculate loss for in-context examples, query and the answer of query equation. The in-context examples are denoted as set $\mathcal{C}_{i-1}$.

For $i > 1$, $\mathcal{C}_{i-1}$

$=\{(x_1,y_1,r_1),\ldots,(x_{i-1},y_{i-1},r_{i-1})\}$

For $i=1$, $\mathcal{C}_0$ is an empty set.

Specifically, we calculate the loss for the sequence $s=\{(x_1,y_1,r_1),\ldots,(x_L,y_L,r_L)\}$ and task description $d$ as follows:

$\mathcal{L}(\theta,s, d)=\frac{1}{L}\sum_{i=1}^L l( f(\{ d,\mathcal{C}_{i-1},x_i,y_i \}),r_i )$

where $l$ denotes the loss function, e.g., cross entropy loss is adopted in our setting. The task description is $d = (a_l,a_u,b_l, b_u, op)$. Similarly, loss for task prediction (task $t=(a,b,op)$) can be re-formulated as:

$\mathcal{L}_t(\theta,s,d)$

$= \frac{1}{L}\sum_{i=1}^L l( f(\{ d,\mathcal{C}_{i-1},x_i,y_i \}),r_i,t ).$

$\bullet$ Thank you for your valuable suggestion, these ablation experiments are indeed important for verification of our results. We have added the following experiments as per your suggestion:

$\hspace{1em}$ $**1. No task description**$: We present the accuracy given no task description and different number of in-context examples in the table below. It can be depicted that the accuracy grows with in-context example number. This table actually refers to zero mutual information in Figure 2A and Figure 2C, and it can be inferred from Figure 2 that model given full info task description always outperforms model given zero task info.

$\hspace{1em}$ $**2. No in-context examples**$: Accuracy(0 ex) in Table given in point 2 lists the model's accuracy given different amount of task info and no in-context examples. When given maximal info (4.6052, referring to totally accurate task description), the model can achieve 0.8641 accuracy, better than all other info level settings, but fall behind models given both full task description and in-context examples. This infers model's ability in understanding task description. Also, it can be seen that under no example setting, the accuracy grows with information gain. The growing trend is relatively tiny given low task info, but speeds up when more task info added. Such performance pattern keeps align with experiments given both task description and in-context examples.

\bullet$$**A more fine-grained synthetic task:** Thank you for your valuable suggestion, and we fully appreciate your concern. We would like to clarify that our synthetic experiment setting can be more fine-grained by simply modifying the value range of $a,b$ in task description, a more detailed explanation are given as follows:

$**1. The setting of task mutual information.**$ The mutual information we utilized here is formulated by $ln(n_{ab}/(r_a \cdot r_b))$, $r_a=a_u-a_l$ and $r_b=b_u-b_l$ stand for the inexact ranges of $a$ and $b$ are implied in task description, and $n_{ab}$ stands for the full number of available $ab$ pairs. Specifically, given maximal upper bound($a_u,b_u$) 10 and minimal lower bound($a_l,b_l$) 1 under the setting in the paper , there can be 100 different $r_a,r_b$ range pairs if $ab$ are restricted to integers.

$**2. Enlarging the value range of$a$and$b$can improve granularity of task info.**$ By enlarging $a$ and $b$, the full number of available $ab$ pairs $n_{ab}$ is increased correspondingly. For example, given a larger maximal upper bound($a_u,b_u$) 100 with minimal lower bound($a_l,b_l$) 1, there can be 10000 different $r_a,r_b$ range pairs, resulting in 100 times more possible mutual info values.

We have sampled 15 mutual info values in our experiment in the paper. Due to time constraints, we regret for not being able to present results with larger $a,b$ range here, but we promise to include results of more fine-grained experiment setting in final version.

Thank you for your valuable comments and suggestions and the encouraging words. We appreciate the time and effort you took to review our paper. Please find below our responses to your concerns and the changes we have made to address them.

$\bullet$ Thank you for your detailed feedback. We have thoroughly revised and polished the paper for clarity and readability.

$\bullet$ We understand your concern about the main experiment. We would like to clarify that the medium task description also helps to improve performance.

We list the exact val accuracy under some experiment settings in Table above, Accuracy(0 ex), Accuracy(4 ex), Accuracy(32 ex) refer to val accuracy of models trained with no in-context example, 4 in-context examples, 32 in-context examples. The results can provide a clearer demonstration of changes in accuracy, that accuracy basically grows with mutual info increasing. Before a certain information threshold (around 3.2 nats concerning results sampled in Table, the accuracy gain remains relatively subtle to be 0.001 between neighbouring info settings. And after this information threshold, the accuracy grows rapidly with information gain. The small accuracy increase before info threshold 3.2 nats is too small to be observed in Fig 2A and Fig 2C. Also, we actually sample several info value sampled after that information threshold (also see Table, referring to task description helps not only at maximal level. We are regret that our notations are not clear enough.

We also explore the reason for model's relative low accuracy and subtle accuracy gain given under threshold task description, and find that the task description will lead the model to ignore the information from in-context examples. As depicted in Fig 2D, the ratio of in-context examples in attention keeps declining with more task description information. On the contrary, the attention ratio of task description increases when more task-related information are given. This indicates that adding task description info will distract model's attention in in-context examples.

$\bullet$ Thank you for your detailed feedback and this is a very interesting point. The maximal possible value for upper bound in task description is 10, while the minimal possible value for lower bound is 1. Such interval seems not large enough to make task description change insignificant.

And we would like to clarify that, according to Table above, task description is helpful whatever the amount of task info and the number of in-context examples given to the model. Especially when no in-context example is provided, the accuracy improves correspondingly with task info gain. Although the accuracy gain remains relatively subtle to be 0.001 before a certain information threshold (around 3.2 nats), an increase in task info does make beneficial improvement.

As for the experiment phenomenon for non-zero task information, we look into the model's attention mechanism, and find that adding task description info will diverse model's attention in in-context examples. This shows that in-context learning is truly affected by adding task description, but in a continuous manner( See Fig 2D, the ratio of in-context examples in attention keeps declining with task info gain). The explanation may be the task description will lead the model to ignore the information from in-context examples.