Where Does In-context Machine Translation Happen in Large Language Models?

We would like to thank the reviewer for their detailed and insightful suggestions.

1. Notations. We have clarified the notations to be much more explicit and removed confusing notation, such as replacing instr_L1L2 with proper terms and removing the usage of $^{\circ,\bullet}$ to be extremely explicit about what is presently masked. Please see general response.

2. Section 4.2 on Inference Efficiency.

there is no detailed comparison of translation accuracy at all.

The translation accuracy that we should have mentioned, is with respect to the ceiling score that we had observed in the context masking experiments (Fig 1). We have proceeded to clarify in writing "Using the example of GPTNeo ($32$ layers), we see from Fig1 that the model is very close to it's ceiling score after processing the examples at layer 20"

3. Discussion on distortion/interdependency of layers.

“masking them would obviously distort the representation of the input anyway. “

We acknowledge that this could be obvious to a reader, and have removed the word “interesting”, and removed the confusing sentence about “interdependency of layers”. Instead we have added more useful discussion around the Llama7b-chat model. We report that the distortion happens strongly when instructions are masked, and there is hardly any distortion when the examples are masked.

4. Revise confusing writing.

The paper already mentions that "Note that this setup is designed to tune the layers for task location. It is highly unlikely that the model can learn translation knowledge from this small amount of supervision", so we cannot take results from such a set-up to say the above finding is correct.

We have revised the sentence to be much more cautious. We revise the claim to be “We show that it is sometimes beneficial to target middle layers of the model which could be associated with task recognition.”

“Finally, we see that when instructions are masked with the examples...”

We acknowledge that this was difficult to read, we have rewritten this to “Finally, we see that \newtext{the behavior of the model is similar when instructions are masked ($\overline{`Instr,Ex`}^{Mask}$) and when no instructions are present ($\overline{`Ex`}^{Mask}$).} “

5. Question about Instructions and Plateu.

In the case of different prompt, I was wondering if the specific layer where the plateaus appears (e.g. 20 for GPTNEO, 25 for BLOOM) will change?

It's an interesting question. We expect that the plateau would not appear at drastically different layers based on what we learnt from the Experiments from Section 4.1, Figure 2. Those experiments showed that even under different number of examples shown, the plateau is reached at almost exactly the same layer. We think that it is unlikely that having a different set of examples would change the plateau point.

Unfortunately we were not able to test this intuition in time for the rebuttal, but would put them in for the next revision (if accepted or not).

We would like to thank the reviewer for their helpful comments and suggestions on our paper.

1. Generalisability of Results. Please see the general response and our updated appendix, where we include additional results on more language pairs and a larger model. We highlight that our conclusions hold across language pairs. For model size we experimented with an Instruction-tuned model, which shows different behavior from the “raw” (not-tuned) ones.

2. Distribution of Attention heads.

“In Figure 5, the head distribution for BLOOM is quite random.”:

While it is certainly the case that the masks learned from L0 regularization are much more evenly distributed than those learned from GPTNeo, we emphasize that they are not completely random: roughly 60% of the masked attention heads occur in the last 30% of the model layers (layers 20-30). Furthermore in Figure 10, we note that masking patterns of fr->en and es->en appear quite similar. The similar pattern was reached with different initialisation and different language direction, lending confidence that the masking pattern could be quite consistent.

We would like to thank the reviewer for their helpful comments and suggestions on our paper.

1. Experiments on additional Language Direction (en<->es). Regarding the concern of our limited MT tasks, as well as the impact of model training, we would like to direct the reviewer to our general response and the appendix, where we address these concerns with new experimental results. We highlight that the overall trends we observe are prevalent in these new experiments as well. For other tasks such as document level translation, we have proceeded to acknowledge this as a limitation of this study in Appendix Section B.

2. Discussion on Attention-head masking (Section 6)

“The section on the importance of attention heads seems fairly preliminary as conclusions differ for the two models investigated”:

In Figure 10 (new), we note that masking patterns of fr->en and es->en appear similar. The similar pattern was reached with different initialisation and different language direction, lending confidence that the masking pattern could be quite consistent for translation. Furthermore, while the learned BLOOM masks are much more distributed across model layers than those of GPTNeo, we emphasize that the high-level trend is similar: the majority of masks occur in the last 30% of model layers (layers 20-30) in both models.

3. Added Citation. We thank the reviewer for noting the missed discussion of Bansal et al 2022, which is relevant to our section 6 and which have adapted our discussion to cite.

4. Question about Fig1 GPTNeo

Do you have any interpretation of the zig-zagging of the green curve in Fig. 1 for GPTNeo prior to reaching stability?

One possibility that we thought of was that the masked out layers which resulted in the zig-zagging were tightly coupled with the previous layer. This could lead to corruption of the intermediate representations. However, this is mostly speculation and we did not investigate further because it did not occur in the other models, although we know it happens consistently across both translation directions of en $\leftrightarrow$ fr and en $\leftrightarrow$ es for GPTNeo.

5. Instruction Tuning

We have added an additional section discussing Instruction Tuned models (LLama7b-chat) in Appendix A.4 Figure 8. We find that In contrast to GPTNeo and Bloom, the Llama model has no difficulty performing translation when the examples are masked, as long as instructions are present ($`Instr`\overline{`Ex`}^{Mask}$). However, when instructions are masked ($\overline{`Instr``Ex`}^{Mask}$), the model performance collapses drastically. This can be explained by the reliance on Instructions for an Instruction-tuned model (\autoref{fig:context_mask_llama7bchat_fren}).}

We would like to thank the reviewer for their thoughtful comments and questions.

1. Language Pairs and Model Sizes. We direct the reviewer to the general response, where we address these concerns, and to the updated appendix where we include our results from these new experiments. We highlight that our conclusions hold across language pairs. For model size we experimented with an Instruction-tuned model, which shows different behavior from the “raw” (not-tuned) ones.

2. Notation. Similarly, we have updated our draft to include changes to the notation mentioned, such as replacing instr_L1L2 with proper terms and removing the usage of $^{\circ,\bullet}$.

We now address concerns that are specific to reviewer tYJG.

3. Experiment masking attention from all input.

“Maybe another interesting experiment is to mask attention from all input (not only instructions and examples)”: where are inputs stop being used ? What is the missing plot between Fig 1 and Fig 4 -- if that included all layers above a given layer --

We do run this proposed experiment in Appendix A2 of the original draft (A5 /Figure 9 in the updated draft), on the en$\rightarrow$ fr language pair for both GPTNeo and BLOOM. We found that a model’s ability to perform the task diminishes much faster, layerwise, when we mask the entire input, rather than just the context, suggesting that attention to the input is still necessary after we have masked out the context. We have added a discussion of these results in the appropriate appendix section (A5).

4. Discussion reconciling all experiments.

“While Figure 1 and Figure 2 seem to display congruent patterns, it's challenging to reconcile them with Figures 3 and 4”:.

On a high level, we can say quite confidently that there appears to be task recognition or critical task layers happening at around the same middle layers of the model. Fig1 and 2 shows task recognition after a certain layer, Fig 3 shows masking of layers across the model is not equal, and Fig4 shows attention heads in later layers can be masked. However while we have tried to explore the title question from multiple angles, the specific details could indeed be hard to reconcile and we hope this work can inspire a deeper investigation.

5. L0 Regularization Optimization.

“Regarding the L0 regularization optimization, there might be several masks that can lead to similar solutions with different patterns.”

In Figure 10 (new), we note that masking patterns of fr->en and es->en appear quite similar. The similar pattern was reached with different initialisation and different language direction, lending confidence that the masking pattern could be quite consistent.

Also, while it is true that other masking schemes may yield different solutions, we note that L0 masks are not biased towards task identification layers. Instead, the masking scheme we leverage acts as a method to determine which attention heads are most important to in-context MT, and which can be discarded.

6. Question about few-shot examples

how relevant are the specific examples selected in few-shot example ?

In our main paper experiments, for few-shot examples we have repeated the experiments with 5 random seeds to remove the effect of specific examples.

7. Question about Different Instructions

Would different instructions to achieve translation generate different pictures ?

We expect that the plateau would not appear at drastically different layers based on what we learnt from the Experiments from Section 4.1, Figure 2. Those experiments showed that even under different number of examples shown, the plateau is reached at almost exactly the same layer. We think that it is unlikely that having a different set of examples would change the plateau point.

Unfortunately we were not able to test this intuition in time for the rebuttal, but would put them in for the next revision (if accepted or not).

8. Question about Other Tasks

would other tasks behave similarly ?

We are not sure how other tasks should behave, and have 1) listed this as a limitation of this study 2) are careful not to overclaim the contributions of this study.