Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?

We thank the anonymous reviewer for their valuable feedback. We respond to each point in turn below.

Weakness 1a: The standard setting for training seq2seq MT models from scratch indeed works with parallel data; however our main experimental setting is distinct from that standard scenario, since for Kalamang we have a grammar book and extremely limited parallel sentences, and we use LLMs in-context to leverage this non-standard data. We emphasise that the purpose of this paper is to revisit claims in previous work, primarily Tanzer et al. (ICLR, 2024) who claim that a single grammar book in-context may suffice for LLMs to learn to translate an (unseen) language. Our findings call these claims into question: we show that LLMs only really learn from the parallel examples within the book. We show this via a thorough analysis of different prompting setups, and further we introduce a novel approach that effectively uses typological knowledge instead of grammar books for grammatical tasks. Our tests on Nepali (and Guarani) show that these findings hold in both XLR and low-resource translation scenarios – LLMs fail to exploit grammatical explanations in general, rather than failing to use the explanations because the language is already seen. Our findings are therefore a contribution on the question of grammar books for LLM-based translation, an area which is receiving increasing research interest (Hus & Anastasopoulos. 2024, Tanzer et al. 2024, Zhang et al. 2024, Ramos et al. 2024).

Weakness 1bi: Regarding generalisation to other low-resource languages, we add experiments on Guarani with a different grammar book (Estegarribia 2020, from UCL Press’s series “Grammars of World and Minority Languages”) to improve our findings’ generalisability. Guarani is similar in task to Nepali, given the similar availability of parallel data, plus a high quality test set (FLORES). Initial results in Guarani show the same pattern of parallel sentences providing most or all useful signal, and few-shot sentences are again the most effective prompt. (See the response to review 5eBi for initial results tables). We will add these results to our updated paper.

Weakness 1bii: It is not entirely clear to us what is meant with “a form that LLMs are not expecting”, however, we use a standard form of grammar text: the Kalamang grammar is from MIT Language Sciences Press which has strict formatting and content guidelines for linguistic texts, and thus we expect this book is representative of grammar books in general. Second, while we cannot comment on what LLMs are expecting, we hypothesise that instruction-tuned models would perform better on prompts that are more similar to their instruction-tuning dataset. Given our discussion on p.10 (where we suggest LLMs learn more effectively from unannotated parallel examples than from grammar principles/explanations), we might suggest these LLMs were instruction-tuned on data including few-shot parallel examples, and not with grammar book-like explanations; whether or not this is the case, our finding holds that current LLMs are unable to exploit the grammatical explanations in grammar books and rely mainly on the parallel sentences. We will highlight this point in our Discussion.

Weakness 1c: Regarding our title, “Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?”, the fact that it includes “really” within a question standardly implies that the speaker doubts the question’s premise. The premise here is the claim by prior work including Tanzer et al. (2024) that prompting LLMs in-context with grammar books enables the model to translate from and into an unseen, XLR language. Our paper revisits these claims, and our title implies that our results doubt these claims, instead showing grammar books do not really help LLMs to translate in this case (but their parallel sentences do).

Weakness 2: We accept that the linguistics-specific topics in this paper may require some more introduction to some readers; to this end, we will further clarify glossing and typology in the Related Work and Methodology in our updated paper, with sufficient information to understand that a) IGT prediction is a well-motivated, grammar-focused task, and b) typological information is a useful and language-independent form of linguistic knowledge. We also provide an excerpt of our typological prompt in Appendix B, Table 6, for the interested reader. Additionally, we note in-depth linguistic knowledge is not required to understand the research questions, experiments, or main contributions of our work. Finally, one of our work’s central goals is to revisit claims made first in Tanzer et al. (2024), an ICLR 2024 spotlight paper; we therefore expect the audience to be appropriate.

We thank the anonymous reviewer for their useful feedback, and respond to the comments below.

Weakness 1: Regarding the generalisation of our results to other low-resource languages, we add experiments on Guarani ($`gug`$) which has a grammar book available of a similar length to Kalamang, and a FLORES test set. Initial results on filtered $BOOK_{p}$ and $BOOK_{\neg p}$ subsets below confirm what we observed for Nepali: most useful signal stems from the parallel data, with the grammatical explanations showing little added utility, and all book settings showing little to no improvement over 0-shot tests. This holds for both Gemini and Llama. Given this, we would expect similar results for other low-resource and XLR languages. Further, while Kalamang is an XLR language, Nepali is more accurately classified as a low-resource language which has some level of parallel data available; and Guarani also falls in this category. This difference is visible in the zero-shot performance of our LLMs (primarily Gemini) on Kalamang against Nepali and Guarani. Testing on Guarani also controls for script since like English it uses Latin script (with additional diacritics), whereas Nepali uses Devanagari script. We will add these results to our updated paper.

English--Guarani

Guarani--English

Weakness 2: While using typological (or language) embeddings seems like a reasonable thing to try, they have been shown to have limited success (Ponti et al. 2019, Ustun et al. 2022) in other NLP tasks. Second, our typological prompt is a drop-in replacement for $BOOK_{\neg p}$, and is used in combination with $BOOK_p$; it is not immediately clear how we would combine embeddings and the textual input $BOOK_p$. We also note that adding complexity to the experimental setup (e.g. using specialised models) runs counter to our goal of using already available, human-readable resources to rapidly adapt a general-purpose model to new language pairs – i.e. in-context learning with LLMs. This comparison with typology/language embeddings is oblique to our main research question, and is a more substantial experiment than running a baseline model, therefore we leave further investigation to future work.

Dear Reviewer 5eBi,

We would like to thank you once again for your helpful feedback, and we would appreciate it if you'd let us know if our response and revisions to our paper address your concerns.

Thank you!

We thank the anonymous reviewer for their helpful feedback. Regarding your comments:

Minor weakness: We will update abbreviations in Table 2 for improved clarity, expanding $BOOK_{p}$ to $BOOK_{para}$, $BOOK_{\neg p}$ to $BOOK_{non-para}$, $P_{book}$ to $PARA_{book}$, and $P_{book}^{IGT}$ to $PARA_{book}^{IGT}$ respectively. The capitalised text indicates the form of the data (i.e. grammar book text, or processed parallel data with or without glosses) and the subscript indicates the subset of this data (e.g. $PARA_{train}$ = processed parallel data, from the training set, not the book). We will add this explanation in our updated paper in 4.1.

Question: Regarding expectations for different model sizes, we would expect similar conclusions that parallel data provides most or all useful translation signal. Further, we know Gemini-1.5-Flash is substantially larger than Llama-3.1-8B (given the smaller Gemini-1.5-Flash-8B model), thus our results suggest larger models show better handling of long-context for translation tasks. We expect models smaller than 8B would show worse performance than Llama-3.1-8B, while models towards 70B and larger may show better results than Gemini-1.5-Flash. Indeed, in the Gemini-1.5 Technical Report (https://arxiv.org/abs/2403.05530), the authors observe translation quality improvements for the larger 1.5-Pro model over 1.5-Flash, suggesting performance on this task, including handling of a long-context grammar book, scales with model size.

Dear Reviewer nVmi,

We would like to thank you once again for your helpful feedback, and we would appreciate it if you'd let us know if our response and revisions to our paper address your comments.

Thank you!