Inductive Linguistic Reasoning with Large Language Models

We would like to thank Reviewer KXR4 for taking the time to review our paper. We appreciate that the reviewer agrees with the value and novelty of our contributions, and for finds our work to have a “good experimental setting and a comprehensive discussion”. We address the points raised in the review in our responses below, and in our revised paper.

The image Figure 1 is very confusing and is really difficult to read as it is of very poor quality.

We’ve incorporated this feedback to revise Figure 1 to be clearer and more directly illustrative of our method, hopefully alleviating any confusion.

Many steps should be carefully explained e.g. the heart section (section 2 introduces the method that should emulate multilingual analogical reasoning). No examples are given in this section and the problem is not formalised, confusing the reader.

Thank you for this feedback -- we’ve now included a discussion of the translation setting of interest in linguistics olympiad problems (“Rosetta Stone” problems) in Section 2, along with an example of such a problem. Figure 1 also illustrates our 2-stage analogical reasoning method, and we address this in the text of Section 2 as well.

The experiments, although many, are poorly introduced and the thread is not understood.

We appreciate this feedback, and have sought to incorporate it to improve our paper. We have added pointers in Section 3 to the respective results sections (in Section 4) where they are addressed. We hope that this makes the thread of experiments easier to follow. Here is the outline of our core experiments:

1. Section 4.1 consists of 4 baseline experiments (zero-shot, few-shot, few-shot with CoT prompting, and few-shot with CoT prompting to induce a full rationale) introduced in Section 3.1.
2. Section 4.2 first compares the 3 methods introduced in Section 3.2. We then compare the best self-generated results of that experiment ("inferred families") against providing a language family label ("oracle families").

How did you conduct the evaluation?

Evaluation is performed for exact match as the primary metric, due to the noted fallacies of ChrF2 and corpus-level BLEU scores in Section 4. All 272 problems were manually evaluated by one of the authors of this work; the annotation was purely for exact match (without partial scoring or other subjective notions for which inter-annotator agreement would prove as a useful signal). The sole necessity of human evaluation in using exact match is due to parsing errors in instruction following that we find with smaller models like Llama-3.1-8B and Aya-8B; to keep the evaluation protocol consistent, this was repeated for all experiments.

Do you plan to release the code publicly?

Yes, we will release the code publicly in the camera-ready release, for the community at large to apply our method as well as to guide future efforts in linguistic reasoning with LLMs in IOL-style problems. Furthermore, given our method is an inference-time intervention to boost performance through prompting, the prompts included in Appendix D should suffice for the purpose of reproducibility.

---

We hope that our revised draft and these responses address your concerns. We would be glad to address any further concerns.

We would like to thank Reviewer SDWx for taking the time to review our paper and for their valuable feedback. We appreciate that the reviewer agrees that our work’s methodology is “valuable” and has “comprehensive experiments” with “clear presentation of results”. Our responses below address the concerns and questions raised in the review.

The paper's contribution is primarily empirical, with limited conceptual innovation. The approach of using analogical prompting to boost performance is not very inspiring, as it mainly involves augmenting prompts with self-generated information [1].

While our method is inspired by analogical prompting, we note that it differs from the evaluation in Yasunaga et al. 2023, by the “distance” from the target problem to those likely seen in these model’s training corpora. For instance, in Yasunaga et al. 2023 [1], which evaluates on mathematical reasoning tasks in the GSM8K and MATH datasets, these models have seen (similar) math problems in both the pre-training and supervised fine-tuning corpora. In Sun et al. 2022 [2], the tasks are knowledge-intensive closed-book question-answering questions in English (NQ, TriviaQA, and HotpotQA), where the answers to similar problems or relevant facts could be expected to be in the model’s knowledge base. In the context of extremely low-resource languages, however, as addressed in our work, prompting for another language in the same family does not consistently yield a high-resource language, per se, or an instance we would necessarily expect the model to have in its parametric knowledge. We have included an analysis on the language families identified (or rather, the similar languages that the model generates on) in Appendix G of our revised paper.

The authors tested their method only on machine translation tasks, overlooking other question formats in IOL, such as multiple-choice and cloze questions. A more suitable benchmark than modeLing would be [2] or [3].

We focus on machine translation tasks -- more specifically, “Rosetta Stone” puzzles -- as these are the primary focus of both the PuzzLing Machines and modeLing datasets as noted in Sections 3.3 and 5.2. Furthermore, the “Rosetta” category constitutes the largest percentage of the problems in LINGOLY, at $46\\%$ of the problems. The problems in LINGOLY are drawn from the UK Linguistics Olympiad (UKLO), and as such may still be susceptible to leakage, in contrast to modeLing, which presents an entirely unseen set of questions. This is reflected in examining our zero-shot results (which never exceed $1.5\\%$ across all models) and the “no context” baseline in LINGOLY, which can be inferred from their exact match and $\\Delta_{NC}$ scores, which is much higher. This would either be attributable to leakage or the models being familiar with (at least some of) the languages being tested upon.

Thank you for bringing the Linguini benchmark to our attention — this was concurrent work that we were unaware of, which appears to have been released shortly (one week) before the ICLR deadline. As we note in section 3, we do not intend to claim that our work addresses all problem types from the IOL competition, which may also require multimodal inputs, but our scope lies with these “Rosetta Stone” puzzles. To this effect, we have added Section 2.1 to discuss the nature of the problems of interest and provide an example of such a problem. Nonetheless, keeping in spirit with expanding the generalizability of our findings, we are currently working on evaluating our method on the LingOly dataset, per your suggestion, which we will include in the camera-ready version.

It is widely known that closely related languages help with cross-lingual transfer [4] [5]. This paper, however, does not seem to provide any novel insights in this area.

While we acknowledge that there are other works that study the impact of closely-related languages for zero-shot transfer, our work does not specify any similar languages in the instruction nor does it perform any training conditional on this knowledge. A key insight of our work is that models can automatically identify exemplars in similar, seen languages, and in fact can do so even better than when a language family label is specified in the prompt (oracle families). This finding is novel (to the best of our knowledge), and as such, differs from prior art. Furthermore, our findings in language models being able to identify the language families of extremely low-resource or nearly extinct languages at a high rate at test-time without any additional training to do so (included in Appendix G of our revised draft), and even constructing synthetic languages similar to language isolates, is a new contribution as far as we are aware.

In Table 1, zero-shot scores are near zero across all models, which is unexpected, given that BLEU metrics are relatively lenient. Any insights into why this might be the case?

Please note that the scores in Table 1 are exact match, not corpus-level BLEU; these are instead included in Appendix A.2. Nonetheless, it is true that the BLEU scores are still near-zero — we find that all models rarely get even a single token correct in the zero-shot setting due to the challenge of these problems. We find this to be another proof of the fact that translation examples of these extremely low-resource languages have not been seen by the model in either pre-training or post-training.

Line 311-313: "Our findings suggest that when equipped with the right tools (analogical demonstrations) from effective multilingual reasoners, strong deducers can thrive.". However, in Table 2, using Aya-23-35B as the generator yields better results than Llama-405B (which performed better in prior evaluations) when GPT-4o is the deducer. Does this imply that Aya excels at language identification rather than machine translation?

This is a great question, and it is challenging to compare the language identification and (general) translation skills of the model. In Appendix G, we introduce a table examining the language families which models identified in producing analogical exemplars. However, we only include GPT-4o and Llama-3.1-405B-Instruct, due to Aya-35B never producing an explicit language family label. That is, while Aya-35B produces analogical exemplars from the same language family as the target language (e.g. "Here are some puzzles translating from and to languages in the same family as Chimalapa Zoque:”), it doesn’t produce the Mixe-Zoque language family as a label anywhere in its response, unlike the other models. However, we do conclude that for the purpose of solving these puzzles, Aya-35B seems to hold more utility for language identification in producing such exemplars (although, as we note, it is challenging to assess the quality of these exemplars without expert annotators), due to the success of the weak-to-strong method, than it does as a deducer in translation.

References

[1] Yasunaga, M., et al. (2023). Large Language Models as Analogical Reasoners. ICLR 2024.

[2] Sun, Z., et al. (2022). Recitation-Augmented Language Models. ICLR 2023.

---

We hope that our response and revised paper addresses your concerns and questions. We would be happy to address any further concerns.

Thank you for your reply, and for the feedback! We include preliminary results on the LINGOLY dataset below; these apply our 2-stage analogical prompting approach with GPT-4o. We compare against the baselines with the respective models as reported in Bean et al. 2024 [1]. The table below is reflected in figure format in Appendix D in our revised paper. We also provide difficulty / category element-wise differences over the baseline, which highlight the improvements yielded through our method. Note that breakthrough is the easiest set of problems and Round 2 is the most difficult, which serves as the UK invitational qualification exam for the IOL. The results below correspond to the exact match scores reported in LINGOLY. Please note that all empty cells correspond to combinations for which problems do not currently exist in the LINGOLY dataset.

Baseline Results for GPT-4o (reported in LINGOLY):

Our Results with GPT-4o, 2-stage analogical prompting:

Deltas (Difference between our result and baseline):

Encouragingly, we find that our results significantly outperform the baseline by a sizable amount across all difficulty levels, and across all tasks. Moreover, the results outperform the Claude-3 Opus state-of-the-art scores reported in the LINGOLY paper on every single setting, with the exception of the Breakthrough Rosetta Stone (which are the easiest problems). Specifically, we find that our 2-stage analogical prompting method enables GPT-4o to solve questions of the monolingual type which it could not before (0% —> 19% and 14%); furthermore, the correctness rates jump considerably for some of the hardest categories over the baseline (1.81x improvement in Round 2 Pattern, 1.96x in Advanced Rosetta Stone, and 3.42x in Round 2 Rosetta Stone). It is especially worth noting that the Round 2 Rosetta Stone results corroborate with our findings on modeLing as reported in our paper. These findings suggest that our method generalizes across both datasets and question types.

A note on the prompting method: we use the same prompts as before, with the addition of the preamble and context provided in LINGOLY for all of the tasks. The context includes both the background on the problem and the exemplars in the target language, as in our evaluation with modeLing. As such, there might be opportunities to further improve the results by optimizing the prompts for the various other tasks introduced in this dataset.

We have added a note in Section 3.3 of the question types included in LINGOLY, which expand beyond the "Rosetta Stone" translation problem style (although, as aforementioned, the Rosetta Stone category still constitutes a sizable fraction of the problems). Furthermore, these preliminary results have been added to a new appendix section (Appendix D; shifting the other sections accordingly) in a revision of our paper which we have uploaded.

Expert Validation

On the point of expert validation — we refer the reader to our limitations, Section 2 on exemplar correctness, and in footnote 3, where we discuss the challenge in introducing a single reliable expert who would be sufficiently familiar with and adept at understanding the typological rarities present in these extremely low-resource languages. Most of the languages in modeLing do not have a consolidated grammar book, to the best of our knowledge (unlike MTOB [2], for which there exists a grammar book for Kalamang), from which one could verify correctness. While we appreciate and acknowledge the importance of a forming a deeper, concept-level understanding of what occurs in the cross-lingual transfer from auxiliary language exemplars, we posit that this would likely require either a global crowd-sourcing initiative (which is practically challenging) or a interpretability-driven analysis to study concept learning, which we find to be beyond the scope of our work.

References

[1] Bean, A. M., Hellsten, S., Mayne, H., Magomere, J., Chi, E. A., Chi, R., Hale, S. A., & Kirk, H. R. (2024). LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. arXiv preprint arXiv:2406.06196.

[2] Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., & Melas-Kyriazi, L. (2024). A Benchmark for Learning to Translate a New Language from One Grammar Book. arXiv preprint arXiv:2309.16575v2.

---

Thank you very much again for your feedback, and we hope our promising results above and responses have addressed your concerns!

What are rationales in the particular case of linguistic problems?

Rationales in the context of these translation task involve word-by-word explanations behind why a word in English maps to one in the target language (or vice versa), as well as any explanations regarding the ordering of words. Here is the exemplar provided for the few-shot chain-of-thought with rationale prompt setting (also included in Appendix E.5):

1. Spanish: ventana roja English: red window 
2. Spanish: ventana azul English: blue window 
3. Spanish: manzana azul English: blue apple 
Using the above examples, translate the following.
Spanish: manzana roja 
EXPLANATION: The first step we notice is that the word “ventana” must mean window because (1) the word “ventana” appears twice between sentences 1 and 2, and (2) the only word that appears twice in the English translation is “window.” Next, we infer that “roja” must be “red” and “azul” must be “blue” by process of elimination. Next, we guess that in Spanish, the noun precedes the adjective because “ventana” comes before “roja” and “azul.” Therefore, the noun in sentence 3 (“apple”) must correspond to the word preceding the adjective (“manzana”) in the Spanish translations. Putting this together, “manzana roja” must mean “red apple” in English. 

ANSWER: English: red apple. 
Now, given the following test phrase, please translate it. Take a deep breath and work on this problem step-by-step in a logical way, using careful analytical reasoning to get the correct result. When you are done with your answer, provide your outputs in the format of **[your answer]**.

---

We hope that our responses address your concerns in the review. Please let us know if you have any further questions!

We would like to thank Reviewer uieZ for taking the time to review our paper, for their valuable feedback, and for finding our 2-stage analogical prompting method “interesting”. We address the concerns raised in the review below and in our revised paper.

There is little in the paper to suggest in the results that any grammar rules have been really learned or what the form of the grammar rules might be.

This is very helpful feedback, and we appreciate the reviewer’s thoughtful response on this point. We acknowledge that the usage of the term “rules” as used in our statement does not quite imply that grammar rules have been rigorously learned, as the reviewer has pointed out. What we intend to say as a conclusion, rather, is on the basis of the evidence that these frontier models first produce token-level mappings between the source and target language for the few-shot exemplars, and token-level mappings between the source and auxiliary languages for the analogical exemplars; then apply these mappings to the test phrase. That is, these mappings are what we intend to refer to as “rules”. Nonetheless, this point is well-taken, and we have rephrased this conclusion accordingly to avoid confusion.

One can gather that at least some of the problems are translation problems. But are they all translation problems? If so, how on an olympiad test would a participant be able to get a good translation for a completely unknown language without any clues? The whole experimental basis of the paper is kind of murky and needs to be cleaned up for those readers who are unfamiliar with the linguistic olympiads.

We focus solely on machine translation tasks -- more specifically, “Rosetta Stone” puzzles -- as these are the primary focus of the modeLing dataset as noted in Sections 3.3 and 5.2. Indeed, in these linguistics olympiad tests, participants are expected to perform translations for often-unseen languages, solely given a list of examples from which to infer surface-level associations and apply pattern matching (deductive reasoning) to solve. We appreciate the feedback on the nature of these puzzles being unclear, and have updated our paper to include Section 2.1, a discussion on the problems of interest in our work and an example of such a puzzle.

I might of missed it but it seems that the paper itself doesn't contain a discussion of what happens when the language family is omitted but the examples are provided. It would have been nice to have a more detailed stucy of the analogical reasoning itself.

(Repeated from general response): We analyze the language family labels produced by Llama-3.1-405B-Instruct and GPT-4o in the first stage of our analogical prompting procedure, prior to identifying specific languages from said family and producing the analogical exemplars. This corresponds to the inferred language families experiments in Figure 2b, where the model is only prompted to produce puzzles from other languages in the same family, without specifying that family. We compare these labels against the oracle labels in the table included in Appendix F, to yield a correctness score; we include these tables in Appendix G.

We also qualitatively discuss models’ behavior in 2-stage analogical reasoning, added to Section 4.2, under “Analogical reasoning boosts frontier models” in our revised draft (repeated here): In the first stage, both of these frontier models correctly identify the language family at a fairly high rate (see tables above), select a few languages from said family, and generate analogical puzzles for those auxiliary languages, as intended. Then, in the second stage, the model walks through the tokens in the test phrase, and analyzes how each is to be translated to the target languages, and then combines them together in the appropriate order based on following the syntactical patterns observed from the given exemplars. Thus, it appears that the model uses the analogical exemplars to better induce the mappings of words in the target language to words in the source language, which it then applies to the target phrase.

We thank Reviewer Entm for taking the time to review our paper, and for their positive review. We appreciate that the reviewer finds our contribution to be an “innovative approach” and our paper to be “well-structured”. Please see below for our responses to the reviewer’s concerns and questions.

Section 4 mentions that each response was manually evaluated to provide exact match scores, but this evaluation process lacks details. Specifically, there’s no mention of how many responses were reviewed, how many LLMs were involved, the number of evaluators, or their inter-annotator agreement. Without this, it’s challenging to assess the reliability of the manual evaluation.

Evaluation is performed for exact match as the primary metric, due to the noted fallacies of ChrF2 and corpus-level BLEU scores in Section 4. All 272 problems were manually evaluated by one of the authors of this work; the annotation was purely for exact match (without partial scoring or other subjective notions for which inter-annotator agreement would be a useful signal). The sole necessity of human evaluation in using exact match is due to parsing errors in instruction following that we find with smaller models like Llama-3.1-8B and Aya-8B; to keep the evaluation protocol consistent this was repeated for all experiments.

Section 5.2 mentions other linguistic reasoning datasets, yet these were not utilized in the experiments. Incorporating additional benchmarks would provide more reliable and generalizable results.

(Repeated from general response): The machine translation tasks -- more specifically, “Rosetta Stone” puzzles -- that form the primary focus of our work are present in a few benchmarks: PuzzLing Machines, modeLing, and LINGOLY. As we note in Section 5.2, modeLing was developed in part due to concerns of leakage of the problems in the PuzzLing Machines dataset, which is older (2020) and whose content may have been included in the vast web-scraping performed for curation of pre-training corpora. This has motivated our selection of the modeLing dataset, which consists entirely of newly written problems by experts to ensure the quality of the problems as well as avoid leakage. By contrast, the problems in LINGOLY are drawn from the UK Linguistics Olympiad, which may still be susceptible to leakage; the authors introduced a “no context” baseline, which is somewhat akin to our zero-shot baselines. However, the “no context” performance is non-zero for most mid-size and large / frontier models, inferred from their exact match and $\\Delta_{NC}$ scores, unlike in modeLing, suggesting that either leakage is present or the models are familiar with the languages being tested upon. This led us to rely on modeLing as our dataset of focus. Nonetheless, we appreciate your feedback on this matter; in the spirit of expanding the generalizability of our findings, we are currently working on evaluating our method on the LingOly dataset, which we will include in the camera-ready version.

The paper briefly mentions that frontier models like GPT-4o and Llama-3.1-405B-Instruct often successfully identify language families. How accurately do LLMs identify language families, and how often do they correctly solve queries when the language family identification is accurate?

The claim we make regarding oracle vs inferred language families is more directly tied to frontier models not relying on language family labels, and leveraging an intrinsic understanding of language similarities to produce useful exemplars that result in performance gains. This is evident through the results with language isolates, wherein the model either chooses a language from a similar family higher in the taxonomy (often on the basis of geographical proximity) or assumes that the language is fictional or invented, and attempts to construct synthetic languages with similar syntactic patterns. That is, the model does not necessarily need to identify the correct language family in order to produce the correct answer. We include an analysis of the language families identified by GPT-4o and Llama-3.1-405B-Instruct in the inferred language families setting (as they often explicitly produce a label, e.g. "Ayutla Mixe belongs to the Mixe-Zoquean language family.”) in Appendix G.

The results show that the mixture setting—where analogical exemplars are generated by one model and applied by another—outperforms the self-generation setting, but the paper does not delve deeply into why this occurs.

Without expert annotations, it is challenging to compare the exemplars generated by the different models, beyond their downstream impact on performance when applied with different deducers. As such, we can surmise from our results that a better deducer model (Llama-3.1-405B-instruct) can make the most out of qualitatively different exemplars.

---

We hope that this addresses the points raised in the review, and we would be happy to address any additional questions!