UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?

We thank Reviewer TCNB for their thorough and constructive feedback. We appreciate the detailed evaluation and the opportunity to address the feedback. Below we address each of the concerns raised, clarifying our motivation, methodology, and revisions for the camera-ready version:

• Critical Issue (Reasoning and Complexity Controls): We appreciate this excellent observation regarding the decomposition of puzzle complexity and characterization of LO puzzle difficulty. Our study also focuses on this challenge by using linguistic features as proxies for complexity, given that LO puzzles are inherently difficult to decompose into reasoning steps or atomic units. As established by Bozhanov et al. (2013), these puzzles "are designed to be solved by deducing linguistic patterns, with other sources of technical complexity increasing difficulty without making them linguistically interesting". Our findings demonstrate that linguistic features exhibit at least a linear correlation with LLM performance. Future work could explore potential super-linear relationships between features and performance, as linguistic patterns across different domains (morphology, syntax, etc.) become increasingly intertwined in more challenging puzzles, such as those in the International Linguistics Olympiad. We will include this motivation for breaking down LO puzzles’ reasoning into features in the camera-ready version for clarity as well.
• Significance of the work: This work represents a foundational step toward probing and enhancing linguistic reasoning capabilities in LLMs. These are crucial because human proficiency in linguistic reasoning and metalinguistics contributes significantly to efficient learning from limited input. Following the frameworks and goals established by Mahowald et al. (2023), McClelland et al. (2020), Beguš et al. (2023), Şahin et al. (2020), and Marcus et al. (2020), improvements in LLM metalinguistic processing capabilities can enhance general performance across diverse tasks. Our work proposes an uncontaminated method for probing these abilities, with findings that can inform more efficient language learning in LLMs and improve their general capabilities.
• State-of-the-art benchmarking: Our model selection prioritized diversity across model families, parameter counts, and reported performance on generic reasoning benchmarks within strict budgetary constraints, rather than maximizing benchmark scores. Consequently, we could not include the Claude family in our initial analysis.
• ITC models: We recently gained access to DeepSeek-V3 and DeepSeek-R1 (ITC model), with the latter demonstrating superior performance while maintaining our observed correlations between model performance and morphological features (Pearson correlation = -0.74***, p < 0.001). These results will be included in the camera-ready version: | Prompt | Prev. Best | R1 | |-----------------------|:--------------:|:------:| | Null Prompt | 47.4 | 60.7 | | ModeLing Minimal | 53.1 | 61.4 | | ModeLing Hand-Tuned | 49.2 | 64.7 | | ModeLing Basic-CoT | 48.7 | 64.9 | | ModeLing Full-CoT | 49 | 60.5 | | LingOly Std | 48.3 | 61.9 |
• Tokenization Experiment: You are correct in assuming that tokenization does not necessarily match morphology-based splitting of words, which is why we added whitespaces manually (Line 280-281: "annotate"). We will clarify this explicitly in the camera-ready version.
• Comparing Tokenization Results: The divergence from LingOly-TOO findings stems from methodological differences. While LingOly-TOO employed automatic tokenizers for character-level processing of unseen languages, our approach utilized morphologically-informed word segmentation based on language-specific rules, resulting in substantial improvements.
• English bias: We acknowledge your important concern about the English bias possibly originating from monolingual models. However, our experiments involve models like GPT 4o, Llama 3.1, etc. which are multilingual. As we test over multiple such models, we observe the bias consistently present in a significant way.
• Multilingual models: You have also pointed out Aya and Gemma as models which are explicitly trained on more balanced multilingual data. We acknowledge that these would be good future additions to our line of study, as they may present interesting results. We will consider them in our future work.
• Data release: We plan to release all the annotations (mappable to data from the publicly available datasets), code, and fixes to the data upon paper acceptance.
• Ethics Statement: We will clearly state in the camera-ready version that we comply with all available licenses and/or have sought authors' explicit permission to use their datasets. We have made minimal changes and will publish only mappable annotations and a list of fixes for full reproducibility.

(contd. below)

(contd. from above)

• Figures 5, 6, 7, 9: These figures are supposed to help readers understand the respective analyses and also give them an idea of what the puzzles look like in context (Figure 2 presents all parts of the annotation process). For academic structure and clarity, in the camera-ready version, we will either put them together in a single figure or use the LaTeX example format to present them inline.
• Figures 8 and 20: We found that "@" separated morphemes yielded results comparable to whitespace separation, hence their exclusion. To address crowding concerns while maintaining clarity, we will present these as stacked bar charts in the camera-ready version.
• Typos: We will fix the following in the camera-ready version: specify LO puzzles & linguistic competency (L97), add a footnote linking WALS to Dryer & Haspelmath et al., and change "correlation. Implying" to "correlation, implying".
• Casual writing: Finally, we will pass the paper through another round of proof-reading to make it more formal and academically worded for the camera-ready version.

References:

• Bozhanov, Bozhidar, and Ivan Derzhanski. "Rosetta stone linguistic problems." Proceedings of the Fourth Workshop on Teaching NLP and CL. 2013.
• Mahowald, Kyle, et al. "Dissociating language and thought in large language models." Trends in cognitive sciences (2024).
• McClelland, James L., et al. "Extending machine language models toward human-level language understanding." arXiv preprint arXiv:1912.05877 (2019).
• Beguš, Gašper, Maksymilian Dąbkowski, and Ryan Rhodes. "Large linguistic models: Analyzing theoretical linguistic abilities of LLMs." arXiv preprint arXiv:2305.00948 (2023).
• Şahin, Gözde Gül, et al. "PuzzLing machines: A challenge on learning from small data." arXiv preprint arXiv:2004.13161(2020).
• Marcus, Gary. "The next decade in AI: four steps towards robust artificial intelligence." arXiv preprint arXiv:2002.06177(2020).

We thank Reviewer Qiv2 for their constructive feedback and recognition of our solid methodology and results. Below we address each of the concerns raised, clarifying our motivations, methodology, and planned revisions for the camera-ready version:

• Excessive Acronyms: We will comprehensively revise the manuscript to reduce reliance on acronyms and provide clear definitions upon first usage to enhance readability in the camera-ready version.
• Relevance: This work represents a foundational step toward uncontaminated probing and enhancing linguistic reasoning capabilities in LLMs. These are crucial because human proficiency in linguistic reasoning and metalinguistics contributes significantly to efficient learning from limited input. Following the frameworks and goals established by Mahowald et al. (2023), McClelland et al. (2020), Beguš et al. (2023), Şahin et al. (2020), and Marcus et al. (2020), improvements in LLM metalinguistic processing capabilities can enhance general performance across diverse tasks. Our immediate findings are relevant for Low Resource Language learning efficiency in LLMs, tokenization optimization, and more.
• Annotations: The high inter-annotator reliability results from our constrained puzzle design, annotation framework with expert annotators, and well-defined linguistic features. Due to space constraints, we will detail the annotation guidelines in the appendix of the camera-ready version for transparency.
• modeLing and LingOly: These datasets are introduced in Lines 63-69 and summarized in Table 1 before their reference in Line 72.

References:

• Mahowald, Kyle, et al. "Dissociating language and thought in large language models." Trends in cognitive sciences (2024).
• McClelland, James L., et al. "Extending machine language models toward human-level language understanding." arXiv preprint arXiv:1912.05877 (2019).
• Beguš, Gašper, Maksymilian Dąbkowski, and Ryan Rhodes. "Large linguistic models: Analyzing theoretical linguistic abilities of LLMs." arXiv preprint arXiv:2305.00948 (2023).
• Şahin, Gözde Gül, et al. "PuzzLing machines: A challenge on learning from small data." arXiv preprint arXiv:2004.13161(2020).
• Marcus, Gary. "The next decade in AI: four steps towards robust artificial intelligence." arXiv preprint arXiv:2002.06177(2020).

We thank Reviewer iDKA for their positive recommendation and recognition of our work as nice with insightful analysis. We address the specific questions below:

• Inflectional vs. Derivational Morphology: We agree that distinguishing inflectional from derivational morphology would be insightful. However, our LO dataset contains too few puzzles that isolate one type of morphological process, preventing reliable statistical analysis (NA/Compound do cover some of these, but it is a small fraction of the data). We will note this limitation and suggest it as future work.
• Tokenization: We did not modify the underlying tokenizer’s vocabulary. Instead, we inserted whitespaces manually to force human‐informed morpheme boundaries (Line 280–281). Because the tokenizer still uses its original word‐piece dictionary, overlap between true morphemes and word‐pieces is incidental and typically low (with GPT-4, we observed only a 0.62 (out of 1) Jaccard similarity between the morphemes and the word-pieces). We will clarify this process in the camera‐ready version.
• Language Swap: We will correct the swapped language names in Figure 6 in the camera-ready version.
• Font Problem: Following the established naming pattern and font style (textsc) in this domain (modeLing, LingOly, Linguini, PuzzLing), we adopted "unveiLing" for our work. The section header at Line 194 removes the "-ing" suffix for grammatical agreement purposes.

We thank Reviewer D2sE for their thorough evaluation, strong acceptance recommendation, and recognizing the thorough experiments and important insights into linguistic reasoning limitations. We address the presentation issues below (and will fix them for the camera-ready version):

• Research Question: We acknowledge this oversight and will explicitly state our research questions to provide clear direction for readers.
• WALS reference: We will add a footnote linking WALS to Dryer & Haspelmath et al. and provide the full form at first usage to ensure proper introduction (World Atlas of Language Structures).
• Figure 1 caption: We will enhance the caption to be more descriptive and accessible for readers unfamiliar with Linguistics Olympiad problems, providing sufficient context for understanding.
• Attributes vs. Features: We use "Attribute" as an encompassing term that includes linguistic features alongside English bias and data constrainedness measures. We will clarify this distinction when "Attribute" is first introduced and conduct comprehensive proofreading to ensure consistent usage throughout the paper.