CLEAR: Understanding the Reasoning Capabilities of Large Language Models

1. The "learning a new language" task feels more akin to in-context learning, emphasizing the models’ imitation and induction abilities rather than the “reasoning abilities” claimed in this paper. While "reasoning" is a term that can describe various tasks and skills, its use here in a “translation-like” task seems unsuitable.
2. The paper devotes excessive space to fundamental concepts. For instance, the section on few-shot prompting in related work and the descriptions of common metrics like BLEU could be more concise. Additionally, sections on logical reasoning in related work, including Logical Reasoning of Large Language Models and Logical Reasoning, are somewhat repetitive and lengthy.
3. The paper is somewhat difficult to follow. The task remains unclear even after reading the introduction, with clarity only starting to emerge in Figure 2. Several typos are present, such as “To‘learn’” in Line 72. Additionally, the text in Figure 1 is too small to read comfortably. Page 8 would benefit from reorganization if it only includes two tables.
4. The analysis is somewhat superficial, focusing mainly on translation direction without offering deeper insights to inform future work. Also, could the unexpected relationship between model performance and question complexity be due to an ineffective measure of complexity?

Overall, I feel the current version seems rushed and does not meet the average ICLR standard.

1. The size of the dataset is limited, which is also reflected on the analysis of prompt complexity. This limitation is noted in the paper but may affect the robustness and generalizability of the findings.
2. There is a potential problem of the entanglement of tokenization and reasoning in this particular tasks. I think further analysis is required rather than just mentioning it in limitations.
3. CLEAR mainly focuses on inductive reasoning, but general reasoning often involves deductive and abductive reasoning. Improving the diversity of the task forms could significantly increase the impact.

1. With repeated exposure to Conlang structures, LLMs may develop specialized reasoning paths tailored to the benchmark, rather than generalizable reasoning skills. The limited dataset could falsely encourage models to adapt to specific patterns in the Conlangs, thereby reducing the benchmark’s efficacy in evaluating general reasoning capabilities.
2. The benchmark focuses on output accuracy without providing insights into the reasoning paths which is crucial for evaluating the reasoning ability. As the grammar and structure of Conlang are relatively simple, including an analysis of intermediate reasoning steps could reveal where models tend to struggle, offering diagnostic insights beyond final translation accuracy. There are related previous works like chain-of-thought, and least-to-most that prompt the model to show explicit reasoning paths. In your evaluation, do you ask the model to generate reasoning paths like in Figure 2, or it only outputs the answer? Some analysis on such paths could improve the quality of the work.
3. As mentioned by the authors, the size and task of the benchmark is quite limited. This would affect the robustness of the benchmark and could make it harder to generalize the findings.

I have one big concern for this article. The lack of details on what the benchmark is testing caused me some issues in understanding what capabilities it is testing. While the authors describe the task and provide a few examples, its small size (140 samples) and the lack of a detailed analysis of what kind of capabilities are required by a model to solve it (the authors mention multiple time logics and, I reckon, compositionality) make it very hard to draw comparisons with existing datasets. Furthermore, the authors do not run concurrent experiments on similar tasks (e.g., a baseline), to show correlation with existing benchmarks. In other words, is this benchmark telling us something about other popular LLMs’ benchmarks? If that’s the case, one can argue that your dataset is a proxy for some high-order reasoning and use your dataset (for example, because one can synthetically create new instances easily and does not suffer from memorisation) instead of one created by human experts.

The article seems rushed (see the paragraph on related work or the methodology, paragraph “Closed Frontier Models”). Further, in lines 269-270, it’s your duty to run the experiments in time and before the deadline; we cannot discount the fact that GPT-4o-mini has a longer response rate, especially considering that your benchmark is very small. It’s better to present organic experiments on all the models or exclude them from the evaluation rather than preliminary results that may not be statistically significant or affected by sample bias.

Figure 1 is confusing and difficult to interpret after reading the first section. Even after reading the methodology, I still do not fully understand what it represents.

Related works are rushed, with a few references missing (line 97) or inconsistent formatting (see 107-108 vs. 122-123). Some sentences are not grammatically consistent (lines 112-13), and others express vague concepts (line 125).