CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation

Thank you very much for your thoughtful comment and for recognizing benchmark design and thorough evaluation. We address your concerns in detail below.

Q1: Potential training strategies or data augmentation techniques that might improve the performance of open-weight models on this benchmark?

A1: Thank you for your comment. Our primary goal is to introduce a challenging and diverse evaluation suite for LLMs and LVLMs, rather than explore novel learning methods. That said, our benchmark is well-suited to support future training research. Crossword puzzles offer a unique advantage: they have verifiable, constraint-based solutions, much like structured tasks such as Sudoku [1], 2048 [2]. This property makes them ideal for reinforcement learning (RL) and other training paradigms where clear reward signals are critical. We briefly discuss this potential in Section 5.4 of our paper.

Additionally, our controllable puzzle generation framework naturally supports data augmentation, a capability that static online sources cannot offer. Even when using the same set of word–clue pairs, we can generate diverse grid configurations that vary in structure, layout, and clue placement.

Reference:

[1] Hrishbh Dalal. Agent 2048: Forging Strategic Gameplay in an AI Through Data, Rewards, and RL. Blog post, March 2024.

[2] Hrishbh Dalal. Teaching Language Models to Solve Sudoku Through Reinforcement Learning. Blog post, March 2025.

Thank you very much for your valuable comment. We will incorporate your suggestions in the final version and address your concerns in detail below:

Q1: Limited Novelty and Contribution

A1: Thank you for the comment. We agree that crosswords have been explored in prior work; however, our contribution goes beyond in several important ways:

• We introduce a scalable and controllable generation framework that enables variation of grid sizes, word-clue pairs, input modality (texts and images), and evaluation settings (different prefill ratios and interactive mode), which prior datasets do not support.
• Crucially, crossword facilitates multimodal reasoning under persistent structural constraints. Unlike other multimodal benchmarks---such as chart understanding [1] or math reasoning [2]---crossword solving requires the model to dynamically and repeatedly refer to the spatial grid structure throughout the problem-solving process. The model must not only interpret clues but also coordinate multiple answers to maintain consistency within the intersection of the grid.
• This creates an environment for evaluating the dynamic interplay between vision and language: models must extract and reason over layout, resolve clues, and ensure that answers conform to both semantic and spatial constraints. Such tight coupling between modalities and reasoning steps is rarely demanded in existing benchmarks.

Reference:

[1] ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. ACL Findings 2022

[2] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. ICLR 2024.

Q2: Opaque Puzzle Generation

A2: Thank you for pointing this out. We will include a more detailed description of our puzzle generation process in our final version and of course release the code. We felt that for the COLM audience the details of the generation process might be of less interest than the implications of the new benchmark for LLM and LVLM research.

In brief, our generation framework uses a heuristic scoring function that encourages desirable properties such as a higher number of horizontal and vertical intersections between words. We also apply post-processing filters, discarding puzzles that fall below a threshold for minimum clue count or exceed a threshold for blocked cells, to ensure quality of the generated puzzles.

While we primarily use grid size to control puzzle difficulty, these heuristics and filters allow us to filter out low quality puzzles. The statistics of the generated puzzles, including clue count, word length, and blocked cell ratios, are reported in Table 1, and serve as a proxy for puzzle complexity and quality

Q3: Lack of Validation: There is no validation demonstrating that the generated puzzles possess comparable quality, difficulty, or structural properties to standard human-created crosswords. Can there be a comparison between the same. This could strengthen this work. Human baselines could also help.

A3: Thank you for the suggestion, and we will discuss this further in the final version and consider human baselines for the future. Complementary prior work such as LR2Bench [1] has directly used human-created puzzles (e.g., from LA Times and Vulture), and reported model performance using S-Acc, a metric conceptually similar to our Word Coverage Rate (WCR). Compared to human-created puzzles, our benchmark seems to be of comparable difficulty. However, while it is clear from prior work that these tasks are difficult for LVLMs, it is unclear what the source of that difficulty is.

In our current work, we thus focus on automatically generated puzzles to enable scalable, controllable, and diverse evaluation across different subjects and grid sizes. This design allows us to systematically vary puzzle difficulty (via grid size and word-clue pairs selection) and supports a wide range of evaluation strategies—including pre-filled puzzle settings and an interactive mode for step-by-step reasoning. While we do not include human-created puzzles in this version, our generation process is designed to preserve key structural properties---such as clue distribution, word length, and blocked cell ratios---as reported in Table 1.

In addition, our benchmark offers a broader set of evaluation metrics. These provide complementary, fine-grained signals beyond overall accuracy, making the benchmark more differentiated and diagnostic for assessing model capabilities.

Reference:

[1] Lr2bench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems. Arxiv 2025.

Thank you for your responses.

Please add a detailed version of A2, A3 in the camera ready paper. Please add A4 in the camera ready version of the paper as well. I have increased my score.

Thank you very much for your thoughtful comment and for recognizing the value of our benchmark design. We address your concerns in detail below:

Q1: Some method details are vague in the main text and require too many trips to the Appendix.

A1: Thank you for your feedback. We will revise the main text to include clearer descriptions of key methods and reduce the need to reference supplementary sections in the final version.

Q2: In the text-only mode, how is the input puzzle formatted? Were ablations on this format performed like the VLM format ablations?

A2: Thank you for the question. In the text-only mode, we format the puzzle grid following the design inspired by ARC-AGI [1]. The grid is represented as a 2D array, where 1 denotes a black (blocked) cell and 0 denotes an empty (unfilled) cell. For example: “empty grid = [[0, 0, 0, 1, 0, 0, 0], [0, 1, 0, 0, 1, 0, 1], [0, 0, 0, 1, 0, 0, 0], [0, 1, 1, 1, 0, 1, 0], [0, 0, 0, 0, 0, 1, 0], [1, 0, 1, 0, 1, 1, 0], [1, 0, 1, 0, 0, 0, 1]]”

Following your suggestion, we also conducted an ablation study using an alternative formatting scheme---a markdown-style grid, where ‘·’ represents an unfilled cell and ‘-’ represents a blocked cell. For example: empty grid =

| | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|---|
|0| · | · | · | - | · | · | · |
|1| · | - | · | · | - | · | - |
|2| · | · | · | - | · | · | · |
|3| · | - | - | - | · | - | · |
|4| · | · | · | · | · | - | · |
|5| - | · | - | · | - | - | · |
|6| - | · | - | · | · | · | - |

We have attached the results for your reference, in summary, there is no significant difference in performance between the two formats.

Reference:

[1] On the measure of intelligence. François Chollet. Arxiv 2019.

Q3: I am at a loss for how to interpret cases where WCR is significantly higher than ICR

A3: Thank you for the question. We would like to clarify the distinction between WCR and ICR:

• WCR measures the fraction of words in the puzzle that are predicted exactly correctly, comparing each word to the ground truth.
• ICR, on the other hand, measures whether the model’s own predictions are internally consistent at crossing points---i.e., whether the same letter appears at the intersecting cell from both the across and down words.

These two metrics capture complementary aspects of model performance. For example, consider a case where two words intersect at one letter:

• The ground truth answers are:
  • Across: ABCD
  • Down: BBEF (with the intersection at the second letter “B”)
• A model might predict:
  • Across: ABCD (correct)
  • Down: CCEF (incorrect)

In this case:

• WCR = 0.5 (one correct word out of two)
• ICR = 0.0 (because the intersection is inconsistent: ‘B’ vs ‘C’)

Such discrepancies often occur with non-reasoning models that treat crossword solving as independent QA tasks, without enforcing structural constraints across answers. This is why we include both metrics: WCR reflects answer accuracy, while ICR diagnoses whether a model reasons holistically over the grid structure. We will make this distinction clearer in the final version.

Thank you very much for your thoughtful comment and for recognizing our work. We address your concerns in detail below.

Q1: That the results will be too specific to crosswords and not really tell us much about LLMs ability to employ constraint-based reasoning

A1: Thank you for your concern about generalizability. Crosswords serve as a canonical constraint-satisfaction problem, and we leverage them as a medium to probe LLMs and LVLMs’ ability to perform constraint-based reasoning. We further show that our framework can generate puzzles drawn from various sources (and even adapt existing benchmarks). This diversity ensures that models are not overfitting to a single corpus or puzzle format, but instead must handle diverse constraint patterns. Thus, although we focus on crosswords, the benchmark provides a meaningful proxy for broader constraint-satisfaction capabilities.

Q2: The word-level results are really just an odd form of Q/A and results on this benchmark don't tell us much more than what we already know with more standard QA benchmarks

A2: Thank you for the comment. We agree that at a surface level, the task of answering individual crossword clues may resemble traditional question answering (QA). In fact, this is the behavior we often observe in non-reasoning models, which tend to treat each clue independently---ignoring the structural dependencies imposed by the grid.

This is precisely why we introduce complementary metrics, such as ICR, which measures whether the model’s predictions are mutually consistent at crossing points---something not captured in standard QA. These metrics allow us to move beyond answer correctness and probe whether models can reason holistically under structural constraints.

Q3: Another minor criticism is the disconnect in the evaluation with earlier work involving actual crosswords.

A3: Thank you for the feedback. Prior work such as LR2Bench [1] has directly used human-created puzzles (e.g., from LA Times and Vulture), and reported model performance using S-Acc, a metric conceptually similar to our Word Coverage Rate (WCR) Compared to human-created puzzles. As shown in their Appendix (Table 6), our generated 7x7 English puzzles are more difficult than 5x5 human puzzles, which more closely mirror the complexity of a standard NYT mini crossword. While we do not include human-created puzzles in this version, our generation process is designed to preserve key structural properties---such as clue distribution, word length, and blocked cell ratio---as reported in Table 1.

We will include the discussion and make it clearer in the final version.

Thank you very much for your valuable comment. We will address your concerns in detail below:

Q1: The novelty and scope of the benchmark may be limited

A1: Thank you for your feedback. We believe that the novelty and importance of the benchmark is supported by two observations: (1) since small crosswords are puzzles that people can do fairly easily, but LVLMs struggle with, this suggests a weakness in current LVLMs and (2) the reasoning required in our benchmark is quite different from that in most other multimodal reasoning benchmarks. We expand on point (2) below.

Our benchmark adopts a crossword-style formulation that requires multi-step, multimodal constraint satisfaction. Unlike many existing multimodal reasoning datasets---often framed as visual question-answering (VQA) tasks [1, 2, 3, 4]---our benchmark combines clues with a visual grid, where QA pairs intersect and mutually constrain each other. Solving one entry often necessitates iterative refinement of other answers, introducing a structured reasoning loop absent from typical VQA formulations. Compared to structured reasoning benchmarks for constraint satisfaction problems [5, 6], which are typically text-only, our task supports both multimodal input generation and vision-language evaluation, enabling a more holistic test of LVLMs.

Moreover, the underlying reasoning paradigm generalizes to real-world scenarios. For instance, embodied agents like warehouse robots or autonomous vehicles must coordinate between visual observations and textual instructions under structural constraints.

Reference:

[1] ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. ACL Findings 2022

[2] DocVQA: A Dataset for VQA on Document Images. WACV 2021

[3] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. ICLR 2024.

[4] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024 Oral

[5] Lr2bench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems. Arxiv 2025.

[6] ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning. ICML 2025.

Q2: The findings offer limited new insights or discussion on deeper failure modes

A2: Thank you for your comment. In our study, we observe:

• OCR orientation failures: LVLMs often mis-extract vertical entries, indicating weak spatial grounding or a bias towards horizontal layouts
• Ineffective self-reflection: Adding self-reflection mechanism yields no noticeable gains, showing that prompting the model (both reasoning and non-reasoning ones) to critique its own reasoning is insufficient for solving crossword puzzles.
• Effect of crossing-letter constraints: When a word has more intersections, reasoning LLMs exhibit higher accuracy, whereas non-reasoning LLMs do not benefit—suggesting that non-reasoning LLMs struggle to leverage these structural constraints.

We will also include some case studies in the final version.

Q3: The paper does not sufficiently explore different training or inference methods

A3: Thank you for your comment. Our primary goal is to introduce a challenging and diverse evaluation suite for LLMs and LVLMs, rather than explore novel learning methods. That said, our benchmark is well-suited to support future training and inference research. In particular, crossword puzzles offer a valuable property: they have verifiable solutions, similar to structured tasks like Sudoku [1] and 2048 [2]. This makes them an excellent candidate for reinforcement learning with verifiable rewards, as we discussed in Sec 5.4 of our paper. Moreover, our controllable generation framework allows for diverse puzzle generation.

Additionally, our interactive mode does represent a novel inference scheme. While the results are preliminary, this provides a good foundation for future work on agentic evaluation, where models make decisions over multiple steps and can receive feedback via function calls to the puzzle generator.

We also explore a self-reflection prompting strategy in Section 5.2. While this particular method did not lead to significant improvements---either for reasoning LLMs or non-reasoning LVLMs---it demonstrates how our benchmark can be used to evaluate the effectiveness of various inference-time scaling strategies.

Reference:

[1] Hrishbh Dalal. Agent 2048: Forging Strategic Gameplay in an AI Through Data, Rewards, and RL. Blog post, March 2024.

[2] Hrishbh Dalal. Teaching Language Models to Solve Sudoku Through Reinforcement Learning. Blog post, March 2025.

Thank you so much for your dedicated review of our paper. We recognize the significant time and effort involved in your review, and we greatly appreciate it. With only 3 days remaining before the conclusion of the discussion phase, we wish to extend a respectful request for your feedback about our responses. If there are any questions or unresolved issues, we are eager to provide further clarification or make necessary revisions. Thank you!

Thank you very much for your valuable comment. We will address your concerns in detail below:

Q1: The vision models are not given the text input also which is a big threat to validity for the hypothesis (vision models are worse). It would be more interesting to see if the vision models are given additional input and how that helps.

A1: Thank you for raising this concern regarding the input modality. To address this, in the submission we conduct experiments where LVLMs are provided with different input formats. Specifically, Table 2 of our main paper includes results from several models (GPT-4o-2024-11-20, Claude-3-7-Sonnet, Gemini-2.0-Flash) under both text-only and image-only input settings. Across these models, performance is consistently lower when provided only with image inputs compared to equivalent textual inputs.

Additionally, in Appendix Section A.6, we present an experiment (please see Fig. 9) comparing two input formats for LVLMs:

• one where the entire puzzle (clues and grid) is rendered as a single image and fed to the model
• another where the grid is given as an image, but the clues are provided in text.

The results show no substantial difference in Word Coverage Rate (WCR) between these two configurations. This suggests that the bottleneck is not merely in the input format, but rather in LVLMs' overall ability to reason over visual input---even when augmented with textural clues.

Q2: The 1 shot setup seems very restrictive and not sure if the evaluation in 1 shot is fair.

A2: Thank you for your comment. Our main evaluation is conducted in a zero-shot setting, as explained in Section 4.1 and Figures 11 and 12 in the Appendix. Models are not given any in-context examples or demonstrations during inference. All models are evaluated under the same zero-shot setting to ensure fairness and comparability.

We agree that solving an entire crossword puzzle from scratch is a challenging task, even for the small puzzles, and performance could be higher in a k-shot setting. As an alternative to doing this, we did explore ways to reduce task complexity. In particular, leveraging our controllable generation framework, we are able to systematically adjust the difficulty of the task by controlling the pre-fill ratio.

Specifically, we consider a pre-fill ratio of 0.5, meaning 50% of the grid cells are revealed to the model, while ensuring that no complete word is revealed.

We have pasted the results below for your reference. Models are evaluated under two settings (0% and 50% pre-fill) on the English 7x7 puzzles. In both cases, we report WCR.

As shown in the table, models consistently achieve better performance when provided with more context via pre-filled cells. We will include these results in the final version.

Q3: Was any qualitative study done on the crosswords to validate the quality of the data?

A3: Thank you for your question. In lieu of a formal qualitative study, we will include several examples of generated puzzles in the appendix of the final version.

In Table 1 of the main paper, we did provide detailed statistics of the puzzles across different subjects and grid sizes, including:

• Average number of words per puzzle.
• Average word length.
• Average number of blocked cells.

These metrics can serve as useful proxies for the structural complexity of the puzzles. Additionally, to ensure the diversity and richness of the dataset, we explicitly enforce that no clue is repeated within the same subject across different grid sizes, or even across subjects.

Firs of all I would like to thank the authors for their response. It helps me understand the work a bit better and clarifies my questions.

Some thoughts:

• The text input study from the appendix should be moved to the main paper. Comparing different input modalities in the same table can be confusing and a threat to validity.
• The 0-shot setting is restrictive. The prefill results are more indicative but the k-shot analysis should be expanded.
• The qualitative analysis will help.
• The interactive mode error handling should be discussed more.