What Makes a Maze Look Like a Maze?

We thank you for the constructive feedback!

Q. Quality of generated schemas.

A. Thank you for the suggestion! For evaluating the quality of generated schema, we added a human study using Prolific, with 20 participants. Each participant was asked 3 questions about each schema. The questions are as follow for an example concept maze:

• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How well do these core components represent this concept on a scale of 1-7?
• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How many components out of the ones listed accurately represent part of this concept?
• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How many more components, if any, would you add to accurately represent this concept?

We collect 3 responses for each of 12 concepts with 20 responses each, and yield a total of 720 human annotated answers.

We see here that (a) participants rated schemas as reasonably representative, (b) 86.3% of the components were considered accurate by participants, and (c) participants suggested adding ~1.6 components on average per schema. The concept that ranks highest for the first question is the concept cell with a rating of 6.15, and lowest as the concept negotiating with 4.50. The concept solar system had the highest number of components to keep at 95%, while the concept negotiating is the lowest again at 61.7%. The concept tic-tac-toe was the most complete, with an average of 1 concept that participants might add, while the concept solar system had an average of 2.65. Though there is still room for improvement, the schemas generated by the LLM are largely accurate and aligned with human expectations. Importantly, none of the schemas were judged as fully invalid, showing the robustness of DSG’s approach. We added a detailed discussion of this human study in Appendix Section C.3.

To address cases where the generated schema introduces additional components not present in an image, DSG relies on the VLM to correctly handle absence rather than hallucinate. For instance, given image 9 in the concept setting up table for two, the grounded DSG schema correctly identifies the absence of two chairs in the scene: {table: dining table, two chairs: absent, table setting: intimate dinner setup}. The VLM then incorporates this absence into its reasoning through the final prompt: “Imagine that the image represents setting up table for two, and the table is a dining table, and the two chairs are absent, and the table setting is an intimate dinner setup. What is the configuration of the seats?” Here, DSG answers correctly with "across".

To conclude, we see that the generated schemas are generally correct, and for cases where specific schema components do not exist in the given image instantiation, we rely on VLMs to detect absence in the image instead of hallucinating.

We thank you for the constructive feedback!

Q. Experiment results for schemas with different numbers of components.

A. Thank you for bringing up this point! We agree that exploring schemas of different levels of complexity is important. Currently, our “knob” for the complexity of the generated schema is the number of components in the example prompt. Our prompt is simple, as we note potential failures of current VLMs in handling such detailed concepts, especially more spatially-focused ones (See discussion in Section 6). We hope that as LLMs become less dependent on the form of the initial prompt, they can decide the schema without being influenced by the bias in the example, and become more accurate in answering more detailed components.

To explore the impact of schema complexity, we conducted an additional experiment varying the numbers of components in the example prompt: 3, 5, 7, and 9. The example prompts (all based on the same abstract concept academia expanded to different levels of detail) can be found in the added Appendix Section A.5.

The results demonstrate that accuracy decreases as the number of components in the example prompt increases, likely due to current VLMs struggling with grounding more specific components. We note that all DSG variants still outperform the GPT-4o baseline. Interestingly, our findings also highlight a tendency for LLMs to generate shorter schemas, even when exposed to more detailed example prompts. Here we show the number components in the prompts and the mean number of components in the resulting schemas, averaged over all abstract concepts.

• Prompt # 3: resulting average 3.00
• Prompt # 5: resulting average 4.66
• Prompt # 7: resulting average 5.75
• Prompt # 9: resulting average 7.00

Notably, although the number of components in the example prompt does affect the average number of components in resulting schemas, this effect is less prominent and the resulting average plateaus. This phenomenon shows that schemas may potentially converge to shorter lengths. We agree that indeed abstract concepts should be deeper and have more components, and vice versa. We note though current VLM limitations constrain schema complexity, as these capabilities improve, DSG’s performance is expected to scale, enabling us to adopt more detailed schemas without sacrificing effectiveness. Thanks to your feedback, we have included a more in-depth discussion of this tradeoff and added the ablation study results to Appendix Section A.5.

We thank you for the constructive feedback!

Q. Complexity of schemas.

A. Thank you for bringing up this point! We agree that exploring schemas of different levels of complexity is important. Currently, our “knob” for the complexity of the generated schema is the number of components in the example prompt. Our prompt is simple, as we note potential failures of current VLMs in handling such detailed concepts, especially more spatially-focused ones (See discussion in Section 6). We hope that as LLMs become less dependent on the form of the initial prompt, they can decide the schema without being influenced by the bias in the example.

To explore the impact of schema complexity, we conducted an additional experiment varying the numbers of components in the example prompt: 3, 5, 7, and 9. The example prompts (all based on the same abstract concept academia expanded to different levels of detail) can be found in the added Appendix Section A.5.

The results demonstrate that accuracy decreases as the number of components in the example prompt increases, likely due to current VLMs struggling with grounding more specific components. We note that all DSG variants still outperform the GPT-4o baseline. Interestingly, our findings also highlight a tendency for LLMs to generate shorter schemas, even when exposed to more detailed example prompts. Here we show the number components in the prompts and the mean number of components in the resulting schemas, averaged over all abstract concepts.

• Prompt # 3: resulting average 3.00
• Prompt # 5: resulting average 4.66
• Prompt # 7: resulting average 5.75
• Prompt # 9: resulting average 7.00

Notably, although the number of components in the example prompt does affect the average number of components in resulting schemas, this effect is less prominent and the resulting average plateaus. This phenomenon shows that schemas may potentially converge to shorter lengths.

We note that current VLM limitations constrain schema complexity. However, as these capabilities improve, DSG’s performance is expected to scale, enabling us to adopt more detailed schemas without sacrificing effectiveness. Thanks to your feedback, we have included a more in-depth discussion of this tradeoff and added the ablation study results to Appendix Section A.5.

Q. Free-response evaluation with Kamalloo’s metric.

A. Thanks for pointing us to this reference! Below, we report free-response results by incorporating one of Kamalloo’s proposed strategies for evaluating free-response answers. In particular, we use zero-shot evaluation via LLM prompting, which includes the ground truth and predicted answer in the prompt, similar to the semantic similarity mechanism, to verify the correctness of the prediction. We use GPT-4 as the base LLM, and compare GPT-4o with DSG.

We see that DSG outperforms GPT-4o overall, with particularly strong results on counting and open-ended questions, and shows stronger signal than with BERTScores. However, performance on binary questions is lower, which we hypothesize is due to some of the components being potentially misleading or ambiguous in binary contexts. We added these evaluation results in Appendix Section A.2.

We thank you for the constructive feedback!

Q. Difference between DSG and chain-of-thought.

A. The key idea behind DSG precisely lies in schema decomposition (e.g., in the form of extracting components: layout, walls, and entry and exit) to guide the VLM to interpret the scene. DSG introduces an explicit guidance to the VLM system, by first generating a decomposition of the target concept, parsing the image, then answering the question. We note that the specific method of processing these questions—whether in a single query or across multiple queries—is not fundamentally important. While in theory, these steps could be merged into a single CoT-style prompt, our experiments show that multi-round execution with explicit grounding steps significantly improves performance in practice.

For example, in Table 3 of the main text, the first row of the ablation shows results when the VLM is given the hierarchical schema for reasoning in a single query, without using the explicit grounding steps. In addition, we ran an experiment based on your suggested prompt, listing out each component and asking the model to think step-by-step: “Imagine that the image represents a maze. Think step-by-step about recognizing the layout, walls, entry exit one-by-one. What is the player in this maze?”. We report results below, and compare (1) GPT-4o, (2) the suggested single-round CoT baseline with schema components, (3) the single-round CoT baseline with the full hierarchical schema, and (4) our full multi-round DSG framework. All results are averaged over 5 runs.

We see that DSG’s multi-round execution outperforms single-round, CoT-style baselines, likely due to its explicit guidance and grounding.

We agree that DSG’s schema decomposition is especially helpful for smaller models, which may lack the reasoning capabilities of larger models like GPT-4o. Thanks to your feedback, we’ve clarified our ablation setup in the updated Section 5 of the main text, and added these experiment results in Appendix Section A.3.

[1] Wei, Jason, et al. "Chain-of-thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.

Thanks for the authors' response, I have increased my score accordingly.

Q. Why grounded schemas are helpful.

A. We hypothesize that the grounded schemas provide structured, holistic context to the VLM (by way of assigning roles to entities in the image), enabling the model to organize information more effectively and improve its performance. This grounded context helps downstream reasoning, even when the query does not explicitly involve a schema component. Importantly, we note the generality of DSG, which does not rely on knowing the query a priori.

Though difficult to analyze the inner workings of the VLM, to address your example, we conducted an analysis of the "What is the player in the maze?" question. Here, the grounded schema explicitly includes components "layout," "walls," and "entry-exit." When we asked the VLM (GPT-4o) to explain its reasoning while using the grounded schema, it provided the following:

"Reasoning: The layout provides structure, which helps identify where pathways and spaces are within the maze. The walls made of branches define boundaries and paths, making it possible to understand where movement might occur within the maze. The entry and exit points indicate potential traversal through the maze. Since no identifiable player (grass, moss, stick, or anything distinct from the environment) is visible within the maze structure itself, the answer is: none."

This reasoning supports our hypothesis: the grounded schema assigns semantic roles to image components, which helps the VLM disambiguate entities in the scene and improve downstream reasoning. In this case, the schema enables the VLM to interpret the absence of a distinct "player" by contextualizing the maze structure. We add this discussion in Appendix Section A.8.

Q. Dataset link.

A. Thanks for your careful feedback regarding the dataset download URL in the supplementary material! To address the concern and avoid any confusion, we have updated the link. We would also like to clarify the following points:

• The URL is embedded for direct downloading, and readers are not expected to see the URL itself. The URL does not appear anywhere in the text of the main paper or the supplementary material.
• The URL does not contain any personal identifiers or link to specific individuals, nor does it reveal any names.
• Similar to how researchers might include images of a university in their work, the inclusion of a university name in a dataset URL does not necessarily identify any author or institution. Thanks!

We thank you for all the comments, which have greatly improved our paper.

Q. Language-only baseline and impact of providing grounded schemas.

A. Thank you for the suggestion! We added two additional language baselines based on your suggestion. The first is a language-only baseline, where GPT-4o only has access to the question itself without input image or schema. The second is a language-with-schema baseline, where GPT-4o has access to the question and the DSG grounded schema (the schema is extracted from the image and provides context), but the final query itself does not have access to the image. We report results averaged over 5 runs below.

Notably, the language-only baseline performs worse than all open-source and closed-source VLMs, showing that the visual input is essential for achieving higher accuracy on the benchmark. We note that the language-only baseline outperforms integrated LLMs with API baselines, as these methods tend to fail in execution.

With DSG’s grounded schema, the language model performs significantly better, highlighting the importance of the holistic context provided by the grounded schema. We see that in open-ended questions, the grounded schema significantly improves performance of the language-only baseline. We hypothesize that this is because the schema helps rule out implausible answer choices by leveraging the grounded components. We add these experiment results and discussion in Appendix Section A.4.

Q. Verification of generated schemas.

A. Thank you for raising this point! To evaluate the correctness and quality of the generated schemas, we added a human study using Prolific, with 20 participants. Each participant was asked 3 questions about each schema. The questions are as follow for an example concept maze:

• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How well do these core components represent this concept on a scale of 1-7?
• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How many components out of the ones listed accurately represent part of this concept?
• Here you are given a concept, maze, and the core components underlying it: layout, walls, entry−exit. How many more components, if any, would you add to accurately represent this concept?

We collect 3 responses for each of 12 concepts with 20 responses each, and yield a total of 720 human annotated answers.

We see here that (a) participants rated schemas as reasonably representative, (b) 86.3% of the components were considered accurate by participants, and (c) participants suggested adding ~1.6 components on average per schema. The concept that ranks highest for the first question is the concept cell with a rating of 6.15, and lowest as the concept negotiating with 4.50. The concept solar system had the highest number of components to keep at 95%, while the concept negotiating is the lowest again at 61.7%. The concept tic-tac-toe was the most complete, with an average of 1 concept that participants might add, while the concept solar system had an average of 2.65. Though there is still room for improvement, the schemas generated by the LLM are largely accurate and aligned with human expectations. Importantly, none of the schemas were judged as fully invalid, showing the robustness of DSG’s approach. We added a detailed discussion of this human study in Appendix Section C.3.

We thank you for the constructive feedback!

Q. Schema detailedness and complexity.

A. Thank you for bringing up this point! We agree that there’s a tradeoff between encouraging more detailed components in a schema and avoiding both (a) components that are less broadly applicable across all image instantiations, and (b) as mentioned by the reviewer, potential failures of current VLMs in handling such detailed concepts, especially more spatially-focused ones. We note that as capabilities of VLMs increase, DSG’s performance will scale accordingly, and we can increase the detail of the schema without making this tradeoff.

In addition, currently our “knob” for the complexity of the generated schema is the number of components in the example prompt. As LLMs become less dependent on the form of the initial prompt, they can decide the schema without being influenced by the bias in the example.

To explore the impact of schema complexity and better understand the capabilities of current VLMs, we conducted an additional experiment varying the numbers of components in the example prompt: 3, 5, 7, and 9. The example prompts (all based on the same abstract concept academia expanded to different levels of detail) can be found in the added Appendix Section A.5.

The results demonstrate that accuracy decreases as the number of components in the example prompt increases, likely due to current VLMs struggling with grounding more specific components. We note that all DSG variants still outperform the GPT-4o baseline. Interestingly, our findings also highlight a tendency for LLMs to generate shorter schemas, even when exposed to more detailed example prompts. Here we show the number components in the prompts and the mean number of components in the resulting schemas, averaged over all abstract concepts.

• Prompt # 3: resulting average 3.00
• Prompt # 5: resulting average 4.66
• Prompt # 7: resulting average 5.75
• Prompt # 9: resulting average 7.00

Notably, although the number of components in the example prompt does affect the average number of components in resulting schemas, this effect is less prominent and the resulting average plateaus. This phenomenon shows that schemas may potentially converge to shorter lengths.

We note that current VLM limitations constrain schema complexity. However, as these capabilities improve, DSG’s performance is expected to scale, enabling us to adopt more detailed schemas without sacrificing effectiveness. Thanks to your feedback, we have included a more in-depth discussion of this tradeoff and added the ablation study results to Appendix Section A.5.

I thank authors for clarifying my queries. I am improving my rating.