ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints

We sincerely appreciate your thorough review and valuable suggestions. We have provided detailed responses to all of your comments and questions. We hope that our clarifications will help you to better understand the specifics of our work and findings.

Regarding Dataset Scale and Robustness

Regarding the statistical significance of the performance differences between models, we report in Table 1 the average of three independent runs, and each model's performance score is appended with a "±" value. Regarding the data regularity, thank you for your valuable suggestions. This is indeed a focus of our future work.

We have attempted semi-automated methods but found that the usability of CP (Crease Pattern) generated by the model is extremely low when reviewed by humans (success rate less than 1%). The cost of using the model to generate and then having humans verify is also very high. We believe that the current MLLM's capabilities are not yet sufficient to handle high-quality semi-automated generation tasks, which will require continuous iteration and optimization of the model version to be realized in the future.

Therefore, our current strategy is to collaborate with established origami communities and databases. We have reached cooperation intentions with creators from several resource websites, including Global Origami Community and Origami Database. We plan to officially cooperate with them to bring in a larger volume of origami model data that has been verified and organized by humans. This approach not only ensures the quality and accuracy of the data but also allows us to expand the dataset more efficiently on the basis of the existing 350 examples.

Looking forward, we believe these two methods are complementary. With the improvement of MLLM capabilities, we will re-examine and deploy semi-automated generation processes, which will allow our community experts to participate in the "human-in-the-loop" process, screening and optimizing the procedurally generated models, thereby balancing efficiency and quality.

Input to Code Generation Task

We agree with your point of view. Providing the model with a "Compiled Flat Pattern" compared to a "Folded Shape Image" does indeed reduce the "real-world" inverse engineering difficulty of the task. However, our goal is to provide a clearer final result, because a "Folded Shape Image" is usually a photo or a 3D render, which itself carries a large amount of perceptual uncertainty, such as viewpoint, lighting, shadows, material texture, and non-rigid deformation. Requiring a model to directly generate precise CP code from such an image would mix the core spatial reasoning challenges with complex 3D visual perception and reconstruction challenges.

We completely agree with your view that generating CP code from a "Folded Shape Image" (possibly with natural language instructions) is a more challenging and more intuitive inverse engineering task. We believe this represents a more difficult and more valuable future research direction.

Our ORIGAMISPACE benchmark can be seen as a key intermediate step in achieving this grand goal. It pioneers the systematic evaluation and improvement of the ability of Multi-modal Large Language Models (MLLM) in core geometric constraints and multi-step spatial reasoning in a well-controlled, perception-free environment. Once a model demonstrates sufficient capability on this foundational task, the next step is to challenge more open and realistic ultimate tasks.

Generalization of RL Agent

Thank you for pointing this out. Regarding generalization, our experimental setup itself is a kind of test. The RL agent is trained on a dataset containing 471 examples. The final results (Table 2) are evaluated on the ORIGAMISPACE benchmark test set, which contains 350 examples. All the origami designs in the test set are completely new to the RL agent during the training process.

Regarding the strategy, in the early stages of training, the model's compilation success rate (CPR) is low. The main feedback comes from the negative penalty for compilation failures. During the middle stage of training, the compilation-related rewards increase rapidly, especially for CP Syntax Error (CSE) and Geometrically Impossible Fold (GIF). We believe this is because these two types of constraints are relatively simple, allowing the model to learn to pass compilation relatively well. In the later stages, the model's CPR gradually stabilizes, but the code quality score, especially the topological similarity (TSS), shows a more noticeable increase, while the geometric similarity (GS) does not change much. This may be because the feedback for the latter is not well-designed, and the required capabilities are more abstract. Furthermore, constrained by the penalty for step length, the number of interaction rounds required to complete the task will decrease. Overall, the model will first learn simpler strategies, such as Euler's formula for planar graphs and Maekawa's theorem, and then try to learn more complex strategies.

Accessibility of Compilers and Environments

Yes, we will publicly release all data, environments (compilers), training and evaluation code, and models in the short term.

Thank you again for your recognition of our work, as well as for your constructive comments and suggestions!

Dear Reviewer,

Could you please check if the authors’ rebuttal adequately addresses your concerns? If so, kindly acknowledge the rebuttal and provide any additional comments. If not, it would be greatly appreciated it if you could engage in a discussion with the authors. Your input at this stage is essential to the review process. Thank you very much for your time and effort!

AC

Dear Reviewer,

According to this year's NeurIPS review policy, "Reviewers must participate in discussions with authors before submitting a Mandatory Acknowledgement", could you please provide additional comments discussing whether the rebuttal addresses your concerns?

Thank you.

AC

We sincerely appreciate your thorough review and valuable suggestions. We have provided detailed responses to all of your comments and questions. We hope that our clarifications will help you to better understand the specifics of our work and findings.

Given that OpenAI's recent multimodal model (o3), which has reasoning capabilities, has performed exceptionally well in similar tasks, it is recommended that the authors include this model here. Even a qualitative comparison would be helpful.

Thank you for pointing this out. We have added the experimental results of o3 on different spatial reasoning tasks, as shown in the tables below:

As can be seen, o3 performs very well on ORIGAMISPACE, comparable to gemini-2.5-pro. We will add these results in the next version.

The function of each crease type (mountain fold, valley fold, etc.) and the complete syntax rules for writing valid code are not clearly explained. This makes the CP code prediction task difficult to understand. The paper would be more helpful if it could break down the semantics of CP code in more detail or provide a specific example.

Certainly. Below is an introduction to the syntax rules for CP code:

A complete CP code consists of four parts: vertex coordinates (vertices_coords), edge-to-vertex relationships (edges_vertices), edge type assignments (edges_assignment), and face-to-vertex relationships (faces_vertices).

• vertices_coords: This defines the positions of all key points on the paper, represented by the 2D coordinates of each point.
• edges_vertices: This defines the line segments, including the paper's boundaries and the internal creases. Each edge is defined by connecting the indices of two vertices.
• edges_assignment: This is the most critical step, representing the property of each "edge." This includes "B" (Boundary), "M" (Mountain fold), "V" (Valley fold), and "F" (Flat fold).
• faces_vertices: This defines all the 2D "faces" that are partitioned by the edges and creases. Each face is defined by the sequence of vertices that make up its contour.

In the next version of the paper, we will provide a more detailed explanation of the CP code, along with an analysis of specific examples.

Thank you again for your recognition of our work, as well as for your constructive comments and suggestions!

Dear Reviewer,

Could you please check if the authors’ rebuttal adequately addresses your concerns? If so, kindly acknowledge the rebuttal and provide any additional comments. If not, it would be greatly appreciated it if you could engage in a discussion with the authors. Your input at this stage is essential to the review process. Thank you very much for your time and effort!

AC

We sincerely appreciate your thorough review and valuable suggestions. We have provided detailed responses to all of your comments and questions. We hope that our clarifications will help you to better understand the specifics of our work and findings.

The dataset contains only 350 samples, which may limit statistical robustness and hinder the training of larger models.

We believe that for this task, the quality of the data is more important than the quantity. The process of origami is extremely rigorous, especially since it requires strict compilation via CP code, where even a minor error can lead to a vastly different result. We previously attempted a semi-automated approach with manual filtering, but the success rate was very low.

Therefore, we have partnered with established origami communities and databases. We plan to access their larger-scale, manually verified, and curated origami model data through official collaborations. This approach not only ensures data quality and accuracy but also allows us to expand our dataset more efficiently beyond the current 350 instances.

Regarding Mathematical Constraints

Certainly, we will further elaborate on the impact and function of mathematical constraints to hopefully resolve your questions.

First, you can think of mathematical constraints in two forms. The first is "hard constraints," as described in Section 3.2 of the paper. These include CP Code Syntax Error (CSE), Geometrically Impossible Fold (GIF), Paper Self-Intersection (PSI), and Ambiguous Fold State (AFS). These four hard constraints must be satisfied; otherwise, the compilation will fail. The second is "soft constraints," which refers to the Constraint Satisfaction mentioned in Section 4.4. At this stage, when the CP Code is compilable, the constraints evaluate the ability to achieve more detailed and self-consistent spatial intelligence. This primarily includes two parts: the verification of specific advanced origami constraints and local flat-foldability conditions, as described in Appendices D.11, D.12, and D.13. The former includes Taco-Taco (TT), Taco-Tortilla (TTo), and Transitivity constraints. During evaluation, the system extracts these advanced constraints from both the generated model and the reference model and then compares the similarity of the two sets. The similarity is calculated as a weighted sum of the overlap of the constraint sets (Jaccard similarity) and the similarity of the constraint counts. The latter assesses a more fine-grained folding state through local vertex metrics.

To make this easier to understand, here are two examples:

• Paper Self-Intersection (PSI) - Hard Constraint: This violates the physical principle that "different parts of an object cannot occupy the same space simultaneously." When the compiler deduces the relative positions and stacking order of the paper's parts, it will report an error if it finds any regions that would overlap or penetrate.
• Taco-Tortilla (TTo) - Soft Constraint: "Taco" refers to a folded flap of paper, while "Tortilla" represents a flat, unfolded surface. The Taco-Tortilla constraint is used to specify this "one-folded, one-flat" stacking order. This decision is critical to the final model's appearance and function. For instance, it might determine whether a pocket is on the outside or inside of a garment, or whether a part of a model is convex or concave. A model that correctly handles TTo constraints demonstrates that it understands more refined spatial operations like "hiding" and "covering."

Now, to answer your two questions directly:

How exactly do the mathematical constraints work?

You can understand it as the compiler performing complex mathematical derivations to calculate whether the CP code can conform to various constraint states and then returning relevant information. This provides a clear, computable feedback signal and evaluation result.

What is the impact of the mathematical constraints? Is it possible to satisfy all of them at each step?

• The impact of hard constraints is that a CP diagram that fails to meet them cannot be folded at all.
• The impact of soft constraints is that a CP diagram that fails to meet them will have an incorrect folding state in some spatial areas compared to the reference.

The role of hard constraints is to assess whether the current state is foldable. The role of soft constraints is to provide a partial reward signal after foldability is confirmed. Therefore, in a multi-step folding process, it is not necessary to satisfy all constraints at every step, as adjustments can be made based on the compiler's feedback. The significance of the constraints is twofold: 1. They provide our evaluation with clear standards and environmental feedback. 2. They provide a simulation that is perfectly consistent with real-world physics.

Reinforcement Learning (RL) Setup

Of course. We will provide more details and plan to open-source all our data, environment, training code, and models in the short term.

Task Structure

• The agent (MLLM) engages in iterative, multi-turn interactions with the environment (the compiler). The compiler returns either a text-based success message or specific error feedback.
• The model first performs planning to generate initial CP code, then reasons through interaction with the environment and decides on the next action, such as adding or deleting a crease.
• This "generate-feedback-modify" loop continues for a maximum of 10 interaction rounds, as set in our experiments.

Training Algorithm & Data

• We employed the TRICO algorithm, an efficient multi-turn reinforcement learning algorithm based on PPO, designed for MLLMs.
• The training data consisted of 471 independent origami datasets.

Training Hyperparameters

• Appendix E provides detailed training parameters, including: training for 10.2 hours on 16 H100 GPUs, with Actor and Critic learning rates of 1×10⁻⁶ and 1×10⁻⁵, respectively, among other relevant parameters.

Performance Change

• In the early stages of training, the model's compilation success rate (CPR) is low, with the primary feedback being a fixed negative penalty for compilation failures.
• In the middle phase, compilation-related rewards increase rapidly, especially for avoiding CP Code Syntax Error (CSE) and Geometrically Impossible Fold (GIF). We believe this is because these two constraints are relatively simple, and the model becomes proficient at passing the compilation stage.
• In the later stages, the model's CPR gradually stabilizes, while code quality scores show more significant improvement, particularly Topological Similarity (TSS). The change in Geometric Similarity (GS) is less stable, possibly because its feedback signal is less refined and the required learning is more abstract. Additionally, influenced by the step penalty, the average number of interaction rounds required to complete a task slightly decreases.
• Overall, the model first learns simpler constraints, such as Euler's formula for planar graphs and Maekawa's theorem, before attempting to learn more complex strategies.

We will open-source all our data, environments, training code, models, and complete training details in the short term.

Thank you again for your recognition of our work, as well as for your constructive comments and suggestions!

Dear Reviewer,

Could you please check if the authors’ rebuttal adequately addresses your concerns? If so, kindly acknowledge the rebuttal and provide any additional comments. If not, it would be greatly appreciated it if you could engage in a discussion with the authors. Your input at this stage is essential to the review process. Thank you very much for your time and effort!

AC

We sincerely appreciate your thorough review and valuable suggestions. We have provided detailed responses to all of your comments and questions. We hope that our clarifications will help you to better understand the specifics of our work and findings.

The use of origami is highly innovative. Could you elaborate on how you see the specific skills tested here (e.g., understanding crease patterns, mathematical constraints) transferring to other complex spatial reasoning domains like robotic manipulation or architectural design?

We believe that the core competency tested by ORIGAMISPACE—multi-step spatial reasoning under strict constraints—is universal across multiple domains. This ability is transferable because it simulates common spatial challenges. For example:

• In robotic manipulation, the sequential folding and mathematical constraints of origami correspond to the multi-step manipulation planning for deformable objects and their physical limitations, respectively.
• In architectural design, predicting a 3D model from a Crease Pattern (CP) is analogous to conceptualizing a building from a 2D blueprint, while the code generation task shares similarities with the parametric design logic in CAD.

To transfer this capability from origami to other domains, we believe the following technical aspects require attention:

• Mapping of Representations: Build a bridge to map problems from the new domain (e.g., point clouds for robotics, CAD files for architecture) into code or symbolic sequences that the model can better understand.
• Generalization of Constraints: Enable the model to understand and apply equivalent physical or design rules in the new domain (e.g., material non-stretchability, structural load-bearing requirements for buildings). This may involve constructing similarly strict constraints and interactive environments across different fields.
• Unified Action Space: For generative tasks like robotic manipulation, the "folding" action in origami needs to be abstracted into more universal "manipulation primitives," allowing the model to learn a unified decision-making policy in the new domain.

The manual collection and verification process for the 350 models must have been very labor-intensive. Have you considered semi-automated methods for expanding the dataset, perhaps by procedurally generating valid CPs and using the compiler to filter them?

Thank you for your valuable suggestion; this is indeed a key focus for our future work.

We did attempt semi-automated methods in the early stages but found that the usability rate of Crease Patterns (CPs) generated by the model was extremely low (less than a 1% success rate) after manual review, while the cost of model generation and human verification was prohibitive. We believe that the capabilities of current Multimodal Large Language Models (MLLMs) are not yet sufficient for high-quality, semi-automated generation tasks. This will likely become feasible only after continued iteration and optimization of model versions in the future.

Therefore, our current strategy is to collaborate with established origami communities and databases. We have reached cooperation intentions with global origami communities and the creators of several origami resource websites, such as the Origami Database. Through official partnerships, we plan to access their larger-scale, manually verified, and curated origami model data. This approach not only ensures the quality and accuracy of the data but also allows us to expand our dataset more efficiently beyond the current 350 instances.

Looking ahead, we see these two approaches as complementary. As MLLM capabilities improve, we will revisit and deploy a semi-automated generation pipeline. We can also involve our community experts in a "human-in-the-loop" process to filter and refine the procedurally generated models, thereby balancing efficiency and quality.

The reinforcement learning results are very promising. Could you provide more insight into the policy the agent learns? For example, does it learn to satisfy certain geometric constraints (like Kawasaki's theorem) first before attempting more complex folds?

In the early stages of training, the model's compilation success rate (CPR) is low, and the primary feedback is a fixed negative penalty for compilation failures. During the middle phase, rewards related to compilation increase rapidly, especially for avoiding CP Code Syntax Errors (CSE) and Geometrically Impossible Folds (GIF). We believe this is because these two constraints are relatively simple, and the model learns to pass the compilation stage more effectively. In the later stages, the model's CPR gradually stabilizes, while the code quality scores, particularly Topological Similarity (TSS), show more significant improvement. The change in Geometric Similarity (GS) is less stable, possibly because its feedback signal is less refined and the required learning is more abstract. Additionally, influenced by the step penalty, the average number of interaction rounds required to complete a task slightly decreases. Overall, similar to what you suggested, the model first learns simpler constraints, such as Euler's formula for planar graphs and Maekawa's theorem, before attempting to learn more complex strategies.

We will open-source all our data, environments, training code, models, and complete training details in the short term.

Thank you again for your recognition of our work, as well as for your constructive comments and suggestions!

Dear Reviewer,

Could you please check if the authors’ rebuttal adequately addresses your concerns? If so, kindly acknowledge the rebuttal and provide any additional comments. If not, it would be greatly appreciated it if you could engage in a discussion with the authors. Your input at this stage is essential to the review process. Thank you very much for your time and effort!

AC