Autoformulation of Mathematical Optimization Models Using LLMs

We appreciate the reviewer’s detailed and thoughtful evaluation.

[P1] Engineering vs theory

Thank you for the thoughtful comment. Our framing of optimization modeling as a three-step process (requirements gathering $\rightarrow$ mathematical model $\rightarrow$ computational model) is intended to reflect the engineering side of the modeling pipeline. We agree that in many academic OR contexts, the emphasis lies in theoretical justifications—such as proving optimality bounds, analyzing relaxations, or exploring structural properties—which are essential to advancing the science of optimization.

While theoretical work is crucial, we also believe our focus remains valuable (further discussed in [P2]). For domain scientists who are not optimization experts, this technology can help democratize modeling by lowering the barrier to entry. For experts, the autoformulator can accelerate prototyping and iteration. When theoretical validation is ultimately required, our system can automate time-consuming or error-prone steps, surface promising candidate formulations, and free up modeling experts to focus on deeper theoretical analysis.

Actions taken. We have updated the manuscript to acknowledge this theoretical dimension more clearly. Specifically, we revised lines L10–L16 R (Introduction) to highlight the role of theoretical modeling and added a discussion in L405–L420 R (Discussion) on future directions that connect autoformulation with theoretical reasoning.

[P2] Real-world utility

Practical utility. We believe that in many domains—such as logistics, healthcare, energy, and manufacturing—there exists a real and growing need to bridge the gap between domain expertise and optimization modeling expertise. Gurobi’s "State of Optimization Report 2023" [R1–R2], based on a survey of 394 commercial users, identified optimization modeling as a rapidly growing area of demand, while also highlighting a significant skills gap. Just as coding LLMs have empowered non-expert programmers to build software, autoformulators can lower the barrier to modeling, reduce translation bottlenecks, and empower domain experts to experiment with formal optimization tasks. For experienced modelers, these systems can accelerate iteration, support exploratory modeling, and free cognitive effort for higher-level design and analysis.

Emerging research & industry focus. Autoformulation is gaining traction in both research and industry. On the research side, recent systems such as Optimus and Chain-of-Experts (both appeared at ICLR 2024) highlight growing interest in the scientific and technical challenges of this space. We note that the benchmarks, tasks, and metrics we used are consistent with those employed in prior works. Industry interest further reinforces the relevance of this problem: (1) Gurobi’s modeling assistant [R3], (2) AWS’s optimization AI tools [R4], and (3) IBM’s research on AI-assisted modeling [R5] all address related aspects of the autoformulation pipeline. We view these developments as strong signals of real-world utility, and our system contributes foundational capabilities to this emerging and impactful area.

Actions taken. Thank you for raising this important point. We have added a new appendix section outlining practical and research use cases of our system.

[R1, R2] stage.gurobi.com/resources/report-state-of-mathematical-optimization-2023/, stage.gurobi.com/resources/report-state-of-mathematical-optimization-in-data-science-2023/

[R3] gurobi-ai-modeling.readthedocs.io

[R4] aws.amazon.com/cn/blogs/industries/optimization-in-the-era-of-generative-ai/

[R5] research.ibm.com/publications/enhancing-decision-making-through-the-integration-of-large-language-models-and-operations-research-optimization-bridge-talk

[P3] Highlighting novelty

We introduced a novel MCTS-based technique tailored to the structure of optimization modeling tasks, grounded in the core challenges [C1-C3], including (1) domain-specific MCTS strategy to enhance hierarchical exploration; (2) SMT-based pruning of trivial equivalences, improving search efficiency by two orders of magnitude; and (3) comparative scoring of candidate formulations, enhancing feedback for search guidance. Empirically, we show that generic, off-the-shelf approaches fail to address key challenges in evaluating formulation quality (S5.2) and eliminating trivial redundancies (Fig. 4). In contrast, our approach outperforms prior state-of-the-art methods—including LLMs specifically finetuned for autoformulation—achieving new best results on formulation correctness across multiple challenging benchmarks.

Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.

We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.

[P1] Addressing challenges

Thank you for raising this point. However, we believe this may stem from a misunderstanding. Please allow us to clarify how our method addresses all three challenges:

[P2] Evaluations

Thank you for pointing this out—we agree with your concern. Our method was originally evaluated using a pass@N metric (i.e., success if any of N iterations is correct), while baseline comparisons were reported under pass@1.

Actions taken. We have re-evaluated ORLM under pass@N by generating N independent samples to match the iteration count (Note: our method still outperforms non-ORLM baselines at N=1). We report in this table this fairer comparison (now added to the appendix), and clarified evaluation metrics in the main text. We observe that our approach maintains a clear advantage even under pass@N. This reflects a core strength of our approach: it performs structured exploration over functionally diverse formulations, which ORLM does not inherently support. We appreciate your feedback, it helped improve the rigor of our evaluation.

[P3] Selection

In our method, each MCTS iteration produces a complete formulation along with a score from our evaluation mechanism, allowing us to select the highest-scoring one if desired. Actions taken. We now report best-of-N accuracy based on score-ranked outputs in this table (i.e., selecting best-of-N formulations using greedy selection). Our selected formulations continue to outperform baselines (specifically, our mechanism selects the best formulation on >90% of problems), supporting the effectiveness of our scoring mechanism.

[P4] MCTS

Thank you for raising this point. We refer the reviewer to our response to Reviewer fWQR ([P1]) for a detailed discussion of why optimization modeling benefits from hierarchical decomposition and MCTS-based structured exploration. Empirical evidence. We include two additional baselines: (1) a Tree-of-Thought using a different tree search strategy, and (2) a naive sequential sampling baseline without tree search. Our method consistently outperforms both, indicating that even at shallow depths, our MCTS framework yields meaningful gains through principled exploration and pruning.

[P5] Additional benchmarks

We agree that broader benchmarking is important. In response, we added experiments on both MAMO (focusing on the more challenging ComplexLP subset) and ComplexOR. Due to time constraints, we prioritized ComplexLP over EasyLP, which is relatively saturated. On both benchmarks, our method outperforms all baselines, further validating its effectiveness. Results have been added to the updated Table 1.

[P6] Other comments

• Baseline results on IndustryOR. We attempted to run Chain-of-Experts and Optimus on IndustryOR using their released code but encountered compatibility issues. As noted by the IndustryOR authors, these methods require non-trivial adaptation. Due to time constraints, we were unable to complete this during the rebuttal, but we plan to include these results in a future version (if accepted).
• App D. App D is not meant to evaluate our method directly, but to explore how solver and formulation choices impact downstream performance. While not central, we included it for context and will trim it in a future version to improve clarity.

Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.

We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.

[P1] Additional analysis

Thank you for this thoughtful suggestion. While objective-value correctness is a standard metric in prior work, we agree that it is an imperfect proxy. To address this, we conducted a targeted expert evaluation on 18 autoformulated problems from the ComplexOR benchmark.

An optimization expert manually reviewed each generated model and assessed the correctness of its major components—(1) decision variables, (2) objective function, (3) equality constraints, and (4) inequality constraints—along with an overall correctness label. This allowed us to identify sources of modeling errors and compare agreements between expert assessments and our objective-value-based proxy.

Analysis. The expert's analysis indicates that the most common sources of error were related to constraint modeling---particularly inequality constraints (e.g., misclassification as equality constraints, incorrect formulation, or omission)---with both equality/inequality constraint errors accounting for over half of all incorrect formulations. Additionally, we observed 82% agreement between the expert’s correctness judgments and the objective-value proxy. In two cases, the expert flagged structural errors despite a correct objective value; in two others, the expert deemed the model correct despite a mismatch in optimal objective values. This suggests that objective-value correctness is a useful but imperfect proxy for full structural accuracy.

Actions taken. We have incorporated these findings into the manuscript and now highlight expert evaluation as a valuable future direction for benchmark development and model assessment.

[P2] Other problem types

Thank you for this observation. Existing benchmarks in this space—NLP4OPT, IndustryOR, MAMO, and ComplexOR—currently consist exclusively of LP and MILP problems, which shaped the scope of our evaluation. Expanding to nonlinear or more complex problems is an important direction for future work, though it first requires the development of suitable benchmarks.

We note that LPs and MILPs remain highly relevant in practice: according to [R1], 61% of real-world OR problems are MILPs and 41% are LPs. These classes also present the same meaningful modeling challenges [C1-C3] for autoformulation, making them a useful starting point.

Additional results. That said, we agree that broader evaluations are valuable. Based on your comment and related feedback, we extended our experiments to additional datasets to further strengthen our evaluations. While these new benchmarks, ComplexOR and the ComplexLP subset of MAMO, still consist of LPs and MILPs, they feature more complex problems and serve as a stronger assessment of generality. Our method outperforms all baselines on both, as shown in our updated Table 1.

[R1] Gurobi State of Mathematical Optimization 2023 Report https://stage.gurobi.com/resources/report-state-of-mathematical-optimization-2023/

Thanks again; we hope the reviewer’s concerns are addressed and welcome further discussions.

We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.

[P1] MCTS motivation

Thank you for the thoughtful question. We interpret this as raising two related concerns: (1) why use a tree-based search framework at all, and (2) why choose MCTS specifically.

(1) Tree-based search rationale. Although the search depth is modest, optimization models naturally decompose into hierarchical components—decision variables, objective function, and constraints—making tree-based search well-suited. This structure enables:

• Clearer credit assignment, isolating which components contribute to success;
• Efficient feedback sharing across subtrees;
• Tractable redundancy pruning via SMT, which operates more effectively at the component level.

These factors contribute to focused exploration, efficient reuse of components, and better search guidance. These benefits are lost when searching directly over complete formulations, resulting in a flat, entangled space.

(2) Why MCTS. We choose MCTS over alternative tree search methods (e.g., DFS, BFS, A*) for two reasons:

• Exploration under uncertainty. MCTS balances exploration and exploitation using value estimates and UCB, making it better-suited for the inherent ambiguity in autoformulation, as it progressively focuses on promising branches. This stands in contrast to deterministic tree-based algorithms.
• Feedback-driven search. MCTS uses backpropagation to update its search policy, whereas other methods follow fixed traversal policies (without adjusting the search from observed outcomes).

Empirical evidence. We point to three existing lines of evidence supporting the efficacy of our MCTS framework:

• Sustained exploration: our method continues discovering novel, functionally distinct formulations over iterations (Fig 5);
• Search efficiency: SMT-based pruning reduces redundancy by two orders of magnitude, improving efficiency (Fig 4);
• Improved search guidance: our comparative scoring method provides more robust feedback, enhancing search guidance (S5.2).

Additional results. Following your feedback, we further isolate our MCTS method's contributions through two baselines:

• A Tree-of-Thought (DFS) baseline with the same hierarchy but no uncertainty guidance or search feedback;
• A naive sequential sampling baseline (same hierarchy) without structured search or pruning (i.e., each component is sampled sequentially, conditioned on the partial formulation).

Results from these additional comparisons are provided in [P2] and are referenced for further analysis.

[P2] Tree-of-Thought

Thank you for this comment. Please find a conceptual comparison below:

Compared to ToT, our approach incorporates (1) structured, domain-specific decomposition, (2) uncertainty-aware, feedback-guided exploration, and (3) symbolic pruning to improve efficiency and reduce redundancy.

Empirical comparison. We include both Tree-of-Thought and the sequential sampling baseline in our updated results. Our method consistently outperforms both across all benchmarks, underscoring the importance of structured decomposition, feedback-driven search, and redundancy pruning. Compared to ToT, MCTS enables more effective exploration by leveraging uncertainty and cumulative feedback to avoid suboptimal branches and refine search trajectories. The comparison with the naive baseline highlights the limits of decomposition alone: without guided search or pruning, performance deteriorates, and manual inspection reveals more invalid or redundant formulations.

Actions taken. We have integrated these additional baseline results into the updated manuscript in Table 1.

[P3] Other comments

• Figure captions: Thank you for this suggestion; we have revised the captions to include more detailed and self-contained explanations.

Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.