Autoformulation of Mathematical Optimization Models Using LLMs

We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.

---

[P1] Autoformulation objective

Thank you for raising these important concerns. Allow us to clarify our original rationale, then outline our planned revisions to improve the connection between our theoretical framework and methodology.

Rationale of probabilistic definition. The probabilistic framework (Eq 2) serves as a generalized formulation that:

1. Captures the inherent uncertainty in translating natural language into mathematical/computational models
2. Explicitly decouples mathematical formulation from computational implementation, recognizing their distinct challenges

Connecting definition to method. Our method directly instantiates this objective (Eq 2):

• $p_\phi$ implements a tree-search model exploring possible formulations, where $\phi$ represents the parameters that are updated through MCTS backpropagation: the value function $V(s)$ and visitation counts $N(s)$. Formally, for the set of all nodes $\mathcal{N}$, $\phi=\lbrace{(V(s), N(s)) | s \in\mathcal{N}\rbrace}$
• $p_\psi$ is realized through our custom-designed deterministic parser, tailored to our mathematical model representation
• $Q(\cdot)$ implements the dual evaluation of formulation correctness (based on LLM comparative scoring) and optimality gap (Eq 4)

Proposed revisions. So far, we have only provided clarifications. While our probabilistic framework is theoretically sound, your review has helped us realize that the current formulation can be confusing for the reader, obfuscating the core connection between the problem definition and methodology. To better align with our method's focus on search, we propose revising Eq 2 to:

$(m^*, c^*) \in \text{argmax}_{m \in \mathcal{M}, c \in \mathcal{C}} ~ Q(m, c; d)$

This revised formulation more explicitly connects our search-based approach while maintaining generality.

Distribution over computation models. Lastly, we wanted to clarify that the second transformation ($p_\psi$), in principle, presents several important challenges that can introduce uncertainty. Most notably, this is reflected in the choice of solvers and their configurations, for example, the configuration of hyperparameters for cutting plane algorithms. Our probabilistic formulation anticipates these challenges and provides a framework for future research to systematically address them, even though we made simplifying implementation choices (deterministic parsing) in this first exploration of autoformulation.

Actions taken: Following your feedback, we have revised the formulation of Eq 2 to improve clarity and expanded the discussion of computational transformation challenges in L195.

I acknowledge that I have read the response from the authors, and appreciate their improvement on this paper. In general, I don't think this is a bad paper. The presentation is clear, the experiment is sufficient, thus I would give a high score if this is a submission to some non-top/application-oriented venues. What really concerns me is that the paper is too "standard", which is not bad for real-world engineering practice, but may be an issue for top conferences like ICLR. I try not to say something like "lack of novelty" in my review, and the authors do made some efforts as their response said, but most of the concepts and techniques are familiar in OR textbooks and other LLM papers. In my very personal opinion, the only major thing that is not so standard is that the "computational representation" stage is represented as a probabilistic model. I know that computational representation can be tricky, and different representations can impact the efficiency especially for large-scale problems, so a model that produces optimal computational representations can be a strength and really interests me. However, the paper does not address this in the method section, which is a pity.

Overall, my opinion for this paper is pretty neutral. I won't be bothered at all if it gets accepted into ICLR.

We appreciate the reviewer’s thoughtful evaluation and positive feedback.

---

[P1] Real-world utility

Thank you for raising this important point. Our work focuses on autoformulation—an automated approach for translating natural language problem descriptions into diverse sets of mathematical and computational models. While our technical focus is on developing this capability, it is crucial to understand both its immediate practical value and long-term potential across different user groups. 

Practical utility: The benefits of autoformulation vary across different user groups: (1) For domain experts without OR expertise (e.g. business owners, healthcare administrators, engineers), it provides an accessible entry point to optimization techniques without requiring extensive OR training. (2) For OR practitioners, it holds the potential to streamline model development by automating routine formulation tasks, allowing greater focus on problem understanding and solution analysis. Much like how software engineers leverage coding LLMs to enhance productivity, autoformulation can serve as both an automation and ideation tool, facilitating rapid exploration of alternative formulations.

Our technical contributions: While autoformulation technology is still emerging, our work makes several fundamental contributions at the intersection of OR and ML: 

• Formal definition of autoformulation as a search problem, with clear characterization of key challenges.
• Novel framework combining MCTS with LLM-based hypothesis generation for systematic exploration under uncertainty.
• Tailored pruning methods using SMT solvers to eliminate redundant formulations, enhancing search efficiency.
• LLM-based evaluation approach for assessing formulation correctness through comparative evaluation.

Our empirical results on IndustryOR, which contains challenging real-world problems, demonstrate significant improvements over existing approaches. Notably, it achieved a $10\%$ improvement over ORLM (an LLM finetuned for optimization modeling), validating the practical potential of our framework.

Looking ahead. While we acknowledge the complexities of real-world optimization modeling (elaborated in [P2]), we see parallels with the evolution of LLM-based coding technologies, which progressed from simple function completion to full application development between 2021 and 2024. We anticipate autoformulation capabilities will follow a similar trajectory of rapid advancement, and our work aims to contribute to the foundations for these future developments.

---

[P2] Real-world complexity

Again, thank you for raising this important consideration. We acknowledge that real-world optimization modeling extends beyond the translation of problem descriptions into mathematical/computational models, and understanding these complexities helps identify which aspects can be effectively automated.

Our autoformulation framework addresses a specific, yet crucial challenge: translating problem descriptions into mathematical and computational models. While this represents just one component of the broader modeling process, solving it alone is important for the reasons mentioned in [P1]. More broadly, the complete modeling process typically involves:

1. Information gathering from diverse stakeholders, often requiring multiple iterations and integration of implicit domain knowledge/conventions (i.e. 'tribal knowledge');
2. Handling complex problem characteristics such as stochasticity, time-varying dynamics, and large-scale variables; 
3. Rigorous validation against real-world data, including sensitivity analysis of modeling assumptions and verification of edge cases; 
4. Continuous communication between technical and business stakeholders to ensure practical utility.

We would also like to use this opportunity to give an example where autoformulation can already play a large role in this process. The optimization problems in engineering systems are relatively 'well-defined'. A concrete example is wireless communications, where there are widely accepted requirements and metrics for measuring system performance (e.g. data rate, delay, energy consumption), commonly encountered decision variables, and the system under optimization is also relatively well understood (derived from physics or by construction of the system). In such domains, autoformulation could play a more comprehensive role in the modeling pipeline.

Actions taken: We have included a discussion of real-world OR modeling complexities in App H.

---

We hope the reviewer’s concerns are addressed and they will consider updating their score. We welcome further discussions.

We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.

---

[P1] Correlation between model generation and evaluation

Thank you for this important observation. We acknowledge that using the same LLM for both model generation and model evaluation could introduce correlations that potentially bias the autoformulation process.

Evaluation strategy. Our method implements two key mechanisms to address this challenge. First, at the global level, we employ a composite evaluation strategy (Eq 4) that combines LLM-based assessment with solver feedback on optimality gap, providing complementary signals from independent sources. Additionally, for complete formulation evaluation, we utilize a ranking-based approach that compares candidates against baseline models rather than relying on absolute scoring, which helps mitigate self-reinforcing biases.

Empirical evidence. Our empirical analysis supports the effectiveness of this approach. The ablation study in Sec 5.2 reveals two key findings:

1. Ranking-based scoring of partial formulations significantly outperforms uniform scoring baselines (Fig 3), demonstrating the evaluation protocol's capacity to meaningfully differentiate between candidate expansions.
2. Comparative evaluation of complete formulations shows strong correlation ($\rho=0.48, p<0.001$) with ground-truth correctness, indicating robust evaluation despite using the same LLM.

Future direction. This challenge reflects a broader open question in LLM research. Recent works have offered tentative evidence that LLMs have the potential to evaluate their own generations and iteratively improve [1, 2]. Our framework could be extended through dedicated generation and evaluation models, fine-tuned evaluators trained on expert feedback, or debiasing techniques using ensemble/mixture-based approaches.

Actions taken: We have included a detailed discussion of these considerations and future research directions in L515.

[1] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. and Yao, S., 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.

[2] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y. and Gupta, S., 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

---

[P2] Diversity in exploration

We appreciate this question. As identified in Sec 2.1, efficient exploration of the vast hypothesis space is a key challenge in autoformulation that our method aims to explicitly address.

Encouraging diversity. Our method incorporates multiple complementary mechanisms to ensure diverse exploration:

1. Hierarchical decomposition structures the search across four levels (variables, objective function, equality, and inequality constraints), enabling focused exploration within each decomposed component rather than searching for entire formulations at once.
2. MCTS framework with UCT scoring dynamically balances exploration-exploitation, adaptively guiding the search towards promising or unexplored regions of the hypothesis space.
3. SMT-based pruning removes trivially equivalent formulations, ensuring computational resources are devoted primarily to exploring functionally distinct solutions.

Experimental results. The efficacy of our method design is evidenced by our empirical results. 

• Fig 4 demonstrates the diversity of generated formulations at each hierarchical level, particularly highlighting how SMT pruning promotes exploration of functionally distinct paths while preventing exponential growth in redundant search efforts. 
• Additionally, Fig 5 (further discussed in subsequent responses) illustrates that additional MCTS iterations consistently uncover new, correct formulations, confirming effective exploration of the solution space rather than redundant sampling or search.

Meaningful diversity. A core premise of our work is leveraging LLMs as dynamic hypothesis generators that assign probability mass over plausible mathematical formulations. While our experimental results validate this assumption by demonstrating LLMs' ability to generate diverse formulations, the search process faces a key challenge: trivial variations due to mathematical equivalences. This underscores the importance of our pruning mechanism in eliminating uninformative diversity and focusing the search on meaningful variations.

---

[P2] Optimality gap and computational efficiency

We appreciate you raising this point, which allows us to highlight why optimality gap and computational efficiency are both formulation and solver dependent. In the table below, we describe the impact of both formulation and solver choice on optimality gap and computational efficiency for different types of optimization problems.

Key observations:

1. For Type I problems, formulation mainly affects computational efficiency rather than optimality.
2. For Type II problems, correct reformulation is crucial for both optimality and efficiency.
3. For Type III problems, both formulation and solver choices significantly impact the trade-off between optimality and efficiency.

Empirical evidence: To support this analysis empirically, we compare the impact of both formulation and solver choice (including 8 solvers) on optimality and solution time. These results are included in the revised App G, and we will briefly summarize them here:

1. Impact of formulation: We observed that convex reformulation results in high-quality solutions with different convex solvers consistently. In particular, a convex reformulation had a 10,000x smaller optimality gap and 1,000x faster solution time than a general-purpose solver on the non-convex original formulation.
2. Impact of solver: Using a specialized convex solver resulted in 1,000x faster solution time than a general-purpose solver.

This empirically confirms the importance of both formulation and solver on optimality gap and computational efficiency. 

Actions taken: Thank you again for this comment, which has helped us improve the paper with further clarifications and discussions. We have updated App G with empirical analysis of the impact of formulation and solvers on solution quality. We have also included the table above into App E.

---

We appreciate the reviewer’s thoughtful evaluation and positive feedback.

---

[P1] Problem definition scope

Thank you for this thoughtful comment. Please allow us to clarify several points about our problem definition and its relationship to our proposed method.

Generalized definition. We introduced a generalized problem definition for autoformulation, as this is a relatively new (and promising) direction at the intersection of ML and optimization modeling. Our framework consists of three key components that together address the autoformulation objective (Eq 2).

Value of general framework. Our proposed method represents one possible instantiation of this broader framework, addressing all three components while focusing primarily on the first component, which we identified as the most challenging and pressing frontier. The generality and completeness of our problem definition serve several important purposes:

1. Clearly delineates the multiple challenges that need to be addressed in this emerging field
2. Facilitates systematic identification of research opportunities, such as developing LLM-based techniques for solver configuration or specialized models for evaluating formulation correctness

We would appreciate any further feedback you may have on this point. 

---