Evo-Step: Evolutionary Generation and Stepwise Validation for Optimizing LLMs in OR

We greatly appreciate your insightful comments and helpful recommendations, which have guided us in improving our work. Our detailed responses to your feedback are provided below.

Weakness 1. The caption of figures is not informed enough and it is difficult to understand the content of diagrams.

We deeply regret that the captions for our diagrams were not sufficiently detailed in the initial submission. To address this concern, we have revised the captions extensively, providing comprehensive explanations for each figure. These revisions will be incorporated into the updated version of our work, with the aim of improving clarity and enhancing the reader's understanding of the diagrams.

Figure 1 illustrates the Complexity-evolving and Scope-evolving strategies. Complexity-evolving entails methods such as constraint modification, parameter adjustment, and objective alteration, all designed to gradually increase the complexity of problems. Newly introduced or modified components are highlighted in red. In contrast, Scope-evolving emphasizes domain adaptation and problem combination, where problems are transformed to fit new contexts. For example, "theme park" is changed to "hospital," and "scorers" are replaced by "nurses." Problem combinations merge different problem elements to create novel scenarios, and alternative formats, such as tables, are introduced to diversify the representations.

Figure 2 provides a detailed depiction of the Evo-Step-Instruct framework. Starting with an initial dataset, seed data is sampled randomly in each iteration, followed by the use of specific evolving methods by the Problem Generators to create new problem descriptions. These descriptions are evaluated by the Description Checker, which identifies and corrects errors to refine subsequent iterations. Qualified descriptions proceed to the Solution Generators, where solutions are produced and validated through the Variables, Constraints, and Program Checkers. These validation stages ensure the logical consistency and correctness of the solutions. Only solutions passing all checks are incorporated into the dataset for further enhancement. All components—checkers and generators—are implemented using LLMs guided by carefully crafted prompts, with specific elements highlighted in red in the figure.

Figures 3 and 4 highlight key meta-prompts used in the framework.

Figure 3 elaborates on constraint modification, describing stepwise adjustments to constraints that increase problem complexity in a controlled manner.

Figure 4 details the constraint checker prompt, showcasing how it validates the consistency between the problem's constraints and its description, with a particular focus on methods such as Big-M to ensure accurate alignment.

Weakness 2. Evolutionary Strategy (ES) typically refers to a type of evolutionary algorithm focused on optimizing complex problems by mimicking natural selection processes. However, I cannot see any element of Evolutionary Strategy in Figure 1, which is claimed as an example of evolutionary strategy.

Thank you for pointing out the misunderstanding regarding the use of "evolutionary strategy." We understand that this term is commonly associated with specific algorithms that optimize problems through natural selection-inspired processes. In our work, the term "evolution" is used more generally to describe a step-by-step methodology for generating high-quality data to improve LLM performance in OR problem modeling.

Our framework draws inspiration from WizardLM’s open-domain instruction fine-tuning and adapts it specifically for OR modeling tasks. We introduce two focused categories: Complexity-evolving and Scope-evolving. Both approaches aim to ensure mathematical validity and logical consistency while enriching the diversity of generated problems.

By focusing on these tailored categories, our framework addresses the unique challenges of OR problem modeling without confusion with traditional evolutionary algorithms. We hope this explanation clarifies our methodology and its intended contributions.

Weakness 3. The citation format is not good and needs improvement.

Thank you for pointing this out. We have thoroughly reviewed the text and identified areas where citation formatting was inconsistent or unclear. For instance, the previous format:

Operations Research (OR) is a valuable discipline for addressing complex decision-making problems, widely applied in fields such as economics, engineering, and computer science (Bertsimas et al., 2019; Pereira et al., 2022; Belgacem et al., 2020).

Has been revised to:

Operations Research (OR) is a valuable discipline for addressing complex decision-making problems, widely applied in fields such as economics, engineering, and computer science Bertsimas et al. (2019); Pereira et al. (2022); Belgacem et al. (2020).

These updates will be fully incorporated in the revised submission.

Question 1. In this work, I cannot see any evolutionary components such as crossover and mutation. How do you evolve and what do you evolve?

We would like to clarify the concept of "evolution" as it is applied in Evo-Step-Instruct, as it differs from the traditional evolutionary algorithms that involve genetic operations like crossover and mutation. In Evo-Step-Instruct, "evolution" does not refer to the genetic operations commonly found in evolutionary algorithms. Instead, it denotes a progressive approach to creating a dataset containing OR problems with varying levels of complexity and diversity, specifically for modeling OR problems, rather than optimizing solutions through iterative selection processes.

The Evo-Step framework employs two core evolution-inspired approaches:

Complexity-evolving: This approach iteratively increases the difficulty of problems by adding constraints, modifying parameters, or introducing new objectives.

Scope-evolving: Through domain transitions and problem combinations, this approach broadens the variety of generated problems, thereby expanding capabilities across different scenarios of the dataset.

The evolving methods in Evo-Step are realized through LLMs to directly generate diverse instances, ensuring logical consistency while progressively enhancing the dataset's complexity and scope. This approach focuses on data generation rather than genetic optimization. We hope this clarification provides a better understanding of the Evo-Step-Instruct methodology and look forward to further feedback.

Question 2. The contribution of this work is not clear or even a bit confused. This work states "increases the complexity of generated problems", why we need to generate problems on our own? A more convincible direction should be that we use LLMs to generate solution to the problems.

We agree that generating solutions with LLMs is a valuable direction. In fact, our work seeks to achieve this goal through an alternative approach by integrating LLMs with solvers, aiming to ultimately improve the overall solution-generation process. Rather than directly using LLMs to produce solutions, we employ LLMs to generate mathematical models and the corresponding programs, which are then used to call solvers to complete the problem-solving process. This method is more reliable, as solvers are specifically designed to handle constraints effectively and ensure the validity of the solutions.

The success of generating mathematical models and programs relies heavily on the quality of training data, as such data helps LLMs learn how to accurately structure solving logic and address problem constraints. However, collecting expert-verified data is labor-intensive and requires significant domain expertise, making large-scale collection impractical. To address this challenge, our Evo-Step-Instruct framework generates high-quality datasets based on a small set of verified examples, significantly enriching the training data.

By fine-tuning LLMs with this enriched data, our models demonstrate measurable improvements amd achieve superior accuracy compared to GPT-4 with well-designed prompts, particularly for more complex tasks. This enables our framework to effectively address real-world optimization challenges while leveraging the strengths of both LLMs and solvers.

We also recognize the need to test this framework across a broader range of optimization tasks and further refine the generated programs for higher generalization, which constitutes a pivotal focus for our future research endeavors.

Question 3. How do you fine-tune the open-source LLMs? What technique do you use?

Details of our fine-tuning process are outlined in Section '4.3 DETAILS,' where we utilize the widely recognized LLaMA-Factory training framework following the Alpaca format template. The input comprises a fixed prompt describing a problem, while the output includes mathematical models and corresponding programs.

We employ the commonly used LoRA fine-tuning technique. For training, we set the learning rate to $1.25\times 10^{-4}$, with 10 and 12 epochs for Mistral-7B and LLaMA-3-8B, respectively. Additional specifics can be found in Appendix A.6.

Question 4. More experiments are required such the the comparison with tradition algorithms (non-LLM approach). LLMs are unnecessarily suited to every task, thus it is important to justify the strength of LLMs in this task.

The automation of optimization modeling saw significant strides with the introduction of NL4OPT in 2022. Traditional methods like tag-BART, the NeurIPS competition winner, fall under the category of pre-trained language models (PLMs). However, tag-BART lacks code generation capabilities and heavily relies on manual validation to ensure the correctness of constraints and objectives. Despite such manual interventions, its performance on NL4OPT remains at 47.9%, notably lagging behind the results of modern LLM-based approaches, including GPT-4 and ORLM, as demonstrated in Table 11.

Table 11: Comparative Performance of Various Methods on NL4OPT

These results highlight the limitations of traditional PLM-based approaches like tag-BART, which are unable to achieve satisfactory performance even with extensive manual efforts. In contrast, LLM-based methods leveraging advanced prompt techniques such as CoT and Reflexion exhibit significant advancements, excelling in both code generation and mathematical reasoning to deliver far superior results.

Our proposed framework, Evo-Step-Instruct, further extends these advancements by fine-tuning smaller LLMs (e.g., 7B and 8B parameter models) on high-quality datasets generated through our method. As shown in Table 11, the fine-tuned Evo-Step models outperform other SOTA approaches, including CoE and ORLM, while avoiding reliance on complex prompt engineering or multi-agent frameworks.

This comparison reinforces the limitations of traditional algorithms like tag-BART while illustrating the transformative potential of LLM-based approaches in effectively bridging the gap between natural language problem descriptions and mathematical optimization models.

Thank you for your response that clarifies the contribution of this work. Importantly, my main concern remains the effectiveness of this overly general framework, since each issue has its own individual characteristics. Misleading terminology that overlaps with the evolutionary computing community also hinders readability, which I think it probably can not be revised at the current stage and needs a thorough revision. I'll therefore keep my current grade.

Thank you for your thoughtful feedback

1) Response to Terminology Concerns

Regarding the concern about “misleading terminology that overlaps with the evolutionary computing community,” we have carefully revised the manuscript to address this issue. Specifically:

(a) Terms such as “Evolutionary Strategies” have been replaced with “Iterative Problem Generation” to avoid overlap with the terminology of evolutionary computing.

(b) Similarly, “Depth Evolution” and “Breadth Evolution” have been changed to “Complexity-Evolving” and “Scope-Evolving,” respectively.

These adjustments align with terminology commonly used in LLM research, as evidenced by recent works [1, 2]. Therefore, we believe the revised manuscript is more precise and contextually appropriate.

2) Response to Concerns about Framework Effectiveness

We understand your concern about the potential limitations of using a general framework for solving distinct problems, especially since each problem has its own unique characteristics. However, recent advancements in machine learning and related fields offer strong evidence supporting the feasibility of unified models tackling diverse challenges effectively.

(a) Cross-Domain Success of Unified Models

In NLP, tasks such as Question Answering [3], Machine Translation [4], and Text Summarization [5] each have their own distinct characteristics. Despite these differences, LLMs [6–8] have demonstrated excellent performance across all of these tasks.

This trend extends beyond NLP. In computer vision [9–11], tasks like image classification, segmentation, and object detection, although unique in nature, can all be effectively addressed by unified models. Furthermore, foundational models in fields like chemistry and medicine [12–14] have proven capable of handling a wide range of tasks in their respective domains.

A noteworthy example is Google's AlphaProof [15], which has showcased the ability of unified models to solve specialized problems, such as those from the International Mathematical Olympiad (IMO). AlphaProof's success in solving algebra and number theory problems demonstrates the potential of a single model to handle highly specialized tasks like mathematical proof.

(b) Emerging Trends in Machine Learning for Combinatorial Optimization

There has been a growing interest in using single models to solve multiple types of combinatorial optimization problems. For example, recent neural solvers have been able to tackle over 10 variants of the Vehicle Routing Problem [16–18], with RouteFinder [19] solving dozens of VRP variants.

Beyond routing problems, frameworks like MAB-MTL [20] have shown promise in addressing diverse problems, including the TSP, the Capacitated Vehicle Routing Problem, the Orienteering Problem (OP), and the Knapsack Problem (KP) with a single model. Unco[21] utilizes LLMs to extract task-specific features from various COPs and combine them with the initial input features. By leveraging an encoder-decoder architecture, Unco unifies the solving process across multiple problems, such as TSP, CVRP, KP, the Single-Machine Total Weighted Tardiness Problem, and the Minimum Vertex Cover Problem (MVC).

Similarly, GOAL [22] has been trained to solve problems like Asymmetric TSP (ATSP), CVRP, OP, KP, and others, showcasing its impressive ability to generalize across a wide range of combinatorial optimization tasks, including 8 others such as the Traveling Repairman Problem, the Prize Collecting TSP, and others.

RedCO [23] is trained with ATSP, 2D Euclidean TSP, Directed Hamiltonian Cycle Problem, and Boolean Satisfiability Problem (SAT), and prove its ability to generalize across problems like the Vertex Cover Problem, Clique Problem, Independent Set Problem, VRP, MIS, and Flexible Flow Shop Problem (FFSP). These examples suggest that solving multiple combinatorial optimization problems with a single model is not only feasible but is increasingly becoming a focal point of research.

(c) Modeling is a simpler task compared to Solving

It is important to distinguish between the tasks of modeling and solving in combinatorial optimization. Modeling involves structuring a real-world problem in a formal mathematical way, which is relatively simpler. In contrast, solving the problem requires substantial computational resources and specialized algorithms to find optimal solutions.

Much like machine translation, which translates text from one natural language to another, our task is to translate natural language descriptions into formal mathematical language. This step is easier than solving the problem itself, as it centers on identifying key components—such as variables, constraints, and objectives—and organizing them into a coherent formal structure.

Given the proven ability of LLMs to handle complex mathematical tasks (e.g., AlphaProof’s performance), we believe that a single LLM can effectively address a wide range of combinatorial optimization modeling tasks.

(d) Empirical Validation of Existing Methods

The task of automated modeling for combinatorial optimization problems has gained significant attention in both the machine learning (ML) and operations research (OR) communities. In the ML field, notable contributions include Chain-of-Experts [24], OptiGuide [25], OptiMUS [26], ORLM [27], and ReSocratic [28]. These works have demonstrated the potential of unified models for modeling a wide range of combinatorial optimization problems. In the OR field, studies such as LM4OPT [29] and others [30] further emphasize the growing interest in this area, showcasing the application of LLMs to solve complex optimization tasks.

In the industrial domain, this research direction has also gained traction. For example:

1. Microsoft’s OptiGuide [25] optimizes supply chain design for Microsoft Azure, showcasing the real-world application of automated modeling in supply chain optimization.
2. Cardinal Operations developed the COLORMind intelligent decision-making platform, based on ORLM [27], which is now being applied in logistics and transportation. (Available at: https://www.shanshu.ai/products/color-mind) (Available at: https://www.shanshu.ai/products/color-mind)
3. Alibaba’s MindOpt Copilot [https://opt.alibabacloud.com/chat] enables seamless workflows from natural language problem descriptions to modeling and solution generation.

We have validated our proposed model, Evo-Step, using diverse datasets such as NL4Opt, Mamo, and IndustryOR, which cover a wide array of problems including routing, resource allocation, assignment, facility location, investment, etc. The performance of Evo-Step across these diverse problems demonstrates the effectiveness of this approach and further supports the feasibility of using a single model for combinatorial optimization modeling.

Through these empirical validations, we further strengthen the case for automating combinatorial optimization modeling with a single unified model, a concept that has gained both academic and industrial support.

We hope that the clarifications above address your concerns, and we look forward to further discussions.

We are truly grateful for your careful review and valuable suggestions, which have guided us in strengthening our work. The following responses address your comments in detail.

Question 1: Could the authors elaborate on Evo-Step’s computational requirements and its scalability for larger datasets or in different domains?

In the data generation phase, Evo-Step-Instruct leverages evolution-based generation and stepwise validation mechanisms to create high-quality datasets. The computational requirements in this stage primarily come from iterative problem construction and validation checks.

Efficiency Problem Construction: The generation mechanism iteratively builds datasets of increasing complexity by reusing solutions from previously generated problems. Instead of re-solving entire problems from scratch, Evo-Step-Instruct focuses only on the newly added or modified components. This significantly reduces computational redundancy and ensures scalability. The validation mechanism focuses on checking specific components (e.g., constraints or objectives) rather than re-evaluating the entire problem, significantly reducing the number of tokens processed per call. These design choices ensure that API usage remains efficient.

Scalability: Evo-Step-Instruct is designed to scale with dataset size. Its generation mechanism operates iteratively, ensuring that only incremental modifications are made to existing problems rather than constructing them entirely from scratch. The validation overhead is bounded for each individual problem, as the mechanism focuses solely on newly added or modified components. The validation has a fixed limit on the number of iterations. This approach not only minimizes redundant computations but also ensures efficient use of resources, making the framework scalable for large and complex datasets.

In the inference stage, Evo-Step directly outputs a complete optimization model based on the input problem description. The generated model is then passed to a traditional solver (e.g., COPT, Gurobi) for obtaining solution. Importantly, Evo-Step does not impose additional computational demands during this phase:

Direct Modeling Output: Evo-Step outputs a fully defined optimization model in one step, requiring neither multi-agent interactions nor iterative reflection to refine solutions. Unlike CoE or Reflexion, which depend on such processes to iteratively improve results, Evo-Step ensures a more efficient inference phase, significantly reducing the computational burden.

No Additional Framework Overhead: Once the optimization model is generated, the computational effort shifts entirely to the solver. Evo-Step’s role ends at the modeling stage, ensuring that its computational requirements during inference are constant and predictable. Leveraging vLLM for inference, Evo-Step supports parallel generation of multiple optimization models through dynamic batching, allowing efficient processing of concurrent tasks. Once generated, these models can be passed to solvers in parallel, further enhancing scalability and enabling Evo-Step to handle large datasets and high-throughput optimization tasks effectively.

Question 2. How adaptable is Evo-Step’s framework for domains outside OR, and what modifications would be necessary?

Evo-Step-Instruct’s framework is fundamentally adaptable to domains outside OR due to its modular design and reliance on general principles of problem generation and validation.

Framework adaptability:

Evo-Step is built on two domain-agnostic principles: iterative Problem Generation and Stepwise Validation Mechanism. The generation mechanism constructs problems by iteratively adding or modifying components, a process that is not tied to OR-specific representations. For example, in WizardLM, a similar strategy is employed to generate instruction-following data by iteratively constructing instructions and responses for LLMs. This demonstrates that Evo-Step-Instruct’s generation approach is well-suited for tasks such as:

1. Machine learning (hyperparameter optimization or dataset augmentation).

2. Natural language processing (automated question generation or summarization).

The validation framework ensures the correctness and consistency of generated problems, a concept that generalizes well across domains. For instance:

1. In natural language processing, validation could verify the semantic coherence of generated instructions.

2. In machine learning, validation might ensure that generated hyperparameters are within feasible ranges.

These foundational principles make Evo-Step-Instruct inherently adaptable to a wide range of structured problem-solving tasks.

Necessary Modifications for Other Domains:

While Evo-Step-Instruct is adaptable, certain domain-specific adjustments are required to maximize its effectiveness: 1) For generation, identifying aspects of the existing problem that can be further evolved, focusing on introducing meaningful complexity or diversity. 2) For validation, specific checks are designed to address different components of the problem, ensuring that all generated aspects meet the necessary requirements and maintain consistency.

Strengths for adaptation:

1. By basing new problem generation on validated existing data, the iterative generation mechanism ensures a baseline level of correctness, avoiding the generation of outlier or nonsensical outputs. This is a key advantage compared to purely generative methods, which often lack such guarantees.

2. Moreover, the separation of generation and validation processes allows each component to be customized independently, enabling freedom in adapting the framework to new domains without disrupting its core functionality.

We deeply value your feedback and thoughtful suggestions, which have significantly helped us refine our work. Each of your comments has been carefully responded to below.

Weakness 1. The presentation is poor especially the diagrams. Their captions are so short to understand what they intend to convey. The caption should be significantly expanded and improved.

We sincerely apologize for not providing sufficiently detailed captions for the diagrams. To address this, we have prepared expanded captions with detailed explanations for each image, which will be incorporated into the revised version to improve readability and ensure that each diagram is more informative for the reader.

Figure 1 demonstrates Complexity-evolving and Scope-evolving approaches. Complexity-evolving includes constraint modification, parameter adjustment, and objective alteration, which incrementally increase problem difficulty. Red-highlighted text represents newly added or modified elements. Conversely, Scope-evolving focuses on domain transformation and problem combination, adapting problems to different contexts. For instance, "theme park" is replaced with "hospital," and "scorers" becomes "nurses." Combination integrates elements from distinct problems, creating entirely new scenarios. Additionally, new formats such as tables are introduced.

To provide clarity on the implementation of these elements, Figure 2 illustrates the Evo-Step-Instruct framework. Given an initial dataset, each iteration involves randomly sampling seed data, after which a specific evolving approach is used by the Problem Generators to create a new problem description. This description is checked by the Description Checker, which provides feedback on detected errors to guide the next round of generation. When the description is qualified, Solution Generators produce a solution. The solution is then validated by the Variables, Constraints, and Program Checkers. Errors are also provided as feedback to help resolve issues. Only solutions that pass all checks are considered qualified. These solutions, along with their descriptions, are added to the dataset for further generation. All the checkers and generators are implemented using LLMs with well-designed prompts, as shown in the figure. The red words highlight special requests of different prompts.

Figures 3 and 4 provide meta-prompts for key components of the framework.

Figure 3 focuses on constraint modification, detailing how to adjust constraints incrementally to increase complexity.

Figure 4 outlines the constraint checker prompt, illustrating how it ensures alignment between model constraints and problem descriptions, with particular attention to techniques like the Big-M method.

We hope these expanded captions will assist readers in gaining a clearer understanding of the key aspects of our work.

Weakness 2. There are some inaccurate expressions. For example, evolutionary strategy specifically refer to a branch algorithm in the evolutionary optimization instead of an arbitrary strategy related to evolution.

Thank you for pointing out the inaccurate expression of evolutionary strategy. To clarify, our approach begins with a simple OR problem and gradually increases its complexity and diversity in a step-by-step manner.The concept of evolution in our framework is inspired by WizardLM, which focuses on tasks related to open-domain Instruction fine-tuning. However, our work specifically adapts this idea to the context of OR problem modeling, introducing two tailored categories: Complexity-evolving and Scope-evolving.

By explicitly focusing on these categories, our framework addresses the unique requirements of OR problems, including the generation of mathematically valid constraints and problem formulations, while avoiding confusion with the term "evolutionary algorithms."

Weakness 3. The problem formulation needs further clarification. Normally, we propose a method/framework to solve a or several problems. However, you are trying to solve a large number of problems simultaneously. How to ensure its effectiveness would be a very serious issue. In addition, the problem to solve is also not clear to readers.

We sincerely appreciate your thoughtful observations about the scope of our work and its relation to the broader field of combinatorial optimization. It is true that most existing studies primarily focus on applying a single method or framework to solve one or a few specific types of combinatorial optimization problems [1,2,3]. However, with the advancement of machine learning models' generalization capabilities, recent research, such as GOAL [4], has begun to explore the potential of using a single model to address a variety of distinct problem types.

Large language models (LLMs), known for their strong generalization abilities [5], hold significant potential for addressing a wide range of problem types. Prior studies, including CoE, Optimus, and ORLM, have already demonstrated the capability of LLMs to address various combinatorial optimization problems, such as planning, routing, allocation, and assignment. These approaches do not directly provide solutions but instead formularize natural language problem descriptions into mathematical models and solver-compatible programs, which are subsequently solved using solvers (like Gurobi and COPT). Notably, CoE and Optimus have shown that LLMs possess inherent modeling capabilities for diverse problems through pre-training, while ORLM has enhanced this ability by generating high-quality data for further fine-tuning.

Building on these advancements, our work leverages the Evo-Step-Instruct framework to generate datasets that further improve LLMs’ capabilities, particularly in terms of generalization. The framework emphasizes three critical aspects:

1. enhancing the diversity of the training dataset to better capture a wide range of problem scenarios.

2. ensuring the dataset includes examples spanning various levels of difficulty.

3) maintaining data quality through precise problem descriptions and accurate modeling.

Our framework specifically emphasizes diversity by generating problems of varying difficulty levels and from different domains, ensuring that fine-tuned models can effectively address a broader spectrum of optimization tasks.

The experimental results demonstrate that Evo-Step-Instruct significantly enhances model performance across various benchmarks, such as NL4OPT, MAMO, and industry, which include problems of different types and complexities.

In summary, by leveraging the inherent generalization capabilities of LLMs and emphasizing diversity and quality in data generation, our Evo-Step-Instruct framework ensures both the effectiveness and adaptability of LLMs in modeling a wide range of OR problems. This targeted effort addresses concerns about their capability to handle multiple problem types while significantly enhancing their performance and robustness.

[1] Bengio, Yoshua, Andrea Lodi, and Antoine Prouvost. "Machine learning for combinatorial optimization: a methodological tour d’horizon." European Journal of Operational Research 290.2 (2021): 405-421.

[2] Cappart, Quentin, et al. "Combinatorial optimization and reasoning with graph neural networks." Journal of Machine Learning Research 24.130 (2023): 1-61.

[3] Mazyavkina, Nina, et al. "Reinforcement learning for combinatorial optimization: A survey." Computers & Operations Research 134 (2021): 105400.

[4] Drakulic, Darko, Sofia Michel, and Jean-Marc Andreoli. "GOAL: A Generalist Combinatorial Optimization Agent Learning." arXiv preprint arXiv:2406.15079 (2024).

[5] Yang, Haoran, et al. "Unveiling the Generalization Power of Fine-Tuned Large Language Models." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Weakness 4. I suggest to use the proposed method to solve some OR problems and compare it with some traditional methods instead of LLM-based method, which can help demonstrate the effectiveness of your method.

The progress of automating optimization modeling began with the competition of NL4OPT in 2022. Traditional methods, such as the tag-BART model that won the NeurIPS competition, belong to pre-trained language models (PLMs) but lack code generation capabilities and require extensive manual validation of constraints and objectives to ensure correctness. Despite these manual interventions, tag-BART's performance on NL4OPT is 47.9%, a result that falls significantly behind the performance of LLM-based methods, including GPT-4 and ORLM, as shown in Table 10.

Table 10: Comparative Performance of Various Methods on NL4OPT

The data underscores that PLM-based models, such as tag-BART, struggle to deliver adequate performance in optimization modeling tasks, even with manual intervention. In contrast, LLM-based methods, particularly those leveraging advanced techniques such as CoT and Reflexion, have made substantial progress in automating both code generation and mathematical reasoning, achieving far superior performance.

Our proposed method further builds on these advancements by fine-tuning smaller LLMs (e.g., with 7B or 8B parameters) using high-quality data generated by Evo-Step-Instruct framework. As demonstrated in the table, our fine-tuned Evo-Step models surpass other SOTA methods, such as CoE and ORLM, without relying on well-designed prompts or multi-agent frameworks.

This performance comparison underscores the limitations of traditional methods such as tag-BART and highlights the promise of advanced LLM-based techniques in bridging the gap between natural language descriptions and mathematical models.

Question 1. How do you fine-tune the LLMs? What technique do you use? Details are lacking.

Details on our fine-tuning process are provided in section '4.3 DETAILS.', where we describe the use of the widely used LLaMA-Factory training framework, utilizing the Alpaca format template. In this setup, the input consists of a fixed prompt with a problem description, and the output is a solution that includes mathematical models and corresponding programs. For specific technique, we use the usually adopted fine-tune method LoRA .

For training, we set the learning rate as $1.25\times 10^{-4}$, and epoches are 10 and 12 for Mistral-7B and LLaMA-3-8B, respectively. The details of training are listed in Appendix A.6.

Question 2. What is the contribution of this work in solving OR problems? It seems to focus on more how to generate problems and validation, which might be more trivival compared to solve problem itself.

Our work contributes to propose a novel high quality training data generation method to train LLMs to have the capacity to formularize real-world problems described in natural language into mathematical models and corresponding solution programs. This step is a significant part of solving real-world OR problems. Only after these problems are transformed into mathematical models can they be solved using OR techniques, including solvers. According to a report from Gurobi[1], 81% of Gurobi solver users hold advanced degrees, and 49% of all respondents major in OR. Such an expertise gap limits the usefulness of operations research in a broader range of practical applications.

Given this expertise gap, we do not view the tasks of problem generation and validation for LLM training as trivial compared to problem-solving. In fact, the current capabilities of LLMs are inadequate for fully automating the optimization modeling of real-world OR problems. Enhancing their modeling capabilities is a substantial challenge that demands urgent attention.

A critical aspect of improving LLMs' modeling capabilities lies in the quality of training data, which fundamentally determines model performance. However, existing fine-tuning datasets revealed a significant number of errors, which can negatively impact model performance if used for fine-tuning. Moreover, previous datasets often lack sufficient diversity, which limits the generalization capability of models to handle a wide range of OR tasks.

To address these challenges, we propose a novel data augmentation framework based on evolutionary methods. During the generation process, we introduce Complexity-evolving to produce data by increasing complexity and refining problem constraints. Additionally, Scope-evolving generates diverse variations of existing problems to cover a broader range of scenarios. To ensure the accuracy of the generated data, we apply stepwise validation mechanism to filter and validate the quality of the augmented dataset.

By enhancing the complexity and diversity of fine-tuning data, our framework enables LLMs to better tackle optimization modeling. The enhanced datasets significantly improve the modeling capabilities of LLMs, making a substantial contribution toward bridging the gap between natural language problem descriptions and effective OR problem-solving.

[1] Gurobi Optimization. matical optimization report, 2023. URL https://www.gurobi.com/resources/report-state-of-mathematical-optimization-2023/.

Question 3. Are all experimental results based on COPT, GUROBI, or are they selecting the best outcomes from either? Additionally, if OMLR is only trained with COPT, is it appropriate to use the original checkpoint directly?

We appreciate the your thoughtful questions and would like to provide additional clarification.

(1) For the prompt engineering methods, we evaluated each method independently using both COPT and Gurobi and reported the best-performing result. For the fine-tuning methods, including Evo-Step and ORLM, all results are based solely on the COPT, as the raw data used for fine-tuning was generated using COPT.

(2) The original ORLM checkpoint was used without modification to ensure fairness and consistency. Since the authors of ORLM only released the checkpoint and 3,000 training samples (rather than the full dataset), it would be impossible to reproduce their results or achieve a fair comparison by re-training or converting their data to Gurobi. Using the checkpoint maintains methodological consistency with the original work, which was exclusively fine-tuned on COPT data, ensuring a fair comparison.

(3) To validate the necessity of fine-tuning and the contribution of our data generation framework, we tested LLaMA-3-8B and Mistral-7B directly (without fine-tuning) on the NL4OPT, MAMO and IndustryOR datasets. The results, shown in Table 8, indicate that both models achieved 0% accuracy across all benchmarks, highlighting that these models alone are insufficient for modeling OR problems. This underscores the importance of high-quality datasets and the advantage provided by our data generation framework.

In contrast, for the prompt engineering methods, larger models such as GPT were used, leveraging their embedded knowledge along with carefully designed prompts. To minimize biases from solver-specific knowledge and ensure fairness, these methods were evaluated separately using both COPT and Gurobi solvers.

Table 8: Performance Comparison of Base and Fine-Tuned Models on Different Datasets

We hope this explanation addresses the reviewer’s concerns.

Question 4. What would be the impact of directly applying the proposed "Stepwise Validation Mechanism" to SOTA LLMs, such as GPT-4, in modeling tasks? Would this significantly enhance accuracy and performance?

To evaluate the impact of the proposed Stepwise Validation Mechanism on SOTA LLMs, including GPT-3.5, GPT-4 and GPT-4o, we conducted experiments on the MAMO Complex dataset. The results are summarized below:

Table 9: Comparison of Stepwise Validation Mechanism and Other Prompt Engineering Methods on MAMO ComplexLP

The results demonstrate that the Stepwise Validation Mechanism delivers consistent improvements across different GPT models when compared to Standard, CoT, and Reflexion methods. For GPT-4, our framework achieves the highest accuracy (42.18%), outperforming all other methods. However, CoE remains superior for GPT-3.5 and GPT-4o, reflecting the strength of its iterative reflection mechanism in these cases.

In contrast, the Stepwise Validation Mechanism emphasizes real-time validation and correction during the modeling process, , avoiding the additional complexity of reflection. This streamlined approach proves particularly effective for LLMs, as demonstrated by its superior performance with GPT-4. Although CoE excels in certain cases, our method offers a robust and efficient alternative.

Additionally, It is important to consider the inherent difficulty of solving tasks directly during testing, as all methods must generate solutions from scratch. However, when used for data generation, the Stepwise Validation Mechanism can reference the solution of the original problem to generate solutions for new problems. By focusing only on the newly added or modified components, the mechanism significantly reduces the modeling difficulty. This advantage is not available during testing, where tasks must be solved entirely independently, but it underscores the potential of Stepwise Validation Mechanism for facilitating high-quality data generation.

Thank you for your insightful feedback and constructive suggestions, which have been invaluable for improving our work. Below, we provide detailed responses to your comments.

Weakness 1. Although the results contribute to specific OR optimization problem modeling, the technique's contribution appears incremental, as the evolution process in data generation has been previously explored.

We acknowledge that the evolution process in data generation has been explored in prior works. However, we believe our contribution addresses critical gaps in these methods, particularly in the context of OR problem modeling.

Challenges in OR Data Generation

Generating high-quality datasets for OR modeling presents unique challenges that general data generation approaches fail to address:

1. Constraint Feasibility: OR problems require mathematically valid and solvable models, where variables, constraints and objectives must align. This is not considered by general-purpose data generation methods.

2. Diversity in Problem Complexity: High-quality datasets should include problems spanning varying difficulty levels to enable comprehensive model training. However, existing methods, including ORLM’s prompt-based generation, lack the ability to control problem complexity, resulting in datasets dominated by either overly simple or overly complex problems.

3. Validation Beyond Syntax: Existing generation usually adopt rule-based postprocessing methods, which can filter out formatting and syntax errors, but is inapplicable of identifying deeper logical or mathematical inconsistencies, which are critical for real-world OR applications.

Our Contributions

To overcome these challenges, our method introduces two key innovations that enhance the quality and applicability of OR datasets:

1. Iterative Problem Generation: We implement a progressive generation strategy, where problems are generated with gradually increasing complexity. This ensures a balanced dataset with problems of varying difficulty, from simple to complex. Such a approach allows us to enrich the training corpus, enabling fine-tuned LLMs to handle diverse problem formulations and adapt to a wide range of real-world scenarios.

2. Stepwise Validation Mechanism: Our framework incorporates a modular validation process designed to ensure that generated problems are not only free from syntax errors but also devoid of deeper logical and modeling inconsistencies. It verifies logical consistency by checking the coherence between constraints, objectives, and variables, ensures solver compatibility through mathematical feasibility checks, and aligns generated problems with real-world OR scenarios. Additionally, the mechanism validates the accompanying code for compatibility with solvers like Gurobi or COPT, eliminating runtime errors and structural inconsistencies. These comprehensive checks guarantee that the datasets are accurate, reliable, and applicable to practical optimization tasks.

Our framework demonstrates significant advancements over SOTA methods across multiple benchmarks. The results highlight Evo-Step-Instruct’s ability to generate high-quality datasets that enhance LLM performance in OR modeling tasks. By enabling LLMs to more effectively translate natural language problem descriptions into accurate mathematical models, we believe this targeted effort has resulted in meaningful progress towards addressing key challenges in OR problem modeling.

Weakness 2. It is recommended to include more comparative studies with SOTA LLMs and provide additional explanations to further validate the results.

Thank you for your thoughtful suggestion. To further validate our results, we conducted comparative studies on the MAMO ComplexLP dataset, involving leading proprietary LLM GPT-4o-2024-08-06 and an advanced open-source LLM Qwen2.5-72B-Instruct. These comparisons provide additional context to the effectiveness of our Evo-Step framework. The results are summarized in Table 7 below:

Table 7: Performance Comparison of Various Methods on MAMO ComplexLP

The results on the MAMO ComplexLP dataset highlight the advancements of both proprietary and open-source LLMs in OR modeling tasks. Proprietary models like GPT-4o demonstrate notable improvements, achieving a maximum accuracy of 54.03% with CoE and consistently outperforming earlier versions like GPT-4 and GPT-3.5. Similarly, open-source models such as Qwen2.5 achieve competitive results, with Reflexion reaching 47.87% and CoE achieving 51.66%. These findings indicate that open-source models are steadily narrowing the gap with proprietary counterparts, even without task-specific fine-tuning.

Despite these advancements, Evo-Step still demonstrates significant superiority, achieving the highest accuracy of 61.61% with Evo-Step-LLaMA-3-8B, surpassing GPT-4o and other baselines. Evo-Step-Mistral-7B also achieves 52.61%, further showcasing the effectiveness of our framework. These results emphasize the impact of Evo-Step’s task-specific training data in elevating model performance across diverse problem formulations.

By generating high-quality, diverse datasets, Evo-Step addresses a key challenge in structured optimization tasks: enabling LLMs to better handle complex problems. The consistent performance of Evo-Step-trained models highlights the importance of integrating precise, task-specific data into fine-tuning pipelines, paving the way for more reliable and effective solutions to real-world optimization challenges.

Question 1. How many queries are utilized during instance generation? What is the total cost? What percentage of samples are discarded during the data generation process?

The instance generation involved 64K queries, and the number of tokens was 179M. On average, each generation iteration required approximately 7.66 queries, with 3.14 queries dedicated to generating and validating the problem description, and 4.52 queries used for solution generation and validation. Of the total tokens, 39M were allocated to generating and validating the problem description, while the remaining 140M were used for solution generation and validation. Additionally, 8,400 generations were conducted, yielding 4,464 samples, meaning that 46.86% of the generated samples were discarded.

Question 2. The authors mention testing both COPT and GUROBI. It is unclear whether they use the same independent data generation pipeline and prompts for each.

In Section 4.2 Baseline, we have already mentioned that for prompt engineering methods, we tested results separately using both Gurobi and COPT, reporting only the best-performing result for each method.

For fine-tuning methods, we exclusively used the COPT solver. This choice was guided by two considerations:

1. The raw data collected for our experiments was based on COPT, making it the natural basis for all derived datasets in fine-tuning experiments.

2. Using COPT ensures a fair comparison with the ORLM method, which also adopts COPT as its solver.

To further clarify this in the manuscript, we plan to revise the Section 4.2 in the manuscript to explicitly state:

"Prompt engineering methods were evaluated independently using both COPT and Gurobi solvers, with the best result reported. Fine-tuning methods, including Evo-Step, exclusively utilized the COPT solver due to the alignment of collected raw data and to ensure a fair comparison with ORLM, which also employs COPT."

We hope this clarification resolves your concern.