OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling

Thank you for the detailed feedback. We address the specific questions:

Regarding Claims And Evidence:

• Problem Length vs. Complexity: We fully agree that scenario understanding is crucial for LLMs. However, more complex scenarios naturally require longer, more detailed NL descriptions (e.g., ROADEF 2012 [1]), challenging both LLM abstract reasoning and long-context processing. Thus, length reflects certain scenario complexity. Our paper's results on MAMO (Fig 3, Table 1) show a correlation between scale/length and reduced accuracy.
• Scenario/Problem Type Coverage: Our seed dataset features over 50 expert-curated problem classes, which we believe provides substantial and representative coverage. As detailed in Appendix A.2, each class is grounded in referenced literature and reflects practical application scenarios. While methods like the evolutionary approach in Li et al. [2] can generate synthetic variations, they often start from a limited set of initial classes (reportedly 8 in that case), whereas our foundation of 50+ manually curated classes is significantly more extensive. Moreover, such synthetic generation typically recombines existing elements rather than creating fundamentally new problem types. We are confident in the breadth of our 50+ classes; further expansion could effectively build upon this rich seed set using augmentation or evolution.

Regarding Methods And Evaluation Criteria:

• Scalability for Massive Instances: Our current pipeline already handles considerable scales effectively (up to ~25k characters, Fig 2). However, for the ultra-large instances (>10MB) you mentioned, direct LLM processing is indeed problematic due to context limits. To address this, our framework can adapt using a Model-Data Separation Format (similar to OptiBench [3]):
  • Format: Instance = (NL Description + Structured Data File). NL has scenario/data needs; Data File has numerics.
  • OptMATH Adaptation: Generate (NL + Data File) pairs. Train AutoFormulator to generate code reading the Data File, validating via OV or bipartite graph check. The (MF, NL, PD) interpretation shifts: NL includes scenario + data ref; PD is the final solver-ready format. This is a promising direction for ultra-large scales.

Regarding Experimental Designs Or Analyses:

• Missing Results & Harder Datasets: We added the requested Optimus results on MAMO/OptMATH Bench and other benchmarks. We also added IndustryOR and OptiBench results. See Table on the response to Reviewer mCmd.
• Regarding Chain-of-Experts and ComplexOR: We do not show the results for Chain-of-Experts (CoEs) [5] and ComplexOR. The input for the CoEs requires each instance to include a structured code template and detailed descriptions of its parameters. Additionally, the ComplexOR dataset format is not suitable for end-to-end modeling tests, as its data format for each problem includes a natural language description without numerics.

Regarding Essential References Not Discussed:

• We agree [4] is pertinent and will add/discuss it in revised Sec 1. While both explore reverse synthesis, OptMATH uniquely emphasizes:
  • Generator-based Scalability: Abstract MF + parameterized generators enable large-scale, diverse PD creation (Sec 3, Alg 1).
  • Rigorous Semantic Validation: Strict Optimal Value (OV) equivalence check (Sec 4.3) ensures high NL-PD correctness (99.6% manual accuracy), beyond just code executability.

Regarding Weaknesses:

• Applicability to Llama: We fine-tuned Llama 3.1 8B on OptMATH-Train, showing applicability. Table of the response to Reviewer mCmd shows significant improvements over baseline Llama. Performances of checkpoints are uploaded in [6] named llama_checkpoints_perf.

Regarding Questions For Authors:

• Data Generation Efficiency: Uses feedback-driven tuning (Alg 1) for controllable quality. PD generation success ~50% (see [6], rate_after_feedback_tuning figure). Rejection sampling (OV matching validation) acceptance is 62.14% (Fig 14: T=1 efficient). 200k valid (NL, PD) samples cost ~$1914 (13.35B tokens, DeepSeek-V3, promotional period), much cheaper than manual creation.
• Number of Constraints/Variables: OptMATH-Train covers wide range: up to ~2500 vars / ~1800 cons, including complex instances. Please refer to optmath_train_under500, optmath_train_linear, optmath_train_log, benchmarks_distribution, benchmarks_under100, and benchmarks_box_plot figures in [6] for a more detailed description.

---

References:

[1] ROADEF Challenge 2012 Subject.

[2] Towards foundation models for mixed integer linear programming.

[3] OptiBench: A Large Language Model Benchmark for Optimization Problem Understanding and Formulation.

[4] OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling.

[5] Chain-of-Experts: When LLMs meet complex operations research problems.

[6] https://anonymous.4open.science/r/OptMATH-Rebuttal-8F5E

Thank you for the detailed feedback. We address the specific questions raised:

1. Regarding Questions 1 & 4: OptMATH-Bench Curation, Distinctiveness, and In-Domain Evaluation

We clarify OptMATH-Bench's curation and distinction from OptMATH-Train to address concerns about evaluating only in-domain capabilities:

• Dual Curation Pathways: OptMATH-Bench was created via two distinct routes. Pathway 1 started with instances rejected by our AutoFormulator (failed OV check), indicating initial difficulty. An "LLM-Committee" (inspired by [1], using diverse powerful models like GPT-4, Claude, Gemini, DeepSeek) then filtered these: (PD, NL) pairs were retained only if at least one and at most two committee members successfully formulated them (passed OV check). This isolated well-posed but non-trivial modeling challenges. Crucially, human OR experts subsequently validated the correctness of these selected pairs and further refined them based on relevance and clarity. Pathway 2 involved experts directly curating challenging problems from external OR literature (journals, textbooks), ensuring methodological/source independence and including known hard problem types (e.g., NLP, SOCP - see Figure 4).
• Addressing "In-Domain" Concern: This dual approach ensures distinction. Pathway 2 uses external sources/methods. Pathway 1 involves significant expert validation. Even if underlying PD distributions overlap, the distribution of (PD, NL) pairs in OptMATH-Bench is fundamentally different due to curation. The LLM-Committee filtering specifically ensures the benchmark represents the hard tail of this paired distribution relative to strong LLM capabilities. The superior performance of our finetuned model on newly added IndustryOR and OptiBench benchmarks (see Table of the response to Reviewer mCmd) further contradicts a purely in-domain evaluation and shows generalization ability.

Proposed Revision: We will revise Section 4.3/6.1 to detail these distinct curation pathways, emphasizing expert roles, external sourcing, the LLM-committee filtering targeting the (PD, NL) hard tail, and problem diversity.

2. Regarding Question 2: Controllable Difficulty Mechanism and Seed Data Validation

We clarify the difficulty control and validation, addressing a misunderstanding about generator training:

• Generator Usage Clarification: We must correct a misunderstanding: we do not train generators. We use pre-defined, parameterized code generators implementing standard MFs from OR literature (e.g., Bin Packing [2], Appendix E.6). Our novelty is controlling their input parameters.
• Feedback-Driven Parameter Tuning (Alg. 1): Difficulty is controlled via an iterative LLM feedback loop (inspired by [3]). The LLM suggests/refines generator parameters based on evaluation feedback (complexity score S(PD), solve time, feasibility) from generated instance batches, steering output towards targets.
• Further Validation: AutoFormulator training uses curriculum learning. Seed MFs/generators are expert-curated from literature (Appendix A.2) to ensure the reasonableness and correctness. PDs are validated for feasibility/solvability (Alg. 1) and OV equivalence (Alg. 2).

Proposed Revision: We will revise Sections 3, 5.2, Appendix A/B to detail the LLM-driven parameter tuning (not generator training), curriculum learning, seed curation, and validation.

3. Regarding Question 3: Use of Benchmarks for Seed Inspiration and Experimental Fairness

We clarify the use of solving benchmarks like MIPLIB and address fairness:

• Solving Benchmarks as Inspiration Only: These were used solely to identify representative OR problem classes (e.g., TSP, Job Shop). Their specific solver-focused PD (lacking NL) were not reused.
• Independent Generator Development: We built new parameterized generators from scratch based on MFs from classic OR literature for each identified class (e.g., Job Shop [4], Appendix A.2).
• Distinction Justifies Fairness: The clear separation – using solver benchmarks only for class inspiration, building new generators from literature, not reusing instances, and testing on different types of benchmarks (Optimization Modeling vs. Solving) – ensures our setup is reasonable and fair. We argue that this strikingly contradicts hacking the modeling benchmarks. We do not use any benchmarks directly.

Proposed Revision: We will revise Section 3 and Appendix A.2 to explicitly state this distinction and the generator development process based on literature.

---

References:

[1] Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions.

[2] Analysis and design of algorithms in combinatorial optimization.

[3] Large language models as optimizers.

[4] The shifting bottleneck procedure for job shop scheduling.

Thank you for your valuable feedback. We appreciate the suggestions for improving the clarity and scope of our work. We address each point below:

Regarding Weakness 1: We will revise the 'Our Contributions' section to more explicitly contrast OptMATH with prior prompting-based methods. Key differentiators that will be emphasized include:

• Specialized & Efficient Models: Fine-tuning on OptMATH-Train yields specialized AutoFormulator models demonstrating superior performance. Additionally, complex, multi-step prompting pipelines (e.g., multi-turn interaction) can sometimes confuse models on simpler tasks, potentially degrading performance relative to complex ones (see Table, OptiMUS on MAMO EasyLP). Furthermore, fine-tuning yields models with low inference costs suitable for large-scale deployment, unlike prompt-based methods requiring continuous, expensive calls to powerful foundation model APIs.
• Beyond Prompting: Unlike methods relying solely on base LLM capabilities via prompting, OptMATH focuses on fine-tuning models using a large-scale, high-quality dataset synthesized specifically for optimization modeling. This fine-tuning approach is complementary to prompt engineering techniques, offering a path to enhance foundational model capabilities rather than just leveraging existing ones.
• Scalable High-Quality Data Generation: Our bidirectional framework systematically generates vast amounts of verified (NL, MF, PD) triplets, addressing the data scarcity that limits fine-tuning approaches.
• Rigorous Semantic Validation: We employ Optimal Value (OV) based rejection sampling, ensuring semantic consistency between the NL description and the problem data, which is more rigorous than checks for mere code executability often used implicitly by prompting methods.
• Controllable Complexity & Diversity: The framework allows generating problem data with targeted difficulty via feedback loops and incorporates diverse problem types.

Regarding Weaknesses 2&3: We now include results on the IndustryOR [1] and OptiBench [2] benchmarks. Our Qwen2.5-32B model, finetuned on the OptMATH-Train dataset, demonstrates consistently superior performance, achieving results comparable to GPT-4 and DeepSeek-V3. Results for OptiMUS, Llama, and ORLM on these benchmarks are also presented. Furthermore, to illustrate the impact of finetuning, results for the baseline Llama and Qwen models (before finetuning) have been added. This updated table provides a comprehensive comparative analysis. Due to time constraints, the reproduced ORLM results were neither tuned nor run multiple times, which explains the drop in performance.

Regarding Weaknesses 4: We agree this is important. The comprehensive methodology, criteria, and structured organization for our seed data generation—including how we utilize benchmark problem structures (e.g., from MIPLIB) to create validated, parameterized instance generators and associated metadata—are detailed in Appendix A.2 (Seed Classes). Recognizing the value of highlighting this in the main text, we will revise Section 3 to include a concise summary of this systematic process and clearly reference Appendix A.2 for the full description.

---

References:

[1] ORLM: Training large language models for optimization modeling.

[2] OptiBench meets ReSocratic: Measure and improve LLMs for optimization modeling.

Thank you for your thoughtful review on the potential areas for improvement. We appreciate the opportunity to address these points:

Essential References Not Discussed: See “Essential References Not Discussed” section of response to Reviewer PCa3

Regarding Weakness 1: Our framework directly addresses this via the rejection sampling mechanism detailed in Section 4.3 and Algorithm 2. Each generated NL description is translated back into problem data (PD') using our AutoFormulator. We then rigorously validate this translation by comparing the optimal objective value (OV') obtained from solving PD' against the optimal value (OV) from the original problem data (PD). Only pairs where OV' equals OV are accepted into OptMATH-Train. Consequently, this process can process NL descriptions that seem to be complex or ambiguous accurately, ensuring that all included data points feature corresponding NL and PD that are demonstrably modelable by an LLM.

Regarding Weakness 2: Please refer to the "Addressing 'In-Domain' Concern" and "Distinction Justifies Fairness" sections of the response to Reviewer LZQX and the table in the response to Reviewer mCmd, where we add new benchmarks. These arguments clearly show that the benchmarks, even OptMATH-Bench, originating from the same PD distribution, do not overlap with OptMATH-Train. These results show the models finetuned on OptMATH-Train generalize to entirely new domains or problem types beyond the curated seed data.

Regarding Weakness 3: Please refer to the "Scalability for Massive Instances" a "Data Generation Efficiency" sections of the response to Reviewer PCa3. Our pipeline is efficient and can easily adapt the data-model separation paradigm to process extremely complex optimization problems.

Regarding Weakness 4: Please refer to the "Scenario/Problem Type Coverage" section of the response to Reviewer PCa3.

Regarding Weakness 5: In Appendix A.2 and A.3, we have given a detailed description of the diversity of seed data and OptMATH-Train. For additional metrics, please refer to the table in the response to Reviewer mCmd. We evaluate the performance of finetuned models on IndustryOR and OptiBench. We also reported the results using pass@8; the higher performance at pass@8 compared to pass@1 indicates a high upper bound on the modeling capability of the finetuned models, signifying that models trained on OptMATH-Train attain a high ceiling for their modeling skills.

Regarding Weakness 6: We agree that this is a critical aspect and will revise the paper to elaborate on our methodology. To clarify, our process is largely automated: we begin with core problem structures inspired by benchmarks like MIPLIB (e.g., TSP, JSS variants) and utilize parameterized instance generators designed around these structures. For instance, our Job Shop Scheduling generator accepts parameters like job/machine counts and operation details to automatically create problem data (PD). Crucially, these generated PD are validated for solvability (e.g., using Gurobi feasibility checks) before being included in our seed set and subsequently translated into natural language by LLMs. To further increase automation and reduce human biases, we can easily adapt the methods in addressing weakness 4 and reduce the amount of seed data to merely 8 generators.

Regarding Weakness 7: Our empirical validation is crucial here: as reported in Section 4.3, manual checks confirmed that our OV-based rejection sampling achieves a 99.6% accuracy rate in capturing the correct problem semantics and ensuring practical equivalence. We find this level of accuracy to be highly effective and practically sufficient for ensuring dataset quality for LLM training. Alternatively, checking graph isomorphism for LP problem structures, inspired by related work [1], could be used for stricter MF equivalence; however, we opted for the OV check due to its operational simplicity and broad applicability across problem types within our framework, ensuring a practical and scalable validation approach.

Regarding Weakness 8: To proactively enhance scalability and mitigate this effect, our framework incorporates specific strategies aimed at maximizing data diversity within OptMATH-Train. As detailed in Section 5.1, we employ extensive data augmentation techniques to generate more varied and non-standard problem instances. Additionally, during the forward modeling phase [Section 4.2], we utilize diverse Chain-of-Thought (CoT) prompting strategies to capture multiple valid reasoning paths and formulation variants. We believe that enriching the training data with such diversity helps elicit stronger performance improvements even on larger models, thereby counteracting the diminishing returns trend.

---

References

[1] OptiBench: A Large Language Model Benchmark for Optimization Problem Understanding and Formulation.