OptiBench: Benchmarking Large Language Models in Optimization Modeling with Equivalence-Detection Evaluation

• ChatGPT-4o is used to propose optimization models, create .json files, and translate solver code into a detailed world problem structure. This could introduce bias when benchmarking other LLMs.
• The only detailed instance described is the Knapsack Problem. I would be very interested in seeing other examples, especially for more complex problems, and supplementary material would be appreciated.

Minor remark: There are some technical errors in the paper, such as broken or missing references and unexpectedly inserted text (e.g., in line 244).

It seems that since GPT-4o proposes the optimization models which are then filtered etc, the dataset might be biased towards problems that GPT-4o already has knowledge about. Is it possible that the dataset is not representative enough of realistic settings that GPT-4o is not performant on? It seems that might limit the applicability of the approach to represent realistic scenarios, especially those in which language-based modeling might fail due to lack of knowledge.

The graph isomorphism test is helpful, but is it possible that it misses reformulations which are equivalent but use different decision variables / constraints? For example, after presolve, the resulting problem is equivalent but has fewer decision variables and constraints. Alternatively, it may be that there are alternative formulations with additional constraints, such as specialized cuts, that improve the optimization model but which wouldn’t be considered correct by this evaluation.

Small comments:

• The text in figure 1 is quite difficult to read as the font is small
• Line 176: why is it “we leverage … “?
• Line 243 has ??, the example of the word problem
• Figure 2 has instanc

1. No datasets or results available. As a benchmark paper, it is essential to make the datasets and benchmark results publicly available to support reproducibility and usability—key criteria for evaluating the usability, effectiveness, and reliability of a benchmark.
2. Some writing issues, including undefined references (Line 243) and incorrect use of quotation marks (please use <`> and <'> in latex for quotation marks).

W1. Writing: I include some detailed comments in the Questions box. In summary, the writing and presentation quality has a ways to go and should be improved before this work is published.

W2. Lack of details on OptiBench problems: The reader is referred to appendix Table 2 for further details on the problems and applications in this benchmark. However, Table 2 only has eight LP types and eight MILP types. These are all standard textbook optimization problems. While appendix Figure 6 features a cargo loading application, it is unclear what other applications are in this benchmark. As a reader, I am left wondering how complex your problems are from a modeling perspective. This makes it impossible to assess whether LLM performance is meaningful for real-world deployment or not.

W3. Use of LLMs to generate the problems: I know that “generative AI” research heavily uses generative models such as LLMs to generate synthetic data for benchmarking purposes. However, in the operations research setting, this seems unnecessary given that the use of LLMs introduces risks of hallucination and other side effects. The authors claim that a quality control process was established using expert operations researchers that went through every LLM-generated problem and discarded spurious ones; there is not enough detail about this process to be able to trust the benchmark’s quality.

Here are 48 distinct, well-defined, well-motivated variants of TSP/VRP, for example: http://webhotel4.ruc.dk/~keld/research/LKH-3/. You can simply generate instances of these problems along with the corresponding natural language description; the papers that introduced these problems also contain further context that you can leverage. The same can be said of scheduling, for example [http://www.schedulingbenchmarks.org/] or maritime inventory routing [https://mirplib.scl.gatech.edu/home] or data center-type problems [see problems 1 and 2 in Datasets at https://www.ecole.ai/2021/ml4co-competition/].

W4. Generality of the proposed equivalence test: It seems that an important aspect of formulation equivalence is ignored in your formal definition A.1. Namely, from a strictly logical standpoint, two optimization models are said to be equivalent if their feasible regions are the same and the objective function values at feasible points are equal. For MILP, this is extremely crucial as for the same problem some formulation may be “weak” whereas another may be “strong”; this typically relates to the LP relaxation’s tightness. The textbook example for this is in Facility location: https://scipbook.readthedocs.io/en/latest/flp.html#weak-and-strong-formulations. Defining equivalence in terms of the variable-constraint graph is clearly limiting in this respect and needs to be addressed explicitly. If your test cannot handle it, that should be made clear. I believe that is the case. It is quite likely that any test for this problem will struggle with this issue.