LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation

We appreciate the reviewer for the valuable and constructive comments. We are grateful for your recognition of LLaMoCo as a significant contribution to the optimization domain, with compelling generalization performance across toy problems and realistic problems. Below, we provide point-by-point responses to address your remaining concerns.

[W1 & Q3, unique techinal challenges]

We would like to clarify that fine-tuning LLMs for specific domains, such as the optimization domain addressed in this paper, presents unique and domain-specific challenges. The primary contribution of LLaMoCo lies in identifying these challenges and proposing novel methodologies to overcome them. Below, we emphasize the unique technical challenges of adapting LLMs for the optimization domain, highlighting our contributions through three key aspects:

Novelty 1: LLaMoCo is the first instruction-tuning framework for adapting general LLMs as an efficient and effective optimization tool. Existing works, such as OPRO [1], primarily rely on iterative prompting of LLMs to optimize solutions. However, these approaches suffer from a) unsatisfactory optimization performance due to the limited domain-specific knowledge of general LLMs. b) inefficient inference modes, which consume an extremely large number of tokens. In contrast, LLaMoCo directly injects optimization knowledge into LLMs through instruction tuning, representing a novel and systematic approach to leveraging LLMs for optimization tasks. The subsequent novelties address the unique challenges of this process and directly target the limitations (a) and (b) of existing works.

Novelty 2: Representing and collecting the optimization knowledge (Section 3.1). We propose a stipulated and unique code-to-code description format to represent optimization problems and their corresponding effective optimizers. This universal representation format facilitates automated data collection and simplifies the adaptation of LLMs to the optimization domain. To collect sufficient optimization knowledge, we have proposed a novel optimization problem generation process which is capable of synthesizing a large number of diverse optimization problem instances, which are further utilized by our proposed automated benchmarking procedure to obtain the most effective optimizer codes—the optimization knowledge.

Novelty 3: Enhancing the training effectiveness (Section 3.2). We explicitly inject the obtained optimization knowledge into general LLMs through efficient instruction tuning. The unique challenges in this process are a) code similarity understanding issues (lines 262-269) and b) data imbalance issues (lines 299-301). We additionally introduce a contrastive warmup for aligning the code similarity, and an example-proportional mixing strategy to re-balance the training data, both of which enhances the training efficiency and stability.

Besides, we have to note that the contribution of LLaMoCo is not limited to the above novel proposals. The dataset construction, the fine-tuning strategy and all observed empirical results would provide in-depth insights and a profound impact on the future development of a combination of LLMs and optimization domain. Following your valuable suggestion, we have refined the above discussion into the related works section in the revised paper to highlight these novelties (lines 125-128, 142-145, 169-173, 181-187, colorred in blue).

[1] Chengrun Yang, et al. "Large language models as optimizers.” arXiv preprint arXiv:2309.03409, 2023.

[Q2, compare with algorithm selection methods]

We argue that comparing LLaMoCo with algorithm selection methods may not be entirely appropriate due to two key distinctions:

1. Code Generalization Advantage: Unlike algorithm selection methods, which merely choose the best algorithm from a predefined pool, LLaMoCo generates complete optimizer source codes. These codes not only specify the selected algorithm but also include the necessary implementation details, ensuring compatibility with various optimization problem descriptions. This level of generalization is crucial for real-world applications, where optimization problems often require customized code beyond the scope of standard algorithm selection.
2. Hyperparameter Tuning: LLaMoCo also performs hyperparameter tuning as part of the optimization code generation process. By tailoring hyperparameter values to the specific problem instance, LLaMoCo provides a level of configurability that algorithm selection methods cannot achieve.

Following your suggestion, we have added some discussion in Section 4.1, lines 349-354 of the revised paper (colorred in blue), where we highlight the above technical differences between LLaMoCo and Algorithm Selection methods. We kindly request the reviewer to check for the updates.

[W2 & Q4, contrastive warm-up]

We are honoured that the reviewer finds the contrastive warm-up component noteworthy enough to merit a standalone paper. However, we clarify that it is an integral part of LLaMoCo, tailored specifically for the optimization domain. In Section 3.2 (lines 262-269), we discuss two domain-specific challenges that necessitate the contrastive warm-up to facilitate the subsequent SFT.

1. In the optimization domain, the number of optimization problems is far more than the number of optimizers. Hence, multiple problems can share the same effective optimizer despite differing descriptions. Without a contrastive warm-up to align the hidden embeddings, the model struggles to generalize well across these cases.
2. Minor changes in the problem descriptions could significantly alter the optimization properties, resulting in different effective optimizers. The contrastive warm-up pulls the hidden embeddings of these similar but distinct problems apart, enhancing the model’s ability to distinguish them.

While we admit the idea of contrastive warm-up could potentially be applied to other domains with similar alignment issues, this should not be a criticism point of our paper. On the contrary, this generalizability reflects the novelty and broad impact of our proposal! Note: we have added the above discussion into the revised paper, Section 5, lines 533-538 (colorred in blue), as a promising future direction for potential practitioners.

Besides, we understand your concern that we only showcase the effectiveness of the contrastive warm-up of LLaMoCo-S on $\mathbb{I}_{eval}$ in the ablation studies (Figure 3). We have expanded these to include performance gain curves for eight more scenarios (three models—LLaMoCo-S/M/L—and three test sets from Table 1). These new results, presented in Figure 6, Appendix E.5 of the revised paper (colorred in blue), consistently demonstrate the learning enhancement through the contrastive warm-up. We kindly request the reviewer to examine these figures.

[Q1, leverage CoT & multi-turn code optimization conversations]

We clarify that under the motivation and vision of LLaMoCo, the current dataset format seems to be optimal. To be specific, the core motivation of LLaMoCo is to provide an alternative way of leveraging LLMs for the optimization domain which could achieve superior optimization performance with minimal computational resources. To this end, we design the dataset format as the prompt-answer pair, which facilitates both end-to-end training and inference. Once trained, LLaMoCo generates the desired optimization codes in a single conversation turn, directly supporting this motivation.

While we acknowledge that constructing a CoT-based training dataset could potentially enhance LLaMoCo's generalization through reasoning dynamics, this approach diverges from our primary goals and requires significantly more human efforts and costs: a) it would significantly increase token consumption during inference, and b) constructing such a dataset would require substantially more domain expertise, whereas our current dataset is fully automated, generated through benchmarking and grid search. We hope this explanation clarifies our design choice. Besides, while we acknowledge that leveraging multi-turn code optimization might potentially enhance the performance of the generated optimizers, it comes with notable drawbacks. First, it requires users to provide additional feedback and guidance to the LLMs, which demands a certain level of expertise. Second, multi-turn code optimization consumes significantly more computational resources (e.g., token usage), making it far less efficient and economical compared to LLaMoCo's single-turn approach.

We thank the reviewer for such valuable comments. We have added the core part of the above discussion into the revised paper, Section 5, lines 533-538 (colorred in blue), as promising directions for further development.

We appreciate the reviewer for the valuable comments. Thank you for recognizing our work as pioneering, the superior performance of LLaMoCo compared to the GPT-4 model in solving optimization problems, and the systematic construction of our training dataset. To address your remaining concerns, we provide the following point-by-point responses.

[W1, add zero-shot evaluation]

We understand the reviewer's concern regarding the importance of testing zero-shot evaluation on a large dataset. However, we believe there may be a misunderstanding. To clarify, the first six problems in Table 3 represent individual engineering problem instances, while the latter two—HPO-B and Protein-Docking—are extensive problem collections containing hundreds of instances. For example, the Protein-Docking collection consists of diverse protein-protein complexes with varying structures, each presenting a challenging optimization landscape. In total, the number of tested problem instances amounts to 6 + 128 + 128 = 262 rather than 8.

To further address your concern, we conducted additional testing of our trained model on a new realistic problem collection derived from the first six problems in Table 3. This collection, proposed by Kumar et al. [1], consists of 57 real-world constrained optimization problems sourced from a diverse range of engineering scenarios. The comparison results (averaged across all 57 problems) between our LLaMoCo and other baselines are presented in the following table:

The results further validate the effectiveness and superior performance of our LLaMoCo. We have added corresponding text content in lines 431-433 and the above results and discussion in Appendix E.2.

[1] Kumar, Abhishek, et al. "A test-suite of non-convex constrained optimization problems from the real-world and some baseline results." Swarm and Evolutionary Computation 56 (2020): 100693.

[W2, add GPT-4 vector search baseline]

We have conducted a baseline experiment based on our understanding of “GPT-4 vector search.” Specifically, we utilized the GPT-4 vector embedding system to generate vectorized embeddings for all prompts in our training dataset. During testing, the tested prompt was also processed through the GPT-4 vector embedding system to obtain its vectorized embedding. We then identified the most similar prompt in the training dataset by calculating the L-2 distance between the embeddings. Finally, the tested prompt, along with the most similar prompt and its corresponding answer from the training dataset, were fed into the GPT-4 model to generate an optimizer code. In this setup, the most similar prompt and its corresponding answer serve as an example of in-context learning. We present the comparison results of this GPT-4 vector search baseline, GPT-4 baseline and our LLaMoCo-L on the test set $\mathbb{I}_{eval}$ in the following table:

Two key observations can be made: a) Providing GPT-4 with an example prompt-answer pair similar to the tested prompt significantly reduces the error rate of the generated optimizer code. b) However, this prompting strategy consumes twice as many tokens as directly prompting GPT-4, making it inefficient—especially when compared to LLaMoCo, which requires only 2.4k tokens to achieve superior optimization performance. This highlights the importance of LLaMoCo in efficiently adapting LLMs to solve optimization problems. We have added the above results and discussion in Section 4.4, Table 4 of the revised paper (colorred in blue).

[W3 & Q2, grid search criteria and correctness]

1. grid search criteria

   We kindly refer the reviewer to Section 3.1, lines 234-240 and Appendix A.2 for all details of our benchmarking criteria. In summary, we have selected 23 representative optimizer commonly used to solve different optimization problems and then applied a two-step procedure to identify the most effective optimizer for each problem instance in our instruction tuning set. Step 1: according to the configuration space we provided in Table 6 of the Appendix, we identify the best-performing configurations of the 23 optimizers for the given problem instances. Step 2: The optimizer that achieves the highest optimization performance by using the found best-performing configuration is labelled as the most effective optimizer for that problem instance.

2. grid search correctness

   Following your suggestions about the grid search granularity we chose (as shown in Table 6 of the Appendix). We conduct two additional benchmarking processes, with half and double granularity of our original setting. For example, if a hyper-parameter holds four optional values in our setting: [0.2, 0.4, 0.6, 0.8], half granularity denotes [0.2, 0.8], double granularity denotes [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], etc. We present the averaged optimization performance of the most effective optimizers on our problem set searched by these two granularities, normalized by our original granularity’s performance, as well as the averaged searching wall time for one problem instance in the following table:

   	half	our setting	double
   performance	71.793%	1	102.344%
   wall time	6s	211s	6379s

   The results reveal an evident tradeoff between the searching effectiveness and the searching efficiency of different grid search granularities. The searching wall time increases exponentially since there are 4-5 hyper-parameters in an optimizer. However, the performance improvement obtained by spending so much computational resources is only 2.344%. We believe this result could validate the selection appropriateness of our grid search granularity. We have added the above discussion in Appendix E.4 of the revised paper (colorred in blue). We also have added some text content in lines 239-240 (colorred in blue) of the revised paper to guide readers for this discussion.

[Q1, combined effect of SFT and contrastive alignment]

We understand your concern since we only showcase the performance gain curve of our LLaMoCo-S on the left of Figure 3. We have added the same performance gain curves of the other eight scenarios (three models LLaMoCo-S/M/L and three test sets in Table 1). These curves are now presented in Figure 6, Appendix E.5 of the revised paper, where we can observe consistent learning enhancement by introducing our proposed contrastive learning warmup. We kindly request the reviewer to examine these figures. We have also added some text content in lines 475-476 of the revised paper to guide readers to review these results.

Dear reviewer #ugM1:

Since the discussion period has been extended, we respectifully request your feedback on our responses. In these responses, we have conducted additional experiments and provided in-depth discussions to address your concerns. If there are any further concerns, we are eager to continue this discussion and address them. We look forward to hearing from you.

Best regards, the authors

There are $N_p = 2570$ problem instances in the testing set $\mathbb{I}_{eval}$ (We have constructed 32570 problems, 30000 of them are used for the instruction tuning, the rest are used for testing). For each problem $p_i$ in the testing set, we feed its prompt to LLaMoCo and obtain the generated optimizer program. We then run the program to optimize $p_i$. There are three cases to compute the performance of LLaMoCo on $p_i$, which we denote as $Perf_i$:

1. If the generated optimizer code is executable with no errors, we run the program to optimize $p_i$ for 5 independent runs. Then the performance is calculated as $Perf_i = \frac{1}{5} \sum_{j=1}^{5} \frac{f_{i,j}^* - f_i^*}{f_{i,j}^0 - f_i^*}$. Where $f_{i,j}^0$ is the initial objective value in j-th run, $f_{i,j}^*$ is the best objective value found by the generated optimizer code, $f_i^*$ is an approximation of the optimum of $p_i$ which we obtain from the large scale benchmarking.
2. If the errors can not be resolved within one turn of debug conversation, we set $Perf_i$ as 0.
3. If there are runtime errors, we prompt LLMs to debug this program. If the errors are resolved within one turn of debug conversation, we calculate a recovery cost $r_i = \frac{L_{err}^i}{L^i}$ as the proportion of lines in the generated codes that need to be revised, where $L_{err}$ denotes the number of error lines, $L$ denotes the total number of lines. We then run the revised program to optimize $p_i$ for 5 independent runs and calculated the performance as $Perf_i = (1-r_i) \times \frac{1}{5} \sum_{j=1}^{5} \frac{f_{i,j}^* - f_i^*}{f_{i,j}^0 - f_i^*}$ . We give a punishment term to the normalized performance to reflect the errors raised in the generated optimizer code.

We provide the exact formula of this performance metric as below:

At last, the performance of LLaMoCo on testing set is the average across all problem instances:

We thank the reviewer for this valuable comments, we will update the Appendix D to make the metric calculation more clear for our readers.

[Q1, impact of dataset size]

We agree with the reviewer that an in-depth analysis of LLaMoCo’s scaling law would significantly enhance the impact of this work, considering both dataset size and model size. To this end, we selected the CodeGen family (350M, 1B, 3B, 7B) as the backbone model for LLaMoCo and conducted experiments with four training sets of varying sizes (1k, 5k, 15k, 30k). The optimization performance of these 16 trained models on the test set $\mathbb{I}_{eval}$ is presented in the following table:

The results above yield several key observations: a) When the dataset size is very small (1k), increasing the model size does not result in performance gains, likely due to overfitting. b) For all model sizes, increasing the dataset size consistently improves performance. c) In summary, both model size and dataset size play crucial roles in determining the final performance of LLaMoCo. We have added the above results and discussion in lines 380-382 and Appendix E.3 of the revised paper (colorred in blue).

[Q2, data quality control]

The quality of the dataset is ensured by including only solvable and non-trivial problems. Specifically, for unconstrained problems, the composition and hybrid construction of base functions follow the procedure outlined in the IEEE CEC 2021 Single-Objective Competition, where the optimum and the optimal objective value of the constructed function are analytically derived. By applying rotation and shifting to the optimum, we modify the optimization landscape, ensuring the solution remains non-trivial. For constrained problems, we further validate solvability by running specialized optimizers from our algorithm pool on the constructed problem instances. These optimizers are executed multiple times (50 runs) to confirm the absence of constraint conflicts, ensuring that each problem instance is solvable. We have included this discussion in the Appendix A.3 of the revised paper (colorred in blue).

[Q3, computational overhead]

The increased token consumption by LLaMoCo can be attributed to two key factors: a) The baseline models we compare against tend to output very simple optimizers and sometimes incomplete codes due to their limited optimization knowledge. In contrast, LLaMoCo often generates more complex optimizer codes to achieve superior optimization performance. b) LLaMoCo includes user-friendly comments above each line of the generated code to help users understand and customize the content as needed, enhancing its flexibility. We kindly invite the reviewer to refer to Figures 7, 8, and 9 in Appendix F for a detailed examination of these two aspects. Nevertheless, as shown in Table 1, the increased token consumption in LLaMoCo is much less than the existing prompt for solution works such as OPRO (2.4k v.s. 115k) and prompt for optimizer works such as GPT-4 Turbo (2.4k v.s. 3.5k), which underscores the effectiveness and efficiency of LLaMoCo.

We hope the above responses could enhance your confidence in our work.

We appreciate the reviewer for acknowledging our LLaMoCo as the first complete framework making a significant contribution to the optimization community. We are pleased that the reviewer finds our paper well-described and effective, with solid experimental results demonstrating both generalization and effectiveness. We hope the following point-by-point responses address the remaining concerns.

[W1, novelty]

We appreciate the reviewer’s concern and would like to clarify the unique innovations in LLaMoCo that set it apart from the referenced works [1], [2], and [3]:

1. We emphasize that while the dataset construction incorporates the synthetic process outlined in [1], we significantly augment this process by carefully introducing random constraints collected from extensive convex optimization literature (as detailed in lines 200–202 and 213-215). This data augmentation ensures that the final dataset includes not only the unconstrained problems from [1] but also a diverse and novel set of constrained optimization problems. This enhancement plays a critical role in improving the generalization performance of LLaMoCo. It also provides valuable insights and inspiration for designing effective training datasets to fine-tune LLMs for potential other optimization tasks in future work.
2. we would clarify the difference between the contrastive learning introduced in LLaMoCo and the approach used in Unixcoder [2]. The contrastive learning in Unixcoder involves two components: a) aligning representations of different modalities by aligning the hidden dropout masks, and b) aligning code fragments with corresponding comments. However, neither of these components is similar to the contrastive learning approach in LLaMoCo. As described in Section 3.2 (lines 262–269), LLaMoCo addresses a novel language alignment task with several unique challenges specific to the optimization field, including a) different prompts (optimization problems) often share similar solutions (optimizers), and b) similar prompts require different solutions. To address them, our proposed contrastive learning warmup aligns the hidden vectors in the final self-attention block of decoder-only LLMs, rather than aligning hidden dropout masks as in Unixcoder. By our efficient contrastive warmup (only 5 epochs), the learning effectiveness of the subsequent SFT process is significantly improved as shown in Figure 3.
3. We argue that the tasks in [3], which focus on daily conversations (a relatively simple domain), are significantly less complex compared to the optimization tasks addressed in LLaMoCo, which are challenging even for human experts. In optimization tasks, constructing well-defined problem descriptions is significantly more challenging than handling daily conversations. To address this, we introduced a templated problem construction process and designed tailored prompt descriptions. Moreover, labelling optimization problems requires expert-level knowledge to identify and fine-tune well-performing optimizers and to write the corresponding code. To ensure robustness, we conducted large-scale benchmarking and grid searches to determine competitive hyperparameters. Additionally, the unique cross-modal challenges described above necessitate an effective learning paradigm, which motivated the development of our contrastive learning warmup method.

We will refine the introduction and methodology sections to highlight these discussions. We thank the reviewer for the valuable suggestions that strengthen our paper and enhance the discussion.

[W2, refining figures and writing]

We have addressed the typos you mentioned in the revised paper. Regarding Figure 1, we have updated it in the revised paper, where we illustrate all sub-components of the instruction tuning process including the dataset construction, benchmarking, and the two-phase fine-tuning in the figure to show the technical novelty of LLaMoCo compared with existing works.

Dear reviewer #1QNj:

Since the discussion period has been extended, we respectifully request your feedback on our responses. In these responses, we have conducted additional experiments and provided in-depth discussions to address your concerns. If there are any further concerns, we are eager to continue this discussion and address them. We look forward to hearing from you.

Best regards, the authors

We appreciate the reviewer for recognizing our LLaMoCo as a significant contribution to adapting LLMs for optimization code generation, supported by our dataset as a valuable resource for future work, an innovative training methodology, and a comprehensive experimental analysis. We hope the following responses address the remaining concerns effectively.

[W1 & Q1, model size & performance discrepancy]

We appreciate the observation! Firstly, we would like to clarify that larger models generally outperform smaller ones across Tables 1 to 3, indicating no overfitting under the given data scale. We acknowledge one exception: in Table 1, the medium model (LLaMoCo-M) outperforms the large model (LLaMoCo-L) on unconstrained optimization problems. This discrepancy likely stems from differences in the base models used during fine-tuning. Specifically, LLaMoCo-S, LLaMoCo-M, and LLaMoCo-L are fine-tuned on CodeGen-Mono (350M), Phi-2 (2.7B), and Code Llama (7B), respectively, demonstrating the generalizability of LLaMoCo across various backbone models. These base model differences may contribute to the observed performance variation.

Notably, we actually have another table used to validate the scaling law of our LLaMoCo. Specifically, the experiment presented in Table 2 uses the CodeGen-Mono model series (ranging from 350M to 7B) serving as the base for LLaMoCo. The results indeed demonstrate that performance improves as model size increases, confirming that larger models achieve significant performance gains under our data scale. We present this table below for your convenience:

Moreover, we would like to clarify that while the results in the table may appear similar, they are based on a normalized performance metric (detailed in Appendix D), which scales optimization performance by the objective value range of the target problem instance. These ranges can be extremely large (e.g., $10^{20}$), causing performance differences to appear smaller than they are. For instance, in Table 1, the small model (LLaMoCo-S) achieves a performance of 81.843%, which seems "similar" to the medium model (LLaMoCo-M) at 83.369%. However, this 1.5% difference reflects a significant performance gap in absolute terms. The reason we normalize the optimization performance is that given various objective value scales in different problem instances, it is unreasonable to compute the absolute average final objective value across problem instances. Additionally, we believe that such normalization improves the table readability of the results.

[Q2, add more baselines]

Thank you for the suggestion! Following your suggestion, we have tested three additional OpenAI models: GPT-4o, o1-mini and o1-preview, on our test dataset $\mathbb{I}_{eval}$. Below, we provide a table comparing their performance with the GPT-4 Turbo baseline and our LLaMoCo mdoels:

From the results, we observe the following:

1. o1-mini/preview v.s. GPT-4o: The o1 models achieve significantly lower coding errors compared to the GPT-4o model, demonstrating their robust coding enhancement capabilities.
2. o1-mini v.s. LLaMoCo: On one hand, the error rate of o1-mini is lower than that of our LLaMoCo, primarily due to o1-mini's black-box training on an extremely large coding task. On the other hand, our LLaMoCo, despite being trained on a much smaller model, achieves greater optimization performance gains while consuming fewer tokens. Furthermore, we analyzed the source codes generated by o1-mini, as we did for the GPT-4 Turbo model in Section 4.4. It was found that o1-mini also tends to generate a specific optimizer, the DE algorithm, for nearly all tested problems. This observation reinforces the core motivation behind LLaMoCo, which is to explore how domain-specific knowledge can be effectively injected into large language models to adapt them for specialized scientific tasks.

We have updated the above discussion into our revised paper, Section 4.4 (colorred in blue). We kindly request the reviewer to examine it.

[Q3, out-of-distribution evaluation]

We would like to clarify that the out-of-distribution evaluation has already been conducted and is presented in Table 3. The tested problems in this table come from diverse realistic scenarios with significantly different problem structures compared with the synthetic problems we used for training. The results demonstrate two key points:

1. The LLMs fine-tuned by LLaMoCo exhibit superior generalization performance to other baselines on the out-of-distribution tasks.
2. The incremental performance trend among the three LLaMoCo models (S, M, L) in Table 3 is consistent with the trend in the in-distribution evaluation in Table 1.

Dear reviewer #mqqR, since the discussion period is extended, we respectifully request you to check the experimental restults and discussion we have added following your constructive suggestions. We look forward to your futher feedback to help us improve this paper. We are open to any suggestions from you. Thanks for your precious time!