L2P-MIP: Learning to Presolve for Mixed Integer Programming

We appreciate the reviewer for the comments on our paper. Thank you for your acknowledgment of our work. We set out below our responses to each of the questions you may concern about.

Q1: Details of the neural net training procedures.

Sorry for the inconvenience, we now add more details of experimental settings in Section 4.1. For the train/test size. We use the train/test split rate as 80%/20% for all datasets. For the easy datasets, we generate 1000 instances for each. For the two industrial datasets, they contain 1000 instances for each. For the medium and hard datasets, we directly split the instances provided by their original benchmarks. Here we list these datasets and their total number of instances: Corlat(2000) [1], MIK(100) [2], Item Placement(10000) [3], Load Balancing(10000) [3], Anonymous(118) [3]. The usage of these datasets is common in existing papers and we just follow them [4,5,6,7]. The number of epochs for training is 10,000. The time of SA to provide a solution for each instance is hours in the medium datasets and days in the hard datasets. Since the training part is totally offline, thereby we do not focus on its efficiency. In the running step, for each instance, our L2P needs less than 0.05s in the medium datasets, and less than 0.5s in the hard datasets, which is efficient enough for practical usage.

Q2: Is there a rule of thumb on estimating the size of required training data?

We appreciate your kind reminder. For the number of data for training, we directly follow the settings in existing work [4,5,6,7], without any specialized optimization. Admittedly, finding the most suitable number of training data to adequately learn is meaningful in the sense of avoiding overfitting. In our reply to your next question, we managed to conduct a series of experiments, trying to release your concerns. As far as we observed, the current need for estimating the size of required training data is not urgent in the field of ML-for-MIP, and we will continue to focus on this topic in our future work.

Q3: Ablation test about adjusting the training data.

Thank you for your valuable suggestion. We have conducted an ablation study by adjusting the training data to 20%, 40%, 60%, and 80% of the original training data. The table below shows the results of the MIRP and CorLat datasets. We plot several figures to better show the trend of L2P and other baselines in Appendix(A.9, Figure 6, page 17) of our revised pdf, please refer to it.

Ratio of training data	20%	40%	60%	80%	100%
MIRP	4.18%	7.74%	10.11%	10.87%	11.12%
CorLat	8.27%	23.17%	30.37%	33.28%	34.74%

We can see that with more training data used, L2P will perform better. But we also notice that after using 80% of the data, the performance of L2P tends to stabilize. Therefore, the review's idea of finding the suitable data size for training indeed makes sense, and we will keep following.

[1] Carla P Gomes, Willem-Jan van Hoeve, and Ashish Sabharwal. Connections in networks: A hybrid approach. In International Conference on Integration of Artificial Intelligence (AI) and Operations Research (OR) Techniques in Constraint Programming, pp. 303–307. Springer, 2008

[2] Alper Atamt¨urk. On the facets of the mixed–integer knapsack polyhedron. Mathematical Programming, 98(1):145–175, 2003

[3] Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmiela, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurIPS 2021 Competitions and Demonstrations Track, pp. 220–231. PMLR, 2022.

[4] Exact Combinatorial Optimization with Graph Convolutional Neural Networks. Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, Andrea Lodi NeurIPS 2019.

[5] Han, Qingyu, et al. "A GNN-Guided Predict-and-Search Framework for Mixed-Integer Linear Programming." The Eleventh International Conference on Learning Representations. 2022.

[6] Wang, Zhihai, et al. "Learning cut selection for mixed-integer linear programming via hierarchical sequence model." arXiv preprint arXiv:2302.00244 (2023).

[7] Li, Sirui, et al. "Learning to Configure Separators in Branch-and-Cut." arXiv preprint arXiv:2311.05650 (2023).

Thank you for your prompt reply and the acknowledgment of our rebuttal work. We also notice the results for the 'Hard: Load Balancing' and 'Hard: Anonymous'. Though our method L2P is guided by SA, L2P did not mean to be always under SA since the testing set does not overlap with the training set, and the characteristic of the neural networks could make it learn better, e.g. the generalization. Therefore, the output presolving of L2P on the testing set may be different from the results of SA, not like in the training. As the searching space of presolving is very large, there are many local optimal points. As a result, SA and L2P may find different presolving points both better than the default presolving, which is very likely to happen due to the vast search space.

For the Load Balancing dataset, actually the average improvement of L2P does not outperform SA, only in the effectiveness part. It is because in some instances the output presolving of L2P happens to improve the PD integral a little, thereby influencing the effectiveness. We use effectiveness (win rate) as a metric because it is common in existing papers, but this kind of disturbance may not be easy to avoid, in this sense, the average improvement is more stable to evaluate the performance.

As for the 'Hard: Anonymous' dataset, this dataset only has 118 instances in total and we use 20 for evaluation, and the split of train and test is provided by the benchmark, not by ourselves. It seems that 11 instances of them have space to be better presolving, and all methods tend to improve them and end up with the same 55% effectiveness. We find that SA does not perform well on this dataset since both SMAC3 and our L2P outperform it in the average improvement. Due to the large scale of this dataset, the running time of SA is tremendous and may happen to miss the better solution. The few number of instances in this dataset may enhance this fluctuation, such as SA just working poorly on these 20 testing instances other than the training instances.

As we mentioned before, the performance of SA on the testing set is not directly related to the performance of our L2P. Due to its tremendous time consumption, the performance of SA may vary and is not suitable for a baseline, as other methods (Random/SMAC3/FBAS/L2P) have a significantly small time consumption enough for practical usage. Admittedly, this phenomenon is interesting and we will try to explore it more in our future work.

Thank you for your valuable comments. We try to answer the questions and hope we can alleviate your concerns.

Q1: It would be interesting to see methods that dive deeper.

We appreciate your suggestion and we also consider that we should take a deeper look at presolving. Regrettably, we can not determine new effective methods for presolving. As far as we know, there is no prior work for using machine learning to discover new presolvers which we believe is valuable yet remains an open problem. Nevertheless, we can try to utilize our L2P to find the effective parts inside existing presolving to some extent.

In Figure 9 on page 20 of our submitted paper, we show the illustrations for improved presolving parameters of different presolvers under multiple datasets. We can see parameters of the improved presolving vary a lot, especially the priority (the order of the presolvers). Meanwhile, we can find that different problem classes show different patterns: some approximately wave-like patterns in each figure. For example, presolver 7 10 11 seem to take more responsibility, as the priority of them tends to be higher. However, in other datasets they become less important, especially in the Anonymous dataset presolver 7 is almost the last presolver to be deployed. In this sense, our L2P can manage to find the effectiveness part in presolving. Admittedly, we consider your suggestion about finding more effective presolving methods to be really important to this area, and we will keep working on it.

Q2: Use something like contrastive learning.

Thank you for your valuable suggestion. As you suggested, the non-optimal results of SA can be regarded as negative samples and the optimal results are positive samples. We did not think of this idea before since we are not familiar with contrastive learning. After our full survey of this topic, especially work from InfoNCE[1], we find one existing work[2] that shares a similar idea, where a contrastive loss is used to encourage the model the distinguish the positive/negative samples, and make predictions similar to the positive samples. We modify their contrastive loss and average it with our original loss after dynamic loss averaging. We test the performance with the variance of the ratio of contrastive loss ranging from 0.0 to 0.2, where 0.0 denotes the original L2P.

Ratio of contrastive loss	0.0	0.05	0.1	0.15	0.2
CorLat	34.74%	34.97%	32.18%	29.87%	25.59%
MIK	3.02%	2.99%	2.77%	2.52%	2.11%

We can see that the improvement of adding contrastive loss is not as significant as we expected. It may be because our dynamic loss averaging module is not designed with the consideration of an extra contrastive loss. Moreover, the outputs of SA vary a lot during the searching procedure, which may disturb the distribution of negative samples. Above all, the idea of using contrastive learning does make sense, but it may be incompatible with the current design for L2P, which needs further research and experiments to find out.

Q3: Re-evaluate based on the similarity of parameters on different instances to quickly collect data.

Thank you for your good question. As we mentioned before, the parameters of the improved presolving vary a lot even if the instances are from the same category, as shown in Figure 9 on page 20 of our submitted paper. In the MIP instances, a tiny variance of the coefficients may result in different performance. Therefore, we consider the implementation of re-evaluate is non-trivial. Actually, as the collection of data by SA is offline, the need to improve its efficiency is not urgent to us. We will keep tracking it if a scenario need such property occurs in the future.

Q4: Mistake in the table.

Sorry for the confusion. We have refined the pdf, please refer to it.

[1] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).

[2] Huang, Taoan, et al. "Searching large neighborhoods for integer linear programs with contrastive learning." International Conference on Machine Learning. PMLR, 2023.

Q4: How crucial are KRL and dynamic loss averaging?

Sorry for the confusion, we have conducted an ablation study in Section 4.4 and Figure 4 of the submitted paper. As shown in Figure 4, we remove the key components of our L2P one by one, which begins with dynamic loss averaging and KRL. As shown in the figure, removing the dynamic loss averaging and KRL will result in a nearly 5% and 10% improvement drop in CorLat, while the total improvement is merely about 35%. We are sorry that we did not emphasize enough about this ablation study. We have refined this section in our paper and please refer to it.

[1] Chen, Ziang, et al. "On Representing Linear Programs by Graph Neural Networks." ICLR 2023.

[2] Brody, Shaked, Uri Alon, and Eran Yahav. "How attentive are graph attention networks?." ICLR 2021.

[3] Shi, Yunsheng, et al. "Masked label prediction: Unified message passing model for semi-supervised classification." IJCAI 2021.

[4] Exact Combinatorial Optimization with Graph Convolutional Neural Networks Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, Andrea Lodi NeurIPS 2019.

[5] Pochet Y, Wolsey L A. Production planning by mixed integer programming. New York: Springer, 2006.

[6] Laporte G. Fifty years of vehicle routing. Transportation science, 2009, 43(4): 408-416.

[7] What's Wrong with Deep Learning in Tree Search for Combinatorial Optimization, ICLR 2022

[8] https://www.mathworks.com/help/optim/linear-programming-and-mixed-integer-linear-programming.html

Thank you for the time, thorough comments, and nice suggestions. We answer the comments and questions point-by-point, clarify the ablation experiments and discuss with the related work.

Q1: Leveraging more modern neural network architectures.

Thanks for this valuable suggestion. We did not think of this point before since GCNN is commonly used in the related work while proposing better GNN architecture is not the focus of this paper. In fact, this is also the case for the ML-for-MIP community whereby the standard GNN is largely used in literature, and there even exists papers challenging the role of GNN in solving combinatorial problems [7]. Hence how to effectively leverage GNNs for MIP is still an open problem.

On the other hand, we notice that RandomGNN [1] is a tailored approach, to add different special features to each node in the process of GNN. It can make GNN better distinguish the nodes and enhance the representation of the MIP instance. We consider this approach could be a possible improvement to existing GNN for our problem. Moreover, as the reviewer mentioned, we try to test the performance of GATConv [2] and TransformerConv [3] as drop-in replacements.

	Original L2P	L2P + RandomGNN[1]	L2P + GATConv[2]	L2P + TransformerConv[3]
CorLat	34.74%	35.11%	34.92%	35.36%
MIK	3.02%	3.08%	2.87%	3.03%

Due to the short time window for response in ICLR 2024, we first test the performances of different GNN backbones on the medium datasets CorLat and MIK. It turns out the replacements of the GNN backbone can actually improve the performance of L2P, but only to a small extent. It may be because the processing of features after GNN is more valuable in L2P. We appreciate the review for the idea of adding a drop-in replacement for the GNN backbone, and we will continue to work in this direction.

Q2: About one concurrent work.

Thanks for your reminder, we are sorry that we did not include this paper in our submitted paper, since it was recently submitted to Arxiv after our ICLR submission. We find that this concurrent work also introduces a new method utilizing reinforcement learning for presolve, while they focus on the LP domain unlike us on the MIP domain. In fact, solving MIP is different from solving LP in many aspects [8]. For example, the branch-and-bound algorithm in MIP involves solving large batches of similar problems efficiently, and the cutting-plane method in MIP adds linear inequalities to the original problem to eliminate fractional solutions. These techniques are easily influenced by the presolving of MIP, but not required by LP.

Specifically, they use RL to tackle the presolve decisions in the process of solving LP. Since the solving of MIP in our work is more complicated than LP, their method may focus on different aspects of the solvers, like the termination point which is critical in LP solvers. However, due to the inefficient sample utilization in RL, the proposed method in this concurrent work may need more time in trial-and-errors and still faces the challenges of sparse reward during training.

Compared to this work, our L2P can take full advantage of samples collected in offline training and the online testing process is more stable and efficient, which is important in practical usage in our opinion. We appreciate the reviewer for mentioning this work, and we have included it in the related work of our revised paper and are thinking of further interactions with our work in the future.

Q3: Performance about other MIP solvers such as GUROBI and CPLEX.

Thanks for your valuable question. Many commercial MIP solvers such as Gurobi and CPLEX are also equipped with kinds of presolving techniques. However, their APIs to adjust the presolving are not exposed since they are commercial solvers and not open-source, and that's why we can not test our proposed method on the commercial solvers. We consider it impossible to directly improve commercial solvers in research papers, as existing papers [4,5,6] mainly conduct experiments on open-source solvers (SCIP) like us. If we are able to cooperate with the commercial solver company in the future, we can also add our method to their internal strategy, integrating our machine learning technologies to help users better solve MIPs.

[1] Xia, Fen, et al. "Listwise approach to learning to rank: theory and algorithm." Proceedings of the 25th international conference on Machine learning. 2008.

[2] Carla P Gomes, Willem-Jan van Hoeve, and Ashish Sabharwal. Connections in networks: A hybrid approach. In International Conference on Integration of Artificial Intelligence (AI) and Operations Research (OR) Techniques in Constraint Programming, pp. 303–307. Springer, 2008

[3] Alper Atamt¨urk. On the facets of the mixed–integer knapsack polyhedron. Mathematical Programming, 98(1):145–175, 2003

[4] Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmiela, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurIPS 2021 Competitions and Demonstrations Track, pp. 220–231. PMLR, 2022.

[5] Exact Combinatorial Optimization with Graph Convolutional Neural Networks. Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, Andrea Lodi NeurIPS 2019.

[6] Han, Qingyu, et al. "A GNN-Guided Predict-and-Search Framework for Mixed-Integer Linear Programming." The Eleventh International Conference on Learning Representations. 2022.

[7] Wang, Zhihai, et al. "Learning cut selection for mixed-integer linear programming via hierarchical sequence model." arXiv preprint arXiv:2302.00244 (2023).

[8] Li, Sirui, et al. "Learning to Configure Separators in Branch-and-Cut." arXiv preprint arXiv:2311.05650 (2023).

[9] Kendall, Alex, Yarin Gal, and Roberto Cipolla. "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[10] Chen, Zhao, et al. "Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks." International conference on machine learning. PMLR, 2018.

[11] Liu, Shikun, Edward Johns, and Andrew J. Davison. "End-to-end multi-task learning with attention." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

Q4: Experimental details.

Sorry for the confusion, we have refined Section 4.1 and clarified our experimental settings. Specifically: For the easy datasets, we generate 1000 instances for each. For the two industrial datasets, they contain 1000 instances for each. For the medium and hard datasets, we directly split the instances provided by their original benchmarks. Here we list these datasets and their total number of instances: Corlat(2000) [2], MIK(100) [3], Item Placement(10000) [4], Load Balancing(10000) [4], Anonymous(118) [4]. The usage of datasets is common in existing papers and we just follow them [5,6,7,8]. The running time is shown in Q3. As for Table 4, the settings are the same as the main experiments in Table 1, including the number of instances and the metric. It was our negligence and we have improved the description of datasets (Section 4.1.1) in our revised paper.

Q5: Default parameters.

Thank you for your advice, we have placed the default parameters in Table 9, please refer to it.

Q6: Neural network representation.

Sorry for the confusion. Here the arrow in Figure 3 denotes the flow of data and how it passes through the inside modules in our framework, we did not mean to derive a network. We have refined it in its caption.

Q7: Graph embedding features.

For the GNN backbone, we just follow the GCNN[5], where the embedding is aggregated across all variables. This GNN backbone is commonly used in the literature[6][7][8] so we did not detailed introduce it in our paper, sorry for the inconvenience. Inspired by other reviewers, we add experiments for trying different GNN backbones, please refer to Q1 of the response to Reviewer zK3o.

Q8: Priority label input.

Thanks for your suggestion. During our attempts, we tested using ground truth priority as labels to see the training results of max-rounds and timing. As a result, the loss of learning max-rounds and timing goes down quicker than before. However, this approach can not be used in the real world since it needs extra ground truth priority information, which leads to an unfair comparison to the methods in our paper. Therefore, we do not regard this method as a baseline.

Q9: Dynamic loss averaging

Sorry for the confusion. We now add more details in Appendix A.5. Specifically, imbalance among multiple tasks is one of the most critical issues to be addressed in multi-task learning, many studies have attempted to balance the convergence rate of different tasks by assigning an adaptive weight to each task. [9,10,11] Inspired by them, the motivation of our Dynamic Loss Average method is to average task weighting over time via adjusting the rate of change of loss for each task. Formally, we define a loss weight: $\omega_k$ for each task: $k$ as: $\omega\_k(t):=\frac{K exp(\mathcal{R}\_k(t-1)/\mathcal{T})}{\sum_{i}{exp(\mathcal{R}_i(t-1)/\mathcal{T})}},$ $\mathcal{R}\_k(t-1)=\frac{\mathcal{L}\_{k}(t-1)}{\mathcal{L}\_{k}(t-2)}$ where $\mathcal{L}_k(t)$ denotes the loss of task $k$ at the $t$ iteration and $\mathcal{R}_k(\cdot)$ calculates the relative training convergence rate of task $k$, $\mathcal{T}$ in the softmax operator represents the temperature that controls the softness of weighting for task-specific loss. In the implementation, temperature $\mathcal{T}$ is set to 2 and $\mathcal{R}_k(t)$ is initialized as 1 when $t=1,2$, but any other effective initialization methods with prior knowledge can also be applied here.

Q10: Other mistakes in the paper.

Thanks for your kind reminder. We have further refined our paper, added the description of PD integral (mentioned by Q1), described the setting in Table 4, corrected several sentences, reorganize the sturture of Section 4 (Experiments), and improved the text of the appendix. For the potential privacy issue mentioned in Section 4.3, it is due to the policy of the company that provides us with private industrial datasets. Due to its policy, its dataset can only be used on their computer, and the code for running must also be authorized. Therefore, deploying SMAC3 and FBAS on their computers may be troublesome. In this sense, the potential privacy is more like business sensitivity.

Thank you for the time, thorough comments, and nice suggestions. We answer the comments/questions point-by-point, clarify our experiment setting and refine our paper as suggested.

Q1: Lack of details

We are sorry for the confusion. The loss functions used in our methods are ListMLE for the priority and Cross-Entropy for the max-round and timing. ListMLE [1] is a loss designed for ranking, which is suitable for learning the priority since the priority denotes the order/rank of the presolvers. We understand that placing the details of the experiment in the appendix is not recommended, therefore, we have reorganized the paper structure and enriched Section 4.1 to make our experimental settings more clear, sorry again for the inconvenience.

For the definition of PD integral, the primal-dual integral in the field of mixed integer programming (MIP) is a measure used to evaluate the performance of MIP solvers. It provides a quantitative assessment of how effectively a solver bridges the gap between the primal (the best feasible solution found) and dual (a bound on the optimal solution) bounds over the course of the solution process. For its mathematics formulations, we recommend the readers to read the documentation of one benchmark we used. The metric section of this documentation describes the calculation of PD integral with informative figures. Here we list a more detailed breakdown, and we also place it in our revised paper:

1. Primal and Dual Solutions in MIP: In mixed integer programming, a primal solution refers to a feasible solution that satisfies all the constraints of the MIP model. The dual solution, on the other hand, is related to the bounds on the optimal solution value. In linear programming, this would be equivalent to the solution of the dual problem, but in MIP, it usually refers to a bound derived from a relaxation of the integer constraints.
2. Gap Measurement: The primal-dual integral measures how the gap between the primal and dual solutions evolves over time. This gap is the difference between the objective value of the best known feasible solution (primal) and the best known bound (dual).
3. Purpose of the Primal-Dual Integral: This measure is especially useful in understanding the efficiency of a solver in converging to the optimal solution. It can highlight whether a solver quickly identifies good feasible solutions and tight bounds or whether it struggles to improve the primal and dual solutions over time.
4. Calculation: The primal-dual integral is calculated by integrating the gap over the time taken for the computation. A lower integral value indicates a more effective solver, as it shows that the solver was able to keep the primal-dual gap smaller throughout its run.
5. Use in Analysis and Benchmarking: This metric is valuable for comparing different solvers or solution strategies in mixed integer programming. It provides insights beyond just the final solution quality or computation time, focusing on the overall trajectory of the solution process.

Q2: Reporting of metrics.

Thanks for your kind reminder. Here, we consider the geometric mean may not be suitable for the task in our paper. In our learning to presolve task, we focus on the solving time/PD integral improvement compared to the default via presolving instead the solving time/PD integral itself. Therefore, we care for the average improvement, while we do not care whether the actual value of the solving time/PD integral is small or large, which already avoids the case of outliers if an instance is too simple or too hard. Moreover, the solving time/PD integral improvement of each instance could be zero or negative, while the geometric mean requires all terms to be positive.

Q3: Comparison and ablation studies.

Thanks for your suggestion. For the running time of methods, L2P/FBAS/SMAC3 shares a similar running time. For each instance, they need less than 0.05s in the medium datasets, and less than 0.5s in the hard datasets, while the SA needs hours in the medium datasets and days in the hard datasets. Here what we want to claim is that the consumption of SA is unacceptable. Since the time usages of L2P/FBAS/SMAC3 are already enough for practical usage, we consider it acceptable that our L2P does not have a significant speed advantage over baselines.

As for the ablation study of KRL, actually we have an ablation study in Section 4.4 and Figure 4 of the submitted paper. As shown in Figure 4, we remove the key components of our L2P one by one, which begins with dynamic loss averaging and KRL. As shown in the figure, removing the dynamic loss averaging and KRL will result in a nearly 5% and 10% improvement drop in CorLat, while the total improvement is merely about 35%. We are sorry that we did not emphasize enough about this ablation study. We have refined this section in our uploaded paper.

Thank you for your prompt reply and kind suggestions. Here we will describe the domain values and loss of the priority, max-round, and timing in our task, based on the official documentation of SCIP (you can search for the word "presolver" on the linked website to find related information).

• Priority: The range of priority of SCIP is [-536870912,536870911], as what we care about is the order instead of the value itself, we just let the neural network output real numbers and use the ListMLE loss to guide the order of them. ListMLE [1] is a loss designed for ranking, which is suitable for learning the priority since the priority denotes the order/rank of the presolvers. It uses a permutation probability model to map a list of scores to a probability distribution and then minimizes the negative log-likelihood loss between the predicted distribution and the ground truth distribution. We recommend the official documentation to see its mathematical formulation, implementation, and examples.
• Max-round: The range of max-round in SCIP is [-1,2147483647], while -1 denotes unlimited. For convenience, we set the max-round task as a multiclass classification in [-1, 0, 1, 2]. Though the range of max-round in SCIP is large, most default parameters are not larger than 1, since the larger values could be included by setting "-1" to some extent. As we formulate it as a multiclass classification, we use the Cross-Entropy loss, which is common in multiclass classification.
• Timing: The range of timing in SCIP is [4:FAST, 8:MEDIUM, 16:EXHAUSTIVE, 32:FINAL], which is already a multiclass classification, and we also use the Cross-Entropy loss.

Though the number of classes in max-round/timing is not large, there are 14 presolvers and the total space is at least (14!) * 4^14 * 4^14 $\approx$ 6.28e+27. Here we use 14! to represent the space of priority but it is actually much larger than this since the output priority of the neural network is continuous value and not constrained.

In our paper, we mentioned the loss functions in the experimental settings. For the domain of priority, max-round, and timing, we did not directly mention them, but you can find them in Figure 7 8 9 and Table 9, where we output the presolving parameters.

For the annotations of Figures 4 and 6, we have modified Figure 6 in the new version of our paper, but for Figure 4 we may need more time to adjust the figure due to the space limit of the main paper. We will manage to fix this as soon as possible. Sorry for the inconvenience.

[1] Xia, Fen, et al. "Listwise approach to learning to rank: theory and algorithm." Proceedings of the 25th international conference on Machine learning. 2008.