SIMULTANEOUS GENERATION AND IMPROVEMENT: A UNIFIED RL PARADIGM FOR FJSP OPTIMIZATION

• Actions (in Section 3.1) are critical, but not defined clearly. I have no problems with actions for construction heuristics, but actions for improvement heuristics are not well defined. In Section 3.2 “Insertion Position Embedding” (P5), the definition of insertion position is undefined clearly, and why the number of choices is (n+m) for each operation. Besides, it is also unclear about why the total number of insertion positions is equal to n×(n+m). For these unclear descriptions, there is no clue to understand the proposed method. Note that in Section 3.2 “Policy Model” (P6), there is no way to understand the description “Obviously, there are at most m different insertion schemes for each improvement decision.”

• Figure 2 is confusing and unconvincing. For example, why is 31 moved to the position after 11, not before 11? If it can also be moved to that before 11, I don’t see the strategy.

• The representation of operations is inconsistent and thus makes it hard to understand how the Insert Position Embedding works (There are $O_{ij}, O_j, O_{j(i)}, O_i$ in the article).

• Lack of test results for public benchmark dataset. With comparison to [29], you should also compare with la(edata) and la(rdata). And you may test on the dataset which is referenced by [29] to improve the reliability of your method.

Presentation comments:

• Lack of spaces in many places. E.g., “Both the generative model and the improvement model will use formula(4) to select the action to be executed in the current state st at step t on their respective feasible action sets.The advantage value function is fitted by a parametric MLP”

• In section 5.1, “ In addition, we also used (ref1),(ref1),”, and “mk [? ]” in Table2. Please carefully check the content.

• In Algorithm1, “E%K” => “e%K”, the second “$EP_I$” => “$EP_G$”, etc. There should be more that you need to find out for fixing.

• There is no data in some places in the tables, such as 40x10 for OR-Tools in Table1 and mk[?] for UB* in Table2.

My biggest concern is that the proposed approach seems to be only applicable to a specific scheduling problem (FJSP) with no variation in terms of constraints. In the abstract, the authors state that:

It is worth noting that this training paradigm can be readily adapted to other combinatorial optimization problems, such as the traveling salesman problem and beyond.

however, there is 1) no empirical evidence to justify the claim and 2) no explanation of how this can actually be done. For instance, how can the improvement step be applied to the traveling salesman problem (TSP), especially considering the reward function? In several combinatorial optimization problems, it is hard to define a step-wise reward function, such as in routing problems such as the TSP. Moreover, the specific design of Section 3 seems to be over-fitted to the FJSP, with new problems requiring a substantial restructuring of the model.

Another important point is that the proposed generation-improvement method is not well justified in terms of performance. There is no ablation study on just using the generator model without any improvement:

From the design of our framework, it can be seen that the generative model and the improved model can run independently, which means that we can only use the generative model to generate a complete solution or use the improved model individually to improve any feasible initial solution generated by other methods (such as random generation or PDRs).

but there is no result about this; in Table 3 only the improvement method alone is shown with other models. How would the generate only perform? Moreover, a natural question would arise, namely why authors decided to go for a potentially more burdensome generate+improve method (in which the generator may potentially be worse due to over-reliance on the improvement model), and not just a generator. In these regards, it would be interesting to see how the same model with the generation part only would do.

The experimental section seems to be lacking some baselines - for instance, Table 1 only compares against OR-Tools and dispatching rules, but not against RGA and 2SGA and the 2 DRL baselines. Also, OR-Tools is missing the solution time, so it is difficult to assess how the proposed approach compares in solution time (given that the quality is already worse than the OR-Tools metaheuristics). Finally, no standard deviation has been reported nor multiple runs.

In terms of the quality of the paper, there is room for improvement. Aside from several typos, the writing feels sloppy, and there are missing references ((ref) in the paper, mk[?] in Table 2 and more), so I would suggest some revision. More importantly, in Algorithm 1:

Store Transition $\tau_{t+1}^g :< \mathcal{S}_t; ~ \mathcal{S}_{t+1} > \text{into} ~EP_I$

I believe this should be, in fact, $EP_I$ , given that $EP_G$ is not being used here.

Finally, no code has been provided to reproduce the results.

• The current manuscript still has room for improvement, including a more detailed explanation of the training.
• The performance evaluation of the proposed framework seems quite limited, especially as the baselines are overly simplified in Table 1.

[Methods and Experiments]

• This paper doesn't provide a clear rationale or justification for the use of various embeddings. Also, there's no ablation study for these design choices.
• The method is evaluated only two public benchmarks, whereas the DRL baseline [1] has been tested on a more extensive set of benchmarks. This raises concerns about the comprehensiveness of the evaluation and potentially limits the generalizability of the proposed method's performance.
• The reported performance of DRL baseline [1] is based on the greedy selection, while the method of this paper leverages sampling for improvement steps. For fairer comparison, the results from both greedy and sampling decoding should be included. Note that sampling performance reported in [1] for v_la task is better than the proposed method, while consuming more computation time.

[Writings]
This paper has significant defects with clarity. First of all, there are too many typos, wrong spacing and inconsistent notaions. Below are some of them:

• page 1: 'PRD' → 'PDR'
• page 2: There are many wrong spacing in Sec. 2, e.g, 'O_i,which', 'operations,O_{ij}', or "...end of production.These two ..."
• page 3: There is a wrong figure reference, '(in figure)'
• page 4: 'avenger' → 'average'
• page 4: GAT has no reference
• page 5: 'avitation' → 'activation'
• page 5: 'M_{ij}' suddenly pops up, which supposedly typo of {A_I}_{ij}, and suddenly A_J is used, which is definitely a typo.
• page 7: In the Algorithm 1, 'EP_I' → 'EP_G' for the transition of generative model.
• page 7: There are several '(ref)'s in Sec 5.1, which should have been replaced by appropriate reference.
• page 8: In the text they say they use Gurobi Solver, but they report OR-Tools in the table.
• Throughout the paper, the authors use abbreviations without declare it, e.g., DRL in page 3, GAT in page 4, and GIM in page 7 (Algorithm 1)

Moreover, the models are not clearly described, which makes it hard to fully understand the algorithm.
For example, in GAT Module section in page 5, it is unclear whether W is shared among different u's or not.
Also, the reward for each model is not stated mathematically, which introduces an ambiguity.

[1] Song, Wen, et al. "Flexible job-shop scheduling via graph neural network and deep reinforcement learning." IEEE Transactions on Industrial Informatics 19.2 (2022)