Order-Preserving GFlowNets

1. Related work. Please see the Common Response: Related Works.

2. RL vs OP-GFNs. Please see the Common Response: Extensive Experiments & RL vs OP-GFNs.

3. More baselines. We think we mainly use the HyperGrid for sanity checking instead of sweeping experiments to compare performances, so we only consider TB here. Later, in the NAS environment, we have implemented and compared OP-TB with various baselines, including FM, DB, subTB. Please see Section E.4.5. To conclude, OP has significant improvements on all the methods. We are currently running additional experiments for the molecular design environments, which we will add as new baselines to the updated manuscript. Please see the Common Response: Extensive Experiments.

4. $R_0$ and exploration-exploitation. Sorry for the confusion. What we wanted to explain is that a high sparsity in the reward function is not good for GFNs to explore different modes. Therefore, small $R_0$ is a harder case for traditional GFN methods to discover the mode on the upper-right corner, see Figure E.4 for details. However, since the OP-GFN only focuses on the ordering, it is independent of $R_0$. We change the term "encourages" to "facilitates".

5. Advanced baselines in NAS. Our OP-GFNs are multi-trial sampling methods. In the NAS environments, we have compared our results against three classes multi-trial sampling methods. 1) evolutionary strategy, e.g., REA; 2) reinforcement learning (RL)-based methods, e.g., REINFORCE, 3) HPO methods, e.g., BOHB. We think these three algorithms is representative of previous baselines. We also include other GFN methods, including (OP-)DB, (OP-)FM, and (OP-)subTB. However, for the widely-used optimization-based methods, such as DARTS, since they are not multi-trial sampling, belonging to a different class of methods from OP-GFNs, we do not include the comparison.

We appreciate your time and effort in reviewing our paper. Given the upcoming deadline for the end of the discussion period, we are wondering whether the added experiments and our responses have adequately addressed your concerns. We are willing to make more improvements based on your further comments.

I would like to thank the authors for the additional experiments, which have clarified my concerns.

Regarding Point 3 (Later, in the NAS environment, we have implemented and compared OP-TB with various baselines, including FM, DB, subTB.), I found the results and the conclusion that "We obeserve that non-OP-TB methods can achieve similar or better performance with OP-TB, which validate the effectiveness and generality of order-preserving methods." very interesting, and better support that the OP method is general, which could further improve the paper. Please incorporate these results in the final version for a more in-depth comparison.

I have therefore raised my score.

Dear Reviewer CCax,

We appreciate your time and effort in reviewing our paper. Since we have a limited 12-hour window for the author-reviewer discussion, we kindly ask for your valuable feedback on our rebuttal. We are willing to make further improvements based on your feedback.

Best, Authors.

Thank you for thoroughly checking our paper and for providing valuable feedback about notations, structure and content presentation. However, with the current paper layout and the given page limit, we see ourselves faced with a lot of restrictions, that make it very hard to restructure the whole paper. Addressing your proposals one-by-one:

Weaknesses:

1. Minor: $\mathcal{D}$ or $\mathcal{T}$. Sorry for the confusion, $\mathcal{D}$ is a typo, it should indeed be $\mathcal{T}$ in Algorithm 1. We have corrected it in the updated manuscript.

2. What $\widehat{R}$ is trained on? $\widehat{R}$ is trained on the order-preserving loss $L_{OP}$, its parameters depend on GFN parametrization used, i.e., the parameters of: 1) $P_F$, $P_B$ and $Z$, 2) $P_F$, $P_B$ and $F$, or 3) $F$, as given in Table 2.1. parametrized by the GFN parametrization, such as $\widehat{R}_{\rm TB}(x;\theta)$.

3. Disentangling algorithms with tricks. Sorry for the confusion. We optionally use backward augmentation and backward KL regularization in the NAS environments, and clearly specify their usage by (-KL) and (-AUG) in the plots or tables. We have observed from Figure 4.2 that the major improvements are from the OP method, and these tricks only yield marginal improvements. More specifically, all 3 variations of OP-GFNs that include selections of KL regularization or training trajectory augmentation are very close to each other. In other environments, we did not include additional tricks when comparing with traditional GFNs but followed the same setup as the original papers. In other words, the only difference is the training criteria.

4. Investigative figures in the appendix. We admit some figures are left in the appendix. However, there is a 9-page limit, so we chose to put the ablation studies and auxiliary results in the appendix. We felt, that discussing the properties of the trained OP-GFN gives a good insight about what kind of generating distribution the OP-GFN learns. Hence, we sacrificed some space to include these theoretic results in the main paper and put the validation of the theoretical results in the appendix.

Questions:

1. Descriptions of GFNs. We succinctly describe the GFNs in Table 2.1. We have made a clearer reference to this table in the updated manuscript.

2. Log piecewise linear. We admit "log piecewise linear" is not a good naming, and might be a confusing term. Following your arguments, we replace it with “piecewise geometric progression”. The result of "piecewise geometric progression" is not as trivial as it sounds, especially when there are candidates of the same objective value $u(\cdot)$, please refer Propositions 2 and 5 for details. % \lukas{However, even if it is trivial it characterizes the distribution that is learned by the OP GFN. Therefore, we feel it is a necessary part of the paper in order to be complete.}

3. Sequence prepend/append MDP. Sequence prepend/append MDP are widely used in molecule generation, such as in Section 4.2, “For the SIX6, PHO4, QM9, sEH environment, we use the sequence prepend/append MDP in Definition 1”. Therefore, theoretical analysis based on this MDP structure is meaningful.

4. Molecular setup. We inherit the basic setup as Shen et al for trajectory balance, and for OP methods, we only change the training loss. Please see Section E.3.2, and Common Response: Extensive Experiments.

5. Definition of -KL and -AUG. We write the names in the main text because it appear in Figure 4.2, we defer the detailed description to the appendix due to the page limit.

6. PC-GFNs vs OP-GFNs in HyperGrid. In Figure 5.1, PC-GFNs indeed attribute some probability mass on the Pareto front points, but not on all such points, such as "currin-shubert", it misses a lot of Pareto front solutions. Please see Figure F.1, this is mainly because "currin-shubert"'s Pareto front is concave, and linear scalarization in PC-GFNs inherently cannot deal with a concave Pareto front. We point out this weakness in Section 2.2. On the contrary, OP-GFNs can recover all the solutions, regardless of the Pareto front shape, and the landscape terminal flow is much more sparse than PC-GFNs.

7. Sanity checking in HyperGrid. Thanks for your advice on sanity checking in the grid-worlds. We perform the sanity checking in HyperGrid in the multi-objective optimization, since only in the hypergrid, the Pareto front can be explicitly calculated. We also perform extensive experiments on real-world environments, including molecular designs and NAS.

8. Details in Section 4, 5. We put the detailed setup in Sections 4 and 5 in the appendix due to the page limit. We hope the experiments' succinct description and reference to the appendix in the main text, as well as "Appendix A: an overview" could help you navigate the detailed settings.

9. RL vs OP-GFNs. Please see the Common Response: RL vs OP-GFNs.

We are thankful to the reviewer for recognizing and acknowledging our work. We will continue working on improving our writing and presentation skills.

We are also grateful for the identification of various sEH setups in different papers. This will be emphasized in Section 4.2 and Appendix E.3 in the updated paper. Besides, we have utilized the fragment-graph-based MDP in "Section 5.4: Fragment-Based Molecule Generation" to analyze OP-GFN's multiobjective performance, where maximizing sEH is one of the objectives.

Thank you for thoroughly checking our paper, pointing out related work, and suggesting improvements to the experimental parts. We try to address your concerns below:

1. Exploration in TB. Thanks for your insightful reference. We will add the paper to the related work. We think the idea of Thompson sampling could be useful in OP-GFNs as well. However, one of the main benefits of our work is that it naturally balances the exploration and exploitation as the training goes, without the need for manually designed exploration steps. Specifically, compared with Thompson Sampling, sampling from $K$ different forward policies increases the computational budget for $\times K$. Please see the Common Response Related Work.

2. Pareto front Sorry for the confusion, the sentence "When the true Pareto front is unknown, we use a discretization of the extreme faces of the objective space hypercube as $P$", is used only in the evaluation instead of the training. When we do not know the true Pareto front, we can only use evenly spaced points on the outermost faces of the $[0,1]^D$ cube to measure the closeness or diversity of the generated samples. In the training, we describe how to label the samples in "Section 3.1, 1) Order preserving". In every step, we sample a batch of terminal states $X$ and label 1 to those which are non-dominated in $X$, and 0 otherwise. We will add this description to make it more clearly in the paper. Please note the local distribution in Section 3.2 is a special case of those in Section 3.1.

3. Value of $k$ in top$k$ We think it might be unnecessary to compare many different $k$s. Since $k$ is only the moving window to reduce the possible variance in measuring the optimal candidates.

4. Plotting against the clock time We follow Nats-Bench paper to plot against the clock time. We think when evaluating the accuracy is costly, it makes more sense to use the clock time than the number of samples to measure the sampling efficiency. When obtaining the reward is easy, such as in the molecular design, we then will use the number of samples.

5. Reward schemes other than GFN-$\beta$. Please see the Common Response: Related Works.

6. Minor: typos. Thanks for pointing out the typos. We have corrected them in the updated manuscript.

Thank you for your well-structured feedback and for pointing out very recent, and strongly related literature. As mentioned in the general response, we added a discussion to the related work section. Concerning your questions:

1. Non-stationarity issues. We find the non-stationary issues are not very serious in the single-objective case. On the contrary, if we do not learn but define a reward in prior, like with GFN-$\beta$, we cannot obtain similar performance like with OP-GFNs. defining a reward in prior by GFN-$\beta$ methods are not as good as the OP-GFNs. Please see Appendix E.4.4. ablation studies. However, In the multi-objective cases, we observe the non-stationary issues arise, and we choose to use the replay buffer to stabilize the training. Please see the second paragraph of Section 5.2. We added a sentence to make this more clear.

2. More competitive baselines From Figure 4.1, Figure 4.2, Figure 5.1, Figure F.1, our algorithm has proven to be more effective than previous methods. We have already include the subTB (Appendix E.4.5) and RL methods (REINFORCE, Figure 4.2) in NAS environments, which are, to our knowledge, strong baselines. Could you please elaborate in more detail, which baselines you would like to see in comparison? For the molecular design, please see the Common Response: Extensive Experiments.

3. Temperature conditioning methods Please see the Common Response: Related Work.

4. Setting higher $\beta$ Using GFN for stochastic optimization, (1) we do not only consider the maximal reward, but we also need to pay attention to the diversity (exploration). For example, in Section 4.2 (molecular design) we also measure the number of different optimal candidates. If the reward is very sparse, the sampler will only visit states with near-zero rewards, and cannot explore further to visit the various different maximal candidates. We validate this in Figure E.4, when setting $R_0=0$, it yields a very sparse reward landscape. As a result, and the sampler can only sample candidates near the initial points. (2) even if we only consider the maximal value regardless of the diversity, a large $\beta$ will still not be a good choice. Since a large $\beta$ might hinder the exploration to optimal candidates distant to the initialization. To point out this problem, we tested various For example, we test various $\beta$ in NAS in Figure E.8. Setting $\beta=128$ does not perform better than $\beta=16$.

5. Replay buffer In the sparse reward landscape, off-policy training has been shown to help stabilize the training. Please refer to Appendix C.1 of the paper "Goal-conditioned GFlowNets for Controllable Multi-Objective Molecular Design" (https://arxiv.org/pdf/2306.04620.pdf). We add an additional sentence in Section 5.2 to state it more clearly.

6. Minor: $u_0$ or $R_0$ We always use $R_0$ to denote the constant in the objective of HyperGrid in the whole paper. Since either $u_0$ or $R_0$ merely represents a constant, we decide to use $R_0$ to align with the whole paper's notations, including the updated Figure E.5.

7. Minor: trajectory augmentation We always use trajectory augmentation in molecule design as suggested in Appendix E.3.2. In the NAS environment, we optionally use trajectory augmentation, and use "-AUG" to if we use this augmentation in the algorithm, see Figure 4.2.

Thank you for reviewing our paper and for providing a specific list how to improve our work. We agree with your assessment and took the effort to run additional experiments, in the hope of raising the score. We address your concerns below:

1. More complex environments. In the submitted paper, we included many complex real-world environments apart from apart from HyperGrid, which includes molecular generation tasks and NAS. Such tasks are standard in evaluating GFNs in previous papers. We mostly use HyperGrid only for sanity-checking and therefore did not consider more complex configurations with larger $H$ and $D$ at first. To understand the exploration of the algorithm better, in the molecular design environments, we record the number of distinct optimal candidates. Please see the Common Response: Extensive Experiments. Furthermore, we additionally consider two high-dimensional settings in HyperGrid: $(D,H)=(3,32)$ and $(D,H)=(4,16)$, please see Section 4.1, and updated Figure E.5. We consider the following three ratios to measure exploration-exploitation: 1) #(distinctly visited states)/#(all the states); 2) #(distinctly visited maximal states)/ #(all the maximal states); 3) In the most recently 4000 visited states, #(distinctly maximal states)/4000. A good sampling algorithm should have a small ratio 1), and a large ratio 2), 3). We confirm that TB’s performance is sensitive to the choice of $R_0$, and observe that OP-TB can recover all maximal areas more efficiently, and sample maximal candidates with higher probability after visiting fewer distinct candidates.

2. Including the detailed balance objective. The submitted paper contains many experiments with detailed balance in the NAS environment in Appendix E.4.5, where we do not observe a significant gain from the trajectory balance. Therefore, we did not include detailed balance in other environments. However, we ran additional experiments and added detailed balance to the molecular design. Please see the Common Response: Extensive Experiments.

3. Adding error bars. We agree with you and add the error bars to the experiments in the main paper, i.e., to the molecular design environments (Please see the Common Response: Extensive Experiments). We observe that the OP-TB's variance is small, and outperforms other methods beyond the error bars (especially in sehstr).

4. A hyperparameter search for the best learning rate We use the Adam optimizer for training criteria and parametrizations (DB, TB, subTB, OP). The Adam optimizer adaptively computes individual learning rates for different parameters. Therefore, we did not finetune the learning rate of the Adam optimizer. For example, in the NAS environment, we use the initial learning rate $10^{-3}$ for network parameters and $10^{-1}$ for $Z$'s parameters.
We think this comparison is fair because it shows that OP-GFN can outperform standard GFN by only replacing the training criterion. In other words, our results are not cherry-picked from the learning rate.

We appreciate your time and effort in reviewing our paper. Given the upcoming deadline for the end of the discussion period, we are wondering whether the added experiments and our responses have adequately addressed your concerns. We are willing to make more improvements based on your further comments.

Thank you for clarifying the concerns and for updating the work. I have revised my score.

We are thankful to the reviewer for recognizing and acknowledging our work.