Local Search GFlowNets

Thanks for providing valuable feedback.

---

W1: About computational overhead

LS-GFN does not introduce any additional computational overhead. Below are the wall clock times for each training round in the QM9 task.

The wall clock time is evaluated with a single RTX 3090 GPU and Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz and 256 GB RAM.

This restriction is in place to ensure that the sample complexity remains consistent with that of the baseline GFN methods.

The calculation of sampling complexity follows this formula: $(I+1) \times M$, where $I$ signifies the number of local search iterations, and $M$ denotes the batch size per training round.

---

Q1: How is the number of steps K picked for backtracking? If it's fixed, is there a way to pick it automatically?

We used $K=\lfloor (L+1) / 2\rfloor$, where $L$ is length of trajecotry. We also provided a hyperparameter analysis in the original manuscript in Appendix B.5. As you mentioned, we can automatically pick $K$ to adaptively balance the wideness of the local search region, which could be a great avenue for future work.

Thanks for the valuable feedback.

---

W1, Q1: The paper does not declare the sampling complexity of the proposed method. The local search may require more sampling, which can lead to an unfair comparison.

Thank you for highlighting this for clarification. All our experiments were conducted under a fair setting, as we used exactly the same number of samples across all experiments. Note this is already outlined in Section 5.3.

The calculation of sampling complexity follows this formula: $(I+1) \times M$, where $I$ stands for the number of local search iterations, and $M$ denotes the batch size per training round.

---

W2: The limitations of the proposed algorithm and potential drawbacks are not discussed in detail.

A limitation of LS-GFN lies in the potential impact of the quality of the backward policy on its performance, particularly when the acceptance rate of the local search becomes excessively low. One immediate remedy is to introduce an exploratory element into the backward policy, utilizing techniques like $\epsilon$-greedy or even employing a uniform distribution to foster exploration within the local search.

A promising avenue for future research could involve fine-tuning backward policy to enhance the acceptance rate of the local search. We plan to incorporate this as a topic in our paper's "Limitations and Further Work" after we get one additional page for the final paper (we now put this into the appendix), and we appreciate your valuable suggestion.

---

Q2: How to decide the local search interaction I? What about the results with different I?

We have conducted an ablation study on the hyperparameter $I$, as now detailed in Appendix B.3.

It's worth noting that across various hyperparameter candidates, namely $I \in \{1, 3, 7, 15, 31\}$, LS-GFN ($I > 0$) consistently outperforms GFN ($I = 0$). This observation suggests that choosing the appropriate hyperparameter $I$ for LS-GFN is relatively straightforward, as it consistently yields superior results.

---

Q3: As PRT is adopted in LS-GFN training, is PER used for RL methods in te experiments?

Taking into consideration your feedback, we implemented reward-prioritized experience replay training in conjunction with the off-policy RL baseline, SQL. It's worth noting that PPO and A2C are on-policy RL methods in which replay training is not directly utilized. Here are new experiment results that compare replay-trained SQL and LS-GFN:

Q1: Why call it “destroy” since the original trajectory isn’t always discarded? A more intuitive name could be “backtrack,” “rewind,” or anything that doesn’t suggest destruction.

We are in agreement on this matter. We have updated our manuscript to incorporate the term "backtrack" instead of "destroy."

---

Q2: The top plots in Figure 5 are strange: it looks like some curves go beyond 100% accuracy. Could you either fix them so we can still see the curves on the plot or explain what is happening?

Please note that Figure 5 is plotted accurately in accordance with the baseline from the paper by Shen et al., 2023, where we have set our accuracy metrics following their guidelines:

$\text{Acc}(p(x;\theta)) = 100 \times \text{min}(\frac{E_{p(x;\theta)}[R(x)]}{E_{p^{*}(x)}[R(x)]},1)$

This provides an upper bound of 100% accuracy. In response to your suggestion, we have included a graph illustrating accuracy beyond the 100% mark in Appendix B.7.

---

Q3: How can LS-GFNs recover from a biased backward policy? In other words, assume the forward policy is fine but the backward policy always backtracks to states that yield the same (high reward) candidate objects — how can LS-GFNs overcome this lack of exploration?

Biased backward policies can indeed pose challenges. In such scenarios, we can employ an $\epsilon$-greedy method to introduce exploratory back-sampling into the biased backward policy. It's worth noting that LS-GFN performs exceptionally well in MaxEnt GFN, even when the backward policy is set to a uniform distribution (i.e., $\epsilon = 1$).

Building upon your valuable feedback, one potential avenue for future research could involve fine-tuning the backward policy to maximize the local search acceptance rate. This approach would aim to ensure that the backward policy identifies diverse intermediate states that lead to improved rewards without exhibiting a bias towards yielding the same candidate objects repeatedly. We appreciate your insightful comment.

---

Q4: Please confirm that lines 7 and 8 in Algorithm 1 aren’t swapped. If they aren’t (which partially addresses my question above), wouldn’t swapping them and extending them further improve exploitation? Maybe this should be added as an ablation as well.

No swapping is involved here. We update every sample to the dataset, irrespective of whether it is accepted or not. The rationale behind this approach is that every sample where the reward is calculated is considered valuable. During the refinement phase, our primary objective is to explore the local region in order to discover high-reward samples rather than determining whether the found sample should be used in training.

In the training process, we employ PRT to sample trajectories in proportion to their rewards from the dataset. Consequently, accepted samples have a higher likelihood of being included during training.

Regarding your suggestion to swap lines 7 and 8, it would entail updating the data only with accepted samples. However, as you pointed out, while this could enhance exploitation, it might excessively hinder exploration. To shed more light on this, here are the new results of our ablation study:

Based on the above results, the original LS-GFN gives higher performances than the swapped version.

---

References

Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving GFlowNet training. In International Conference on Machine Learning, pp. 30956–30975. PMLR, 2023

Thanks for your valuable comments.

---

W1-1: While the method itself is simple and well presented, it remains a little unclear the relative importance of different components. For example, the exposition focuses on the trajectory-balanced objective — does this mean LS-GFNs don’t work with different training objectives?

LS-GFN is a versatile technique that can be effectively employed for a wide range of GFN objectives. We have already successfully applied LS-GFN to various GFN objectives, including Sub-trajectory balance (SubTB), detailed balance (DB), and maximum entropy GFN (MaxEnt), as demonstrated in Table 1 and Table 2 in our original manuscript.

---

W1-2: The paper also mentions building on top of Shen et al., 2023 which introduces prioritized replay training (PRT) to GFNs, but this method doesn’t appear as a baseline — so how much of the improvements are due to PRT and how much is due to the local search?

For a fair comparison, we consistently use "PRT" for training and "SSR" for parameterization in all GFN baselines. Additionally, for direct comparison purposes, we assessed our approach against Shen et al., 2023, denoted as "GTB" in Figure 5 and Figure 6.

It's important to note that Shen et al., 2023 introduced three distinct techniques: "PRT" (pertaining to dataset sampling), "SSR" (concerning GFN parameterization), and "GTB" (associated with the guided trajectory balance objective).

---

W2: In addition to the hyper-parameter (number of revisions with local searches), I wished the number of backtracking steps and the acceptance rate were also studied — how much tuning did they require in order to get these good results?

We've already conducted experiments on hyperparameter $I$ in Appendix B.3, as well as explored the acceptance rate in Appendix B.6.

It's worth noting that across various hyperparameter candidates, namely $I \in \{1, 3, 7, 15, 31\}$, LS-GFN ($I > 0$) consistently outperforms GFN ($I = 0$). This observation suggests that choosing the appropriate hyperparameter $I$ for LS-GFN is relatively straightforward, as it consistently yields superior results.

---

W3: Wall-clock time is never mentioned — how much overhead does the refining process of LS-GFNs incur? How about in terms of sampled states (instead of training rounds)?

There's negligible additional wall clock time overhead.

Here are the wall clock times for each training round in the QM9 task. Wall clock time is computed as the mean of three independent seeds. To remove the bias from warmup, we compute the wall clock time per round after 10 initial training rounds. There is no meaningful wall time difference between the two methods:

The wall clock time is evaluated with a single RTX 3090 GPU and Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz and 256 GB RAM.

The wall clock time remains consistent because we always use the same number of samples per training round ($M \times (I+1) = 32$, $M$: batch size, $I$: number of local searches) for every method across all experiments. This choice is particularly crucial considering that evaluating the reward per sample is a significant bottleneck (in some applications, one could spend a day for reward computation), especially in real-world scenarios like bio and chemical discovery tasks. By keeping the sample complexity uniform, we ensure that the computational overhead remains stable.