Evolution guided generative flow networks

Dear reviewer kQAn,

Thank you for your comments! The insights you provided in the reviews are invaluable, and we would like to thank you for your time crafting this review.

In what follows, we address your comments one by one.

---

W1. The method introduces additional complexity in the form of the evolutionary optimization, but does not analyze why such a decision would improve performance. The evolutionary algorithms applied are well-known, and the combined algorithm boils down to an ad-hoc fitting of EA and gradient descent sequentially. The added complexity results in slower training speed, as mentioned in the paper. This paper would strongly benefit from a more principled look at the training dynamics of GFlowNets, and a stronger opinion on why evolutionary algorithms help learning.

Thank you for your observation. To address your concern, we have performed additional experiment in the context of increasing size of the tasks. Figure 22 of the Appendix H reports the results. From the figure, it should be clear that our claim that EA is more resistant to the length of the trajectory holds true, as we can see that the mean reward of the population still rises quickly despite the increasing difficulty of the tasks, compared to the star agent.

Now, why does high reward samples matter? EA's are naturally diverse and explorative, so the high reward samples are also diverse. These samples go to the PRB, where they are sampled for off-policy training for the star agent. This is important, because the added diversity acts as a trajectory length regularizer for the PRB (we show this in the Figure 9 of the Section 5), making sure the star agent is trained sufficiently for all length of the trajectory, and the high-reward samples ensure better training signals.

---

Q1. Can GFlowNets be applied to more traditional generative modeling tasks? (e.g. images, etc).

While the current state of GFlowNets does not apply to such scale of large number of variables, GFlowNets have been applied to much auxiliary tasks such as pretraining and fine-tuning[1], learning discrete latent variables[2], and text-to-Image diffusion alignment[3].

---

Q2. In Figure 4, it would help to clear up which of the labelled methods are RL, MCMC, or GFlowNet variants.

Thank you for your suggestion! Our current change should reflect your suggestion.

---

Q3. In page 2 paragraph 2, it would be good to re-clarify what TB stands for, and introduce these prior objectives together.

Thank you for your suggestions. We have revised our draft to address your comments.

---

Q4. How are neural network weights mixed in the crossover step? Is this an important detail, or is a naive strategy good enough?

We use a naive strategy. For clarity, we have included the algorithm of our strategy in the Appendix I.

---

References

1. Pan, Ling, et al. "Pre-Training and Fine-Tuning Generative Flow Networks." The Twelfth International Conference on Learning Representations.
2. Hu, Edward J., et al. "GFlowNet-EM for learning compositional latent variable models." International Conference on Machine Learning. PMLR, 2023.
3. Zhang, Dinghuai, et al. "Improving GFlowNets for Text-to-Image Diffusion Alignment." arXiv preprint arXiv:2406.00633(2024).

Dear reviewer yNGR,

Thank you for your comments! The insights you provided in the reviews are invaluable, and we would like to thank you for your time crafting this review.

In what follows, we address your comments one by one.

---

W1. EA only provides diverse experiences to be sampled. Comparing the performance of the population to the star agent to validate the assumptions.

While it is true that EA provides diverse experiences to be sampled, EA also provides high-reward diverse samples for PRB to sample from. In the Figure 22 of the Appendix H, we compare the rewards between the population and the star agent, showing that the population achieves high mean reward quicker than the star agent.

---

W2. Comparison based on the number of evaluated trajectories to assess the sample complexity.

Thanks for bringing this point. For fair comparison, we keep the number of evaluations similar, if $S$ and $S'$ is the on-policy samples by other algorithms and EGFN, respectively, it should be enough to make sure that

$S \approx S' + \mathcal{E}k.$

Given your feedback, we also have experimented with number of trajectory evaluations on the X axis comparing GFlowNets and EGFN on Figure 21 in Appendix H with the same hyper parameters of our experiments. We still see significant improvement of EGFN, compared of GFlowNets.

---

W3. EGFN performs slightly worse in the more generic tasks despite being computationally more intense

It is not clear which generic tasks EGFN performs slightly worse. Besides, given GFlowNets is a sampling algorithm, it is necessary to point out that it is also succeptible to mode collapse--thus not just the speed performance, but also the mode finding capability is important. Finally, we also believe the conversation of using EA for sampling algorithm such a GFlowNets is valuable to the community too, despite its computational intensity.

---

W4. GAFN and MARS missing a short introduction, explanation, or comparison

Thank you for your feedback. Based on your feedback, we have added a short introduction of all baselines, in the Appendix G.

---

W5. Elaborate sparsity levels shown in Fig. 4

Thank you for pointing it out. We have fixed it now in our updated manuscript.

---

W6. p.3 l.109f.: abbreviations FM, DB, and TB should be introduced first.

We introduced the abbreviations in line 42 and 81 in the Section 1. Please let us know if this is not sufficient.

---

W7. Alg. 1, l.181: P^_F should be P_F? Alternatively, the reason the star agent is used to evaluate the population should be elaborated on.

Thank you. We have edited our algorithm accordingly. It was supposed to be a machanism to store the online trajectories of the star agent to the replay buffer, which our current edit reflects.

---

W8. Alg. 1 l.182f.: vars for online and offline trajectories should differ

We agree. Our current change reflects your observation.

---

Dear reviewer 2fJP,

Thank you for your comments! The insights you provided in the reviews are invaluable, and we would like to thank you for your time crafting this review.

In what follows, we address your comments one by one.

---

W1. In its current form, the paper contains several ambiguous or incorrect claims in Section 2.1.

1-1. Lines 93 - 102: The DAG structure should be defined first to clearly specify the action space according to the DAG’s edges. Additionally, the phrase “sample proportionally to different peaks of reward” in line 101 (and line 34) is misleading.

To incorporate your feedback, we have written the beginning paragraph of Section 2.1.

1-2. Line 112: The notation $F(\tau)$ has never been defined but is used to define $F(s)$.

To address your feedback, we have added the definition.

1-3. Line 118: There’s an incorrect use of the prime ( ` ) symbol. Moreover, the equation $P_F(s'|s, \theta) = F(s\rightarrow s')$ is inaccurate. The RHS should be divided by F(s) . A similar issue appears in lines 124-125.

Thank you for pointing them out. Our revision should now reflect your observation.

1-4. Line 133: The expression $\sum_x{R(x)} = \sum_{s:s_0 \rightarrow s \in \tau \forall \tau \in \mathcal{T}}{P_F(s|s_0;\theta)}$ needs more explanation. At first glance, it doesn’t seem to hold generally.

Total flow is the sum of all forward flow from starting state $P_F(s_0|s, \theta)$ where there is an edge between $s_0$ and $s$. Or, it could be thought of the sum of all terminal rewards upon convergence.

---

W2. 1. I’m unclear on why EGFN improves credit assignments. The star agent in the EGFN framework uses conventional learning objectives like DB or TB, and I couldn’t find any specific design element that enhances credit assignment. From what I understand, EGFN’s main advantage is its evolving population of GFNs, which provides more diverse experiences for the star agent to learn from. This should enhance exploration, which is especially beneficial in sparse environments.

EA also provides high-reward diverse samples for PRB to sample from. In the Figure 22 of the Appendix H, we compare the rewards between the population and the star agent, showing that the population achieves high mean reward quicker than the star agent. It can also be seen that with increasing difficulty, while the star agent's top 10% reward starts to fall, the EA population stays more consistent with finding high-reward samples for PRB, given that it is a black-box optimization method.

---

W3.1. Experiment Setup (Reward Calls): Were all algorithms given the same number of reward calls? All learning progress figures use training steps as the x-axis, but I suspect EGFN might use additional reward calls per training step due to the rewards needed for fitness calculation (line 173). However, in real-world applications where reward evaluation is costly (e.g., in vitro experiments), sample efficiency is often more critical than learning efficiency [3, 4]. Therefore, I recommend including results with a fixed number of reward calls, especially for biochemical sequence generation tasks.

EGFN uses 36 calls. Other baselines use 32, except for PPO which doubles the reward calls since we double the amount of on-policy samples due to not having off-policy samples.

However, in real-world applications where reward evaluation is costly (e.g., in vitro experiments), sample efficiency is often more critical than learning efficiency [3, 4] Therefore, I recommend including results with a fixed number of reward calls, especially for biochemical sequence generation tasks.

Indeed it is true that sample effeciency is important for in vitro experiments. However, in many active learning experiments where a surrogate reward is used, learning efficiency is more important. To address your feedback, we have included preliminary results with number of trajectory evaluations on the X axis comparing GFlowNets and EGFN on Figure 21 in Appendix H with the same hyper parameters of our experiments. We still see significant improvement of EGFN, compared of GFlowNets. We will include similar results for the biological experiments in the final revision.

---

W3.2. Line 304 and 898: The paper states the number of modes for the hypergrid task is $2^D$, but this doesn’t seem correct. There are indeed $2^D$ reward “regions” if a region is defined as a collection of adjacent modes. However, the actual number of modes could be $M*2^D$, where $M$ represents the number of modes in each region, potentially increasing with $H$.

Thank you for your careful observation. This is indeed the correct general number of modes. For our experiments, we found $M$ to be equal to 1. Our revised draft reflects your observation.

I apologize for the late response and appreciate your efforts on the rebuttal.

Here are some additional comments on the updated manuscript:

• I cannot see the revision history. Is there any way to make the previous version visible to reviewers?

• Remaining errors

  • line 102: A Directed Acyclic Graph (DAG) includes a tree as a special case. In a tree, there is only one path leading to each state.
  • line 119: I suggest using $\mathbb{R}$ for the set of real numbers instead of $R_{\geq 0}$, as it could cause confusion with the notation for reward.
  • line 127 (Eq.3): The flow-matching GFlowNet requires parameterization of the edge flows. The forward policy $P_F(\cdot | \cdot, \theta)$ is derived from these edge flows, as mentioned in my previous review. In its current form, the equation still seems incorrect. Typically, the flow-matching loss is defined using the edge flows.

• Regarding Point W2: To my understanding, finding high-reward samples is not directly related to the credit-assignment problem. Credit assignment involves "distributing the credit of success among the multitude of decisions involved" [1]. For more on credit assignment from a GFlowNet perspective, please refer to [2, 3].

• Regarding Point W3.1:

  • From Figure 21, it appears that EGFN and GFN do not show significant differences in terms of the number of modes discovered.
  • If learning efficiency is important, why not compare the algorithms based on execution time?
  • I believe that comparing algorithms in terms of sample efficiency offers a fairer assessment, as is common in many GFlowNet studies [3, 4].

[1] Minsky, Marvin. "Steps toward artificial intelligence." Proceedings of the IRE 49.1 (1961): 8-30.
[2] Pan, Ling, et al. "Better training of gflownets with local credit and incomplete trajectories." International Conference on Machine Learning. PMLR, 2023.
[3] Jang, Hyosoon, Minsu Kim, and Sungsoo Ahn. "Learning energy decompositions for partial inference of gflownets." arXiv preprint arXiv:2310.03301 (2023).
[4] Lau, Elaine, et al. "Qgfn: Controllable greediness with action values." arXiv preprint arXiv:2402.05234 (2024).

F6. From Figure 21, it appears that EGFN and GFN do not show significant differences in terms of the number of modes discovered. If learning efficiency is important, why not compare the algorithms based on execution time?

A6. We thank the reviewers for such detail-oriented feedback. We would like to point the reviewer to the end of the Figure where EGFN's mode count is pointing upwards, while GFlowNe's mode count seems to converge.

Indeed, this is also evident from the L1 error of the Figure where EGFN clearly dominates GFlowNets. The results show with similar number of evaluated samples, EGFN models the reward distribution better, which makes it more likely to find more modes.

With a similar note, we would also like to make it clear that by learning efficiency we specifically mean how well the algorithm can model the reward distribution.

The authors in [5] state "The challenge of mode collapse manifests in GFlowNets as well. A GFlowNet may become fixated on a particular mode during its training phase, motivating various off-policy exploration techniques to enhance discovery of modes during training."

Evidently, as a model better learns the distribution, it should be more likely to discover more modes as we see with our upward trend with EGFN compared to the convergence of GFlowNets. It is especially useful when the model that we use is a surrogate.

F7. I believe that comparing algorithms in terms of sample efficiency offers a fairer assessment, as is common in many GFlowNet studies [3, 4]

A7. We thank the reviewer for their feedback. Our Figure 21 is an effort to offer the assessment, which we plan to extend to for other experiments--something we could not do during the author-reviewer discussion period.

We also believe the results in Figure 21 should convince the reviewer that the results in our other experiments should also hold with that requirement. Indeed, our method only exceeds the GFlowNets baseline by 4, which is mainly due to our design choice to keep the number of online samples a multiple of 16, e.g., 16 for EGFN (with 20 for population) and 32 for other baselines.

5. Krichel, Anas, et al. "On Generalization for Generative Flow Networks." arXiv preprint arXiv:2407.03105 (2024).

F1. I cannot see the revision history. Is there any way to make the previous version visible to reviewers?

A1. As authors, we cannot edit the visibility of the revisions. That said, please let us know which part you'd like to know the before and after of and we can provide a prompt description.

Now that we are aware of that, our responses below will attempt to explicitly mention the changes we made. We thank the reviewer for letting us know.

F2. line 102: A Directed Acyclic Graph (DAG) includes a tree as a special case. In a tree, there is only one path leading to each state.

A2. Thank you for pointing it out. Indeed, we missed the special case as trees are a special case of the DAGs and we do not consider that as "tree-structured DAGs (autoregressive generation) are equivalent to RL with appropriate entropy regularization or soft Q-learning and control as inference", as mentioned by [1].

To address this, we have changed "There exist different paths leading to the same state in the DAG, except for the root, which has no parent." to " We specifically consider DAGs that are not tree-structured, thus there exist different paths leading to the same state in the DAG, except for the root, which has no parent."

F3. line 119: I suggest using for the set of real numbers $\mathbb{R}$ instead of $R_{ \ge 0}$, as it could cause confusion with the notation for reward.

A3. We agree with the reviewer and added this change from $R$ to $\mathbb{R}$ in our manuscript.

F4. line 127 (Eq.3): The flow-matching GFlowNet requires parameterization of the edge flows. The forward policy is derived from these edge flows, as mentioned in my previous review. In its current form, the equation still seems incorrect. Typically, the flow-matching loss is defined using the edge flows.

A4. To address the reviewer, we have now defined our loss using estimated edge flows.

Particularly we have changed "where $F(s'\rightarrow s'') = R(s)$ if $s \in \mathcal{X}$. Using a estimated distribution over children $P_F(s'|s,\theta)$ and an estimated distribution over parents $P_F(s'|s'',\theta)$,..." to "To achieve the criterion, using an estimated edge flow $F_\theta : \mathcal{E} \rightarrow \mathbb{R}^+$, ..."

References

1. Malkin, Nikolay, et al. "Trajectory balance: Improved credit assignment in gflownets." Advances in Neural Information Processing Systems 35 (2022): 5955-5967.

F5. To my understanding, finding high-reward samples is not directly related to the credit-assignment problem. Credit assignment involves "distributing the credit of success among the multitude of decisions involved" [1]. For more on credit assignment from a GFlowNet perspective, please refer to [2, 3].

A5. We now understand the source of confusion and will try to answer as clearly as possible.

The reviewer's claim that "finding high-reward samples is not directly related to the credit-assignment problem". However, our work tackles the credit-assignment problem in long trajectories and sparse rewards.

Indeed, credit assignment in long trajectories and sparse rewards is difficult with the current advances, as in [6] the authors conclude, "TB trades off the advantage of immediately providing credit to early states with the disadvantage of relying on sampling of long trajectories...".

This problem is evident in other methods too, as the authors in [7] discuss, "... methods in RL use bootstrapping to address this issue but often struggle when the time horizons are long and the reward is sparse"

This being the case, we do not tackle the credit assignment problem, rather address it in the difficult cases and utilize current credit assignment methods for our method. Our evaluations in Figure 9 show that the mentioned cases make the training trajectory lengths skewed, and utilizing EA helps it to be more balanced by improving diversity.

Next, our evaluations in Figure 22 state that our assumption of EA immune to our problem is valid and can indeed bring better samples for improved training in the difficult cases.

6. Malkin, Nikolay, et al. "Trajectory balance: Improved credit assignment in gflownets." Advances in Neural Information Processing Systems 35 (2022): 5955-5967.
7. Khadka, Shauharda, and Kagan Tumer. "Evolution-guided policy gradient in reinforcement learning." Advances in Neural Information Processing Systems 31 (2018).

Thank you for your response.

" We specifically consider DAGs that are not tree-structured, thus there exist different paths leading to the same state in the DAG, except for the root, which has no parent."

This seems a bit misleading. If your method isn't specifically designed for non-tree DAGs, it might not make sense to limit your work to "DAGs except trees." It might be better to remove this sentence, if it is not necessary.

Our evaluations in Figure 9 show that the mentioned cases make the training trajectory lengths skewed, and utilizing EA helps it to be more balanced by improving diversity.

In this sense, I still believe this work is more related to exploration, and thus some of the claims about better credit assignment seem overstated.

We would like to point the reviewer to the end of the Figure where EGFN's mode count is pointing upwards, while GFlowNe's mode count seems to converge.

This feels like a somewhat naive and careless evaluation. Also, given the increased complexity of EGFN, I don’t think the performance gain is significant enough to justify the extra complexity.

---

(Additional comment) In Figure 5, the graph for GFN and EGFN are both blue-colored, and it is hard to distinguish them.

---

Overall, I think this work still has many aspects that need improvement. I’m maintaining my score.