Towards Improving Exploration through Sibling Augmented GFlowNets

We thank the reviewer for reviewing our work and providing helpful feedback. We are excited that they found our work well written, easy to follow and novel, as well as found our experiments and included baselines useful. We provide further clarifications below.

• Experimental environments: In order to address difficult exploration settings, we include 4 hard tasks in this work: (a) Hypergrid with zero-reward, (b) Hypergrid with very low difficult to explore rewards, (c ) auto-regressive task of bit sequence generation, and (d) molecule generation with a large state space of the order of $10^{16}$ and between 100 and 2000 actions depending on the state. For the Hypergrid task, we also include larger hypergrids to allow much longer trajectories than the previous works.
  While we agree that the RL environments suggested by the reviewer are indeed difficult to explore; in this work, we do not consider any of the RL frameworks because the current paradigm of GFNs (a) does not aim to maximize returns but instead samples from a target distribution (proportional to the rewards) and (b) only operates for non-cyclic settings of a DAG. Extensions of GFNs to more standard RL settings is a separate, and interesting, future research problem.
• GFlowNets vs RL: It would be fair to characterize GFlowNets as a particular kind of RL on DAG-MDPs with a number of additional assumptions, but these differences enable new (off-policy) training objectives not equivalent to existing RL methods, which is why we see the two as distinct. There are also a lot of parallels between RL and GFlowNets and topics such as exploration and off-policy learning are common to both; however, GFlowNets mainly focus on turning an energy function into a sampler or a generative model and they do so by sampling one action at a time through a forward policy. The core objective of GFlowNets is to match a reward distribution (not maximize it) and turn it into a sampler, while RL mainly focuses on expected reward maximization. Even if GFN is within the broad family of RL and it is true that GFN methods take inspiration from RL across multiple fronts, it is important to understand this distinction and especially the fact that we are not trying to achieve the same objective. The closest RL cousin to GFN is entropy-regularized RL, and their objective coincides in some settings, where there is only one path to a given state to form a tree and not a graph, which is generally not the case in many problems.
• Using a Critic: While it is possible to use a critic or a target network to our work and to GFlowNets in general, we did not use it to provide a fair comparison with other baseline methods and to mainly focus on the core problem of exploration and intrinsic rewards in this work. Additionally, while this double parameterization can help, GFlowNets can usually be optimized in a stable manner.

We hope this helps in clarifying any questions the reviewer might have. We are happy to provide further clarification to any other pending concerns and suggestions and to further improve our work.

We thank the reviewer for their feedback and review of our paper. We are excited that they found our paper well written, clearly presented, and novel. Please find answers to the comments and questions below.

• Other intrinsic rewards: We agree that a variety of intrinsic rewards exist; what we are set out to show through this work though is that training two policies, one to explore, one to behave, is advantageous in a GFlowNet setting. One can imagine then choosing their intrinsic reward of choice. RND is a simple yet effective choice as an intrinsic reward, and other methods, such as NovelD, are based on it. We include NovelD in Sec 10.3 and Fig 12 to show support for other intrinsic rewards. Further improvements and design of different types of intrinsic rewards and investigating their benefits for GFlowNets, are interesting research questions and would be a useful extension of our work.
• Hard exploration environments: To evaluate on hard exploration tasks, we consider 4 difficult-to-explore settings in this work. Specifically, we (a) expanded the reward configuration in the hypergrid environment to include very low and zero reward regions to separate out modes to make exploration more difficult, (b) added the molecule generation task in Sec 5.3 that has a large state space of the order of $10^{16}$ and between 100 and 2000 actions depending on the state and is closer to a practical & hard exploration problem, (c ) evaluated on the auto-regressive bit sequence task that is still an unsolved task in terms of finding modes by the existing methods. These settings are considerably hard to explore settings, as also evident by the failure of the existing baseline methods, while our method shows considerable improvements in terms of exploration in all of them.
• Hyperparameter ablation: The main hyperparameter for our method is the weighing of the intrinsic reward term, as in RL. We keep all other hyperparameters pretty much constant across our experiments. However, we thank the reviewer for this suggestion and will consider adding a discussion around this intrinsic reward weighting hyperparameter to the final version of the paper for completeness.
• SA-GFN and other SOTA methods such as NovelD: The proposed SA-GFN integrates seamlessly with NovelD through a simple plug and play in the term $r^i$ for intrinsic reward in Eq 2. For this reason, NovelD is complementary to SA-GFN and can be incorporated into SA-GFN as shown in Sec 10.3 and Fig 12. We simply chose RND through all of our experiments due to its simplicity, among other reasons mentioned in our first comment above on “other intrinsic rewards”.
• Sibling network training: The Sibling Network does not use the data from Behavior Network and its main purpose is to collect exploratory data which is then relabeled and sent to the Behavior Network. Sibling network is indeed doing exploration, but is not sufficient by itself as can be seen from the “RND” baseline in which a single network using RND rewards does not learn that well.
• Training frequency of Sibling Network and Behavior Network: We indeed use a 1:1 ratio, and haven’t explored alternatives, but note that because of a dual network architecture, the training frequency of the two networks can be made completely independent and asynchronous of each other. In fact, doing so could enable an efficient parallelized training of the SA-GFN. For example, the exploratory data collected by the Sibling Network (SN) can simply be collected in a replay buffer and asynchronously sent to the Behavior Network (BN) for its training. We don’t use replay buffers in this work to provide a fair comparison with baselines, but using one is possible and can further help with efficiency.

We hope this helps. We are happy to answer any other pending questions from the reviewer and to improve our work.

We thank the reviewer for their insightful review of our work. We are enthused that they found our work to be well motivated and clearly presented as well as found the experiments to be comprehensive.

We provide further clarifications below.

• Reward setting from Section 5.1.2: While we take the hypergrid domain from previous works [1, 2], we expand on it by including a more difficult to explore reward setting to focus more on exploration in this work. To do so, we consider a very low value of the reward hyperparameter R0 in the non-zero reward setting that makes non-mode regions sparser and difficult to explore. Specifically, the reward distribution configuration: R0=10−5, R1=1.0, R2=3.0, is hard to explore and has not been considered in the previous works for Hypergrid and lets us focus on the exploration problem.
• No of steps in Fig 4: We chose the current values to keep it consistent with the previous works [1,2,3]. However, we will consider increasing the number of steps in the final version of the paper.
• Calculation of intrinsic reward: The intrinsic reward calculation is exactly the same as done in RL by taking a predictor network and a fixed target network such that the representations from the target network are distilled into the predictor network.
  While we consider the intrinsic rewards on all states, the framework can also work with only terminal intrinsic rewards. When only terminal intrinsic rewards are considered, the second term for intrinsic rewards (in Eq 2) will now have a relatively lower value (as compared to the current form in which we add up the rewards through all timesteps). Therefore, an appropriate change in the hyperparameter $\beta_i$ will allow providing the right weighting and preference to the intrinsic rewards to encourage exploration.
  In this work, one of the reasons for our choice of using intrinsic rewards from all timesteps is to reward the policy on taking novel paths to a given object creation. This is especially useful for GFlowNets that support a DAG structure and multiple ways to generate an object.
• L1-error plots in Fig 4: Harder hypergrid configurations, such as a longer grid horizon, a higher dimension or a difficult reward configuration, can make exploration difficult. As we can see in Fig 4, as the difficulty of exploration increases (e.g. with horizon >= 136 or number of dimensions = 4), the baseline methods can sometimes diverge and/or are not able to learn in a stable manner, causing the L1 to increase. For easier configs, the behavior is much more stable.
• Decoupled setting in RL: This idea of decoupling the exploration policy network to efficiently train a second, main target policy can be used for exploration in RL as well, provided our RL policy can be trained in an off-policy manner. Specifically for navigation tasks, where both exploration and safety can be essential, an exploratory policy could be used to collect exploratory data which can then be stored, relabeled (e.g. using HER) and sent to the main RL policy that can safely learn from this data.
• Other edits: We will make sure to correct the incomplete references and the indices of metrics in 5.2 - thank you for mentioning them.

We hope these help in clarifying any questions that the reviewer might have. Please let us know if there are any other comments or suggestions and we would be happy to address them and further improve our work.

[1] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation.
[2] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets.
[3] Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability.

We thank the reviewer for their feedback and for reviewing our work. We are excited that the reviewer found our work relevant, interesting and novel. We provide further context on the questions asked below:

• Background on GFlowNets: Since some of the previous works [1,2,3] provide a good background on GFlowNets and due to the page limits, we kept this part short and concise. However, in the final submission, we will use an appendix to expand the background section to make the document self-contained for the readers.
• RND and alternative schemes: We agree several other methods exist, and in this work, we mainly used RND because it is a simple, yet a very competitive method for exploration. In addition, to show compatibility with the other alternatives, we provide comparison with an alternative scheme, Noveld [4] in Section 10.3 and Fig 12 in the Appendix. Our setup can also use other forms of exploration rewards, ranging from different types of pseudo counts and other novelty based methods, by simply using them in the intrinsic reward term, $r^i$, in Eq 2 of the Sibling Network.
• RND for GFlowNets: In our work, we show that more powerful intrinsic rewards (with a separate network to learn about novelty), and in particular RND, can help GFlowNets, thus confirming the hypothesis that investing compute in exploration can be a crucial ingredient for GFlowNets to realize their potential, i.e., discover yet unseen modes of the target distribution. Further improvements are an interesting research direction and can be considered an extension of this work, but would we have to speculate, we suspect that methods that prioritize modal exploration over coverage would benefit GFlowNets more - this is because GFlowNNets are set up to capture modes and can discover new ones through the generalization property of neural networks, whereas a covering/uniform exploration method might just waste some time within a mode.

[1] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation.
[2] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations.
[3] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets.
[4] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan-dong Tian. Noveld: A simple yet effective exploration criterion