Random Policy Evaluation Uncovers Policies of Generative Flow Networks

Thanks to the reviewer for the time and effort in the feedback. We would like to clarify the novelty and significance of our contributions as follows:

Q1: Concerns regarding insights: Theorem 4.1 didn't bring anything new about the forward policy.

While the reward transformation is a key component in our algorithm, our work’s novelty and impact extend far beyond this step. The focus of our work is not to connect the 'value functions and flow functions in the GFlowNet', but to address a critical gap in the literature by bridging GFlowNets and standard (non-MaxEnt) RL, an underexplored and overlooked connection that holds broad implications for both theory and practice. Unlike previous works that necessitate flow-matching or entropy regularization [1,2,3], we demonstrate that a GFlowNet forward policy with reward-matching properties can emerge naturally from standard RL components like policy evaluation.

The core of GFlowNets' research is not about defining forward policies in different formulations ($\pi(x) \propto R(x)$), but rather developing effective methods to learn these policies that sample proportionally to rewards. Proposition 1 in [1] explains why MaxEnt RL (Buesing et al., 2019 Haarnoja et al., 2017) are biased by the number of trajectories leading to each terminal state in non-injective environments (i.e., $\pi(x)=\frac{n(x)R(x)}{\sum_{s' \in X}n(x')R(x')}$). In contrast, our Theorems 4.1-4.2 provide distinctive contributions, serving as the foundation of RPE to demonstrate how a standard RL component (policy evaluation) can surprisingly achieve reward-matching properties without requiring specialized and sophisticated flow-matching objectives or entropy regularization. This fundamentally redefines how GFlowNets can be understood and implemented.

• Novel theoretical contributions: We uncover an unexpected bridge between standard (non-MaxEnt) RL and GFlowNets, showing that a basic component in RL, i.e., policy evaluation, can naturally achieve the same reward-matching property as in GFlowNets with minimal modifications. Our work is not a trivial reformulation or mere reward transformation, but offers a fundamentally new insight into the capabilities of standard RL components. Our analysis demonstrates that GFlowNets’ core mechanisms can be reinterpreted through the lens of random policy evaluation in standard RL. This reshapes our understanding of GFlowNets’ learning dynamics, resolving an open question in the field.
• Algorithmic contributions: Based on our main theoretical results in Theorems 4.1-4.2, we develop a simple yet effective RPE algorithm using only policy evaluation, which eliminates the need for complex flow-matching conditions [1,2,3]. Despite its simplicity, our extensive experiments show that RPE matches or exceeds the performance of well-established GFlowNet methods [1,2,3], validating both our theoretical insights and practical utility.
• Expanding understanding: Our work provides novel insights into how reward-matching properties can naturally emerge from policy evaluation of a random policy under appropriate reward transformation. This expands the important understanding that was overlooked and opens up promising directions for future research at the intersection of RL and GFlowNets.

In summary, our work makes substantial contributions by revealing a novel, simpler way to achieve reward-matching policies in GFlowNets through standard RL components under appropriate conditions.

[1] Bengio et. al., Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation, NeurIPS 2021

[2] Bengio et. al., GFlowNet Foundations, JMLR 2023.

[3] Malkin et. al., Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022

Q2: Concerns regarding the assumption in Theorem 4.2

Thanks for your question. Please refer to our response to Reviewer ghFJ's Q1 for a detailed discussion due to space limitations.

Q3: Questions about Appendix A.

Thanks for your question. We will elaborate on more details in Appendix A in the revision to explain the connections between the flow/value functions in GFlowNets, MaxEnt RL, and our RPE approach.

Q4: About proof for Theorem 4.1

Thanks for your question. We clarified in Appendix B (lines 674-675) that Theorem 4.1 represents a special case of the more general Theorem 4.2. For the sake of conciseness and to avoid redundancy in the presentation, we chose to omit the explicit proof of Theorem 4.1 in the appendix (the proof can be readily derived by following the same logical framework presented in the proof of Theorem 4.2, with $B(s_{i+1}) = 1$ and there is exists exactly one path from $s_0$ to any state). To ensure completeness and facilitate better understanding, we will include the proof for this special case for Theorem 4.2 in the revision.

We hope these clarifications effectively address your concerns and highlight our key contributions. We welcome any further questions or discussions.

We thank the reviewer for the useful feedback and positive assessment of our work, and for noting that our work builds a simple, practical, and novel algorithm that bridges RL and GFlowNets. We carefully address your concerns as follows:

Q1: Questions about Max Entropy RL baselines

Thanks for your insightful question! Our choice of Soft DQN and Munchausen DQN as the max-entropy RL baselines was guided by prior research [1], considering the discrete action spaces in our experimental settings. While SAC was originally designed for continuous control problems and would require discretization in our context, Soft DQN is naturally suited for discrete spaces, with Munchausen DQN representing an enhanced variant with documented performance advantages.

To directly address the reviewer's concern about SAC, we conduct additional experiments incorporating SAC as a baseline on the RNA1 generation task, following the cleanRL's implementation. The results illustrated in Figure 2 in https://anonymous.4open.science/r/RPE-added-experiments-57A9 show that SAC achieves comparable performance to Soft DQN in terms of mode discovery but is less efficient in terms of accuracy, which is consistent to prior research [1] demonstrating that discrete SAC typically exhibits inferior performance compared to Soft DQN and Munchausen DQN in discrete action spaces for similar tasks. Notably, our proposed RPE method achieves superior performance across all metrics against these established baselines.

[1] Tiapkin et. al., Generative Flow Networks as Entropy-Regularized RL, AISTATS 2024

Thanks for your valuable feedback and for a positive assessment of our work! We carefully address your concerns as follows:

Q1: Concerns about path invariance scaling factor $g$

Thanks for your question. We would like to emphasize that this condition is satisfied in a broad range of domains of GFlowNets literature, including most standard benchmarks and important application areas: 1) all tree-structured problems naturally satisfy this condition, e.g., sequence/language generation [1,3], red-teaming [2], and prompt optimization [4]; 2) non-tree structured DAGs where the number of parent and children states remains independent of the specific trajectory to the state, like the widely-studied set generation tasks[4], and real-world applications including feature selection[5], recommender systems[6], experimental design optimization[1], portfolio construction, etc. The key requirement is that the task can be modeled as a DAG where the ratio of children states or parent states remains consistent regardless of the specific path taken to reach the state. These examples demonstrate that the condition applies to a wide range of practical generative tasks within the GFlowNet framework.

However, as noted in Section 4.1.2, we acknowledge that RPE's simplicity imposes limitations in certain environments, such as the toy HyperGrid task with boundary conditions. Yet, this simplicity is precisely what makes our findings both surprising and significant. Our work provides the first rigorous characterization of when GFlowNets and policy evaluation align, bridging a critical gap in understanding the relationship between (non-MaxEnt) RL and GFlowNets. This fundamental insight unlocks new possibilities and advances both fields by leveraging their complementary strengths.

Despite being based on basic policy evaluation principles for a uniform policy, our method achieves performance comparable to many well-developed GFlowNet algorithms while maintaining remarkable simplicity across diverse environments, making it a valuable foundation for future exploration in GFlowNets and RL research.

Q2: Questions about training variance and stability

Thanks for your question. As demonstrated in Figures 3,4,5,6 in the main text, RPE exhibits both consistently improving performance and remarkably stable learning throughout the training process. It is noteworthy that RPE has a fundamental algorithmic design difference (advantage): it evaluates a fixed uniform policy $\pi$, while both standard GFlowNets and MaxEnt RL algorithms need to estimate flows/values for continuously evolving policies. This distinctive approach helps RPE eliminate key sources of non-stationarity that can contribute to training instability.

To ensure statistical reliability, all experimental results are averaged across multiple random seeds following evaluation standards in previous works [7]. For the RNA1 generation task, RPE demonstrates the lowest standard deviation among all baseline methods (please see Table 1 in https://anonymous.4open.science/r/RPE-added-experiments-57A9). The reduced variance of RPE can be attributed to reduced complexity in the learning objective through fixed policy evaluation and simplified parameterization that learns only the flow function $F_{\theta}$, different from privious state-of-the-art GFlowNets method like trajectory balance (TB) [7], which can incur high variance due to the reliance on Monte-Carlo estimation techniques.

Q3: Performance of RPE beyond training iterations 1e4.

To further evaluate RPE's sustainability beyond the standard training regime, we extend our experimental analysis by training for 1e5 iterations (10$\times$) on the RNA1 task, and compare it against the most competitive algorithms from both Soft RL and GFlowNets paradigms, including Munchausen-DQN and TB in this task. The results presented in Fig.1 in https://anonymous.4open.science/r/RPE-added-experiments-57A9 demonstrate that RPE maintains its performance advantage throughout the extended training period, which consistently outperforms these top-performing baselines, validating its effectiveness as a stable and high-performing approach.

References

[1] Jain et. al., GFlowNets for AI-Driven Scientific Discovery, Digital Discovery, 2023

[2] Lee et. al., Learning diverse attacks on large language models for robust red-teaming and safety tuning, ICLR 2025

[3] Hu et. al., Amortizing intractable inference in large language models, ICLR 2024

[4] Yun et. al.,Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation, CVPR 2025

[4] Pan et. al., Better Training of GFlowNets with Local Credit and Incomplete Trajectories, ICML 2023

[5] Ren et. al., ID-RDRL: a deep reinforcement learning-based feature selection intrusion detection model, Scientific Reports 2022

[6] Liu et. al., Generative Flow Network for Listwise Recommendation, KDD 2023

[7] Malkin et. al.,Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022

Thanks for your valuable suggestions and feedback! We carefully address your concerns as follows:

Q1: $\tau$-dependence regarding transformed rewards $R(x)g(\tau,x)$

Thanks for your question! This concern is addressed by the assumption in Theorem 4.2: "For any trajectories $\tau_1$ and $\tau_2$ that visit $s_t$, they must satisfy the condition $g(\tau_1, s_t) = g(\tau_2, s_t)$." Under this assumption, the transformation from $R(x)$ to $R'(x)=R(x)g(\tau,x)$ is well-defined, since the transformed reward function $R'(x)$ remains consistent regardless of the specific trajectory taken to reach the terminal state $x$ (and this property naturally holds in a number of standard GFlowNet benchmarks -- Q2). We will also update the notation to eliminate any ambiguity.

Q2: questions about the condition g(τ_1,s_t)=g(τ_2,s_t)

Thanks for your question. Indeed, this condition is a necessary assumption for the validity of Theorem 4.2.

We remark that this assumption can be satisfied in a number of standard GFlowNets problems. Please refer to our response to Reviewer ghFJ's Q1 for a detailed discussion.

Q3: Definition of P_B

Thanks for the question. This is not a typo, and $P_B$ is defined as $P_B(s|s')=\frac{F(s\to s')}{F(s')}$ following the GFlowNets literature (Eq. (17) in [1]), representing the probability of having come from state $s$ given that we are currently at state $s'$ (defined in terms of the flow).

Q4: Concerns regarding motivations and contributions.

The motivation for connecting GFlowNets and policy evaluation stems from a critical gap in the current understanding of GFlowNets' relationship with RL frameworks. While previous research has extensively explored connections between GFlowNets and various ML methods, their link to RL has largely been limited to MaxEnt RL [2,3] via modified objectives with entropy regularization. However, the connection between GFlowNets and standard (non-MaxEnt) RL remains largely unexplored, despite the shared foundation of sequential decision-making. This gap limits cross-disciplinary insights (e.g., leveraging RL’s efficiency for GFlowNets or enriching RL with GFlowNets’ diversity).

Our work fills this crucial gap and makes the following key contributions: (1) Surprisingly, we find that the flow functions in GFlowNets naturally correspond to value functions in uniform policy evaluation. This insight leads to our key finding that reward-matching effects, previously achieved through complex GFlowNets algorithms [1,4], can be accomplished through simple policy evaluation. (2) Based on this theoretical foundation, our proposed RPE method achieves competitive performance with reduced complexity. (3) Our work challenges the prevailing belief that GFlowNets are intrinsically tied to MaxEnt RL and provides a fundamentally new perspective on GFlowNets learning dynamics.

In addition, traditional policy evaluation is merely a component for policy improvement; RPE reframes it to generate diverse, high-quality candidates via uniform policy evaluation with transformed rewards, expanding its utility to domains requiring both quality and diversity.

Q5: relations to Soft Policy Iteration (SPI) and the "control as inference" framework

Thanks for your insightful question. Our work shares conceptual connections with SPI, while establishing key distinctions that highlight the novelty of our work: (1) SPI necessitates iterative cycles of policy evaluation and policy improvement to derive the desired policy, while RPE evaluates only the uniform policy, transforming its value function to flow functions to achieve reward matching. (2) SPI and 'control as inference' framework both explicitly incorporate entropy regularization terms to ensure policy stochasticity, while RPE achieves superior reward-matching performance through standard policy evaluation utilizing a summarization operator, eliminating the need for explicit regularization. Our experimental results demonstrate RPE's superior performance compared to both SoftDQN and MunchausenDQN (which are advanced soft RL algorithms for discrete environments) in terms of the standard GFlowNets evaluation protocol. We will add detailed discussions between them in the revision.

Q6: Comparison on a standard RF setting

Since the goal of our work is to validate that a simple RPE algorithm can achieve competitive reward-matching performance as existing complex GFlowNets, we evaluate RPE and extensive baselines using well-established benchmarks in the GFlowNets literature [8,9], e.g., TFBind/RNA/molecule generation, which requires a critical balance between diversity and quality.

[1] Yoshua Bengio et. al., GFlowNet Foundations, JMLR 2023.

[2] Tiapkin et. al., Generative Flow Networks as Entropy-Regularized RL, AISTATS 2024

[3] Tristan et al., Discrete probabilistic inference as control in multi-path environments, UAI 2024

[4] Bengio et. al., Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation, NeurIPS 2021