Summary: Thank you for recognizing the novelty of our approach in applying GFlowNets to stochastic environments. We understand the concerns regarding theoretical justification, experimental setup, and comparison with related work, and we have addressed these points in detail in the revised manuscript.

1. Theoretical Justification for Equation (3)
   We have added a detailed derivation for Equation (3) in Section 4.1, explaining why replacing the transition dynamics with the entropy ratio is possible. The derivation includes an explanation of how this replacement retains the properties of a valid probability distribution, ensuring that the dynamics are properly modeled even in stochastic settings.

2. Marginal Distribution and Reward Function Matching
   To address this concern, we have included a proof in Appendix A showing how the marginal distribution over terminal states matches the reward function, thereby validating the GFlowNet objective in a stochastic environment.

3. Comparison to Related Work (Jiralerspong et al., 2024)
   Our work is different from the work of Jiralerspong et al. (2024), showing that our proposed approach is closer to Stochastic Generative Flow Networks (Pan et al 2023) performs better in terms of mode discovery and reward quality in stochastic environments.

4. Plot Readability
   All plots have been updated to ensure consistent color usage across different methods, and the text size has been increased for better readability without the need for zooming.

Response to Questions

• Justification for Entropy Ratio in Equation (3): The entropy ratio represents a balance between high and low entropy states, helping maintain an appropriate level of exploration in stochastic environments. Section 4.2 now includes a more detailed explanation of why this approach is theoretically sound. Explained in the previous review in more detail.
• Natural Example of a Stochastic Environment: We provided the example of TFBind8, where the stochasticity introduced through transition dynamics mimics the inherent uncertainty in biological environments, highlighting the need for a stochastic approach.

Formulas Explained

• Equation (3): $P(s_{t+1} | (s_t, a_t)) = \frac{H_{\text{high}}(s_{t+1})}{\gamma H_{\text{high}}(s_{t+1}) + (1 - \gamma) H_{\text{low}}(s_{t+1})}$ This equation represents the transition dynamics in stochastic environments, where $H_{\text{high}}$ and $H_{\text{low}}$ represent densities associated with high and low entropy states, respectively. The parameter $\gamma$ adjusts the relative influence of these densities.

• Equation (9): (See the previous review comment above in more detail) $L(\theta) = - \frac{1}{N} \sum_{n=1}^{N} \left( z_n H(\pi_\theta(a_n | s_n)) \log r_\gamma(s_n) + (1 - z_n) H(\pi_\theta(a_n | s_n)) \log(1 - r_\gamma(s_n)) \right)$

This summation reflects the practical dynamics loss computed over sampled state-action pairs, integrating entropy ratios into the optimization process.

• KL Divergence Minimization: $D_{KL} = \sum_{s,a,s'} \pi_\theta(s, a|s') \left[ \log \pi_\theta(s, a|s') - \log F(s, a) - \log r_\gamma(s') \right]$ This KL divergence objective ensures the consistency of forward and backward flows, aligning the stochastic transition dynamics with the entropy-based policy adjustment.

Feel free to ask for more clarification and we really appreciate your feedback.

Summary: We appreciate your feedback and acknowledge the areas where the original manuscript lacked clarity. Below, we provide detailed explanations

1. Demonstrating Stochastic Domains We have added a discussion on real-world problems that are inherently stochastic, such as biological sequence generation. This highlights scenarios where stochastic models are crucial for capturing the full diversity of outcomes, which deterministic models cannot handle effectively.
2. Misattribution of $\alpha$-Randomness :The reference to $\alpha$-randomness is to differentiate it from sticky actions Machado et al. (2017). We explain $\alpha$-randomness in our approach involves probabilistic action selection.
3. Graphs and Visual Clarity: updated
4. Reproducibility of Experiments: see Supplementary Materials

Response to Questions

• Definition of $H_{\text{high}}$ and $H_{\text{low}}$: These terms are explicitly defined in Section 5.1 as densities related to high and low entropy states, representing the stochastic behavior of state transitions.
• How $\gamma$ is Tuned: $\gamma$ is dynamically adjusted during training based on the variance in state transitions to balance exploration and exploitation, as explained in Section 7. Equation (9) represents the practical form of the dynamics loss used during training: $L(\theta) = - \frac{1}{N} \sum_{n=1}^{N} \left( z_n H(\pi_\theta(a_n | s_n)) \log r_\gamma(s_n) + (1 - z_n) H(\pi_\theta(a_n | s_n)) \log(1 - r_\gamma(s_n)) \right)$
• What Are We Summing Over?
  The summation in Equation (9) is over all sampled state-action pairs from the empirical distribution during training. Specifically, the training process samples $N$ different state-action pairs, denoted by $(s_n, a_n)$, from trajectories generated by the current policy. Here: - $z_n$ is a binary indicator that denotes the occurrence of a specific type of state-action pair. If the sampled transition is considered exploratory, then $z_n = 1$, otherwise $z_n = 0$.
  • $H(\pi_\theta(a_n | s_n))$ represents the entropy of the policy $\pi_\theta$ for the given state-action pair, which measures the randomness in selecting action $a_n$ at state $s_n$.
  • The term $r_\gamma(s_n)$ is the entropy ratio, which adjusts the probability of each state-action pair based on the weighted contributions of high- and low-entropy states. This summation provides a loss function that penalizes the policy for deviating from expected entropy-weighted transitions, helping to balance exploration (high entropy) and exploitation (low entropy) during training. Equations (3) and (7) are related but represent different aspects of entropy estimation:
• Equation (3):$P(s_{t+1}|(s_t, a_t)) = \frac{H_{\text{high}}(s_{t+1})}{\gamma H_{\text{high}}(s_{t+1}) + (1 - \gamma) H_{\text{low}}(s_{t+1})}$ This equation describes the transition dynamics in terms of entropy densities. It shows how the next state $s_{t+1}$ is determined by a mixture of high- and low-entropy components, where $\gamma$ is the exploration-exploitation trade-off parameter. The numerator $H_{\text{high}}(s_{t+1})$ represents the density of high entropy states, while the denominator is a weighted sum involving both high and low entropy.
• Equation (7):$r_\gamma(s) = \frac{H_{\text{high}}(s)}{\gamma H_{\text{high}}(s) + (1 - \gamma) H_{\text{low}}(s)}$ Equation (7) represents the entropy ratio used for optimization. It is equivalent to Equation (3) in the sense that both involve the ratio of high-entropy and low-entropy contributions. However, Equation (7) is used in the optimization objective to measure the likelihood of a state being selected based on its entropy characteristics. Essentially, Equation (3) defines how transitions are modeled, while Equation (7) provides the quantitative metric for adjusting the exploration during training.

Equation (6): defines the flow entropy for the policy $\pi$:

$H(\pi) = \mathbb{E} \left[ \sum_{t=0}^{T-1} H(\pi(\cdot | s_t)) \right] = \sum_{s \in S} \mu_\pi(s) H(\pi(\cdot | s))$

• Flow Entropy Definition:
  • $H(\pi(\cdot | s)) = -\sum_{a \in A(s)} \pi(a | s) \log \pi(a | s)$ represents the entropy of the policy at state $s$, which measures the uncertainty or randomness in the actions selected by the policy.
  • The expectation $\mathbb{E}$ is over all possible states in a trajectory, considering the entire time horizon from $t = 0$ to $T-1$.
• What Does $\mu_\pi(s)$ Correspond To?
  $\mu_\pi(s)$ represents the state visitation frequency under policy $\pi$. It indicates how often state $s$ is visited when following policy $\pi$. The flow entropy is computed as a weighted sum of the policy's entropy at each state, with $\mu_\pi(s)$ serving as the weighting factor. This ensures that states that are visited more frequently contribute more to the overall entropy, which is crucial for understanding the exploration behavior of the policy over time.

Hi, the plots are still inconsistent! In figure 2 and 3, the colors between (a) and (b) are not the same, nor are the included lines (where did SAC go for (b)?), nor is the legend order. Unless there is a good, explained, reason, all plots should have same comparison lines. Unless there is a good reason, all legends should look exactly the same, especially for (a) and (b) in the same figure. I also don't see confidence intervals except in Figure 5. Maybe you forgot to update those plots?

Furthermore, you removed the setting of $\alpha$ from the line name in Figure 5, meaning the reader does not have enough information about what you're plotting. The trouble before was that you include the seed name in the plot, which is ugly and confusing, but now it's imprecise.

The paper still incorrectly attributes random-actions to Machado et al, as I said in the first response.

There are now two places (below Eq 7, and below the definition of $r_\gamma$ where you have the exact same definitions of $H_{high}$ and $H_{low}$ -- that's not necessary. Also, those quantities are used much earlier than they're introduced, which is very confusing. Additionally, $H$ depends on the policy, not only the state, and this is not made clear in the the notation.

Sadly, the numerous presentation issues have made it challenging for me to really dive into the technical contribution. But overall, as other reviewers said, the paper does not make a very clear case that this method is theoretically well-founded, and the Equation 3 somewhat comes out of nowhere (instead of being derived). This paper needs a lot of structural work before it is ready for publication, and hopefully upon re-submission it's easier for the next reviewer to understand and therefore substantially critique.