Flow of Reasoning: Training LLMs for Divergent Problem Solving with Minimal Examples

Thank you for the positive and insightful comments!

Response to Weaknesses

W1: The benchmarks in this paper do not require much divergent thinking. Consider lateral thinking.

Response-W1: Thank you for your thoughtful feedback and suggestions. We address your concerns and provide additional insights into our work below.

Concern about Unsuitable Benchmarks

Firstly, we respectfully argue that benchmarks like Game24 and Rubik's Cube are indeed suitable for evaluating the diversity of our results, as they often involve multiple trajectories to achieve a single goal. For example, as shown in Figure 1 of our paper, both Game24 and Rubik's Cube exhibit diverse solution paths, making them effective benchmarks for assessing the exploratory nature of our method. Additionally, although PrOntoQA is not allowed to have a diverse solution, it (and other benchmarks) still benefits from our method’s ability to avoid getting stuck in incorrect trajectories (please refer to the Qualitative Results in the General Response). Specifically, FoR encourages effective exploration in the search space during inference with repeated samplings, which increases the likelihood of finding correct and diverse trajectories.

Additional GSM8K Experiments

To further address your suggestion, we conducted additional experiments on GSM8k math problems which are deemed more open-ended than the tasks we've done. Following RAP[4], we define an action as an intermediate sub-question to solve the problem and a state as the history of all intermediate sub-question-answer pairs.

We perform experiments in a 2-shot setting, comparing FoR with Supervised Fine-Tuning (SFT), CoT, CoT with self-consistency (CoT-SC), and RAP [4]. For each problem, we sample 4 solutions, considering success as achieving at least one correct solution when calculating accuracy. For training, we use the last 50 examples from the GSM8k dataset. The baselines are implemented based on [4]. Since no established metrics exist for evaluating the diversity of open-ended mathematical reasoning, we manually annotated 50 test examples that have multiple reasoning trajectories to assess the similarity between reasoning trajectories, determining whether two reasoning paths were semantically equivalent or not.

As shown in the table below, the results demonstrate that FoR is effective on GSM8k, achieving higher accuracy than all baselines while maintaining the diversity of reasoning trajectories. These findings further validate the generalizability of FoR across different reasoning benchmarks.

Discussion of Suggested Works

We appreciate your recommendation to consider the works in lateral thinking [1, 2, 3].

Upon review, we think that:

• [1] investigates humor generation, emphasizing the ability to interpret figures or questions with multiple plausible meanings.
• [2] and [3] focus on riddle-solving QA tasks that require reasoning about unexpected or unconventional answers.

While these works emphasize lateral thinking—focusing on looking at problems from new perspectives and defying preconceptions associated with the right-brain hemisphere [2,3]—our work aligns with vertical thinking, which involves logical, sequential reasoning typically linked to the left-brain hemisphere [2,5]. Our approach emphasizes exploring diverse solutions within a structured, logical framework rather than breaking away from it. Unlike lateral thinking, which defines creativity as the ability to defy conventional frameworks and generate surprising content, our work defines creativity as the diversity of reasoning paths within the context of multi-step problem-solving tasks (e.g., math) with a clearly defined and known goal state. Therefore, we believe these works are orthogonal to ours, as they address fundamentally different aspects of creativity and reasoning.

References

[1] Let’s Think Outside the Box: Exploring Leap-of-Thought in LLMs with Creative Humor Generation, CVPR24

[2] Lateral Thinking Puzzles for Large Language Models, EMNLP23

[3] RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge, ACL21

[4] Shibo Hao, et al. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, Singapore. Association for Computational Linguistics.

[5] Waks, S. Lateral Thinking and Technology Education. Journal of Science Education and Technology 6, 245–255 (1997).

Dear Reviewer,

This is a gentle reminder that the discussion period is closing soon. We've responded to all concerns, providing detailed clarifications and new experimental results. Please feel free to let us know if you have any further questions. We'd greatly appreciate it if you could consider raising scores and/or engaging in further discussions. Thanks!

Dear Reviewer,

We noticed that you reduced your score from 6 to 5, and we’d greatly appreciate it if you could share any additional concerns or feedback that caused the change. We have responded to your initial questions in the original review and provided new experimental results on more "open-ended" problems (e.g., GSM8K). If there are any new issues you’d like us to address, we’d be happy to do so.

Thank you for your time and consideration!

Dear Reviewer,

We’d like to thank you again for your suggestion on the related work. In our updated PDF, we have cited and added more discussion of the suggested work. Due to the time and space limit, it's challenging for us to make this change to the main paper now. We currently included it in the section of “Lateral Thinking and Vertical Thinking” in Appendix A, and we will make sure to move to the main paper in the camera-ready version. We look forward to more discussion!

Dear Reviewer, This is a gentle reminder that the discussion period will be closing soon. We have addressed all concerns by providing detailed clarifications and updated experimental results. Please don't hesitate to reach out if you have any additional questions. We would greatly appreciate it if you could consider raising the scores and/or engaging in further discussions. Thank you!

Thank you very much for your insightful comments and suggestions! Below, we address each of comments in detail.

Response to Weaknesses

W1.1: lack of clarity regarding any original insights or core observations the authors provide about large language models in the context of reasoning with diversity. Throughout the paper, it seems that all key observations, points, and techniques are primarily derived from existing works, without presenting novel contributions from the authors.

Response-W1.1:

Please refer to the Qualitative Results in the General Response to see the insight of our methods. Also, please refer to the contributions part in the General Response.

W1.2: LLMs inherently exhibit randomness and uncertainty in their reasoning processes, which could naturally encourage exploration across diverse reasoning paths.

Response-W1.2:

We would like to clarify that, though LLMs exhibit randomness during generation (e.g., with random sample decoding), it does not lead to diverse and high-quality reasoning paths. Indeed, our experiments have done extensive study of LLMs with various diversity-enhancing decoding strategies, such as Nucleus Sampling, Typical Sampling, and diverse beam search (Line.287-290). Results show these common methods fail to generate diverse and accurate solutions at the same level as our proposed method (see Table.1). Our proposed approach instead generates solutions with probabilities proportional to the solutions' "goodness" (i.e., reward). This mechanism produces diverse and high-quality solutions.

W1.3: In additional papers, such as TR [1] and CR [2], they all build a DAG toward getting solutions.

Response-W1.3:

Thank you for pointing out these works. We'd like to clarify that the works TR and CR that build DAGs still aim to find the max-reward solution instead of diverse high-quality solutions. Thus these works are similar to the methods we've compared with, such as RAP and ToT. Our approach achieves better accuaracy and diversity (e.g., Table.1). In addition, the TR and CR methods use revisions and multiple rounds of prompting, which are orthogonal and complementary to our method.

W2: Why sampling in such a way can generate not only diverse but also high-quality reasoning paths? What is the logic between this strong conclusion and its previous and subsequent sentences?

Response-W2:

Sampling in proportion to reward, as used in our method, facilitates the generation of both diverse and high-quality reasoning paths by aligning the sampling process with the reward distribution [2, 3]. This mechansim encourages to discover multiple solutions, each evaluated and sampled based on its "goodness" as determined by the reward. The exploration for diverse solutions also makes the model more robust and generalize better. A concrete example is analyzed in the Qualitative Results in the General Response: traditional methods like SFT fails to explore alternative reasoning paths, becoming stuck in specific incorrect trajectories. In contrast, our method explores and uncover alternative valid solutions with high rewards.

W3: This paper introduces too many existing methods, ideas, equations, and conclusions from prior works. However, the way the authors organize these elements does not integrate them effectively, resulting in a lack of clear and logical explanations to support an understanding of their own contribution.

Response-W3:

Please refer to the Contribution part in the General Response for a summary of our contributions. In particular, we're the first to formulate multi-step LLM reasoning as Markovian flow. This formulation enables us to adapt and integrate existing successful approaches (e.g., GFlowNets objectives, local search strategies) to our new setting of multi-step reasoning. We believe the abilility to generalize and integrate existing successful approaches (instead of having to re-invent all technical components from scratch) is a key advantage of our framework. We will make this more clear in the revised paper.

W4: Some of the questions posed in the paper are unclear.

Response-W4:

Q4.1: "in Line 188, why the distribution directly equals to $F(\tau)/Z$?"

A4.1: This follows directly the definition of a normalized probability distribution in flow-based models (e.g., Equation (3) in [1]). More specifically, the reason the distribution equals $\frac{F(\tau)}{Z}$ is that $F(\tau)$ represents the flow associated with trajectory $\tau$, and $Z$ is the total flow (partition function), which normalizes $F(\tau)$ over all possible trajectories. This normalization ensures that the resulting value is a valid probability distribution, as probabilities must sum to 1 across all trajectories.

Intuitively, think of $F(\tau)$ as assigning a "weight" or "importance" to each trajectory $\tau$. Dividing by $Z$, which is the sum of these weights across all trajectories, converts these weights into probabilities that reflect the relative likelihood of each trajectory.

Q4.2 "in Line 191, where is the $P(s_0)$ that is the prior in this step-wise distribution?"

A4.2: $P(s_0)=1$. This is because, by the definition of the Markovian flow, all trajectories start with the same initial state $s_0$ [2]. That is, the probability of a trajectory passing through the initial state $s_0$ is 1. We will make this clearer in the revised version.

Q4.3: "in Line 202, when F is defined for the state flow, it's unclear why $F(s_n)$ is to be equal to R."

A4.3: As explained in the response to W4, the diversity-promoting reasoning is achieved by sampling reasoning trajectories with probabilities proportional to the reward. By definition of state flow, $F(s_n)$ represents the cumulative flow reaching the terminal state $s_n$. The equality $F(s_n) = R$ ensures that the terminal state flows are directly aligned with the reward distribution. This alignment ensures that the trajectories leading to $s_n$ are sampled in proportion to their contribution to $R$.

W5: The experiment section does not provide sufficient persuasiveness for the paper.

Response-W5:

Q5.1: "The main content should provide a more specific and detailed explanation of how to fine-tune LLMs according to the proposed objectives."

A5.1: We’ve described the fine-tuning objective and efficient approximation in Section 3.2. The experiment section describes specific instantiations of the fine-tuning components in each task, such as the reward design. Appendix D further provides the detailed algorithm and all prompts used in fine-tuning to elaborate on the finetuning procedure. We’ll make the details clearer.

Q5.2: "How the authors define diversity is unclear to me."

A5.2: Thank you for your feedback. We report the average number of unique correct solutions among the $n$ (e.g., 20) sampled trajectories (Line.302). This means that for each problem, we calculate the number of different correct trajectories out of $n$ samples. We then compute the average of these values across the test set, which serves as our Diversity metric. We have also included more detailed definition of the diversity metric in Appendix C.1. We will ensure this explanation is clearer in the next revision.

Q5.3: "But how the two trajectories can be compared when there are many ones?"

A5.3: We use exact string match to determine uniqueness between two trajectories. This comparison works because the output of our tasks are in a certain standardized format.

Q5.4: "Thus, during training, how can you ensure that the LLM can generate two different paths from one state?"

A5.4: During training, we encourage the model to generate different trajectories in a "soft" way, by setting a high temperature ($\alpha=1$) that increases randomness of LLM generation, as well as using local search (Section.3.2.2) which further promotes exploration of different trajectories. Crucially, the training objective uses reward to give feedback to these trajectories, to make sure the model learns to generate not only diverse but also high-quality trajectories.

References

[1]. Malkin, Nikolay, et al. "Trajectory balance: Improved credit assignment in gflownets." Advances in Neural Information Processing Systems 35 (2022)

[2] Bengio, Yoshua, et al. "Gflownet foundations." The Journal of Machine Learning Research 24.1 (2023)

[3] The GFlowNet Tutorial

Dear Reviewer,

This is a gentle reminder that the discussion period is closing soon. We've responded to all concerns, providing detailed clarifications and new experimental results. Please feel free to let us know if you have any further questions. We'd greatly appreciate it if you could consider raising scores and/or engaging in further discussions. Thanks!

Dear Reviewer, This is a gentle reminder that the discussion period will be closing soon. We have addressed all concerns by providing detailed clarifications and updated experimental results. Please don't hesitate to reach out if you have any additional questions. We would greatly appreciate it if you could consider raising the scores and/or engaging in further discussions. Thank you!

Thank you for answering my questions!

I am pleased to see that some of my concerns have been acknowledged and solved, and I hope the authors will continue to polish and improve the paper as indicated in their response.

However, I personally believe this submission has some weaknesses, leading me to conclude that a score of 5 is fair. Some of my concerns and identified weaknesses can be found in the 'Further Comments' section above. Here is the final conclusion paragraph as a quick snapshot:

"In summary, I hold my view that this work, which claims too many firsts and breakthroughs but lacks the necessary definitions and discussions of related work, uses too much content from existing work (formulas, definitions, mathematical explanations), and directly presents the best numerical results but lacks convincing discussions and insights, does not reach the borderline ICLR. Therefore, I believe that 'marginally below the acceptance threshold' is a fair assessment, particularly after thoroughly reviewing this submission multiple times."

Therefore, I am relatively confident that 'marginally below the acceptance threshold' is my final assessment after thoroughly reading the paper multiple times and engaging in discussions with the authors.

Thank you very much!

Thank you for your comments. We believe our previous responses have addressed your questions directly. Here we’d like to re-emphasize key points from our previous replies. We’d appreciate any further comments on the basis of our previous responses to make progress in the discussion.

• Our paper has already cited, discussed, and empirically compared existing works on LLM generation diversity works, such as [1], as well as many decoding methods including sampling with varying temperatures, Nucleus Decoding, Typical Decoding, and Diverse Beam-Search [2,3,4] (See Table.1). As discussed in several places in the paper (e.g., Line.92 and Line.144), these methods (including the four papers you mentioned) use token-level modeling of LLM generation. However, such token-level approaches are less effective for handling complex multi-step reasoning tasks that require generating reasoning trajectories consisting of multiple steps (each comprising multiple tokens). Our work introduces higher-level modeling at the granularity of reasoning steps for better promoting diversity in this multi-step reasoning setting. As discussed in Introduction section (Line.95), this effectively combines the best of the previous token-level diversity-seeking approaches (particularly GFlowNets) and the recent multi-step reasoning methods such as Tree-of-Thoughts and RAP (which search at the step-level for max-reward trajectory), while overcoming their respective limitations.

Therefore, as clarified earlier, our focus is on promoting diversity in LLM multi-step reasoning by designing a novel approach tailored to this specific setting. While general token-level generation can be interpreted as "inherently performing multi-step reasoning," this perspective overlooks the potential for developing more effective methods specifically suited to the multi-step reasoning setting.

• Thanks for acknowledging MCTS is not equivalent to Markovian Flow. We’ve summarized our "novelty" in the General Response. As again clarified in our previous response, adapting the classical Markovian Flow formalism to the new setting of multi-step LLM reasoning opens the door for us to connect/adapt successful existing approaches (proposed in other settings) to address this significant new problem effectively. We’d like to highlight that the ability to re-purpose and generalize successful existing approaches to new problem setting (instead of having to re-invent all details from scratch) is a key advantage of our work. We’d be happy to discuss if there are different perspectives on the expectations of "novelty".

• We’ve explained, in both the initial and follow-up responses, the “insights” of why our approach improves both accuracy and diversity. We’ve explained the intuition behind our method and also provided a qualitative example (in General Response) to illustrate this (i.e., the exploratory nature of FoR encourages diverse explorations and helps avoid the reasoning getting stuck in bad steps, thus improving accuracy while producing diverse solutions). We’d appreciate more discussion on the basis of our responses.

[1] Hu, Edward J., et al. "Amortizing intractable inference in large language models." International Conference on Learning Representations (2024) Oral.

[2] Holtzman, A., et al. "The Curious Case of Neural Text Degeneration." International Conference on Learning Representations. (2020)

[3] Meister, C., et al. 2023. "Locally Typical Sampling" Transactions of the Association for Computational Linguistics. (2023)

[4]Vijayakumar et al., "Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models" AAAI (2018)

Thank you very much for your insightful comments and suggestions! We address each of comments in detail below.

Response to Weaknesses

W1：Although a variety of tasks were evaluated, all of the tasks are relatively structured and uses constrained action spaced (such as STACK, UNSTACK, PUT, PICKUP in BlocksWorld). In contrast, reasoning tasks that are useful to real users are typically much more open ended, with a more diverse variety of both questions and potential answer formats. In order to show that the proposed approach is scalable to these more open ended reasoning tasks, it would be great to also evaluate on benchmarks like MATH or GSM8k.

Response-W1:

Thanks for your suggestion. We add an additional experiment of GSM8K to evaluate the performance of our method on a open-ended reasoning task. We follow [1] to define an action as an intermediate sub-question to solve the problem and a state as all the history intermediate pairs of a sub-question and its answer.

We conduct experiments with 2-shot settings, and compare them with supervised fine-tuning (SFT), CoT, CoT with self consistency (CoT-SC), and RAP. For each problem, we sample 4 solutions and the success is indicated as long as 1 solution is correct. For training, we construct the training dataset with the last 50 examples in the GSM8K training set. The implementation of baselines refers to [2]. Due to the lack of established evaluation metrics for assessing the diversity of open-ended mathematical reasoning, we manually annotate 50 test examples to evaluate the similarity between reasoning trajectories, determining whether two reasoning trajectories are semantically equivalent or not.

The results are shown below: FoR shows effectiveness on GSM8k and exceeds the accuracy and diversity of all baselines, which demonstrates the potential of FoR for extending to more open-ended reasoning tasks.

As shown in the table below, in the task of GSM8k, FoR outperforms all baselines in both accuracy (by an absolute improvement of 4.7% over SFT) and diversity (by an absolute improvement of 0.13 over SFT). These results highlight the potential of FoR for extending to more open-ended reasoning tasks.

W2: Use a personalized reward function for each task. Not practical when training general-purpose LLMs on a wide range of different tasks.

Response-W2:

We would like to clarify that the reward functions used in our paper do not heavily rely on human design. Specifically, they consist of a standard success-or-failure reward and a value that that measures the goodness of an action to achieve a goal, which is provided by an additional LLM or Environment (Line.351, Line.450-451). This reward formulation is consistent with the previous RL framework for LLM reasoning (e.g., RAP [1]).

We would like to note that in the additional GSM8k experiment, we only use a standard success-or-failure reward during training, which is also effective, as demonstrated in the response to W1. Overall, our approach is applicable to a wide range of tasks as long as a reasonable and reliable reward function/model is available, similar to the common RL approaches [3] applied to LLM reasoning .

W3: Great to show qualitative examples of the different responses generated by each approach, and highlight how they differ from standard approaches

Response-W3:

Please refer to the Qualitative Results in the General Response.

References

[1] Hao, Shibo, et al. "Reasoning with Language Model is Planning with World Model." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[2] Hao, Shibo, et al. "LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models." First Conference on Language Modeling.

[3] Wang, C., Deng, Y., Lyu, Z., Zeng, L., He, J., Yan, S., & An, B. (2024). Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283.

Dear Reviewer,

This is a gentle reminder that the discussion period is closing soon. We've responded to all concerns, providing detailed clarifications and new experimental results. Please feel free to let us know if you have any further questions. We'd greatly appreciate it if you could consider raising scores and/or engaging in further discussions. Thanks!

Dear Reviewer, This is a gentle reminder that the discussion period will be closing soon. We have addressed all concerns by providing detailed clarifications and updated experimental results. Please don't hesitate to reach out if you have any additional questions. We would greatly appreciate it if you could consider raising the scores and/or engaging in further discussions. Thank you!

Thank you so much for your feedback!

"use the best hyperparameters for baselines when conducting comparisons"

We agree with your comment that during comparison, the best hyperparameters for baselines should be used. This is indeed what we've done in the paper. For example, in Table.1, we've compared with SFT under different temperatures (0.1, 0.5, 1.0). We've also tried different decoding algorithms (e.g., DBS, Nucleus, Typical) to see which setting is the best for the SFT baseline. Results show that our method outperforms SFT with the best hyperparameter configuration.

On GSM8K above, SFT (temperature=0) achieves better accuracy than our method, but SFT (temperature=0) can produce only one solution at maximum, while ours aims at generating more diverse solutions. We see diversity as a major advantage of our approach.

New results

As stated in the last response, the new results were obtained by averaging over 5 runs with different random seeds. So the numbers could be slightly different from (and are more robust than) previous results from only one run.

Low-data regime

Thanks for the comment! Low-data setting is an important scenario in practice. In addition to finetuning-based methods, we've also compared with prompting-based methods that are commonly used in low-data settings (e.g., see Table.1). Results show our method outperforms both finetuning- and prompting-based methods. We thus see our ability of learning to produce diverse solutions from only limited examples as an advantage. We'll add more discussion and results especially in the settings when more (e.g., thousands of) data are available. Thanks again for the suggestion.

Q6: The proposed method seems to not include search during inference. Why then is its runtime significantly higher than other COT/SFT methods that also do not include search?

Response-Q6:

Following LLM-Reasoners [5], our method (and the baselines like RAP [6]) utilize another LLM (e.g., in BlocksWorld) or additional environment (e.g., in Rubik's Cube) to perform the state transition $T$ (Line.218-220) while the CoT/SFT method only predicts the action series. This results in different inference times for these methods.

Q7: For baseline evaluation, do the prompts explicitly include keywords to encourage diverse responses? If not, this could be worth investigating.

Response-Q7:

We did not include these keywords in our prompt to reflect their common usage in practice and to ensure a fair comparison across all baselines. We follow your suggestion and add a diversity-encouraging prompt as instruction:

• “Please carefully understand the goals and initial states, then come up with diverse solutions and think outside the box.”

We evaluate multiple baseline methods using LLama-3-8B as the base model, following the exact same settings described in our paper. The results for BlocksWorld are reported below. The numbers in parentheses indicate the performance difference compared to the original prompt without the diversity-encouraging instruction.

We observe that diversity-encouraging prompts for the CoT and SFT baselines lead to improvements in both diversity and accuracy, with average absolute gains of 0.03 and 5.11%, respectively. However, FoR still outperforms them, achieving average absolute improvements of 0.19 in diversity, 9.46% in creativity, and 34.93% in accuracy compared to the best baseline for each metric. These results will be included in our next version.

References:

[1] Bengio, Yoshua, et al. "Gflownet foundations." The Journal of Machine Learning Research 24.1 (2023).

[2] Madan, Kanika, et al. "Goal2FlowNet: Learning Diverse Policy Covers using GFlowNets for Goal-Conditioned RL."

[3] Wang, C., Deng, Y., Lyu, Z., Zeng, L., He, J., Yan, S., & An, B. (2024). Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283.

[4] Lou, Xingzhou, et al. "Uncertainty-aware reward model: Teaching reward models to know what is unknown." arXiv preprint arXiv:2410.00847 (2024).

[5] Hao, Shibo, et al. "LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models." First Conference on Language Modeling.

[6] Hao, Shibo, et al. "Reasoning with Language Model is Planning with World Model." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[7] Hu, Edward J., et al. "Amortizing intractable inference in large language models." The Thirteenth International Conference on Learning Representations 12 (2024) Oral.

W3: The method’s core approach—sampling reasoning paths proportionally to rewards—faces inherent challenges in many real-world decision-making problems, where exploration is difficult, and rewards are sparse. In scenarios where only a binary success/failure reward is available, for example, the proposed method may converge toward behavior similar to traditional reinforcement learning.

Response-W3:

For tasks where ground-truth dense rewards are unavailable (such as Blockworlds and Game24 in our experiments), we acknowledge that this presents difficulties for traditional GFlowNets and also existing RL algorithms like PPO. To address this, we employ augmented rewards (see "Reward Design" sections in every task) which help make the reward signal more dense, and incorporate the technique of local search to expand the exploration of trajectories during online training (Section 3.2.2 and Appendix D and E). Our ablation study (Section 4.7) demonstrates that these techniques effectively enhance exploration during training, mitigate the challenges of sparse rewards, and indeed lead to greatly improved performance and diversity than traditional RL (e.g., Table 1).

We acknowledge that for tasks or areas beyond the internal knowledge of LLMs, the augmented reward signal may be less effective—a challenge that similarly affects RL methods. Future work could explore incorporating measures of uncertainty or reliability in LLM knowledge into reward design. For example, uncertainty-aware rewards [4] could adaptively guide exploration in unfamiliar areas, enabling the model to better handle tasks with sparse or unreliable reward signals.

Response to Questions

Q1: Why does only the Game of 24 domain use a different model?

Response-Q1:

We use LLaMA-2-13B for Game of 24 as it achieves reasonably good performance across all comparison methods. We are currently running additional experiments using LLaMA3-8B (the same model as in other tasks) on the Game of 24 and will include these results in the revised paper.

Q2: Could the authors clarify which components are borrowed from existing works and which are newly proposed in this method? Many core components (sampling proportional to rewards, trajectory balance objective, adaptive efficient exploration) seem to be derived from existing approaches.

Response-Q2:

Please refer to the Contribution part in the General Response. Our method models the multi-step reasoning problem as a Markovian flow. This new formulation of multi-step LLM reasoning allows us to adapt and integrate existing successful approaches. As mentioned above, we believe the ability to incorporate existing techniques is an advantage of our formulation.

Q3: Could the authors clarify the usage of examples in the test dataset? Is it used for few-shot prompting during inference, or is it only used for calculating creativity metrics?

Response-Q3:

We didn’t use the test data as part of the few-shot prompting prompt. And yes, the test set is only used for calculating the creativity scores.

Q4: In Line 301, what is the value of N for each domain? The choice of N seems to not be specified for domains like Blocks World or Game of 24.

Response-Q4:

We would like to clarify that the value $n$ refers to the number of samples generated during inference, as described in the Setup section (Section 4) of each task. For example, $n$ is 8, 20, and 40 for the 2, 4, and 6-step cases, respectively, for BlocksWorld (see Line.323) and $n$ is 20 (see Line.373) for Game of 24.

Q5: While it's clear that each fine-tuning method shares the same offline dataset, has the size of the online dataset also been controlled across methods?

Response-Q5:

Yes, we sample the same amount of online data for training FoR and the baselines like SFT+PPO and SFT + GFN-CoT, we will make it more clear in the experimental setting in the revision.

W1.3.3 It would also be interesting to analyze how diverse reasoning paths may contribute to FoR’s effectiveness in out-of-distribution tasks.

Response-W1.3.3:

Thanks for the insightful suggestion! By exploring multiple high-reward trajectories and matching the reward distribution, FoR develops a richer understanding of the underlying task structure, rather than overfitting to specific patterns seen during training (e.g., maximizing reward as in RL). This diversity facilitates the model to adapt to novel scenarios on OOD problems, as it can leverage alternative reasoning paths that may still align with the new data distributions. [2,7]

To further assess performance on out-of-distribution (OOD) settings, we conduct experiments on the BlocksWorld task (Section.4.2). Specifically, we train the model using FoR and SFT on a 2-step training set and evaluate them on a 4-step test set, and train the model on 4-step training set and evaluate them on the 6-step test set. This allows us to analyze their generalization on OOD problems. For prompting-based baselines, we use 2-step and 4-step examples as demonstrations, respectively.

According to the table below, FoR maintains highest accuracy (71.43%) on OOD tasks compared to other methods like CoT and SFT, which range from 9.52% to 14.28%. FoR also achieves greater diversity (by an absolute improvement of 0.2 over SFT), highlighting its superior generalization and solution exploration capabilities.

W2: The core setup of the algorithm relies on Markov assumptions and a Markovian flow structure, which are suitable only for domains that can accommodate state/action flows under these constraints. However, many complex real-world reasoning tasks do not necessarily align with the Markov assumption, having significant partial observability challenges or making it challenging to define clear “state” and “action” flows.

Response-W2:

The Markovian assumption in our approach, that the next state is only based on the previous state, is indeed inspired by the common Reinforcement Learning (RL) formulations. The same state-action formulation in RL has been successfully applied to various LLM reasoning tasks [3] (e.g., with proper definition of “state” to include necessary reasoning history). Consequently, our formulation inherits this compatibility, allowing it to be applied across a wide range of LLM reasoning scenarios.

While being as general as RL approaches, we highlight that the key advantage of our approach over traditional RL formulations is its diversity-seeking nature by encouraging exploration and sampling of diverse reasoning paths.

Thank you very much for your insightful comments and suggestions! Below, we address each of the comments in detail.

Response to Weaknesses:

W1.1: The motivation behind the method lacks clarity. While the authors focus on framing the problem as a Markovian flow and sampling reasoning paths using GFlowNets, it is unclear why these specific elements are crucial or effective for addressing the issue of diverse reasoning paths.

Response-W1.1:

The cited prior work referenced in Line.138 [1] demonstrates that GFlowNets are specifically designed to generate diverse and high-quality samples with probabilities proportional to their rewards. This mechanism encourages exploration because it allows the model to assign probabilities to all trajectories based on their “goodness” (reward), rather than focusing solely on a single optimal path. Therefore, GFlowNets can explore a broader solution space, discovering multiple ways to complete a task during sampling. This contrasts with methods like SFT or PPO (Line.131), which tend to constrain the diversity of generated solutions due to reward-maximization. Building on this intuition, our work is the first to formulate multi-step LLM reasoning as Markovian flow, which makes it possible to adapt GFlowNets approaches into our new settings for diverse reasoning paths. We will revise this part to be more clear in the next version.

W1.2: The authors could consider presenting the motivation and broader rationale behind key design choices before delving into technical and mathematical specifics.

Response-W1.2:

Thanks for the suggestion! We would first like to note that one of contributions of our work lies in formulating LLM reasoning as a Markovian flow. This formulation enables us to connect to existing successful approaches (proposed in other settings) and adapt them to the new multi-step LLM reasoning setting. We would like to highlight that the ability to re-use and generalize existing successful approaches to our new problem setting (instead of having to re-invent all details from scratch) is a key advantage of our work.

Below we clarify motivations of each component, and will make this clearer in the revised paper.

1. Why trajectory balance?

As noted in Line.233, the trajectory balance objective has demonstrated greater efficiency than other learning objectives, like Detailed Balance (DB) mentioned in Appendix B, Line.207. We’ll add more details in the next version.

2. Why log-variance approximation?

Estimating $Z$ in the trajectory balance loss requires an additional model beyond the LLM policy, which is computationally cumbersome. As noted in Line.246, log-variance approximation provides a practical and effective alternative.

3. Why local search?

The vast trajectory space and sparse rewards make generating high-reward samples consistently challenging. Local search, as mentioned in Line.262, helps explore high-reward regions in the sample space. The ablation studies in Section 4.7 further validate the effectiveness of this approach.

W1.3.1: Benefit from a case study illustrating the diverse reasoning paths generated by FoR, which other methods fail to find, would be informative.

Response-W1.3.1:

Please refer to the Qualitative Results part in the General Response.

W1.3.2: a discussion on why FoR achieves such high performance while focusing on diversity would be valuable.

Response-W1.3.2:

FoR achieves high performance through its inherently exploratory nature [2], which uncovers multiple ways to accomplish a task, with each solution being sampled in proportion to its "goodness" or reward. This ensures diverse, effective solutions are consistently represented. As shown in the Qualitative Results part in the General Response, SFT samples often get stuck on specific incorrect trajectories, repeatedly failing to explore alternative solutions, whereas our method effectively discovers multiple valid trajectories, ensuring diverse exploration and avoiding over-reliance on a single trajectory.

Dear Reviewer,

This is a gentle reminder that the discussion period is closing soon. We've responded to all concerns, providing detailed clarifications and new experimental results. Please feel free to let us know if you have any further questions. We'd greatly appreciate it if you could consider raising scores and/or engaging in further discussions. Thanks!

Thank you for your thorough response to my questions. I find the added experimental results insightful, and the responses comprehensively address my primary concerns. My sole remaining reservation is that the paper incorporates multiple components that might challenge readers' understanding of each component's contribution to the main scientific argument. Nonetheless, the authors have conducted extensive ablation studies that effectively demonstrate the significance of these components, substantiating their critical role in the proposed algorithm. Overall, the paper presents solid empirical evidence supporting the algorithm's effectiveness and makes a significant contribution to an important yet underexplored domain of LLM reasoning, i.e. the exploration of diverse reasoning paths in LLMs. I will recommend the paper for acceptance and adjust my score accordingly.

Thank you very much for adding additional explanation. Thanks to the extension of the ICLR review process, I now have more time to thoroughly read the paper and the rebuttal again.

I am concerned about the "Novel Problem," as it is directly related to my Weakness1.1. More concerns will be outlined in the review block unless the authors address my other questions directly here.

It may introduce the risk of over-claiming by directly emphasizing and highlighting decisive conclusions such as "an area previously unexplored" and "We are the first to introduce the problem".

It may be better to provide a very narrow and precise definition of diversity; otherwise, repeatedly emphasizing that it is the first in the community may be perceived as unfair.

For example, considering the diversity explored in existing works [1][2][3][4], these studies have all contributed to the community by highlighting the importance and application of diversity in LLMs to some extent.

Therefore, I encourage the author to provide a stricter and more narrowly defined explanation of diversity and to exercise greater caution when making conclusions such as "unexplored" and "first."

It may even lead to greater misunderstanding and confusion when the authors emphasize, "We are the first to formulate multi-step LLM reasoning from a Markovian flow perspective." The main reason is that Markovian flow is not a novel idea but a well-established domain that has been explored for many years. Moreover, in the context of LLMs, numerous works have applied Monte Carlo Tree Search (MCTS) to formulate or organize the step-by-step reasoning process of LLMs. We can say that "they all formulate multi-step LLM reasoning from a Markovian flow perspective."

Therefore, once again, it may be unfair to directly conclude that "we are the first" without a very clear and narrowly defined declaration. Personally, I believe that the risk of over-claiming may significantly reduce the persuasiveness of the paper.

[1]. DLCRec: A Novel Approach for Managing Diversity in LLM-Based Recommender Systems, WSDM 2025

[2]. LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity, EMNLP 2024.

[3]. Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions, 2024.

[4]. Diversity of Thought Improves Reasoning Abilities of LLMs, 2023.

You refer to the contribution and the qualitative results as the answer to the first weakness proposed by the reviewer h3sb. However, after carefully reviewing the authors' responses, I believe that the current answers may be insufficient to clearly convey the major insight or observation derived from this submission. When discussing insights, it may not be merely about integer numbers or demonstrations but rather the ideas and conclusions that this paper offers and how they contribute to the community. For example, when focusing on the diversity of LLMs, what is the impact of this new objective on accuracy, and to what extent? Is there a tradeoff between the two, or does improving diversity lead to higher accuracy? What are the underlying reasons or observations behind these outcomes? What are the in-depth insights into the corresponding designs?

For the final question, what confused me was that, upon reading the paper, many concepts and observations discussed appeared to originate from existing works, leading the authors to cite numerous papers in each part of the design section. Additionally, the authors made only minor direct modifications. As a result, as I previously mentioned, these limitations make the paper resemble more of a technical implementation rather than an academic contribution.

The numbers presented by the authors in the paper and the rebuttal are certainly an important aspect, but it is even more important to present new findings and observations that can inspire other researchers in the field.

With more insights and discussions presented in the paper, the numbers would become more convincing.

I believe that the lack of in-depth insights and well-organized motivations in the method design sections is likely why many reviewers assigned a negative score to the 'presentation' category. Reading the paper feels more like reading a technical report on a project.