Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search

Dear reviewer YpLW,

Many thanks for your professional, detailed, and valuable reviews. We are encouraged by your recognition of CoMCTS as a novel approach, the valuable resource of the Mulberry-260K dataset, and the strong performance of Mulberry models. We will do our best to address your concerns one by one.

---

W1: Discussion of the Uniqueness and Distinction of CoMCTS

Thank you for raising this concern. We respectfully believe that LE-MCTS and our paper are concurrent works (completed within the same week). Given its relevance and excellence, we will include a discussion of the differences between our work and LE-MCTS, and cite it appropriately in our revised paper. Below, we summarize the key distinctions between CoMCTS and LE-MCTS:

1. Different objectives. Our work uses CoMCTS to generate a novel, effective, and reflective reasoning dataset for MLLM training, whereas LE-MCTS focuses solely on inference for LLMs.
2. New dataset constructed by CoMCTS: We use CoMCTS to build a large-scale reasoning and reflection dataset, which we hope will benefit and facilitate further research in the community.
3. Different MCTS design. In each iteration, our model generates a complete reasoning path, followed by pruning error nodes, as detailed in Lines 146-162. We believe this approach is more suitable for LLM/MLLM compared to traditional step-by-step generation, making data generation more efficient. Unlike classic MCTS methods in domains like Go or multi-agent systems that require step-wise search due to external feedback, reasoning tasks with MLLMs allow for generating an entire reasoning chain at once, reducing redundant computation and accelerating the search process. Additionally, our experiments show that CoMCTS significantly reduces the average search iterations compared to LE-MCTS. ||Le-MCTS|CoMCTS| |-|-|-| |Average Search Iteration|32.8|12.7|
4. Reflection design. We introduce a new method that leverages error sibling nodes to construct reflection data. To our knowledge, this paper is the first to explore the reflective capabilities of MLLMs through tree search and collective knowledge.

---

W2: Clarification of CoMCTS results

The performance improvement from MCTS alone appears marginal. For instance, in Table 2, GPT-4o direct achieves 58.2, while GPT-4o + CoMCTS reaches only 63.8—a modest gain given the increased computational overhead.

Thank you! We believe there may be some misunderstanding regarding the value of GPT-4o + CoMCTS in Tab 2. The improvements brought by CoMCTS primarily result from collaborative search with multiple models, as shown in the last row of Table 2 (58.2% -> 80.3%) and the search success rates in Table 4 . In Table 2, the “GPT-4o + CoMCTS” setting uses only a single model, but the MCTS configuration is identical to that of CoMCTS, such as the reasoning and error localization prompts. So we named it GPT-4o + CoMCTS. We will revise it to provide more clear clarifications and will change “GPT4o + CoMCTS” to “GPT-4o + MCTS”.

The more significant improvement seems to stem from ensembling the Qwen2-VL-72B model rather than MCTS itself.

We conducted experiments comparing the search success rate using MCTS with a single GPT-4o, a single Qwen2-VL-72B, and CoMCTS with GPT-4o + Qwen2-VL-72B. The results below show that the search performance of individual models is limited (i.e., 63.8 for GPT-4o, 56.1 for Qwen2-VL-72B), and the main performance gain comes from collective intelligence (i.e., 74.6 for GPT-4o + Qwen2-VL-72B) rather than from a single powerful model. Besides, we have provided the direct prediction results using only Qwen2-VL-72B in Table 9 of Appendix D.1. These results show that the reasoning capability of Qwen2-VL-72B alone is limited (48.3), further demonstrating that the benefits of CoMCTS primarily stem from collective intelligence rather than from a single strong MLLM.

---

W3: Effectiveness of collective learning group and computation overhead

Thank you!

The effectiveness of CoMCTS hinges on the diversity and quality of the models in the collective learning group.

We respectfully clarify that a single MLLM for reasoning is ineffective, which motivates us to leverage collective intelligence for collaborative reasoning. Employing stronger models for collaborative reasoning and error localization can further improve the results of CoMCTS (see our response to Reviewer H5ne’s Q2 for details), highlighting the long-term potential of collective intelligence and collaborative learning for future advances.

Additionally, the reliance on multiple models for collective learning and iterative tree search increases the computational overhead.

Regarding computational overhead, inference with smaller models such as Qwen2-VL-7B is relatively fast (1s per inference), and GPT-4o API calls are also highly efficient. Although larger model Qwen2-VL-72B is more resource-intensive, utilizing vLLM for inference can significantly accelerate the process (2-3s per inference). While utilizing multiple models for search inevitably increases computational overhead, we believe that a more important benefit is the improvement in the quality of the generated reasoning data. When training the Mulberry model with data searched via CoMCTS, using the same base MLLMs and a comparable data scale, the results outperform both LLaVA-CoT and LLaVA-Reasoner, as shown in Tab 1, Tab 6, Tab 16, and in our responses to reviewer GUCu's W1 and reviewer H5ne’s Q1.

---

Q1: Innovations of CoMCTS and its Relation to Other MCTS Methods

Thank you! We would like to kindly clarify that we discuss the relationship of our paper to other MCTS methods in Appendix E.1 and to other reasoning MLLMs in Appendix E.2. Additionally, we will include the discussions from W1 in the revised version of our paper.

Below, we summarize the key contributions of CoMCTS and highlight the differences compared to previous MCTS methods:

1. we introduce collective learning into MCTS and propose CoMCTS, which leverages collective knowledge to collaboratively conjecture, search and identify effective and reflective reasoning paths for MLLMs, significantly improving search effectiveness and efficiency. To our knowledge, this is the first work that explores collective learning with MCTS for MLLMs.
2. Its joint expansion mechanism allows CoMCTS to combine reasoning trajectories from multiple MLLMs through iterative search, constructing a unified reasoning tree with diverse and complementary nodes. This leverages the strengths of different models while avoiding traps in low-quality, homogeneous nodes within a single MLLM's reasoning space.
3. In each search iteration, CoMCTS generates a complete reasoning trajectory and selects the last correct step as the new starting node, skipping intermediate steps to significantly reduce search time without sacrificing effectiveness. Furthermore, collective knowledge enhances error localization by enabling mutual validation among models.
4. Using CoMCTS, we search for effective, reflective reasoning paths for multimodal inputs and construct Mulberry-260k, a dataset containing rich, explicit reasoning trees for each question.
5. To our knowledge, Mulberry is the first work to explore the reflective capabilities of MLLMs through tree search and collective knowledge.

---

Q2: Measuring CoMCTS search phase overhead

Thank you for your question. Below, we provide detailed estimates of the total token usage, GPU hours, and API costs incurred during data generation in the CoMCTS search phase.

---

Q3: Statistics on the contribution of each model in CoMCTS

Thank you for your valuable suggestions. We agree that providing statistics on the contribution of each model to CoMCTS search is important and informative.

the similarity in sampling between different models

To compute similarity, we first encode the reasoning paths generated by different models into embeddings using the all-MiniLM-L6-v2 model. We then calculate the similarity between these embeddings. The results are shown in the table below.

or the proportion of each model in generating correct paths

In addition, we also report the contribution of each model to the generated correct paths, as shown below.

---

Q4: CoMCTS results without smaller models

Thank you!

Is it possible to achieve good performance without introducing smaller models, for example, by only using GPT-4o and Qwen2-VL-72B?

We agree that only using larger foundation models can achieve good performance. However, incorporating smaller models provides additional benefits with minimal computational overhead. As shown below, adding two small models leads to a 5.6% improvement in search success rates.

For Qwen2-VL-72B and Qwen2-VL-7B, their behaviors might be similar due to shared training data—does integrating both models still provide an advantage?

The similarity analysis from Q3 also shows that while Qwen2-VL-7B and Qwen2-VL-72B generate relatively similar reasoning responses, they are not identical. They can still collaboratively contribute diverse ideas.

---

Thank you again for your insightful and professional comment, which made our work more complete and solid! If there are any further questions, please let us know. If you feel all questions have been addressed, we would be grateful if you could kindly consider re-rating our work. Thanks!

Thank you once again for your thoughtful review, which has greatly improved the quality and clarity of our paper.

In the revised version, we will highlight the innovations of CoMCTS and the Mulberry models, and we will include a discussion of the differences between LE-MCTS and CoMCTS. Furthermore, we will incorporate the results and discussions from the rebuttal period into the revised paper. If you have any further questions or suggestions, please feel free to reach out. We will remain actively engaged in the discussion until the end of the rebuttal period.

Dear Reviewer GUCu,

Many thanks for your professional review! We are pleased that you have recognized our key contributions: 1) the effectiveness and efficiency of CoMCTS; 2) reasoning and self-correct capabilities of Mulberry; 3) high-quality tree-structured multimodal reasoning datasets Mulberry-260k. Below, we tried our best to respond to your concerns and suggestions one by one.

---

W1: Additional evaluation of CoMCTS on pure-text LLM & More results on mathematics benchmark.

Thank you for your valuable feedback! We are going to respond to this concern in two parts:

Although CoMCTS itself is not specific to MLLMs, the evaluation is conducted solely on MLLMs.

Additional evaluation of CoMCTS on pure-text LLM. As stated in Lines 29-30 and 36-37 and throughout the paper, our work is motivated by the limited reasoning capabilities of existing MLLMs, which prompted us to design CoMCTS for collaboratively searching high-quality reasoning data for MLLMs. Accordingly, this paper focuses on multi-modal reasoning data search and MLLM training, and our evaluations have thus been conducted on MLLMs. However, to address your insightful question, we conducted an experiment using 100 cases sampled from DAPO-Math, showing that collective intelligence also improves search success rates in pure text reasoning tasks using LLMs, as presented in the table below.

From the outset of Section 1, the research assumes an MLLM-based setup, but it would be necessary either to include evaluations on tasks like mathematics or to provide a justification for not addressing such tasks.

More results on mathematics benchmark. We agree that evaluating mathematics tasks is both necessary and important for assessing the reasoning abilities of MLLMs. To this end, we would like to kindly clarify that we have reported the performance of Mulberry models on mathematics benchmarks such as MathVista, DynaMath and MM-Math in Table 1. In response to your suggestion, we have further expanded our evaluation to include additional mathematics benchmarks, including MathVision, MathVerse and WeMath, as shown in the table below. The results show that Mulberry models consistently achieve substantial improvements over base models and reasoning MLLMs across all evaluated mathematics benchmarks. We believe these results further demonstrate the effectiveness of our approach. Thank you! We will include these results in the revised version.

---

W2: Effect of collective intelligence vs. single powerful MLLM (Qwen2-VL-72B).

Thank you for raising this concern. We conducted experiments comparing the search success rate on 100 samples using MCTS with a single GPT-4o, a single Qwen2-VL-72B, and CoMCTS with GPT-4o + Qwen2-VL-72B. The results below show that the search performance of individual models is limited (i.e., 63.8 for GPT-4o, 56.1 for Qwen2-VL-72B), and the main performance gain comes from collective intelligence (i.e., 74.6 for GPT-4o + Qwen2-VL-72B) rather than from a single powerful model. Additionally, we have provided the direct prediction results using only Qwen2-VL-72B in Table 9 of Appendix D.1. These results also show that the reasoning capability of Qwen2-VL-72B alone is limited (48.3), further demonstrating that the benefits of CoMCTS primarily stem from collective intelligence rather than from a single strong MLLM.

---

Lastly, thank you very much for your constructive feedback and suggestions. We will incorporate these points into the revised paper. If you have any further questions, please feel free to let us know. We will be available throughout the rebuttal period.

Dear Reviewer H5ne,

We appreciate the reviewer’s valuable comments and feedback. Thank you for recognizing the clear organization of this paper and for highlighting the contributions of the Mulberry-260K dataset and Mulberry models to future research in multimodal reasoning. We are also grateful for your recognition that even small multimodal models can benefit from enhanced reasoning and reflective abilities. We have carefully responded to each of your concerns and suggestions below, and will revise the manuscript accordingly.

---

W1: The novelty and insights of CoMCTS and Mulberry Models.

Thank you for your question! We believe our work, including CoMCTS and Mulberry Models, offers unique innovations. Below, we first summarize the key contributions of our paper, and then highlight the differences between CoMCTS and other MCTS-based methods, as well as between Mulberry and other reasoning MLLMs.

Contributions: First, we introduce collective learning into MCTS and propose CoMCTS, which leverages collective knowledge to collaboratively conjecture, search and identify effective and reflective reasoning paths for MLLMs, significantly improving search effectiveness and efficiency. To our knowledge, this is the first work that explores collective learning with MCTS for MLLMs. Second, we construct Mulberry-260K, a valuable resource for step-by-step reasoning and reflection research in MLLMs. Third, we develop Mulberry, a series of MLLMs with strong step-by-step reasoning and reflection abilities. Fourth, extensive experiments validate the superiority of our methods across various benchmarks.

Relation to Other MCTS Methods: We summarize the contributions of CoMCTS and highlight the differences compared to previous MCTS methods (e.g., Traditional MCTS, ReST-MCTS): (a) CoMCTS introduces collective learning into tree search for MLLMs, enabling more effective and efficient reasoning path searching and learning. (b) Its joint expansion mechanism allows CoMCTS to combine reasoning trajectories from multiple MLLMs through iterative search, constructing a unified reasoning tree with diverse and complementary nodes. This leverages the strengths of different models while avoiding traps in low-quality, homogeneous nodes within a single MLLM's reasoning space. (c) In each search iteration, CoMCTS generates a complete reasoning trajectory and selects the last correct step as the new starting node, skipping intermediate steps to significantly reduce search time without sacrificing effectiveness. Furthermore, collective knowledge enhances error localization by enabling mutual validation among models.

Relation to Other Reasoning MLLMs: We summarize the contributions of Mulberry models and highlight the differences compared to previous reasoning MLLMs: (a) Using CoMCTS, we search for effective, reflective reasoning paths for multimodal inputs and construct Mulberry-260k, a dataset containing rich, explicit reasoning trees for each question, which is then used to train the Mulberry models. (b) Unlike pre-defined stage-based methods (e.g., LLaVA-CoT, LLaVA-Reasoner), CoMCTS searches step-level reasoning data of varying lengths, enabling Mulberry to perform flexible step-by-step reasoning. (c) To the best of our knowledge, Mulberry is the first work to explore the reflective capabilities of MLLMs through tree search and collective knowledge.

We hope these clarifications help to highlight our contributions and the novelty of our work.

---

Q1: Supplementing the values in Table 1.

Thank you for your suggestions. Some values are missing because the original papers did not report results on certain benchmarks, which resulted in incomplete values in our Table 1. Many thanks for your feedback. We have now conducted additional evaluations on these models ourselves to provide a more comprehensive comparison. During the rebuttal period, we present a part of results in the table below, which shows that the Mulberry models outperform other reasoning MLLMs across the overall benchmarks using same base MLLMs. The remaining missing values will be included in the revised paper.

We believe offering these missing values would improve the completeness of our comparisons. Thanks!

---

Q2: CoMCTS results with four strong foundation models.

Thank you for your question! In the table below, we present results of CoMCTS using four strong foundation MLLMs (GPT-4o, Claude-3.5 Sonnet, Qwen2-VL-72B and InternVL2-40B). As model strength increases, CoMCTS is able to generate more accurate reasoning steps and error localization, resulting in higher search success rates. These findings also highlight the long-term potential and scalability of CoMCTS as foundation models become increasingly powerful. We will include these discussions in the revised paper.

---

Q3: Open source CoMCTS code, data, and models.

Thank you for raising this point. We would like to clarify that we will release all resources, including the CoMCTS search code, training data, training scripts, model checkpoints, and evaluation code, to facilitate reproducibility and further research.

---

Q4: Addressing writing problems at L68 and L123–125.

Thank the reviewer for pointing out these writing issues, which help improve the clarity and quality of our paper. We have removed the extra space at L68 and revised the citation in L123–125 to use \cite{} as recommended in the revision.

---

Lastly, thank you so much for helping us improve the paper and we appreciate your open discussions! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!

Thank you again for your review, which has greatly improved the quality of our paper.

In the revised version, we will highlight the novelty and insights of CoMCTS and the Mulberry models, and we will include all additional experimental results. Furthermore, we will open-source all resources to support and facilitate future research. Please feel free to reach out if you have any further questions or suggestions.

Dear Reviewer Wx49,

We appreciate your valuable comments and suggestions. We are encouraged by your recognition of our contributions in constructing a large-scale multimodal reasoning dataset, the strong performance of the Mulberry models, and the clear utility of each CoMCTS component shown in our ablations. We have carefully considered all your suggestions and will revise the manuscript to include a more detailed discussion on token usage and improve the overall organization as suggested.

---

W1: Comparison of necessary token usage.

Thank you for your suggestion. We agree that including a comparison of token usage will help provide a more thorough understanding of this area. Therefore, we provide a comparison between CoMCTS and other CoT data construction methods regarding search token usage per question, search success rate, and token usage per sample for successfully finding the correct answer, as shown in the table below. Although CoMCTS consumes more tokens during the search phase, it achieves higher search success rates and data quality, leading to superior performance of the Mulberry models.

Additionally, we manually evaluated the correctness of reasoning paths in 100 cases generated by CoMCTS and the direct prediction method (e.g., LLaVA-CoT). The results show that by leveraging collective error localization, CoMCTS significantly improves the accuracy of intermediate reasoning steps.

---

W2: Better organization of the experimental results.

Thank you very much for your valuable suggestion. We agree that incorporating the contents from Appendix C.1 and C.3 into the main paper could improve the clarity and comprehensiveness of our experimental comparisons. In the revised version, we will move the results and analyses from Appendix C.1 and C.3 into the main text, and we will update Table 1 to clearly indicate that Mulberry is compared with both base models and other reasoning MLLMs using the same backbone. We believe these changes help to better highlight the effectiveness of our approach and address your concern.

---

Many thanks for your professional, detailed, and valuable reviews! Please let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period.

We're glad to hear that your concerns have been adequately addressed. We appreciate your professional and constructive feedback which made our work more solid and clear.

We'll revise the paper based on our discussions to better present Mulberry. If you have any questions or suggestions, please feel free to comment here. Thank you.