Responses to Reviewer YDqj [Part 3/3]

---

(Continued from the previous page. W2&W3)

W4: "Can FTBI further improve performance upon stronger open-source baselines?"

Based on your suggestion, we conduct the fine-tuning experiment using LLaVA-NeXT, aiming to investigate whether a more advanced MLLM can enhance the performance of the FTBI model. The selected base model is llama3-llava-next-8b, and the training dataset is LLaVA-NeXT's visual instruction tuning dataset. The new results are presented as follows.

From the table, incorporating detection information improves LLaVA-NeXT's performance on benchmarks related to object detection and text recognition. Moreover, the LLaVA-NeXT version of the FTBI model demonstrates superior overall performance compared to both the original LLaVA-NeXT and the TFI model. These results align with the experimental conclusions presented in our paper.

Thank you for your suggestion. We have included this subsection in Appendix D.1 of our revised paper.

---

W5: The Two Paradigms to Incorporate Detection Experts

We have compared these two paradigms in our related works (lines 150–187). As mentioned, the former approach is often similar to multi-agent methodologies, being plug-and-play and requiring no training. Whereas the latter often necessitates training for new adapters and may require substantial amounts of data. However, with ongoing advancements in this field, the need for training data is becoming less extensive. So it will not be a disadvantage in the future.

When determining the research focus of our paper, we have conducted comparative experiments on these two approaches. Specifically, we evaluated three methods: (1) directly converting the outputs of detection models into textual description, (2) using a newly initialized MLP layer to map the output features of detection models , and (3) fusing detection features with CLIP features using a six-layer Cross Attention mechanism. Based on these methods, we incorporated the outputs of detection models into MLLMs and conducted re-training of LLaVA-1.5-7B through both pre-training and fine-tuning. Finally, we evaluated these methods on the VQA-v2, TextVQA, MME-Cognition, and POPE benchmarks, with the results summarized as follows:

Under the same training conditions, the text-based approach achieves the best performance, effectively transferring the outputs of detection models to the MLLM and enhancing its capabilities. In contrast, due to the limited amount of training data, the newly initialized structures of the MLP and Cross-Attention methods cannot be adequately trained, resulting in suboptimal performance. Therefore, based on our comparisons, we believe that conducting adaptive training research centered on the text-based approach is more likely to yield significant results.

Many researchers in the relevant field adopt the former approach for their studies, as it is more straightforward and delivers noticeable results. By simply incorporating external text descriptions into the MLLMs, they can significantly improve the performance of MLLMs. We conduct experiments around the former because there has not been a comprehensive comparison between training-free and adaptive training methods using this approach, and we hope our findings will inspire researchers who use the former approach.

Besides, the references you provide are highly valuable, and we have included them in the related works section of our revised paper.

---

Closing Remarks

We sincerely appreciate the valuable time you have dedicated to reviewing our work, as well as the insightful feedback you provide. As per your suggestions, we invest considerable time and effort into conducting additional experiments. We hope these experiments address your concerns and lead you to raise the score. Thank you once again for your time and effort.

Responses to Reviewer YDqj [Part 2/3]

---

(Continued from the previous page. W1)

W2: "Would using the LVIS-trained detector improve FTBI performance?"

As per your suggestion, we conduct the training-free and fine-tuning experiments using Co-DETR-LVIS, aiming to investigate whether a closed-set detection model with a broader detection range could further enhance the performance of the FTBI model. The new results are as follows:

We can derive the following points from the table:

• Under the training-free condition, the TFI model based on Co-DETR-LVIS performs worse than the DINO-based TFI model across almost all benchmarks. After analysis, we believe that this is because Co-DETR-LVIS introduces more noise compared to DINO, as it detects a significant number of redundant objects.
• After fine-tuning, the MLLM gains the ability to mitigate the noise introduced by Co-DETR-LVIS. Consequently, the FTBI model based on Co-DETR-LVIS achieves comprehensive performance improvements over its TFI counterpart. This observation is consistent with the conclusions presented in our paper.
• Furthermore, when comparing the FTBI model based on Co-DETR-LVIS with the FTBI model based on DINO, it is evident that the Co-DETR-LVIS-based model performs significantly worse, exhibiting inferior results across all nine benchmarks.

In summary, detection models capable of identifying a wider range of objects do not necessarily improve the performance of the FTBI models. We think this is because many of the objects they detect are redundant and may instead introduce noise, leading to a decrease in performance scores. We have included these additional experiments in Appendix D.6 of our revised paper.

---

W3: "The proposed method with an open-set detector is similar to VisualCoT."

The purpose of introducing the experiment related to Grounding DINO is not to propose a new method, but rather to verify that our fine-tuned model's effectiveness can still be maintained after replacing the detection model. Grounding DINO introduces a lot of noise, and it generates information differing from DINO's due to the differences in detection range. In light of this, our aim is to validate that the fine-tuned model can retain its denoising capabilities and understanding of specialized detection information**, after the detection model being replaced without additional training**.

Here, we aim to expand on the ablated factor for our systematic study on detection information infusion, rather than proposing a new method.

(To be continued.)

Responses to Reviewer YDqj [Part 1/3]

---

Dear Reviewer YDqj, we sincerely thank you for your recognition of the novelty and comprehensive experiments in our work! Following your constructive suggestions, we conduct several additional experiments, which require considerable time and effort. Below, we provide detailed responses to each of your comments, with the hope that this will encourage you to lean toward the acceptance of our paper.

---

Response Overview

1. More Experiments

• W1: We conduct experiments using YOLOv5N and YOLOv11L, and compare the impact of detector performance on the performance improvement of MLLMs.
• W2: We conduct experiments based on Co-DETR-LVIS to explore the impact of a broader object detection scope on MLLM performance.
• W4: We conduct experiments based on LLaVA-NeXT and validate that our conclusions remain effective on a stronger MLLM.

2. Clarification of the Reviewer's Misunderstanding

• W3: We clarify that the experiment involving an open-set detector is for investigating whether the training effects brought by the fine-tuning can be inherited, rather than proposing a new method.
• W5: We explain that the two paradigms of integrating detection models do not have a clear advantage over each other. Additionally, we clarify why our work focuses on the first approach.

---

W1: The Impact of Detector Performance

The selected object detection model, DINO, achieves a high mAP score of 58.5 on the COCO benchmark. It is challenging to identify another object detection model that offers significantly better performance than DINO while maintaining a comparable fast inference speed. To address your concern, we conduct the FTBI experiments again based on two object detection models with markedly different performance levels for comparison. They are YOLOv5N (mAP 34.3) and YOLOv11L (mAP 53.4). The new results are presented as follows:

We can summarize the following points:

• Under the training-free strategy, for general VQA capabilities, YOLOv5N introduces noise to LLaVA-1.5-7B, resulting in performance degradations. In contrast, YOLOv11L, due to its superior performance, introduces minimal noise and thus has little negative impact.
• Regarding the object detection-related capabilities, both YOLOv5N and YOLOv11L bring performance improvements under the training-free strategy. However, the improvement from YOLOv5N is noticeably smaller than that from YOLOv11L, which can be attributed to the different model performance.
• Furthermore, after fine-tuning, both two versions of the MLLM achieve comprehensive performance improvements, surpassing the original LLaVA-1.5-7B. Moreover, the model with YOLOv11L consistently outperforms the model with YOLOv5N across all benchmarks.

Therefore, high-performing object detection models can indeed bring more performance gains to MLLMs than low-performing object detection models. Thank you for your suggestion! We have included this discussion in Appendix D.3 of our revised paper.

(To be continued.)

Responses to Reviewer JV9J [Part 3/3]

---

(Continued from the previous page. W2&W3)

W4.1:"Why is the retraining strategy not training visual encoders?"

The retraining strategy refers to training from scratch as advanced MLLMs do, rather than directly retraining off-the-shelf models. All MLLMs must undergo training from scratch, and our retraining strategy simply refer to this process.

In our work, we do not train the visual encoder because the baseline we use, LLaVA-1.5-7B, also keeps the visual encoder frozen during training. To address your question more thoroughly, we conduct additional experiments as requested, where we unfreeze the visual encoder. We repeat both the retraining and fine-tuning processes. The new results are presented as follows, where "TVE" denotes training with the visual encoder unfrozen.

As shown in Table 1, even with the visual encoder being trained, the performance of the training-free, retraining, and fine-tuning strategies aligns with the patterns summarized in our paper. Specifically, the RBI model outperforms the training-free model, while the FTBI model further surpasses the RBI model. Moreover, the fine-tuned model achieves the best performance in 9 out of 10 benchmarks while training with the visual encoder unfrozen.

Furthermore, Table 2 presents the performance of RBI and FTBI models when the detection information is not dynamically incorporated during inference. It demonstrates that, under the condition where the visual encoder is unfrozen, the fine-tuned model still maintains comparable performance to the original LLaVA-1.5-7B, while the RBI model performs worse than the original model. This indicates that the fine-tuning strategy better balances the contributions of the image encoder's outputs and the detection information, thereby facilitating a more effective understanding of detection cues. These findings are consistent with the conclusions presented in our paper.

Thank you for your valuable comment. We have included this discussion in Appendix D.5 of our revised paper.

---

W4.2: "... the different backbone is used for retraining"

We note a misunderstanding by reviewer regarding this point. Both the retraining strategy and the fine-tuning strategy utilize the Vicuna-1.5 backbone. The term "LLaVA-1.5" in the figure refers to the Vicuna-1.5 architecture as well, as the LLM backbone of LLaVA-1.5 is also based on Vicuna-1.5.

We distinguish between "Vicuna-1.5" and "LLaVA-1.5" in the figure for clarity:

• For the retraining strategy, the LLM backbone is initialized with the original weights of Vicuna-1.5. Because we need to replicate the pretraining and fine-tuning stages of LLaVA-1.5 from scratch.
• For the training-free and fine-tuning strategies, "LLaVA-1.5" represents that we use the Vicuna-1.5 weights derived from LLaVA-1.5 to initialize the LLM backbone, which have already been fine-tuned on the LLaVA-1.5 dataset.

We have detailed this distinction in our paper, specifically in lines 258–279.

---

Closing Remarks

We would like to express our sincere gratitude for your suggestions and queries. We hope that our responses have addressed your concern and that you will consider re-evaluating our paper. Please do not hesitate to contact us if you have any further inquiries or recommendations. We appreciate your time and feedback immensely.

Responses to Reviewer JV9J [Part 2/3]

---

(Continued from the previous page. W1)

W2: "... overlooks the effectiveness of training-free methods."

We don't overlook the efficiency of training-free methods. In lines 51-53 and 182-187 of our paper, we clearly state that training-free approaches are simple, resource-efficient, and cost-effective. In our paper, we objectively conduct a comparative analysis of the performance of three different training strategies. The conclusion that the performance of training-free models is inferior to that of the trained models is based on objective experimental results.

Regarding resource consumption, our fine-tuning strategy uses the visual instruction tuning dataset of MLLMs, which is relatively small in size and does not require excessive training time. Given the significant performance improvement, this approach is highly cost-effective. Additionally, in Section D.1 of the appendix, we demonstrate that fine-tuning Qwen-VL with LLaVA-1.5's small-scale fine-tuning dataset (665K) achieves excellent results. This suggests that fine-tuning does not necessarily require the model's original dataset, and high-quality VQA datasets with small scales are also effective. Furthermore, we try removing the detection data of the LLaVA-1.5 fine-tuning dataset (from 665K to 450K), and the resulting model still performs excellently. The results are presented in the table below, and the corresponding model name is "LLaVA-1.5-7B-FTBI-FNDI". The data size is already very small, and we can further explore reducing the data size to improve efficiency even more.

---

W3: "The novelty of the proposed methods is limited..."

Our work focuses on comparing the performance gains brought by the training-free strategy and the adaptive training strategies for employing detection models to assist MLLMs, rather than proposing a new method. The training approach we select aligns with the training paradigms of advanced MLLMs. Specifically, the introduction of LoRA modules achieves good results in both the pre-training and fine-tuning stages of MLLM training. This training method is sufficient to enable an objective comparison of different training strategies.

Most research on deploying detection models to assist MLLMs is based on a training-free approach. To our knowledge, there are no exisiting studies that systematically investigate the comparison between the training-free and adaptive training methods. With the aim of inspiring further research in this area, we first explore this aspect. Our work is innovative.

(To be continued.)

Responses to Reviewer JV9J [Part 1/3]

---

Dear Reviewer JV9J, we sincerely appreciate the time and effort you have dedicated to reviewing our paper, as well as the valuable feedback and suggestions you provide! However, after carefully reading your response, we believe you may have misunderstood certain aspects of our work. In light of this, we address your concerns one by one in our rebuttal and hope that you will consider re-evaluating our paper.

---

Response Overview

We give a quick summarization of our response and present more details later.

1. More Experiments

• W4.1: We explain that we don't train the visual encoder because the baseline model we choose, LLaVA-1.5, does not train it either. To address the reviewer's concern, we unfreeze the visual encoder and conduct additional experiments, which yields results consistent with the conclusions of our paper.

2. Clarification of the Reviewer's Misunderstanding

• W1: We explain the significant differences between our paper and related works such as MiniGPT4-v2 and VisionLLM.
• W2: We clarify that we don't overlook the efficiency of training-free methods, and we explain that the adaptive training in our paper does not consume significant training resources.
• W3: We explain that our work is the first systematic investigation into the impact of adaptive training on deploying detection models to assist MLLMs, which is an innovative contribution to the field.
• W4.2: We clarify the reviewer's misunderstanding and explain that all training strategies use the same LLM backbone, specifically Vicuna-1.5.

---

W1: "The key insights may seem to reiterate the existing observations of the related papers."

The focus of these works is different from ours. MiniGPT4-v2, VisionLLM, and Shikra (mentioned in our paper) all enhance their datasets with a large amount of object detection data, as well as use special tokens such as "<det>" to perform object detection downstream tasks. They don't deploy independent detection models during training or inference. In contrast, our work focus employing off-the-shelf vision detection models for real-time detection information generation, similar to the multi-agent approach.

Another key distinction is that the detection information introduced by MiniGPT4-v2 and VisionLLM is completely accurate since it is derived from datasets. However, in our work, the detection models may occasionally produce errors, introducing noise that affects the training-free model.

In short, our research investigates whether adaptive training can help MLLMs better identify noise in real-time detection information and more effectively leverage the outputs of additional detection models to enhance VQA performance. This is fundamentally different from the objectives of related works like MiniGPT4-v2 and VisionLLM.

(To be continued.)

Dear Reviewer JV9J,

Thank you very much for taking the time to read our responses and re-evaluate our paper! We are sincerely grateful for your decision to increase your rating!

We notice that the reviewer still have concerns regarding “the motivation behind why we should focus on using off-the-shelf detection models rather than choosing the widely adopted baselines.” Below, we provide a detailed explanation to address this point, and we hope it will help clarify the reviewer's concerns.

---

1.No Absolute Superiority Between the Two Paradigms:

• First and foremost, we want to clarify that we do not claim that deploying detection models to assist MLLMs is inherently better than choosing those widely adopted baselines. Instead, our paper aims to conduct systematic comparative experiments based on the former paradigm, fostering the understanding of the impact of adaptive training on its performance.
• Whether by deploying off-the-shelf detection models or by introducing special tokens for object detection downstream tasks, both approaches enhance the object detection capabilities of MLLMs and enable them to better focus on detailed information within images. Specifically, if we integrate special tokens into object detection data during MLLM training, as done in MiniGPT-v2 and VisionLLM, the MLLM itself could learn to understand these special tokens and follow their guidance to perform end-to-end object detection reasoning. Alternatively, deploying independent detection models to generate real-time detection information as input could enable MLLMs to directly access image details through context-enhanced text. Each approach has its unique strengths, and neither is inherently superior to the other.

---

2.The Role of Detection Information Differs Between the Two Paradigms:

While both paradigms involve detection information, the role of this information differs significantly:

• The method of deploying detection models allows MLLMs to receive real-time detection information during both training and inference. This type of detection information encompasses the locations of all detectable objects in the image, containing rich details about the image.
• The special token method, which does not deploy detection models, requires manual input at the input stage. Such information is typically limited to a single object or a small number of objects, serving primarily as task guidance. Next, this approach requires the MLLM to output detection information in a formatted manner, guiding the MLLM to complete the detection task.

Thus, in the paradigm of deploying detection models, detection information assists MLLMs in downstream tasks by providing useful detection details. In contrast, in the special token paradigm, detection information usually acts as a signal to indicate that the task involves detecting specific targets and guides the MLLM to finish the detetion task. While both paradigms infuse detection information, their functions differ: the former serves as an auxiliary function, and the latter acts as an instructive function.

---

3.Our Study is a Pioneering Work, Offering Inspiration for Further Research:

• Deploying independent detection models (or models for other downstream tasks) to generate context-enhanced text for assisting MLLMs is straightforward and effective. Furthermore, the deployed models are interchangeable, which provides excellent scalability. Considering the strong points, an increasing number of researchers are investigating this paradigm and working based on it.
• Nevertheless, many researchers tend to adopt training-free strategies. The impact of adaptive training, however, remains an important area of investigation. Therefore, we conduct systematic experiments based on the training-free and adaptive training strategies in this paradigm, as there has not been a comprehensive comparison between them.
• Our findings demonstrate that the adaptive training strategy indeed outperforms the training-free strategy. Additionally, we confirm that fine-tuning with only a small amount of high-quality VQA data can also lead to improved performance, and the performance gain is still preserved even after replacing the detection models. As a pioneering study in this area, we have uncovered many valuable insights, and we hope our findings will inspire researchers in related fields.

---

We newly submit a revision of our paper. We have included these discussions into the Related Works Section and Appendix D.7.

We hope that our responses adequately address your comments and help strengthen your confidence in the acceptance of our paper. If you have any additional concerns or queries, we warmly encourage you to share them with us. Thank you once again for your valuable time and feedback!

Best, authors

Dear Reviewer JV9J,

We would like to know if you have any further questions regarding our latest discussion points. We hope that our responses have clarified the misunderstandings you raised. We greatly appreciate the time and effort you have invested in reviewing our paper. Thank you!

Best, authors

New Responses to Official Comment by Reviewer JV9J [Part 1/2]

Dear Reviewer JV9J,

We notice that you have not replied to our latest response. We sincerely hope that our response has addressed your remaining concerns. In addition to the previous response, below, we now provide more examples and references to support our arguments.

---

1. Comparison of Detection Information Examples: MiniGPT-v2, VisionLLM, and Ours.

We further compare the detection information examples of MiniGPT-v2, VisionLLM, and our method, providing a more detailed illustration of the significant functional differences between them.

• MiniGPT-v2

The question and answer format of MiniGPT-v2 is similar to the following example. The question is: "[grounding] please describe this image as detailed as possible," and the answer is: "<p>A crepe</p> {<38><51><90><78>} sits on <p>a white plate</p> {<29><45><100><83>}...".

MiniGPT-v2 introduces a special token "[grounding]" to indicate that the current task involves object location marking. Additionally, special symbols such as "<p>," "</p>," and "{}" are introduced to construct detection information. As shown in the example, the detection information in MiniGPT-v2 appears in the answer and is specifically used to guide the model in completing the "grounding" task (next token prediction). The MLLM follows this detection information format to output grounding results.

• VisionLLM

The question and answer format of VisionLLM follows a structure similar to the example below. The question is: "Identify the objects in <image> that belong to {'What is the child eating?': <c0>, 'red gamepad': <c1>} and draw a bounding box around each one...", while the corresponding answer is: "The bounding boxes are [(<c0>, 226.4, 229.8, 363.1, 347.4), (<c1>, 441.1, 183.5, 538.6, 269.9)]."

As shown in the example, VisionLLM introduces special tokens, such as "{}," "<c0>," and "[]," to structure detection information. These tokens are designed to specify the task, guiding the MLLM to output object location information in the required format (next token prediction).

• Ours

The question and answer format of our method follows a structure similar to the example below. The question is, "Here are the central coordinates of certain objects in this image: 2 people: {[0.25, 0.12], [0.11, 0.43]}, 1 cake: {[0.42, 0.32]}... What number is on the cake?" The corresponding answer is, "The number on the cake is '9'."

Our detection information consists of the number and locations of all detectable objects within the image and is incorporated into the MLLM's input. Unlike the detection information in MiniGPT-v2 and VisionLLM, our detection information includes the fine-grained location details of all detectable objects, which serve as auxiliary information to support the reasoning process of MLLMs.

• Comparison

MiniGPT-v2 and VisionLLM primarily use special tokens to instruct MLLMs on how to format their outputs as detection information. In this case, the special tokens and the detection information mainly serve as task guidance, guiding the MLLM to finish the grounding tasks.

In contrast, our approach utilizes detection information to supplement MLLMs' inputs with detailed visual information from images. By collecting all detectable objects within an image, we create enhanced contextual content to support and enrich the MLLMs' capabilities.

Therefore, the role of detection information in MiniGPT-v2 and VisionLLM differs from ours, as theirs serves as guidance, while ours functions as supplementary assistance.

(To be continued.)

New Responses to Official Comment by Reviewer JV9J [Part 2/2]

---

(Continued from the previous page.)

2. A Further Explanation of the Research Motivation.

Here, we further explain the motivation behind our current research. First, let us review some recent related studies: GLEE [1] enhances MLLMs' performance by feeding textual object queries into the backbone LLM. P2G [2] and IVE [3] employ detection agents to generate textual grounding clues for improved reasoning. Power-LLaVA [4] utilizes an object detector to produce textual class and location information to assist the MLLM in generating high-quality outputs. VLPrompt [5] leverages an object detector to generate target names and infer relationships, thereby aiding MLLMs in reasoning tasks. While these models all adopt the approach of deploying detection models to generate grounding information for MLLMs, none of them have explored adaptive training.

Our work systematically investigates the impact of adaptive training on the performance of the approach that deploys detection models to assist MLLMs. Our findings demonstrate that the adaptive training strategy indeed outperforms the training-free strategy. Moreover, we confirm that fine-tuning with only a small amount of high-quality VQA data can also lead to improved performance, and this performance gain is preserved even after replacing the detection models. We hope that our experimental results will inspire researchers in related fields to consider introducing appropriate adaptive training to enhance model performance.

[1] General object foundation model for images and videos at scale.

[2] Plug-and-play grounding of reasoning in multimodal large language models.

[3] Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models.

[4] Power-LLaVA: Large Language and Vision Assistant for Power Transmission Line Inspection.

[5] Vlprompt: Vision-language prompting for panoptic scene graph generation.

---

May we kindly ask if you have any further questions regarding your remaining concern? We would greatly appreciate it if you could provide us with your feedback at your convenience. Thank you very much for taking the time to review and consider our comments.

Best, authors

Responses to Reviewer j1zf

---

Dear Reviewer j1zf, we sincerely thank you for your time and efforts in reviewing our paper, as well as your feedback for this work! However, we have carefully read all your comments and believe there may be some misunderstandings regarding our work. We address each of your concerns, hoping to clarify our paper and encourage you to lean toward its acceptance.

---

Foreword

We would like to point out the key misunderstanding here. Our paper is not focused on dataset construction, but rather on exploring the integration of detection models—similar to Multi-Agent—into the MLLMs for real-time assistance. We focus on deploying vision detection models, rather than generating detection data. Additionally, we investigate the impact of applying training strategies when the detection models are introduced, as well as demonstrate how the synergy between MLLMs and vision detection models, combined with tailored training techniques, can achieve performance improvements.

---

W1.1: "The superiority of fine-tuning over training-free methods has been well-established in the literature."

Related works on fine-tuning are primarily based on introducing new datasets or modifying the architecture of models. To our best knowledge, there has been no previous work systemcally and specifically studying the effect of fine-tuning with incorporated additional detection models.

Existing research on introducing detection models typically considers training-free methods. With that in mind, it's meaningful to propose a work that demonstrates how adaptive training can lead to performance improvements with additional detection models incorporated. Our work can inspire researchers to explore better model performance while they employ vision detection models to assist MLLMs.

---

W1.2: "The comparison with retraining from scratch adds limited value."

We respectfully disagree with the reviewer's opinion. First, it is important to clarify that the retraining strategy does not imply retraining a pre-trained model; rather, they represent training from scratch as the advanced MLLMs do. In practice, all MLLMs must undergo training from scratch, and our retraining strategy simply refer to this process. Therefore, such a comparison is highly relevant to practice.

Furthermore, without incorporating a comparison with the retraining strategy, we would miss several meaningful insights presented in our paper. For instance, we demonstrate that first training an MLLM without the detection models infused, followed by fine-tuning it with the detection models infused, enables a better balance between the image encoder's output and the detection information. This valuable finding is derived from the comparison with the retraining strategy.

Thus, the comparison of the retraining strategy is valuable as it can help us understand the practical implications of adaptive training.

---

W2: Ambiguities in Dataset Construction

We would like to reiterate that our work does not focus on dataset contributions. The paper does not introduce any new datasets, and therefore, there is no discussion of data processing methods or dataset construction. Instead, our work introduces additional detection models (similar to Multi-Agent), which generates real-time detection information to assist MLLMs in reasoning.

Regarding the format of the detection information derived from the detection models, it follows related Multi-Agent or MLLM works, such as P2G [1], Power-LLaVA [2], and CogVLM [3]. The experimental results in these papers have already validated the effectiveness of the format as "object + coordinates". Moreover, as demonstrated by the experiments in our paper, this approach is indeed effective.

Furthermore, we have provided a clear explanation of the "infusion" strategy within our paper. For instance, Figure 2 (lines 216-227) clearly illustrates how the textual detection information passes through the LLM's embedding layer before being concatenated with the image features at the embedding level. Additionally, Section 3.2 (lines 245-254) provides a detailed and specific description of the infusion process.

---

[1] Plug-and-play grounding of reasoning in multimodal large language models.

[2] Power-llava: Large language and vision assistant for power transmission line inspection.

[3] CogVLM: Visual Expert for Pretrained Language Models

---

Closing Remarks

We sincerely appreciate the valuable time you have spent in reviewing our paper. We respectfully ask you to take another look at the merits of our work. We hope our responses have addressed your comments and can enhance your confidence in the acceptance of this paper. If you have any additional concerns or queries, we warmly invite you to share them with us. Thanks again!

Responses to Reviewer Excq [Part 1/4]

---

Dear Reviewer Excq, we greatly appreciate your recognition of the practical relevance and the robust evaluation of our work! Thank you for your constructive and detailed review. We make the following detailed responses point by point to address your comments with substantial experimental support. We hope the responses will convince you to lean more toward the acceptance of our paper.

---

Response Overview

We first summarize our response and show more details below.

1. More Experiments

• W2: We evaluate our models on two DocumentVQA benchmarks, DocVQA and InfographicVQA.
• W3: We conduct experiments based on YOLOv5N and YOLOv11L to investigate the impact of detection model accuracy on MLLM performance.
• Q2.2: We remove the detection data from the instruction tuning dataset and repeat the FTBI experiment, aiming to investigate whether the model can still maintain a broad language comprehension capability.

2. Clarification of Certain Aspects of Our Paper

• W1 & Q1: We have already discussed the training cost in our paper. And we add a statement of the memory usage.
• Q2.1: We explain that the discussion of model overfitting is already in our paper.

---

W1 & Q1: Time Consumption and Memory Requirements

We have already detailed the training costs in Appendix B.6. The reviewer may have overlooked it. Specifically, on four A100 GPUs (80GB), the time consumption is as follow:

• For pretraining LLaVA-1.5-7B, the time increases from 6 hours to 11 hours.
• For pretraining LLaVA-1.5-13B, the time increases from 11 hours to 17 hours.
• For fine-tuning LLaVA-1.5-7B, the time increases from 16 hours to 22 hours.
• For fine-tuning LLaVA-1.5-13B, the time increases from 26 hours to 33 hours.

Regarding the memory requirements, deploying detection models results in an additional GPU memory usage of up to 4GB in each GPU compared to not deploying detection models.

Considering the performance improvements achieved, these time costs and memory costs are relatively minor and cost-effective. Moreover, researchers can opt for detection models with faster inference speeds, lower memory usage, and higher efficiency, further reducing the resource consumption and improving model performance.

We have made these details clearer in the revised version of our paper and marked them in red.

(To be continued.)

Responses to Reviewer Excq [Part 2/4]

---

(Continued from the previous page. W1&Q1)

W2: Evaluations on DocumentVQA

Following your suggestion, we evaluate our models on two well-known DocumentVQA benchmarks, DocVQA and InfographicVQA. The new results are presented in the two tables below. The first table compares the performance of the TFI, RBI, and FTBI models on the DocVQA and InfographicVQA benchmarks. The second table compares the performance of the RBI and FTBI models on the same benchmarks without inputting detection information during inference.

As shown in Table 1, the deployment of detection models, particularly the OCR model, leads to a significant score improvement on DocVQA. Furthermore, models with adaptive training noticeably outperform training-free models . Specifically, the FTBI models surpass the RBI models, which in turn outperforms the TFI models. This suggests that the adaptive training enables MLLMs to better leverage the input detection information, resulting in improved performance.

Table 2 presents a comparison between the RBI models and the FTBI models in the absence of textual detection information. As shown, the performance of the RBI models is significantly inferior to that of the FTBI models. While the FTBI models, without detection information, perform similarly to the original LLaVA-1.5. This demonstrates that the fine-tuning strategy allows MLLMs to effectively balance the weights between the image encoder output and textual detection information, thereby preserving the comprehensive VQA capabilities. These results are consistent with the findings in our paper.

Thank you for your valuable suggestion! Incorporating the DocumentVQA benchmarks enhances the validation of our experimental conclusions. We have included this point in Appendix E.3 of the revised paper.

(To be continued.)

Responses to Reviewer Excq [Part 3/4]

---

(Continued from the previous page. W2)

W3: Impact of Detection Models with Varying Accuracy

The outputs of low-performance detection models often include noise, which can adversely affect the following MLLM. Following your suggestion, we respectively employ a low-performance detection model YOLOv5N and a high-performance detection model YOLOv11L (replacing only the object detection model DINO while keeping the PaddleOCR unchanged) and conduct both the training-free and fine-tuning experiments again. Next, we compare the adaptability of the training-free and fine-tuning strategies.

The new results are presented in the table, from which the following conclusions can be drawn:

• Under the training-free strategy, for general VQA capabilities, YOLOv5N introduces noise to LLaVA-1.5-7B, resulting in performance degradations. In contrast, YOLOv11L, due to its superior performance, introduces minimal noise and thus has little negative impact.
• Regarding object detection-related capabilities (POPE & MM-Vet), both YOLOv5N and YOLOv11L bring performance improvements under the training-free strategy. However, the improvement from YOLOv5N is noticeably smaller than that from YOLOv11L, which can be attributed to the difference in model performance. This suggests that the training-free strategy exhibits poor adaptability to low-performance detection models.
• Furthermore, after fine-tuning, both two versions of the MLLM achieve comprehensive performance improvements, surpassing the original LLaVA-1.5-7B. The results align with the conclusions in our paper, demonstrating that the fine-tuning strategy enables the MLLM to better differentiate between noise and useful information and more effectively interpret specially designed detection information, leading to performance enhancement.

These results indicate that the fine-tuning strategy is more robust and better able to handle the erroneous information introduced by low-performance detection models compared to the training-free strategy. This further supports our experimental conclusions. We have included this additional experiment in Appendix D.3 of our revised paper. Thank you for your suggestion.

---

Q2.1: "Is there a risk of overfitting?"

In our paper, we have already included a discussion on model overfitting, which is in Section 4.3 & 4.4. As shown in Table 2 in our paper, models under retraining strategy tend to overfit. These MLLMs overly focus on textual detection information, neglecting the image encoder's output, which leads to a decline in performance on comprehensive VQA benchmarks. Furthermore, as demonstrated in Table 3, the models under fine-tuning strategy do not exhibit overfitting. They strike a good balance between textual detection information and the image encoder outputs, ensuring high performance on both general VQA and object detection benchmarks.

(To be continued.)

Responses to Reviewer Excq [Part 4/4]

---

(Continued from the previous page. W3&Q2.1)

Q2.2: Fine-tuning on Tasks Unrelated to Detection

Following your suggestion, we conduct fine-tuning experiments using data unrelated to detection tasks, and investigate whether the FTBI model can still maintain broader language understanding capabilities under this training configuration. Regarding the new fine-tuning dataset, we remove samples related to "coordinate" questions (object detection samples) and eliminate all text recognition samples from the original LLaVA fine-tuning dataset. Consequently, the number of samples decreases from 665K to 450K. The new experimental results are presented in the table below, and the corresponding model name is "LLaVA-1.5-7B-FTBI-FNDI".

From the table, it is evident that even without fine-tuning on detection-related data, the FTBI model still demonstrates strong performance, significantly surpassing the original model and the training-free model. Moreover, its results are only slightly below the version fine-tuned with detection data. These results indicate that, even without fine-tuning on tasks related to detection, the fine-tuned model is still capable of maintaining a broad range of language understanding abilities.

Thank you for your suggestion. We have included this experiment in Appendix D.4 of our revised paper.

---

Closing Remarks

We want to express our deepest gratitude for the constructive suggestions and queries you provide. We dedicate a lot of time and effort to conducting additional experiments based on your recommendations, and we hope that you can consider an increase in the rating if these response with new results effectively address your comments. Thank you!

Dear Reviewer Excq,

The discussion period will end in just a few days, and we truly appreciate this valuable opportunity to engage with you.

We notice that you haven't responded to our replies. We believe that your insightful comments, combined with the additional experiments we newly conduct, have helped us further improve our paper. We hope that the new results and responses effectively address your comments, and we kindly ask you to consider a potential increase in your rating. If you have any additional questions or concerns, please feel free to reach out to us. Thank you!

Best, authors