Octopus: A Multi-modal LLM with Parallel Recognition and Sequential Understanding

Q1: The motivation of combining DETR with MLLM is not new. Several previous works have explored the nearly same design. In particular, LLaVA-Grounding have the same design by introducing the Detr decoder into the output of LLM.
A1: Our motivation is not combining DETR with MLLM. Our motivation (and key contribution) is integrating the parallel recognition and sequential understanding into one LLM, and organizing them into a “parallel recognition → sequential understanding” hierarchy. This is fundamentally different and even opposite to previous "MLLM + DETR'' methods because they all follow the "understanding → recognition'' regime, i.e., first concept understanding and then recognition.

Our novel hierarchical MLLM framework has several advantages, compared to previous "DETR + MLLM'' methods, including:

• The parallel recognition results can further augment the sequential understanding ability of current MLLMs. In contrast, in previous "MLLM + DETR'' methods, the recognition does not benefit understanding.
• Our DETR head can be used separately for recognition tasks, resulting in high efficiency. In contrast, previous "MLLM + DETR'' methods is more computational expensive because they require full inference over the entire LLM.

---

Q2: The technical novelty is limited. The meta-architecture is the same as previous works. [2] is on arXiv even one year ago. However, the authors adopt the nearly same idea but without a citation. This is disrespectful for the community.
A2: We respectfully disagree. Though our method shares some similarity ("MLLM + DETR''), the architecture and underlying mechanism are fundamentally different:

• Architecture: as explained in the Q1-Answer, our Octopus is a novel "parallel recognition → sequential understanding'' MLLM framework, while [2] adopts the popular "understanding pivot → recognition head'' regime.
• Mechanism: [2], as well as some recent methods [1,3], employs an MLLM as the understanding pivot to call a recognition head, so that the recognition head can receive and understand free-style human instructions. The LLM benefits the recognition head. In contrast, our Octopus uses bottom LLM layers to assist the recognition and in turn uses the recognition results to prompt the following understanding. The LLM and the recognition mutually benefit each other. We will add the above comparison to the manuscript.

---

Q3: Several ablation studies are missing. For example, the effect of pre-training data and the effect of transformer layers in DETR.
A3: Thanks. The pretraining data consists of REC and Flickr30k and the ablation is shown in the table below. During rebuttal, we further increased the pretraining data by adding Visual Genome (3rd row) and observed another round of improvement.

The impact of the number of transformer layers in DETR is presented in the table below. We observe that increasing the number of transformer layers results in a slight improvement in accuracy from 4 to 6 layers, reaching an optimal performance at 6 layers. Considering training efficiency, we use 4 layers in our experiments.

---

Q4: No failure case analysis and no improvement analysis, compared with late fusion decoder.
A4: We guess the “late fusion decoder'' you mentioned is the ”understanding $\rightarrow$ recognition'' methods [1,2,3]. Compared with late fusion decoder, our method used relatively fewer LLM layers (16 out of 32 layers) for informing the DETR head with the object-of-interest. It is possible that in some extreme cases where the object-of-interest is very obscure, our model do not capture the user intention and missed the object.
The advantage of our approach, on the other side, is that the recognition is much more efficient for recognition (REC, in particular). More importantly, our recognition results will be injected into the following LLM layers to assist the understanding. In contrast, the late fusion mode does not benefit understanding.

---

Q5: What about using LLM to predict the box in the end?
A5: Directly using LLM to predict the position is the "sequential recognition'' manner introduced in Line 29 to 32 in the manuscript and is inferior to ours. One important motivation of this paper is that the sequential recognition manner has low efficiency. Table 4 in the manuscript shows that our method significantly improves the efficiency by about 4 $\times$ faster. Table 1 in the manuscript shows that our method has higher accuracy (e.g., 2.01 on RefCOCO) in comparison, as well.
Reviewer Up36 kindly reminds there is another manner, i.e., using localization tokens, as in Kosmos-2. This manner has relatively lower REC accuracy (-5.9 acc. on RefCOCO test set) and is inferior to our method. Please refer to Reviewer Up36 Q2 for details.

Dear Reviewer,

Thanks for reviewing this paper! The authors have provided rebuttal to address your concerns. Could you have a look and let the authors know if you have further questions?

Thanks, Your AC

Q1: The comparison in this paper are not comprehensive and sometimes unfair to some baseline methods. It would be better to compare Octopus and LLaVA using the exact same training data. Moreover, in Table 3, some methods from Table 2, such as LLaVA and Qwen-VL, are not included.
A1: Thank you for your advice. We have trained LLaVA with the same source of training data as Octopus (by adding the grounding data). The results in below Table show Octopus surpasses LLaVA with same training data.

We will add the results of Qwen-VL and LLaVA into Table 3 in the manuscript. As shown in the below table, our method surpasses LLaVA and QwenVL-Chat on MMB, SEED, MM-V, and POPE benchmarks, but is lower than LLaVA on the LLaVA$^W$ benchmark.

---

Q2: Some MLLMs, such as PaLI and Kosmos-2 introdices localization tokens to fullfill grounding ability. What are the performance and efficiency advantages when comparing Octopus with these methods (introducing new localization tokens)?
A2: Thank you for your advice. We compare using localization tokens and our Octopus in the table below. It is observed that our Octopus achieves higher accuracy (e.g., + 5.9 test acc.) and is slightly faster. The results are reasonable: the localization tokens have coarse spatial resolution (7 $\times$ 7 pixels per token) and is thus inferior for REC. We used the Kosmos-2 code for implementing the localization code and adopted the same backbone (i.e., LLaVA-1.5) plus the same REC training data for a fair comparison. We did not use PaLI because PaLI has not released its code yet.

---

Q3: In Line 200, it is stated that after bipartite matching, the response is modified by replacing the bounding boxes with the index tokens. Does this mean that for one sample (image, instruction, and response), it requires dual forward passes (first image + instruction + queries, then image + instruction + queries + modified response)? As the authors mentioned in Line 201, Octopus introduces non-negligible training overhead.
A3: Yes, our model requires two forward passes to generate the correct response. However, the training overhead is relatively low (only ~10%) because the first pass requires no gradients computation and is efficient.

---

Q4: Considering the unfair comparisons in Tables 2 and 3, and given the slight performance advantages in Table 1, the advantage of "Parallel Recognition" does not seem significant.
A4: The advantage of "parallel recognition" on REC is non-trivial, regarding both efficiency (about $4\times$ faster in Table 4) and accuracy (+1.76 over the baseline on Refcoco val). Please kindly note that we have already made the comparison fair by using the same training data during rebuttal.

As for understanding task (Table 2 and 3), the superiority does seem relatively small. This is because many understanding tasks are not closely related to recognition. During rebuttal, we add another experiment by selecting closely-related sub-tasks and observe larger improvement (+1.7 VQA v2). Please refer to R1-Q1 for details.

---

Q5: How are the newly introduced object queries initialized? Does this affect the final performance?
A5: We guess your question is about the training initialization for the object queries. At the start of training, we use the standard Gaussian initialization (the default PyTorch initialization) for these object queries (as well as for all the other vocabulary embeddings). After training convergence, all the object queries use the learned embeddings for each inference.

---

Q6: Would it be better to use a pre-trained DETR decoder from off-the-shelf publicly available checkpoints?
A6: Thanks. Utilizing a pre-trained DETR decoder can benefit training, as validated by our experiments in Stage-2 (L219). A pre-trained decoder helps in quickly adapting the DETR decoder to the LLM layers. If we do not use DETR pretraining, it takes a longer period (1.5$\times$) to achieve similar results.

Dear Reviewer,

Thanks for reviewing this paper! The authors have provided rebuttal to address your concerns. Could you have a look and let the authors know if you have further questions?

Thanks, Your AC

Q1: Missing many experiments. This paper propose a framework for both detection and understanding. However, the authors only report the performance of results on VQA MLLMs. They only report inference time and fps on RefCoCo and Flickr30k, not performance.
A1: Table 1 and Section 4.2 in the manuscript already provide performance (acc.) on RefCOCO, as well as other two datasets, i.e., RefCOCO+ and RefCOCOg. These are the three most popular datasets for the referring expression comprehension (REC) task. We did not employ Filcker30k because its annotation does not satisfy the standard REC requirement, i.e., binding the bounding box with the corresponding object. No prior methods employed Flicker30k for REC evaluation yet either.
Given this limitation, during rebuttal, we compute the Rank-1 accuracy as the alternative evaluation protocol, by computing the accuracy between the ground truth boxes and predicted boxes pairs of IoU > 0.5. We observe that our method improves the baseline, e.g., +2.4 (59.3 to 61.7) R1 score on Flicker30k test set.

---

Q2: The compared methods are not newly proposed and current SOTAs, should add more recent methods for comparison, such as LLaVA 1.5, LLaVA-next and etc.
A2: The requested LLaVA-1.5 result is already listed in Table 2 in the manuscript ("LLaVA"). We will revise the notations from "LLaVA" to "LLaVA-1.5" to avoid confusion. Table 2 in the manuscript clearly shows that our method surpasses LLaVA-1.5, e.g., +0.7 on VQAv2 and +1.3 on GQA.

We did not compare LLaVA-Next because it is not an official academic paper, but rather a technical blog. It does not release its training data and code, either. Since our method is not restricted to any specific MLLM baseline, it can be plugged into LLaVA-Next and is potential to achieve some improvement, as well.

---

Q3: What is the performance of the proposed method on Flickr30k and RefCOCO?
A3: As explained in the responses to Q1, Table 1 in the manuscript already evaluated RefCOCO (+2.01 compared to Shikra baseline). Flicker30k is not quite usable for the standard REC task, so we use an alternative evaluation protocol and observe consistent improvement (58.9 $\rightarrow$ 61.2 rank-1 accuracy). Please refer to the response to Q1 for details.

Q1: More experimental investigations shall be conducted to support the motivation of this paper, especially more quantitative analysis.
A1: Thanks. During rebuttal, we conducted one more experiment according to your suggestion. This experiment along with the already-existing ones in Table 1 to 4 in the manuscript can well support our motivation, i.e., the "parallel recognition $\rightarrow$ sequential understanding'' hierarchy brings mutual benefits between recognition and understanding, and is efficient. The details are as below:

Existing experiments. Table 1 and Table 4 in the manuscript show that our method improves open-world recognition (REC) accuracy (+2.01 on RefCOCO) and efficiency (5$\times$ faster). Table 2 and Table 3 show the recognition results then benefit the understanding (+0.7 on VQAv2).

The new experiment in rebuttal. The improvement on understanding task seems relatively small in Table 2 and Table 3, mainly because many understanding evaluation objective is not closely related to recognition. Hereby, we investigate some closely-related recognition tasks. We selected a subset (173621/214354) of the VQAv2 dataset consisting of "What?" (87868), "How many?" (23372), and "Is?" (62381) type questions, where grounding is needed to recognize the referred objects. We compared the results of our methods against the baseline methods on this subset.

As shown in the above table, the advantage of Octopus on this subset is more apparent, which further validate our motivation that parallel recognition can further improves the sequential understanding of MLLM.

---

Q2: DETR framework is too old for visual recognition, it is more proper to introduce some efficient architecture for this part, e.g deformable-detr.
A2: Thanks. During rebuttal, we substituted the DETR backbone with Deformable-DETR pretrained on COCO-2017 dataset, and trained Octopus with Deformable-DETR on LLaVA-Instruct, REC and Flickr30k datasets. Results are evaluated on RefCOCOg val and test sets, as shown in the table below. We observe further improvement, i.e., +0.5 and +0.4 on validation and test set, respectively. Moreover, during training , we observed that Deformable-DETR significantly accelerates training in earlier stage. We will release the code for both DETR and the deformable DETR implementations.

---

Q3: The writing of this work can still be improved, and typos still exists.
A3: Thank you for your advice, we will continue to improve the writing and fix typos in the revised version.