Unleashing Region Understanding in Intermediate Layers for MLLM-based Referring Expression Generation

Thank you for your attention to our work and your positive acknowledgment of our idea. We have open-sourced the code. We will address your concerns below.

1. low efficiency. We would like to hightlight more about probing-based estimation that is designed to allivate this issue. While the cycle-consistency-based quality ranking incorporates the RES model, adding to the computational load, our probing-based estimation method simplifies this process. By transforming the REG-RES cycle into a straightforward set of importance weights (please refer to Figure A in our common response), we eliminate the necessity for RES in decoding. The proposed method and its resulting importance prior weights can effectively reduces MLLM hallucinations by using combinations of intermediate layers. We have validated this approach in our experiments, as shown in Table 3 of the PHD benchmark. We hope this could partially address your concern.

2. Ablation study. Thanks for your precise and valuable feedback. We have reported some subset-related ablation experiments in the table below. The range of candidate layers is [0, 7]. As for the “full-D”, we calculate the layer importance weights on the full dataset, and then integrate these weights into decoding. “full-R” denotes cycle-consistency-based ranking on the full dataset. “top” and “avg” denotes the best/average results we tested on different sampled 1/8 subset.

In the first row, we report the average result of 10 different random seeds for 1/8 subset sampling, which is also the result shown in Table 1 of the main paper. The second row reports the best results for the 1/8 subset. The results indicate that the randomness of the sampled subsets affects the performance of our proposed decoding strategy in reducing hallucinations. In other words, the quality of the subset has a certain impact on the decoding outcome.

However, assessing the quality of subsets during probing is also an intractable problem. One possible solution is to use an unsupervised clustering method (e.g. K-means) to first cluster the multimodal features (extracted by CLIP or embedding layers of MLLM) of the entire dataset, and divide the subsets based on different centroids, then calculate and store the inter-layer weights in an "importance weight bank." During inference, we can compute the distance between the new query and these centroids, selecting the set of weights from the closest centroid for inter-layer combination during decoding. This strategy needs careful designing, and we consider it a future extension. Besides quality, by comparing the second and third rows, we also find that compared to the 1/8 subset, the 1/16 subset shows less stable de-hallucination and generation performance.

3. Different RES model. We appreciate for the detailed comments. In the following Table , we ablate the impact of different RES models utilized in the proposed cycle-consistency-based quality ranking (CCR). We adopt another MLLM-based RES model LISA to score the sentence quality of intermediate layers.

From the table, we can observe that the choice of the RES model affects the CCR. The RES task itself involves transforming a language query into a pixel-level visual representation of an object. Given that GLaMM performs slightly better than LISA in RES, we can therefore infer that the more robust this transformation is completed, the better the performance of the CCR in quality ranking.

4. Explanation of symbols. Thanks for the question and we apologize for the confusion. $\alpha$ is a hyperparameter in [0, 1] that truncates the next token distribution of $p(y_t|y_{<t})$. It helps split out the tokens whose score is lower than a proportion of the highest score. Larger $\alpha$ entails more aggressive truncation, keeping only high-probability tokens, whereas smaller $\alpha$ allows tokens of lower probabilities to be generated [Li et al., arXiv:2210.15097].

$\mathcal{V}$ in Line 171 denotes the vocabulary set.

$n$ and $N-1$ are different. $N$ denotes the number of layers of MLLM model, 32 in our case, and $N-1$ represents the number of remaining layers except the final layer. The lowercase $n$ is dynamic and refers to the number of candidate layers in experiments, which we chose as 8 in our implementation.

Thank you once again for your insightful comments. We deeply appreciate your guidance and found great resonance with your perspectives. With the discussion period nearing completion in less than two days, please feel free to share any final comments at your convenience.

Thanks for your feedback. We have made the code open source at the link attached to the abstract. We will address your concerns below.

1. Increase computational overhead. Thanks for the comments. In the cycle-consistency-based quality ranking, the RES model is incorporated as an auxiliary, leading to increased computational overhead. However, we need to emphasize that our proposed probing-based estimation method simplifies the cycle between REG and RES into a set of importance weights (refer to Figure A in common response). It removes the need for RES during decoding, and mitigates the MLLM hallucinations through combinations of intermediate layers. This result has been demonstrated in our experiments, on the Table 3 PHD benchmark.

2. More related work. Thank you for your suggestion. Based on your advice, we would like to summarize some related works on inference-time decoding strategies for mitigating hallucination in large language models (LLMs). Specifically, our work represents an inference-time decoding strategy to mitigate hallucinations, which utilizes the latent knowledge within the intermediate layers without additional training. There are more details in the code link. We hope this could help you better understand the positioning of our work.

3. Undefined terms. Thanks for your feedback. HW denotes the flattened encoded visual token numbers. Q denotes the number of tokens that the model used to encode the region prompt.

4. The choices used in visualization. Thanks for the question. We first observed through the early-exit method that outputs from different intermediate layers tend to vary, and some may even contain more descriptive content than the final layer. Subsequently, we attempted to quantify how much potential different intermediate layers have for better region understanding. PCA and Wasserstein distances are two common analytical methods used. PCA reduces the dimensionality of high-dimensional latent features while preserving the principal components, which not only speeds up the visualization process but also allows features from different dimensions to be reduced to the same dimension for distance calculation. We use Wasserstein distances as a measure of distance because it evaluates the distance between two distributions from the perspective of transportation cost. Compared to other methods used to measure distances between distributions, it requires fewer samples [M. Arjovsky et al., ICML2017].

5. How to quantify hallucinations. Thanks for the question. According to your suggestion, we would like to provide a detailed elucidation of the hallucination metric calculation and summarize it as follows: In our study, we utilized two approaches to quantify the severity of the hallucination:

1. The first approach is based on a widely adopted metric CHAIR, which directly counts the number of hallucinatory descriptions generated by the model. We use it for the sentences of REG task. It relies on a reference expert table, providing the scope of the explicit object, and quantifies the object hallucination by calculating the ratio of "the objects mentioned but not in the expert table" to "all objects mentioned in a description". The result is reported in Table 1 & 2.
2. The second approach we adopted is prompting MLLM and counting the average ratio of the number of answers that do not fall into the hallucinations to all answers. We utilize the PHD benchmark (a challenging extension of POPE) to achieve this. Similar to POPE, since prompted questions are all interrogative sentences in this benchmark, it can be directly concluded whether the description is hallucinatory or not just from calculating the accurate "yes" or "no" answers generated by the model. The result is reported in Table 3. For more details, please refer to the code link.

6. The specific improvements and failure case.

1. specific improvements: By extracting information from the intermediate layers of MLLMs, our method generates sentences that are more descriptive while reducing hallucinations. Specifically, Table 1 shows an increase in the METEOR metric, indicating higher quality sentence generation, and an improvement in the CHAIR series metrics, showing a reduction in hallucinations. Table 3's improvements on the PHD benchmark further demonstrate that our decoding strategy effectively mitigates hallucination.

2. failure case: Thank you for the question, and we provide a failure case as Figure C-(6) in the common response. In this case, the referred object is the black dog on the left. It can be seen that during the generation process, the model initially describes the referred object in detail, but eventually a hallucination occurs, mistaking the state of another dog (looking at the cake) for the state of the referred dog. This also illustrates the unique challenges of region-level understanding.

7. Is there any Transformer-based region encoder? Thank you for the question. In the context of our paper, the region encoder is used to encode region prompts for the MLLM. To our knowledge, MLLMs that perform REG have not yet employed transformer blocks for encoding region prompts. Shikra [K. Chen et al., arXiv:2306.15195] converts regions into coordinates, represented using natural language numbers. Ferret [H. You et al., ICLR2024] uses CNN blocks and point sampling for encoding. GPT4ROI [S. Zhang et al., arXiv:2307.03601] and GLAMM [H. Rasheed et al., CVPR2024] employ ROI pooling methods, and Osprey [Y. Yuan et al., CVPR2024] utilize CNN blocks for multi-scale mask encoding. We speculate that such designs may aim to reduce computational complexity, as transformer modules require a larger amount of data to fit. Given the vast knowledge underlying pre-trained visual encoders (CLIP) and the MLLM base (LLAVA), the region prompts need to align with the existing knowledge. A lightweight region encoder could reduce the optimization difficulty.

It is great to know that our responses are helpful, and thank you for your positive feedback and support for this work! Your suggestions have helped improve this work and we will incorporate them in the revised manuscript.

Thank you for your detailed review and constructive comments. The code is accessible through the link indicated in the abstract. We noticed that there might have been some misunderstandings regarding to our proposed "Probing-based importance estimation" method, and we hope this response will better convey our decoding strategy to you. We will address your concerns below.

1. Reference format. Thanks for the suggestion. We have changed it to numerical style in the revision. Additionally, we would like to summarize some related works on inference-time decoding strategies for mitigating hallucination in large language models (LLMs), hoping this can better help you position our work. Specifically, our work is in line with CCS [C. Burns et al., ICLR2022], ITI [K. Li et al., NeurIPS2023], and DoLa [Y.-S. Chuang et al., ICLR2024], which are all aiming to utilize the existing knowledge in trained models through representational operations to enhance the authenticity of the output. Compared to these works, we uncover the intermediate layers of multimodal LLMs and, as far as we know, are the first to propose using a combination of intermediate layers for region-level caption generation. There are more details in the code link.

2. The architecture seems simply a combination of REG and RES models. Structurally, the cycle-consistency-based ranking method we proposed indeed relies on the combination of REG and RES. RES serves as an auxiliary tool we utilize to filter the intermediate layer information unleashed afterward. Additionally, we would like to highlight some key points beyond the architecture:

1. We designed an inference time decoding strategy, which can unleash the latent knowledge inside the intermediate layers without additional training.
2. The key observation is: compared with the output of the last layer, the sentences of intermediate layers can sometimes provide more discriminative descriptions for the referred object, which motivates us to design a cycle between REG and RES to localize the desired information.
3. Besides the REG and RES cycle process, we also proposed a probing-based estimation method that can obviate the need for RES, directly blending estimated weights into the existing model to enhance its anti-hallucination efficacy.

3. The trade-off between granularity and accuracy. Thank you for your detailed comment and we apologize for any confusion caused. We would like to clarify that the trade-off discussed is inherent to the REG task itself. The more extensively a model describes, the higher the likelihood of errors, as demonstrated by the CHAIR-related metrics in Table 1. As descriptions become lengthier, both $\text{CHAIR}_I$ and $\text{CHAIR}_S$ metrics deteriorate, indicating an increase in hallucinations during detailed description generation. To alleviate this issue, we unleash information from the intermediate latent knowledge to broaden the space of choice. Then a RES model as a listener is used to evaluate and select proper output. This "unleash-then-eliminate" approach helps us get rid of the intrinsic constraint of detailedness and accuracy. Table 1 demonstrates the efficacy of our proposed method. In Table 2, our goal is to investigate the effects of candidate layers from different zones using our method, and to illustrate the properties of various regions within a trained MLLM. We have not explored such a trade-off across different layers.

4. Open source. Thank you for your interest in our implementation. The code has been uploaded.

5. Final architecture of Probing-based estimation, and the number of parameters. We have noticed that there might be some misunderstandings about the approach we have proposed. We have introduced a decoding strategy for inference-time, not a new model that requires training, hence no new parameters have been added. The Probing-based estimation method involves the following steps:

1. First, sample a subset as the probing set.
2. On this probing set, perform cycle-consistency-based ranking and calculate scores for each sample in each layer.
3. Calculate an inter-layer importance weight on this subset, which is mainly based on frequency counting of each candidate layer being the optimal layer.
4. Combine the importance weight prior with the J-S divergence to represent the probability of each layer being selected (the higher the importance, the more likely it is to be chosen).
5. Sample from this hybrid importance distribution to decide which layer to use for decoding the next token, until the generation finishes. This approach compresses the “cycling” process into a set of importance weights, thus facilitating convenient decoding. We would like to emphasize that the experiments in Table 3 have demonstrated that such a set of weights can be directly applied to MLLM to alleviate hallucinations, without the need for RES models and the cycling process, enhancing the practicality of our method. We also provide a visualization of importance weights in Figure A (listed in common response).

6. The metrics of Table 3. These refer to the prompt types of PHD benchmark. For better illustration, we provide some prompt and generation examples in Figure B (in common response). This benchmark sets ten different types of questions, which are composed of five different tasks, with each task featuring two modes of questioning. We refer you to the 'Datasets and Metrics' section and the newly added Appendix C for details on the different tasks. The two modes of questioning include Neutral mode and Misleading mode. The prompts of the former only include the original question, while the prompts of the latter would accompanied by misleading descriptions. More details are available in the code link.

Thank you once again for your valuable comments and thorough review. We have tried our best to clarify the concerns on the paper. Your detailed feedback has enhanced the clarity of our explanations. With the discussion period nearing completion in less than two days, we would appreciate it if you could let us know if any aspects remain unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

Thank you for your positive acknowledgment of our work. We are glad to notice your interest in the latent information of the intermediate layers. We hope our responses below will partially address your concerns.

1. Suboptimal compared to the training-based methods.

Yes, we proposed an inference-time decoding strategy that utilizes existing knowledge in trained models through representational operations to enhance the authenticity of the output. If the training is allowed, there are two potential areas within our framework where performance could be improved through training:

1. If more descriptive datasets are available, the selection among intermediate layers could depend not only on the RES model but also on the ground truth sentence.
2. A linear layer could be designed to learn the mapping from different scores across layers to the importance weights of layers. We still need to emphasize that although training-based methods might yield better results, as you mentioned, the proposed "unleash-then-eliminate" method is training-free, which also avoids the need for additional data and training.

2. Computation load.

Thanks for the feedback. We agree that the computational intensity of the cycle ranking could potentially affect the decoding speed. This is an inherent challenge when integrating more complex algorithms to improve the accuracy of the generation. We would like to highlight that in probing-based estimation, we compress the cycle process into a set of layer importance weights which could directly be merged into the original decoding process, thus enhancing the practicality of our method.

3. The RES model is imperfect.

Thanks for the precise comments. We acknowledge that our method depends on the performance of the RES model. We hope that more powerful RES models in the future will help alleviate this limitation. In the table below, we have ablated the impact of the existing MLLM-based RES on our method. Given that GLaMM performs slightly better than LISA in RES, we can therefore infer that the more robust the RES model is, the better the performance of cycle-consistency-based in quality ranking. “full-R” denotes cycle-consistency-based ranking on the full dataset.

Thank you once again for your insightful comments! We deeply appreciate your precise feedback and truly resonate with your perspectives, which have been instrumental in enhancing our work. With the discussion period nearing completion in less than two days, please feel free to share any final remarks at your convenience.

Thank you so much for your encouraging feedback and for upgrading your score to a "strong accept." We are truly grateful for your recognition of the efforts made in our rebuttal. It's great to hear that our responses have successfully addressed your concerns. We're committed to continuously improving our work, taking into account your insights and those from other reviewers. Thanks again for your supportive words!