LMEye: An Interactive Perception Network for Large Language Models

Q4: In my view, I think there is no essential difference between RVII and LoRA.

A: Thank you for your insightful question. To clarify the distinctions between RVII and LoRA, we focus on three aspects: integration approach, adaptation dynamics, and role in the decoding process.

Distinction in Integration Approach Regarding LLMs as an agent, The RVII module is designed to obtain the dynamic visual information based on the interaction between LLMs and visual information. LLMs first understand the basic image content and human queries in the first-stage perception process, then sending its request signal to obtain the visual information what they desire according to its understanding. The dynamic enhancing visual information are used in the second-stage thinking (response generation) process. LoRA integrates additional parameters within the LLM and alters the internal dynamics of the LLM by tweaking its parameters, used to focus on valuable visual and textual information in the internal operations of LLMs. The influential LLaMA-adapter V1/2 has made significant progress in this regard. In contrast, RVII acts as an external intermediary, processing visual inputs based on real-time human queries before they are explicitly fed into the LLM to generate final response. RVII's design preserves the LLM's original structure, treating it as an agent, highlights the importance of external and adaptable multimodal integration and make up for the information loss caused by one-step visual input. Of course, we believe that using LoRA to further tune LLMs will enhance the performance of our model.

Dynamic vs. Static Adaptation Another key difference lies in the nature of adaptation. RVII dynamically interacts with visual data, adjusting its processing based on the context of human queries. This is in contrast to LoRA's adaptation, where the incorporated changes are fixed post-training and visual information do not dynamically alter based on the human query. RVII's dynamic adaptation allows for more context-sensitive and relevant integration of visual information, enhancing the LLM's response accuracy to specific human queries. Similarly, based on your suggestions, we can also learn from the recently proposed Image-Blind (we will introduce and compare it in the next version) and Llama-adapter work, introduce the second stage of training and use the LoRA adapter in large models to improve the operation of two types of visual information and textual query during decoding progress.

Role in Decoding Process While it is accurate that in LLMs, each token is decoded based on previous tokens, including visual tokens and human queries, RVII adds a nuanced dimension to this process. RVII continuously refines the visual input, ensuring that at each step of the decoding, the visual information remains contextually aligned with the evolving user query. This ongoing adjustment keeps the visual data relevant throughout the interaction, providing a more sophisticated and tailored integration than static models. In summary, RVII's external, dynamic, and context-aware approach to integrating visual information with LLMs offers distinct advantages over one-step inputting static image feature in LLMs, particularly in terms of real-time adaptability and relevance to specific user queries. The LoRA mechanism plays a crucial role in refining the interaction between textual and visual inputs, ensuring a more seamless and effective multimodal integration. Hence, we believe that the advantages of LoRA and RVII could be integrated together. We plan to use LoRA technical to further improve the adaptation of LLMs on multimodal input and performance, yet we also need consider another question, that is remaining the original capability of LLMs to process text-only NLP tasks.

Thank you for your suggestions and comments on our work. We will answer your questions one by one in two pages.

Q1: Explore the contribution of RVII module. For example, the impact of RVII module under the same data and training process.

A: The RVII module is designed to enhance the model's ability to dynamically interact with visual information based on human queries. To evaluate its contribution, we refer to Tables 3 and 4, where we compare the performance of models with and without the RVII module (notably LMEye (xx)*). These tables show that incorporating RVII results in significant improvements for several LLMs in tasks like VQA/OK-VQA/VQA with long answers. Moreover, when comparing LMEye (FlanT5-BLIP-2-XL) with Instructblip (FlanT5-BLIP-2-XL), both of which utilize similar multimodal instruction data, the key difference lies in the inclusion of the RVII structure in our model. This comparison, as shown in the results on evaluation benchmarks, also underscores the effectiveness of the RVII module.

To further explore the contribution of RVII module, we conducted an additional experiment. Here, we fine-tuned LMEye (FlanT5-BLIP-2-XL)* using identical instruction data but without the RVII module. The results of this experiment, detailed in the following table, clearly demonstrate the significant contribution of the RVII module to the model's performance.

Q2: The training parameters in the instruction tuning phase?

A: During the instruction fine-tuning stage, our model utilizes the RVII system along with several linear feature projection layers, which collectively amount to approximately 105 million parameters, with RVII alone comprising about 100 million. Recent findings indicate that for image understanding tasks, it's possible to reduce the number of RVII layers without noticeably affecting performance. By halving the number of RVII layers, we can significantly decrease the total count of trainable parameters, thereby streamlining the model while maintaining its effectiveness.

Q3: Since the visual information needs to be fed into LLM twice, will the model run slower?

A: Thank you for raising this point. Intuitively, feeding visual information into the LLM twice suggest a slower process. In practice, the impact on processing speed is minimal. This is due to the efficient design of our model which optimizes the integration of visual data. The length of visual tokens is set to 32 and we only input one <img-q> token as the probing token to obtain the request signal. For encoder-decoder T5 model, we only use the encoder as the the first-stage basic perception and request signal generation. Time is mainly spent in the token generation phase. Moreover, the additional time taken for processing is offset by the significant improvements in accuracy and context relevance of the output. In summary, while there is a slight increase in processing time, it is negligible compared to the enhanced performance and quality of results achieved by our model and we will contiguously optimize the efficiency according to your question.

Thank you for your appreciation and acknowledgment of our work. Below, we will address your constructive comments individually.

Q1: Give a short description of LMEye (BLIP-2-FlanT5-XL) and provide more clear data introduction.

A: LMEye (BLIP-2-FlanT5-XL) is a derivative of the FlanT5 large language model, operating on an encoder-decoder architecture. It employs the pre-trained vision encoder along with the Q-former to serve as the image encoder, translating image representations into the language space. Specifically, we utilize the T5-Encoder for image comprehension and request processing, employing a probe vector labeled as '<img-q>' to prompt signals essential for the large model's visual information acquisition. Subsequently, we leverage the RVII mechanism to gather visual information relevant to the aforementioned model requests. This process enriches the original image data with dynamically enhanced visual information. The amalgamation of this enhanced visual data and the original image information is then fed into the T5 model.

In terms of multimodal instruction data, our approach encompasses datasets akin to those utilized by InstructBLIP, such as VQA and OK-VQA. However, our adaptation goes further by integrating additional sources beyond the initial scope, including artwork analysis and LLaVA data.

Q2: The difference between the LMEye variants for the encoder-decoder and the decoder-only language models.

A: The distinction between the LMEye variants in encoder-decoder and decoder-only language models lies in their operational mechanisms. Encoder-decoder LLMs leverage the encoder specifically to grasp input information, enhancing model inference efficiency by focusing on relevant data. On the other hand, decoder-only LLMs utilize the entire language model to accommodate evolving visual input and human queries. However, our recent advancements have led to an enhanced variant that integrates the foundational layer of decoder-only LLMs for comprehending both image and query data. Moreover, we've implemented the RVII (small size) module across the upper layers of the language model. This strategic integration empowers the language model to continually assimilate dynamically enriched visual information as it delves deeper, akin to imbuing deep thinking with a visual dimension. Consequently, this visual data becomes an integral component of the calculation process within LLMs, augmenting their cognitive capabilities.

Q3: Expand LMEye to Video understanding.

A: This question delves into a fascinating area of exploration. Extensive experimentation has convincingly showcased the remarkable scalability of LMEye across various large language models, hinting at its potential extension into the realm of video processing. The approach involves training a dedicated video encoder and converter, leveraging RVII's ability to engage with prolonged encoded signals. Our methodology encompasses aggregating encoding information from brief video frames and subjecting it to rigorous benchmark testing for video action and procedural comprehension.

The outcomes of our experiments substantiate the feasibility of extending LMEye's capabilities to comprehend video content. Notably, our findings highlight not only the successful extension of this model but also its commendable performance compared to Video-related MLMs. This progress indicates promising avenues for further advancements and applications in this evolving field.

Q4: Introduce visual information request during text generation

A: We are actively considering the integration of visual information retrieval mechanisms into our text generation process. Our current focus is on the development of specific tokens dedicated to retrieving pertinent visual data during generation. This approach mirrors our successful implementation of Tool API tokens and aligns with our ongoing efforts to enrich the generation process with relevant external information sources. Your feedback has guided us towards this promising avenue, and we are committed to exploring and implementing this enhancement in our future work.

Thanks for your comments. We will respond to your comments below.

Q1: The so called "interactive perception model" is actually a standard and common technique used everywhere.

A: While the interactive perception is a concept that has been used in various research domains, our implementation within LMEye exhibits several distinctive and innovative aspects, particularly in its integration with Large Language Models (LLMs).

LLM-Centric Approach: LMEye is uniquely engineered to function synergistically with LLMs. Here, the LLM serves as a central processor, orchestrating a dynamic and efficient interaction with the external visual environment. This is a significant deviation from previous models, which predominantly focused on static visual summarization.

Dynamic Interaction: Unlike the methods you mentioned, such as NMN in VQA, LMEye empowers LLMs to actively initiate, gather, and process visual information in real-time, in response to ongoing human instructions. This approach aims to advance the frontiers of AI-Generated Content (AIGC) research by fostering interactions that are more aligned with human intentions and responsive to changes in the environment.

Performance and Efficiency: Our comprehensive experimental evaluations demonstrate that LMEye not only significantly enhances zero-shot performance across various multimodal benchmarks but also accomplishes this with fewer parameters than other advanced multimodal large models. This indicates a substantial improvement in both practical effectiveness and efficiency, setting LMEye apart from existing methodologies.

Q2: The network/module themselves are also pretty common, and do not convey any significant "direction-shifting" messages.

A： Based on your comments, I would like to clarify and emphasize the innovative aspects of LMEye.

Clarification on Network/Module Commonality: It's essential to specify which aspect of the network or module is deemed 'common.', which makes us confused. LMEye indeed builds upon foundamental large language and vision models. However, the uniqueness lies not in the individual components but in their novel integration and application.

Novel Bridging Approach: The core contribution of LMEye is its innovative methodology to bridge a significant gap in the existing framework of Large Language Models (LLMs) and the visual environment. Unlike conventional methods that input visual information in a static manner and depend on a pre-tuned attention mechanism, LMEye introduces a dynamic and interactive approach RVII. This allows for the integration of visual information in real-time, responding adaptively to a range of human queries.

Dynamic vs. Static Processing: This dynamic processing contrasts sharply with the common static input models. LMEye's approach is not just about focusing on key visual information through attention mechanisms but about continuously updating and integrating this information based on evolving human instructions.

Empirical Evidence of Superiority: Our experimental results substantiate the effectiveness of LMEye. They demonstrate improved performance over existing methods, with less parameters.

Thank you for your valuable suggestions and comments, which will undoubtedly enhance the quality of our manuscript. Due to page limitation, we will use two pages to respond to your comments.

Q1: Suggestions for Improvement in Figure 1 and Writing.

A: We appreciate your feedback on Figure 1 and acknowledge the need for its enhancement. To address this, we plan to revise Figure 1 to more clearly delineate and label each component of the LMEye system, ensuring that it accurately reflects the architecture and functionality discussed in the text. This revision will aim to enhance the figure’s clarity and its utility as a visual aid for readers. Additionally, we recognize the importance of clear and correct writing for effective communication of our research. To this end, we will conduct a thorough review and revision of the manuscript to correct grammatical errors, including the specific instance you mentioned on page 2.

Q2: Ablations in MME and SEED-Bench benchmarks; Ablations without RVII module in LMEye (BLIP2) in Tables 3 & 4.

A：In our prior paper, we highlighted the efficacy of the RVII module through a comparative analysis involving LMEye (BLIP), InstructBLIP, and BLIP, based on FlanT5-XL. InstructBLIP, by integrating multimodal instruction data to fine-tune the Q-former from BLIP, demonstrated superior performance across various tasks compared to BLIP itself. It's essential to note that the structural composition of InstructBLIP and BLIP remains unchanged. However, LMEye (BLIP) introduces a distinct component known as the Request-based dynamic visual information interactive module (RVII). While trained with instruction data akin to InstructBLIP, LMEye (BLIP) without RVII essentially aligns with the architecture of either BLIP or InstructBLIP. By juxtaposing the performances of these variations in Tables 1, 2, and 3, we can ascertain the specific impact of the RVII module.

To address your constructive feedback, we conducted experiments by training LMEye (BLIP) without the RVII module using our code and instruction data. The subsequent experimental results, outlined in the following table, present the significance and role of the RVII module in enhancing performance. All models are based on the same vision and language models (FlanT5-XL).

Q3：Vision Model and Language Model with their corresponding parameters.

A：Thank you for your valuable suggestion. We have collected relevant data as shown in the table below. We will also update specific parameter statistics in the next version.