Towards General Continuous Memory for Vision-Language Models

Rebuttal

We sincerely appreciate the reviewer’s thoughtful feedback and are grateful for the encouraging remarks. Below, we reply to all the concerns mentioned in the weaknesses and questions section.

Weakness-1: Unclear Model Generalization

The effectiveness of the proposed method is primarily demonstrated on Qwen2, leaving it unclear how well the approach generalizes to other models.

Reply-to-Weakness1:

Our method is effective for equipping models other than Qwen2-VL to improve the performance, even helpful for augmenting Qwen2-LLM with multimodal knowledge.

• Improved Performance on Qwen2-VL and Qwen2.5-VL: In our paper we evaluate effectiveness of CoMEM on Qwen2-VL and Qwen2.5-VL (Tables 3, 4 in paper). Here we add result table1 by comparing with up-to-date RAG methods. The result shows that CoMEM significantly improves performance over both the original models and strong RAG baselines, demonstrating consistent gains across different models.
• Multimodal Knowledge Transfer to Text-only LLM: In Section 4 (Transferability Study to LLMs), we demonstrate that a text-only LLM (Qwen2.5-Instruct) can leverage the multimodal knowledge provided by our CoMEM using Qwen2.5-VL as the encoder. The result is in Table2. It shows that CoMEM is a powerful and general bridge between VLMs and LLMs, empowering the visual reasoning ability of LLMs, and validating the generalizability of our approach.

Table1: Performance Comparison Across Datasets (Accuracy %)

Table2: Transferability Study of Vision-Language Memory Encoded by CoMEM on LLMs (Accuracy %)

Weakness-2: Motivation is not shown in Result clearly

The paper motivates the method by stating that RAG may increase context length and degrade performance, but Table 3 shows only modest or inconsistent performance changes with RAG. Highlighting which samples involve long contexts in Table 3 could help better connect the empirical results to the stated motivation.

Reply-to-Weakness2:

We sincerely thank the reviewer for this insightful suggestion and will report the average context length for each benchmark with or without our method. These clearly show that RAG makes all the inputs of these benchmarks become long-context, approximately 15× times (see Table 3 below). As long context understanding is a hard problem for LLMs, the performance is also affected:

• RAG performance is unstable due to the long context: While RAG occasionally improves performance (e.g., Qwen2-VL on OKVQA: 36.3 → 41.9), it fails to deliver consistent gains across benchmarks, and even degrades the performance (e.g., Qwen2.5-VL on AOKVQA: 39.8 → 34.9). It indicates that although the additional long context contains useful reference data, the lengthy context makes it difficult for VLM to read and use this information.
• CoMEM performs better across different benchmarks, benefiting from its compression ability: CoMEM provides consistent gains on most of the benchmarks, while using much fewer tokens. This reinforces our claim that our unified memory is not only more efficient but also reliable across diverse tasks.

Table3: Token Number and Accuracy Across Datasets

Rebuttal - Part 1

We thank the reviewer for the detailed and constructive feedback. We respectfully reply to all the concerns raised in the weaknesses and questions section below.

Question-1: Unclear Memory Mechanism

The proposed method is presented as a memory system, but key aspects typically associated with memory—such as information storage, updating, and retrieval—are not clearly defined or illustrated. From a high-level perspective, the method appears more like a LoRA-based fine-tuning framework with retrieved image/text inputs, rather than a complete or standalone memory architecture.

Reply-to-Question1:

We appreciate the reviewer’s thoughtful question and would like to clarify our design and contributions as follows:

• Memory Encoder, Not a Full Memory System: Our work proposes a memory encoder module, rather than a complete memory system. Its primary goal is to encode external knowledge into compact continuous embeddings that can be flexibly integrated into the VLM to enhance its capabilities.

• Plug-and-play and Generalizable Memory: As demonstrated in Table 3 in the paper, the proposed memory encoder is effective across different backbones, such as Qwen2-VL and Qwen2.5-VL. Furthermore, Table 1 below shows that our continuous memory can successfully transfer visual and textual knowledge to text-only LLMs. This indicates that the continuous memory representations are broadly compatible and can effectively preserve transferable multimodal knowledge for augmenting any LLM.

• Simple Yet Effective Retrieval: While our current implementation adopts a CLIP-based retrieval mechanism, consistent with prior work, our method already demonstrates strong performance. We believe that incorporating more advanced retrieval strategies could further enhance the results.

• Storage Efficiency: Our approach offers a significant storage advantage over traditional designs. The following Table 2 shows the average context length for each benchmark with or without our method. It clearly shows that the discrete token format is approximately 15x times larger than the continuous memory format. By storing compact continuous embeddings instead of original images or discrete token sequences, we can substantially reduce memory and storage requirements.

Table 1: Transferability Study of Vision-Language Memory Encoded by CoMEM on LLMs

Table 2: Token Number and Accuracy Across Datasets

Question-2: Inference Process Ambiguity

It is unclear how the memory embeddings interact with the VLM during inference. The claim that “we keep the VLM’s parameters frozen during inference, while continuous memory embeddings are directly used for downstream tasks” lacks clarity. More detailed explanation of the inference workflow and the contribution of memory in prediction is needed.

Reply-to-Question2:

We apologize for the confusion; there was a typo in the original statement. The correct description is: “We keep the VLM’s parameters frozen during training, and only train the LoRA adapters and the Q‑Former.” Below we clarify both the training and inference processes:

• Training stage:

  • Trainable Components: (i) LoRA adapters (low‑rank update matrices) in selected layers of the VLM encoder and (ii) the Q‑Former, which compresses multimodal knowledge items into 8 continuous memory embeddings each.
  • Frozen Components: All original VLM parameters are frozen.

  This design updates only 1.2% of parameters, following a parameter‑efficient fine‑tuning strategy (Section 3.3, “Parameter-efficient Fine-tuning”).

• Inference stage:

  • The trained Q‑Former generates continuous memory embeddings for retrieved multimodal knowledge items. These memory embeddings are then prepended to the input embedding sequence of the frozen VLM(Section 3.2, “Plug‑and‑Play Mechanism”). The VLM then performs normal autoregressive generation, attending to the memory embeddings and the query jointly.
  • Therefore, the continuous memory is consumed by the VLM’s self‑attention layers during decoding, which provide retrieved knowledge to guide prediction. This design allows the memory module to be plug‑and‑play without altering the base VLM’s parameters. As shown in Table 2 in the paper, this paradigm achieves the lowest training cost among most state‑of‑the‑art RAG models while still delivering superior performance.

Question-3: Limited Comparison Baselines

In the SOTA table, only vanilla RAG is compared. The authors should conduct a more comprehensive comparison with other RAG-based approaches to better validate the effectiveness of their method.

Reply-to-Question3:

Thank you for your suggestion. We agree that stronger and more diverse baselines are important for a fair evaluation, and we have selected SOTA ones in our paper for comparison.

• Competitive RAG methods: In Table 3 (in paper), we have compared four highly relevant and latest multimodal RAG-based methods: Wiki-LLaVA, RORA-VLM, EchoSight, and ReflectiVA.

• Adding More Experiments on RAG Baselines: To address your concern, we have: (1) Re-implemented EchoSight and ReflectiVA (the only open-sourced methods among the four) and tested it on all benchmarks (MRAG, OKVQA, AOKVQA, ViQUAE) to provide a comprehensive evaluation. (2) Included more relevant RAG-based methods for comparison:

  • Wiki-LLaVA[1]: Uses hierarchical retrieval from Wikipedia to add external knowledge for visual question answering.
  • RORA-VLM[2]: Enhances robustness with two-stage retrieval and adversarial training to filter irrelevant content.
  • EchoSight[3]: Applies a visual-first retrieval followed by text-image reranking for better encyclopedic knowledge integration.
  • ReflectiVA[4]: Uses reflective tokens to decide when and what external knowledge to retrieve.
  • REVEAL[5]: Pre-trains a retrieval-augmented VLM with a large multimodal memory for knowledge-intensive tasks.

  The results are shown in the following Table 3. This extended evaluation demonstrates that CoMEM consistently outperforms open-source multimodal RAG methods (Echosight, ReflectiVA) across tasks, and is competitive with or superior to closed-source systems like RORA-VLM, REVEAL, etc. These results further validate the generality, scalability, and effectiveness of our proposed memory-based method.

Table 3: Performance Comparison Across Datasets

[1] Davide Caffagni et al. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR 2024.

[2] Jingyuan Qi et al. RoRA-VLM: Robust Retrieval-Augmented Vision Language Models. In 2024.

[3] Yibin Yan . EchoSight: Advancing Visual-Language Models with Wiki Knowledge. In EMNLP 2024.

[4] Federico Cocchi et al. Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. In CVPR 2025.

[5] Ziniu Hu et al. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. In CVPR 2023.

Thank you very much for considering our responses! We truly appreciate your recognition of our efforts to address your suggestions, and we’re grateful for your decision to raise the score. Your support and constructive comments are very encouraging and helpful in improving our work.

We also want to kindly remind you to remember to raise your score, as it is quite valuable and important to us. Thanks again for your time and support!

Rebuttal

We sincerely thank the reviewer for the insightful comments and appreciate the positive feedback! We reply to all the concerns raised in the weaknesses and questions part.

Weakness-1: Limited Theoretical Justification

Lacks analysis of why continuous embeddings outperform discrete tokens or how compression affects performance.

Reply-to-Weakness1-A:

For continuous embeddings and discrete tokens, it is hard to propose a proof that conclusively establishes the superiority of either approach. Instead, we demonstrate the intuitive and practical advantages of continuous embeddings from several perspectives.

• Conceptual alignment with human cognition: Cognitive‑science research shows that humans store and manipulate knowledge at the level of abstract concepts, not at the level of exact pixel patterns or phoneme sequences. When you think of “the Great Wall” or “the Eiffel Tower,” you effortlessly summon a coarse, high‑level mental image while discarding countless low‑level details such as the precise arrangement of every brick or rivet. Therefore, the continuous embedding based memory better mirrors the way our brains compress and retrieve multimodal information.
• Continuous embeddings better support high compression: as continuous embeddings can densely encode the information, which prevents the great increase of context length (see Table 2). It is rather helpful for VLMs to read and understand massive multimodal data. For the example of RAG scenario, here we list the context length in three settings: Original model(no retrieval) , Qwen2-VL+RAG and our CoMEM method. We can clearly see that although we only retrieve 10 examples in the context, the context length increases by approximately 15× when using RAG. It will cause the VLM to suffer from the long context understanding problem. In contrast, our method converts the retrieved examples into continuous embeddings, which leads to fewer than 100 additional embeddings to the input sequence. It avoids the long context understanding problem and also reduces the cost of encoding such long input sequences.
• Better empirical results of continuous embedding: since continuous embeddings generally have higher compression rate, their rather short context length allows the model to process multimodal information more effectively. In our method, we jointly compress both image and text knowledge into a unified continuous latent space, which enables better cross‑modal integration and achieves superior performance on most benchmarks compared to state‑of‑the‑art methods. We also add results comparison with up-to-date RAG methods. We also added comparisons with up‑to‑date RAG baselines, and CoMEM consistently delivers the best results across all benchmarks:

Table 1: Performance Comparison Across Datasets

Table 2: Token Number and Accuracy Across Datasets

Reply-to-Weakness1-B:

Comparison of different compression rates: We conducted an ablation study by varying the number of memory embeddings (4, 8, 16, 24) on the InfoSeek benchmark, to study the effectiveness of different compression rates. As shown in Table 3 below, 8 embeddings consistently achieve the best overall performance across both unseen question and unseen entity settings.

• 4 embeddings do not perform well due to limited capacity—it is difficult to encode complex multimodal and multilingual knowledge with such a compact representation.
• 16 and 24 embeddings perform slightly worse than 8. While they offer more representational capacity, they also introduce redundancy and noise, making them harder to train effectively. These longer embeddings require more data to properly align with the frozen inference model, reducing training efficiency.
• 8 embeddings strike an optimal balance between compactness and expressiveness. They are sufficient to capture relevant knowledge while remaining lightweight, easy to integrate, and efficient to train.

These results suggest that 8 embeddings represent an effective inflection point, supporting our design choice in CoMEM and emphasizing the importance of tuning memory size for both performance and training efficiency.

Table 3: Performance Across Different Embedding Sizes on Infoseek

Weakness-2: Narrow Benchmarking

Focuses heavily on VQA tasks; testing on broader tasks (e.g., captioning) would strengthen claims of generalizability.

Reply-to-Weakness2:

Following the suggestion of the reviewer, we evaluate the generalizability of CoMEM on a caption generation task using the COCO 2014 dataset. We randomly sampled 100 image-caption pairs from the test set and used CLIP to retrieve relevant image-caption pairs from the training set. We then compared three setups: (1) Original model (no retrieval); (2) RAG-style retrieval (retrieved image-caption pairs prepended in prompt); (3) Our method (retrieved image-caption pairs encoded into continuous memory). We tested both Qwen2.5 and Qwen2, and report standard captioning metrics below in Table 4. We obtain the following conclusions:

• Better precision and fluency: BLEU-1 / BLEU-4 measure n-gram precision; METEOR balances precision and recall with synonym matching; higher scores indicate better word accuracy. CoMEM consistently improves BLEU-1, BLEU-4, and METEOR across both models, indicating more fluent and accurate captions.
• Substantial gain in content relevance: CIDEr scores use tf-idf weighted n-grams to measure relevance to ground-truth captions. It's increased significantly in Qwen2.5-VL with CoMEM (from 0.24 → 0.64), showing better content relevance to ground-truth captions.
• Improved semantic similarity: ROUGE-L captures the longest common subsequence with the reference, reflecting surface-level overlap. BERTScore-F computes semantic similarity using contextual embeddings. Memory-augmented models generate captions with high ROUGE-L and BERTScore-F scores, showing stronger lexical overlap and semantic similarity.

These results confirm that CoMEM generalizes well beyond QA tasks, offering an effective and lightweight memory mechanism for open-ended generation tasks like image captioning.

Table 4: Evaluation Metrics for Qwen2.5-VL and Qwen2-VL on COCO Captioning task

Question-1

Why do continuous embeddings work better than discrete tokens? The results show clear advantages but lack explanation. Is it because of higher compression efficiency or better compatibility with the VLM's attention mechanism? A simple theoretical analysis or comparison of different compression rates (e.g., 8-dim vs 16-dim embeddings) would make this more convincing.

Reply-to-Question-1:

Please refer to Reply-to-Weakness1-A and Reply-to-Weakness1-B

Question-2

Does this work only for QA tasks? All current experiments are on question answering. Could you test it on generation tasks like image captioning? Even one small experiment on COCO captioning would prove the method's generalizability.

Reply-to-Question-2:

Please refer to Reply-to-Weakness2

Rebuttal - Part 1

Weakness-1: Lacks Clarity in its Writing

The paper lacks clarity in its writing, with many terms left unexplained. For example, Figure 1 does not clearly explain what the attention-based compression rates refer to.

Reply-to-Weakness1:

We appreciate the reviewer’s feedback and will add further clarifications to improve the readability of the paper in all figures and tables.

First of all, we define the compression rate as:

Here, we provide details for the experiment in Figure 1.

• The continuous memory module first encodes each retrieved item into continuous embeddings. We then compute the average attention scores across all layers within the VLM. Embeddings with the highest attention scores are retained with a compression rate, and the rest are discarded (same as described in Section 2.1, Memory Methods - VLM-as-Memory+Attn).
• Figure 1 demonstrates that, even with this simple token‑selection strategy, VLM‑as‑Memory methods outperform the baseline at high compression rates. This figure provides the intuition and preliminary evidence for our proposed method; therefore, we kept the explanation brief in the original submission.

To further clarify and prevent any possible ambiguities, we provide the following additional explanations:

• Table 1 (in paper): This table compares various training‑free memory methods across three benchmarks (InfoSeek, OK‑VQA, A‑OKVQA). Methods include vanilla RAG, token pruning (FastV), and our VLM‑as‑Memory variants. The table shows that naive RAG can degrade performance due to context length overload, while our VLM‑as‑Memory methods consistently outperform token‑based methods because continuous embeddings align more naturally with the VLM.
• Figure 3 (in paper): This figure analyzes our model’s long‑context robustness. We vary the number of retrieved image‑text pairs (x‑axis) and plot accuracy (y‑axis) for RAG and our method on InfoSeek. RAG performance drops when more than 30 items are retrieved because the increased context length introduces attention dilution and ambiguity. In contrast, our method remains stable across retrieval sizes since the compressed continuous memory does not excessively increase input length, demonstrating better scalability for long‑context scenarios.

Weakness-2: Novelty of the framework

The methodological novelty over prior Q-Former and RAG frameworks is limited and incremental.

Reply-to-Weakness2:

In this paper, our target is to validate the effectiveness of using continuous memory for augmenting VLMs. Concretely, Q-Former is the technical way for achieving this goal, while RAG is a typical scenario for evaluating our approach. Follows are more detailed discussion about our contribution.

• Our key contribution is to construct an information-rich and plug-and-play continuous memory with minimal cost. To achieve this goal, we try to use simple and easy-to-follow techniques to verify whether the continuous memory is effective for improving VLMs. According to our extensive experiments, we validate that we can compress each multimodal item into just 8 continuous embeddings via the VLM itself, and the compressed embeddings are more helpful than SOTA RAG methods to better improve the performance.
• Our design is meaningful and opens a new direction with many possibilities:
  • Firstly, it turns a VLM into its own memory encoder and achieves strong semantic alignment without modifying the base model.
  • Secondly, our approach supports both multimodal and multilingual reasoning, and has shown strong performance across a broad range of tasks—including knowledge-intensive VQA, image captioning, and reasoning tasks—demonstrating its generality. | Method | Knowledge-Intensive VQA (Average) | Image Captioning (COCO) | Reasoning (MMMU) | |-------------|-------|------------------|-----------| | Qwen2.5-VL | 30.8 | 0.39 | 32.22 | | +RAG | 27.6 | 0.43 | 31.48 | | +CoMEM | 35.4 | 0.46 | 37.41 |
  • Thirdly, our memory also acts as a bridge between VLMs and LLMs, enabling the transfer of visual knowledge to pure language models. | LLM | InfoSeek: Unseen-Q | InfoSeek: Unseen-E | InfoSeek: All | OVEN: Query | OVEN: Entity | OVEN: All | Avg. | |---------------------------|--------------------|---------------------|----------------|--------------|---------------|-------------|--------------| | Qwen2.5-Instruct | 5.0 | 4.8 | 4.9 | 2.4 | 0.1 | 1.3 | 3.1 | | Qwen2.5-Instruct + RAG | 13.4 | 10.3 | 11.9 | 1.8 | 2.7 | 2.2 | 7.0 | | Qwen2.5-Instruct + CoMEM | 29.3 | 27.4 | 28.3 | 6.8 | 7.7 | 7.2 | 17.8 |
  • Importantly, our method opens up new possibilities such as grounding LLMs with vision, storing and accessing large-scale memory efficiently, and supporting long-horizon reasoning and planning in agent-based scenarios.
• Our simple and effective designs well prove our concept, and also reduce the cost: It has low training and inference cost with small inference latency, and the encoded continuous memory can be reused across tasks. We achieve strong performance using only 15.6k self-synthesized training samples and fine-tuning just 1.2% of parameters, making it both data and compute efficient, especially suitable for low-resource or cross-lingual settings. Across benchmarks and scenarios (knowledge-intensive VQA, image captioning, and reasoning tasks), our method consistently outperforms original vanilla RAG, up-to-date RAG baselines, confirming the effectiveness, robustness, and generalizability of our method.

Weakness-3 Unfair Comparison to RAG

The evaluation should include the latest works in RAG, rather than relying solely on a vanilla baseline.

Reply-to-Weakness3:

Thank you for your suggestion. We agree that stronger and more diverse baselines are important for a fair evaluation, and we have selected SOTA ones in our paper for comparison.

• Competitive RAG methods: In Table 3 (in paper), we have compared our method with four highly relevant and latest multimodal RAG-based methods: Wiki-LLaVA, RORA-VLM, EchoSight, and ReflectiVA.

• Adding More Experiments on RAG Baselines: To address your concern, we have: (1) Re-implemented EchoSight and ReflectiVA (the only open-sourced methods among the four) and tested them on all benchmarks (MRAG, OKVQA, AOKVQA, ViQUAE) to provide a comprehensive evaluation. (2) Included more relevant RAG-based methods for comparison:

  • Wiki-LLaVA[1]: Uses hierarchical retrieval from Wikipedia to add external knowledge for visual question answering.
  • RORA-VLM[2]: Enhances robustness with two-stage retrieval and adversarial training to filter irrelevant content.
  • EchoSight[3]: Applies a visual-first retrieval followed by text-image reranking for better encyclopedic knowledge integration.
  • ReflectiVA[4]: Uses reflective tokens to decide when and what external knowledge to retrieve.
  • REVEAL[5]: Pre-trains a retrieval-augmented VLM with a large multimodal memory for knowledge-intensive tasks.

  The results are shown in the following Table 1. This extended evaluation demonstrates that CoMEM consistently outperforms open-source multimodal RAG methods (Echosight, ReflectiVA) across tasks, and is competitive with or superior to closed-source systems like RORA-VLM, REVEAL, etc. These results further validate the generality, scalability, and effectiveness of our proposed memory-based method.

Table 1: Performance Comparison Across Datasets

[1] Davide Caffagni et al. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR 2024.

[2] Jingyuan Qi et al. RoRA-VLM: Robust Retrieval-Augmented Vision Language Models. In 2024.

[3] Yibin Yan . EchoSight: Advancing Visual-Language Models with Wiki Knowledge. In EMNLP 2024.

[4] Federico Cocchi et al. Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. In CVPR 2025.

[5] Ziniu Hu et al. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. In CVPR 2023.

Dear Reviewer,

Thank you again for your initial review. We have carefully addressed your concerns in our rebuttal and provided detailed explanations along with additional experimental results to support our claims.

We would greatly appreciate it if you could review our response and share your feedback. If there are any remaining concerns that require further clarification, we would be more than happy to provide more details and have a further discussion.

We also wanted to note that the other reviewers have responded positively to our rebuttal and have updated their scores accordingly. We sincerely hope to receive your feedback as well before the discussion period ends.

Best regards, The authors of Paper Towards General Continuous Memory for Vision-Language Models

Dear Reviewer,

During the rebuttal phase, we initially provided part of our response in the main rebuttal section, and included additional details in the Author AC Confidential Comment section, which we now realize is not visible to reviewers. To ensure you have access to the full context of our response, we are appending the missing portion below.

Weakness-4: Trade-off between RAG and Continuous Memory

RAG can be used in a training-free paradigm, whereas continuous memory typically requires training to adapt to the inference-time VLM.

Reply-to-Weakness4:

Thanks for your suggestion. We agree that continuous memory requires training to adapt to the inference‑time VLM. However, a key advantage of our methods is that it significantly minimize both adaptation cost and training overhead:

• One-time training with cross-task generalization: Unlike task‑specific fine‑tuning approaches, our framework requires only a single, lightweight training of the Q‑Former and LoRA adapters, which then generalizes across all downstream benchmarks. This design removes the need for retraining when switching tasks, thereby improving scalability and practical usability in multi‑task scenarios.
• Minimal adaptation cost: We explicitly designed the framework for parameter‑efficient adaptation. Only ~1.2% of parameters are trained, and the base VLM remains frozen. This one‑time training cost is significantly lower than most SOTA RAG models that require larger adaptation modules or resource‑intensive fine‑tuning.

Weakness-5: Need an Ablation Study

The paper lacks an ablation study to justify why using 8 embeddings is sufficient for the memory.

Reply-to-Weakness5:

Thank you for the suggestion. We conducted an ablation on the number of memory embeddings (4, 8, 16, 24) using the InfoSeek benchmark. As shown in Table 2, 8 embeddings consistently yield the best performance across both unseen question and unseen entity settings.

• 4 embeddings underperform due to limited capacity for encoding complex multimodal and multilingual knowledge.
• 16/24 embeddings introduce redundancy and noise, making training less efficient and harder to align with the frozen inference model.
• 8 embeddings offer the best trade-off between expressiveness and compactness—sufficiently powerful, yet lightweight and efficient to train.

These results suggest that 8 embeddings represent an effective inflection point, supporting our design choice in CoMEM and emphasizing the importance of tuning memory size for both performance and training efficiency.

Table 2: Performance Across Different Embedding Sizes on Infoseek

Weakness-6: Latency

Inference latency of the whole framework is not evaluated.

Reply-to-Weakness6:

We appreciate the reviewer’s suggestion and test the latency of baselines and our method. Concretely, we evaluated the throughput (tokens per second) for each model on 100 random sampled instances. The higher throughput indicates faster inference speed. All experiments were conducted on a single NVIDIA H100 (80 GB), and the results are summarized in Table 3 below:

• The findings show that CoMEM maintains competitive inference speed as the original model, because it does not significantly increase input length. For RAG, as it expands the input by up to 15× and causes a proportional increase in attention cost, the inference speed decreases a lot. Since our CoMEM adds < 100 continuous memory tokens, this keeps inference cost close to that of the base model while improving accuracy.
• Additionally, our training cost is the smallest among state‑of‑the‑art RAG models (Table 2 in paper). We train only lightweight LoRA adapters and the Q‑Former module (~1.2% of total parameters), while keeping the base VLM frozen. This parameter‑efficient design minimizes both compute and memory requirements during training.

Table 3: Latency

Dear Reviewer,

We truly appreciate the time and effort you’ve already dedicated to reviewing our work. During the rebuttal phase, we invested significant effort in running additional experiments and providing detailed explanations to directly address your concerns.

All other reviewers have kindly responded with constructive feedback and positive updates. We sincerely hope to hear your thoughts as well — your feedback is very important to us. If our responses have resolved your concerns, we would be grateful if you could consider updating your score accordingly. If there are any remaining questions or points you'd like us to clarify, we would be more than happy to continue the discussion.

Thank you again for your time and consideration.

Best regards, The authors of Paper Towards General Continuous Memory for Vision-Language Models

Thanks for authors' detailed and timely rebuttal. All of my questions have been answered with good answers and extensive qualitative results, and I am happy to raise my score.