Response to Reviewer 8cuF

Thank you for your detailed feedback! We address your comments below.

Q1: Concern about reference set construction: Academic benchmarks use predefined sample types, but real-world scenarios may not allow task type selection before inference. It might become necessary to store multiple types of reference sets simultaneously, increasing storage and computational demands.

1. Storage Cost: We appreciate this concern. However, our reference sets store only lightweight metadata—question text and routing weights—totaling just 3.24 MB in Parquet format, while images are loaded dynamically from Huggingface.
2. Memory Cost:As shown in Reviewer HT7U Q1, the additional memory and computational overhead remain minimal.
3. Scalability: Using a lightweight classifier to select a relevant subset of the reference set for each test sample, ensuring that optimization is performed on a much smaller and targeted set. (Please see Reviewer HT7U Q5)

Q2: A filtering mechanism could be required to identify the appropriate reference set before applying R2-T2, adding complexity to the framework.

Thank you for your question. A similar point was addressed in Reviewer HT7U Q5 regarding training a classifier. Please refer to that response for details.

Q3: Additionally, the effectiveness of R2-T2 heavily depends on the choice of neighborhood within the reference set, with only similar samples contributing meaningfully to the optimization process. What happens if no sufficiently similar samples exist in the reference set? What if an incorrect reference set is chosen (e.g., using OCR data for knowledge-based VQA)? Would R2-T2 still maintain its strong performance under these conditions?

Thank you for raising this important point. Please refer to Reviewer bdGh Q2 for no sufficiently reference samples. As for mismatched reference sets, we conducted experiments using an OCR-based subset (ST-VQA & DocVQA) as the reference for R2-T2 on SQA-IMG (knowledge-based VQA), yielding the following results:

These results highlight that R2-T2 provides significant gains (from 83.5 to 88.3) with a task-relevant reference set but offers minimal improvement (83.8) when the reference is mismatched, emphasizing the critical role of reference set selection.

Q4: Efficiency considerations: The framework introduces additional storage costs, particularly due to the embedding model's memory overhead (e.g., 7B parameters).

Thank you for raising this important point regarding storage and memory overhead. In addition to FLOPs, we now provide GPU memory usage comparisons (see table below).

Our R2-T2 framework with the nv_embed_v2 model requires 27GB of GPU memory and achieves 80.7% accuracy, while the base MoAI model uses 18GB at 74.5% accuracy. Importantly, smaller embedding models like all_mini_v6 (20GB total, 77.5%) or Stella-En-1.5B-V5 (22GB total, 78.5%) still provide significant accuracy gains with lower memory overhead. This demonstrates a scalable trade-off: users can select larger embeddings for maximum performance or opt for smaller models to balance efficiency and accuracy.

Q5: Scalability—The reference sets could grow significantly, especially when handling diverse or unpredictable tasks.

Thank you for your insightful question. We address scalability by

1. Using a lightweight classifier to select a relevant subset of the reference set for each test sample, ensuring that optimization is performed on a much smaller and targeted set. (Please see Reviewer HT7U Q5)
2. We employ parquet compression techniques and caching strategies to reduce storage and memory consumption effectively. (Please see Q1)
3. By leveraging efficient similarity search frameworks like FAISS, we significantly improve the computation speed of neighbor retrieval even when the overall reference set is large. (Please see Reviewer HT7U Q1)

Q6: Similarities to RAG & in-context learning—The method shares similarities with RAG, and could in-context learning achieve comparable performance?

Thank you for your question. Please refer to Reviewer HT7U Q3, where we discuss both RAG-related considerations and the role of in-context learning in comparison to our approach.

Response to Reviewer 1y8A

Thank you for your detailed feedback! We address your comments below.

Q1: Generalization to OOD samples—The method relies on a reference set with similar questions. How does it perform on truly out-of-distribution (OOD) cases?

Thank you for your question. Our approach is designed as a zero-shot reference method. We deliberately do not use any benchmark data for reference, meaning that our test questions is out-of-distribution (OOD) relative to the reference set. And there is no overlap between Benchmarks and Reference Set (Please see Reviewer bdGh Q1)

For the questions are not similar with reference samples: Our Mode Finding (section 3.3 in paper) strategy does not strictly rely on identical questions but rather on the proximity of routing weights in the expert space. Even for literally different questions, if their routing weights are close to those of reference samples, it indicates that similar experts are needed. Therefore, as long as the underlying expert requirements are similar, our method generalizes well even to not similar cases.

Q2: Computational overhead and scalability—Larger reference sets may introduce significant computational costs. Have the authors analyzed this in detail?

Thank you for your question. Please refer to Reviewer 8cuF Q5 for your concern about computational overhead and scalability.

Q3: Potential mismatches—If a reference question closely overlaps with a test question but belongs to a different type, how does the method avoid incorrect matches?

Thank you for this insightful question. In theory, two questions might have similar surface forms even if they are of different types.

However, in practice, VLM questions tend to be short and less complex compared to those in LLMs, and our tasks come with sufficiently detailed descriptions that help disambiguate their semantic intent. This means that even if a few words are similar, the overall context captured in the embedding still reflects the true task type.

Furthermore, in scenarios where subtle differences might lead to confusion, our method can be augmented with a chain-of-thought (CoT) prompt. In this variant, we prompt the model to outline a few high-level reasoning steps, and then we use the resulting CoT output as the embedding for neighbor search. This additional step helps ensure that the underlying reasoning process—and not just the superficial wording—is captured, further reducing the chance of mismatches.

Response to Reviewer HT7U

Thank you for your detailed feedback! We address your comments below.

Q1: Significant increase in computational and memory costs.

R2-T2 does not require significant increase in computations and memory because it only optimizes a low-dimensional routing weight vector (e.g., 6-dim for MoAI). Except NGD, the other two R2-T2 strategies are gradient-free and do not require backpropagation. Hence, compared to PEFT (LoRA, prompt, prefix tuning), re-training the MoE model or routers, R2-T2 is much more efficient. Moreover, R2-T2 significantly improves model accuracy, especially on challenging downstream tasks, making the modest additional cost worthwhile.

We can further reduce R2-T2's cost by:

• The embedding vectors can be pre-computed and cached (less than 1GB), which substantially reduces the runtime overhead during deployment.
• We use fast similiarty search tools such as faiss to search the neighbors and compute similarity efficiently.
• Our approach is flexible, allowing users to adjust the kNN neighborhood size and the number of iterative steps for a better trade-off between accuracy gains and computational cost effectively.

Q2: The method has only been adapted for a single-level routing decision, lacking the complexity of SOTA MoE models with independent routing across multiple layers.

Our focus is on visual-centric tasks, where the primary bottleneck of most VLMs is the limited capability of a single visual encoder. In MoE VLMs such as MoAI and MoVA, MoE is applied only at the visual encoder, by replacing it with <10 experts, making a single-level routing decision sufficient for effective feature extraction. Extending to multi-layer routing—common in LLM MoE—would introduce much more computational overhead without addressing the core challenge of constrained visual expert capacity.

Q3: The framework assumes access to a large-scale reference set, making comparisons in Table 3 unfair.

Because MoAI and MoVA do not support interleaved input and ICL/RAG, we use Qwen-VL as the base model. 1.For RAG, we retrieve similar reference samples based on embedding similarity and use them as few-shot demonstrations. 2.For ICL, we randomly choose the same task demonstrations with correct answer in reference set as few-shot demonstrations.

RAG improves only modestly (61.7% → 64.1%) and ICL to 62.9%, while R2-T2 boosts MoAI from 74.5% to 80.7%, showing it leverages the reference set more effectively.

Q4: Please evaluate additional baselines, such as ensemble voting, noisy router sampling, few-shot prompting, or a simple RAG approach.

We evaluated these baselines:

• Ensemble Voting – Enabled dropout during inference and performed 10 forward passes per sample, aggregating predictions via majority voting.
• Noisy Routing Ensemble – Added Gaussian noise to the router’s output, ran 10 forward passes with different noise realizations, and applied majority voting.

Few-Shot Prompting – Please see Q3

RAG-style Retrieval – Please See Q3

As shown, ensemble-based methods provide only marginal gains (≤0.9%), whereas R2-T2 delivers a substantial improvement of +6.2%. These results validate the effectiveness of R2-T2 beyond what additional compute alone can achieve.

Q5: The user must specify the task type to select the appropriate reference set. Can this process be automated using a classifier?

Automating task type selection could improve usability and efficiency, so we conducted experiments to explore this approach. We pre-annotated our reference set with three task types (visual understanding, knowledge reasoning, OCR) and extracted 4096-dimensional task embeddings using NV_Embed_V2. Then, we trained a lightweight logistic regression classifier using embedding of reference set, which only introduces $10^4$ – $10^5$ FLOPs per inference. Given a test sample, the model predicts its task type and selects the corresponding reference subset for test-time adaptation.

Our results show that this automated selection reduces computational overhead while maintaining accuracy, with only a marginal drop from 80.7% to 80.4%. These findings demonstrate that task classification is a viable strategy for improving efficiency with minimal impact on performance.

Response to Reviewer bdGh

Thank you for your detailed feedback! We address your comments below.

Q1: Possible data contamination—subsampled reference datasets (e.g., VQA-V2, MathVista) may overlap with evaluation benchmarks (e.g., MMBench, TextVQA), potentially inflating results. How was this addressed?

We appreciate the reviewer’s concern and have conducted a rigorous analysis to ensure no data contamination. Our process follows a two-step screening approach:

1. Question Similarity Check – We first computed cosine similarity between evaluation benchmark questions and reference set questions using the NV_Embed_V2 embedding model. Samples with a similarity score >0.95 were flagged for further inspection.
2. Image Similarity Check – For flagged cases, we applied CLIP to measure image similarity. Only samples where both question similarity and image similarity exceeded 0.95 were classified as potential overlaps.

Through this analysis, we found no overlapping samples between the reference set and evaluation benchmarks. These results confirm that our performance gains are not due to data leakage but stem from the effectiveness of our proposed method.

Q2: Impact of reference set choice—how does the method perform when reference samples have limited coverage for certain task types (e.g., rare spatial reasoning)? What is the effect of reference set size (e.g., 1K vs. 10K samples)?

We appreciate this insightful question and have conducted further experiments to analyze these factors.

1. Limited Task Coverage: We evaluated R2-T2 on 3DSRBench, a dataset focused on rare spatial reasoning cases. Despite lower reference coverage for these tasks, R2-T2 still delivers a 4.5% improvement over the base model, demonstrating its robustness.

2. Reference Set Size: We varied the reference set size and observed that even when reduced to 1/10th of its original size (e.g., 1K instead of 10K samples), R2-T2 still provides a notable boost (+2.9%). However, when reference samples are randomly selected (without ensuring task relevance), the improvement is minimal (+0.3%), underscoring the importance of well-curated reference sets.

These results confirm that while reference set quality and coverage affect performance, R2-T2 remains effective even with smaller but carefully selected reference sets.

Q3: More evaluation results are needed on important benchmarks, such as MMMU and ChartQA.

Thank you for the suggestion. We have evaluated R2-T2 on MMMU and ChartQA, with results summarized below:

These results demonstrate that R2-T2 consistently improves performance across diverse multimodal tasks, reinforcing its robustness and generalizability.

Q4: While the improvements presented are compelling, the significant increase in FLOPs shown in Table 4 raises concerns. Additionally, a comparison of latency is needed.

Please refer to Reviewer HT7U Q1 regarding computational cost. To directly address latency, we conducted experiments on an RTX A6000:

While R2-T2 increases latency by 3.3×, we believe the substantial accuracy gains justify the trade-off. Moreover, optimizations such as reference set pruning and efficient kNN search can further reduce overhead.