LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

We thank the reviewer for the feedback and comments.

R-o1yY-W1: REC performances.

R-o1yY-A1: Indeed, the performances of many REC methods are better than the performances of LLM-wrapper, and we thank the reviewer for the references [1],[2],[3] that we will add to the paper.

However, this observation misses the key focus of our work: the aim of LLM-wrapper is not to replace or outperform traditional white-box learning and fine-tuning methods for the REC task, as clarified in the paper (Lines 27, 64, 88, 352). Instead, our method is designed as an alternative solution for scenarios where traditional white-box learning and fine-tuning approaches are not feasible, which includes:

• When there is no access to a model's architecture or weights, such as in the case of proprietary or closed-source models (Lines 17, 50–52).
• When expertise in designing fine-tuning objectives and optimizing hyperparameters specific to each VLM is lacking (Lines 48–49).
• When using fine-tuning APIs is not an option, for instance, due to concerns about sharing private data (Lines 52–53).

Additionally, LLM-wrapper provides unique advantages over classic white-box fine-tuning:

• Transferability: Once fine-tuned on one VLM, LLM-wrapper can be directly transferred to another VLM without additional adaptation.
• Ensembling capability: LLM-wrapper can seamlessly learn to combine outputs from multiple VLMs, leveraging their strengths for improved performance.

R-o1yY-W2: Additional dataset comparisons.

R-o1yY-A2: Thank you for the suggestion to expand the experimental evaluation.

In response, we evaluate LLM-wrapper on the HC-RefLoCo dataset, which features longer and more complex text queries than RefCOCO/+/g, with 84.4 words per query on average. Since HC-RefLoCo lacks a training set, we transfer an LLM-wrapper trained on RefCOCOg using the protocol described in Section 4.3 (starting L.408).

LLM-wrapper achieves a significant +18.8 (resp. +18.4) P@1 improvement on the val (resp. test) split over zero-shot Florence-2 (see Table R.11 below), showing its ability to handle complex queries, adapt to more challenging datasets, and reaffirming its transferability. We will add these results to the revised paper.

Table R.11. LLM-wrapper performance on the HC-RefLoCo benchmark (when fine-tuning on RefCOCOg), vs zero-shot Florence 2.

Regarding RefClef, we note that 99.7% of its queries have fewer than three words, with 76.5% consisting of just a single word. These short queries are less aligned with LLM-wrapper’s strengths in reasoning over complex expressions involving relations, spatial cues, or negations. Thus, we do not anticipate substantial improvements on RefClef. For Flickr30K Entities, the black-box adaptation with LLM-wrapper is ongoing. We aim to include those results in the revised paper if time allows, or in the final version.

. This method could be a viable option when the weights or architecture of LLMs are not available. However, existing LLMs such as GPT-4, Claude, OFA, and MDETR have demonstrated their ability to handle various vision-language tasks, including REC. Moreover, some of these models have publicly released their code. In extreme cases where none of these LLMs are effective, could you provide a specific example and present experimental results for such scenarios? Based on the authors' response, I decide to raise the rating to 5.

Dear Reviewer,

As promised, we have incorporated your feedback and added clarifications and results to the revised paper, where changes are marked in blue. Below, we point to the specific sections where the changes addressing your remarks and questions have been implemented.

R-o1yY-A2: Our answer has been integrated to the revised paper, with the new Table 6 and new paragraph at the end of Section 4.3 (Paragraph: "Transferring a trained LLM-wrapper to new datasets.").

As the author-reviewer discussion period continues, we would greatly value your feedback on whether our responses have sufficiently addressed your concerns. If you have any further comments, please feel free to share them with us.

Best regards,

Answer Part 1/2

We thank the reviewer for the insightful feedback and comments.

R-DAad-W1: Double inference cost.

R-DAad-A1: We acknowledge this limitation of our approach. LLM-wrapper doubles the inference cost because it integrates LLM reasoning with the generalist VLM. However, our method is compatible with techniques that reduce LLM computation costs, such as FlashAttention and Quantization (L.204). We appreciate this comment and will include this limitation in the revised paper.

R-DAad-W2: LLM-wrapper is highly dependent on the performance of the chosen VLM.

R-DAad-A2: We agree that LLM-wrapper's performance depends on the quality of the chosen VLM. As acknowledged in the paper (L.42-43, L.159), LLM-wrapper leverages the ability of detection-oriented VLMs to localize most query nouns with reasonably accurate bounding boxes and labels. This assumption is reasonable given the significant improvements in generalist VLMs, which now perform well in grounding objects from text queries. This dependency is fundamental to our approach, and if a VLM fails to meet this assumption, LLM-wrapper's effectiveness would also diminish. We will further acknowledge this in the limitation section of the paper.

R-DAad-W3: Applying LLM-wrapper on other VLMs, like Ferret, KOSMOS-2, and Qwen-VL.

R-DAad-A3: We thank the reviewer for this suggestion. As the proposed VLMs (Ferret, Kosmos-2, Qwen-VL) are pre-trained for the REC task, we do not anticipate significant improvements with LLM-wrapper. Indeed, these VLMs are already designed to handle referring expressions and grounding tasks effectively. Nevertheless, we evaluate Kosmos-2 with and without LLM-wrapper on the RefCOCOg validation set. The LLM-wrapper here uses a Llama3 model trained to adapt Florence-2 (following our transfer protocol, similar to Section 4.3), providing a lower bound for what we could achieve with direct black-box fine-tuning of Kosmos-2 with LLM-wrapper, which is currently under investigation. While the improvement is small, it demonstrates that LLM-wrapper can slightly enhance the REC performance:

Table R.7. Results when transferring LLM-wrapper: Florence-2 → Kosmos-2.

To gather additional results regarding the use of LLM-wrapper on VLMs that are already REC-adapted, we add in Table R.8 below results of LLM-wrapper applied to Florence-2 Large FT and GDRec (SwinB), both of which are already fine-tuned on REC data. Further adapting them with LLM-wrapper produces the following results:

Table R.8. LLM-wrapper results when applied to VLMs, already fine-tuned for the REC task.

While the improvements are modest, it is notable that LLM-wrapper does not degrade performance and, in some cases, provides a slight boost. This highlights its compatibility with state-of-the-art VLMs. We will include these results in the revised version of the paper.

Answer Part 2/2

R-DAad-W4: Analysis on the number of entities in the referring expression.

R-DAad-A4: We appreciate the reviewer's ablation study suggestion. To perform this study, we count the average number of words and noun groups per query for all of our datasets (in particular, for their val splits). Noun groups were identified using SpaCy’s noun_chunks method (e.g., the sentence "Green plant behind a table visible behind a lady's head" yields three noun groups: "Green plant," "a table," and "a lady's head"). The results are summarized in the Table R.9 below (note that we add numbers regarding the HC-RefLoCo dataset, with implementation details discussed in R-DAad-A5):

Table R.9. Text query complexity, number of candidate boxes and LLM-wrapper's P@1 on all our datasets (statistics and P@1 scores are computed on val splits).

LLM-wrapper demonstrates robust performance across datasets, showing no clear correlation between the number of entities in referring expressions and either the P@1 score or the performance boost. Interestingly, on the challenging HC-RefLoCo dataset, which features long queries with many entities (more than 20 noun groups per query on average), LLM-wrapper maintains strong reasoning capabilities and performs well on the numerous boxes proposed by the VLM (more than 17 per prompt on average). We will include this discussion in the revised paper.

R-DAad-W5: Evaluation of LLM-wrapper on longer expressions with more entities, such as HC-RefLoCo.

R-DAad-A5: At the reviewer’s suggestion, we evaluate LLM-wrapper on the HC-RefLoCo dataset, which includes longer expressions with more entities than the RefCOCO/+/g series. Since HC-RefLoCo has no training set, we transfer an LLM-wrapper trained on RefCOCOg to this dataset, with a similar protocol as described in Section 4.3 (starting L.408).

LLM-wrapper achieves a significant +18.8 (resp. +18.4) P@1 improvement on the val (resp. test) split over zero-shot Florence-2 (see Table R.10 below), showing its ability to handle complex reasoning, adapt to datasets with more challenging expressions, and stressing again the transfer abilities of LLM-wrapper. These results will be added to the revised paper.

Table R.10. LLM-wrapper performance on the HC-RefLoCo benchmark (when fine-tuning on RefCOCOg), vs zero-shot Florence 2.

Dear Reviewer,

As promised, we have incorporated your feedback and added clarifications and results to the revised paper, where changes are marked in blue. Below, we point to the specific sections where the changes addressing your remarks and questions have been implemented.

R-DAad-A1 and R-DAad-A2: We added a "Limitations" paragraph in the Conclusion.

R-DAad-A3: Our answer has been integrated to the revised paper in Section 4.2 and further expanded upon in Appendix A.

R-DAad-A4: Our answer has been integrated to the revised paper in Section 4.3 and further expanded upon in Appendix B.2

R-DAad-A5: Our answer has been integrated to the revised paper, with the new Table 6 and new paragraph at the end of Section 4.3 (Paragraph: "Transferring a trained LLM-wrapper to new datasets.").

As the author-reviewer discussion period continues, we would greatly value your feedback on whether our responses have sufficiently addressed your concerns. If you have any further comments, please feel free to share them with us.

Best regards,

Thank you for your feedback and suggestion for a deeper analysis of entity counts within queries.

We provide initial results on RefCOCOg val, showing that performance remains robust as the number $K$ of words in queries increases:

Table R.14. P@1 of LLM-wrapper (adapting Florence-2 zero-shot with Llama3 8B) on RefCOCOg val, grouped by $K$.

We will expand this analysis using HC-RefLoCo, which contains longer and more complex expressions, and refine our study to assess the impact of multiple entities or bounding boxes from the same category in the final paper.

Answer Part 1/2

We thank the reviewer for the insightful feedback and comments.

R-f3st-W1: White-box experiments use SwinB, but zero-shot baselines and LLM-wrapper experiments use SwinT.

R-f3st-A1: We agree that using different backbones introduces variability between the two sets of experiments. However:

• SwinT (not trained for REC) and SwinB (trained for REC) are the only two publicly available models and checkpoints (L.260-261) from Grounding-DINO's official code.
• Our goal is not to compare directly with white-box fine-tuning, as we are working in a black-box setting (L.27, L.64, L.88, L.352). Moreover, comparing our results to the upper-bound performance of white-box fine-tuning is unfavorable to us, as we apply our method to the smaller backbone (SwinT).
• Despite using the smaller backbone, after black-box fine-tuning with LLM-wrapper, we can achieve performance that is close to the white-box fine-tuning on the larger backbone (SwinB) for RefCOCOg (L.352-356).

R-f3st-W2: Results of the Transfer experiments on 300 samples (Table 5).

R-f3st-A2: As requested, we have added results for the transfer experiments (Florence-2 → GDrec, GDrec → Florence-2) evaluated on the same 300-sample subset used for GD-1.5. These results are reported in the column P@1 Val (Subset 300) in the Table R.5 below and align directly with Table 5 of the paper. The new results are marked with #.

Table R.5. Results of LLM-wrapper when using different VLMs’ outputs during fine-tuning and inference, in P@1 on RefCOCOg ‘umd’ splits.

We observe that results on the 300-sample subset are consistent with those on the full RefCOCOg validation set. This shows that the small 300-subset is a rather good proxy of the dataset's overall difficulty, and that the performance gains of GD-1.5 adapted with LLM-wrapper are indeed significant. We will include these results in the revised paper.

R-f3st-W3: For REC, models with a similar number of parameters fine-tuned using the same amount of bounding boxes can even get better results.

R-f3st-A3: LLM-wrapper relies on bounding box annotations, and while models fine-tuned with such annotations often perform better, this is only feasible in a white-box setting where access to the model’s architecture, weights, and training expertise is available. LLM-wrapper, by contrast, addresses the black-box scenario, where such access is not possible.

Training a model from scratch using REC data and annotations is another option, but this approach requires expertise in model design and would likely fall short of achieving competitive performance. Indeed, recent SOTA REC models benefit from training on large-scale, diverse datasets such as Objects365 and Visual Genome, which include hundreds of thousands of images and complementary tasks. This rich training environment provides robustness and generalization, which cannot be achieved with REC data alone.

Instead, we believe that leveraging off-the-shelf generalist VLMs is a practical approach, since these models benefit from the scale and task diversity provided by their pre-training datasets. However, without access to their internal architecture or training parameters, we explore a black-box adaptation approach in this work. LLM-wrapper effectively adapts such generalist VLMs for REC, offering a practical and resource-efficient solution for restricted-access scenarios.

Answer Part 2/2

R-f3st-Q1: Are the reported results for dataset X obtained with LLM fine-tuning on corresponding training set for X?

R-f3st-AQ1: The results reported in all experiments are obtained using LLM-wrapper fine-tuned exclusively on the corresponding training data for each benchmark, not on a combined set of samples from all three benchmarks. We will clarify this point in the revised manuscript.

In addition, this question inspired us to explore LLM-wrapper's zero-shot domain transfer capabilities, across datasets. Using the Florence-2 model adapted with LLM-wrapper on RefCOCOg-train, we evaluate its performance on RefCOCO and RefCOCO+ val/test sets. The results (Table R.6) show performance gains of 12.7/11.9 points on RefCOCO val/test and 7.3/6.6 points on RefCOCO+ val/test compared to zero-shot Florence-2. Although slightly below black-box fine-tuning directly on target datasets, these results highlight LLM-wrapper's ability to learn transferable knowledge. New results compared to the original paper are marked with #.

Table R.6. Results of LLM-wrapper in P@1 when using different datasets for training and evaluation, compared to our original setting (same dataset from training and evaluation).

This transferability reduces the need for multiple fine-tuning datasets and training processes. It also enables LLM-wrapper to be applied to datasets without training splits. For instance, as detailed in our response R-DAad-A5 to reviewer DAad, we used an LLM-wrapper trained on RefCOCOg to evaluate our method on the proposed HC-RefLoCo benchmark, which only has validation and test splits. These findings will be added to the revised paper.

Dear Reviewer,

As promised, we have incorporated your feedback and added clarifications and results to the revised paper, where changes are marked in blue. Below, we point to the specific sections where the changes addressing your remarks and questions have been implemented.

R-f3st-A2: We have revised Table 5 and integrated our answer in Section 4.3 (Paragraph: "Transferring a trained LLM-wrapper to a new VLM").

R-f3st-AQ1: Our answer has been integrated to the revised paper, with a clarified Section 4.2 (L.300-301), a new Table 6 and a new paragraph at the end of Section 4.3 (Paragraph: "Transferring a trained LLM-wrapper to new datasets.").

As the author-reviewer discussion period continues, we would greatly value your feedback on whether our responses have sufficiently addressed your concerns. If you have any further comments, please feel free to share them with us.

Best regards,

Thank you for your response and for highlighting what you find most interesting: LLM-wrapper zero-shot transfer ability, enabled by its black-box and semantic-aware adaptation, which avoids the overfitting of white-box fine-tuning and leads to better generalization.

We understand the concern about black-box fine-tuning given the availability of open-source models adapted for REC. While LLM-wrapper's true strength lies in adapting VLMs that are not pre-adapted for the task, it also complements existing white-box adaptation methods. As shown in Appendix A, LLM-wrapper achieves up to +2.4 P@1 improvements even on VLMs already fine-tuned for REC.

Answer Part 3/3

R-q6Km-W4: Failure case analysis. Does the LLM adhere to producing only the best box as the output?

R-q6Km-A4: Indeed, a failure case arises when the LLM does not adhere to the rule of producing only the index of a candidate bounding box as output. In such cases, the LLM may output non-integer values or indices that are out of range of the candidate boxes' list. However, these issues are very rare (<0.5%). In Table R.4 below, we report the percentage of invalid outputs, when Florence-2 is adapted with LLM-wrapper, with either Mixtral or Llama3. In these rare failure scenarios, we use a simple fallback strategy: the best-ranked box from the zero-shot VLM is used as prediction.

The main failure case of LLM-wrapper, described in L.497-500, occurs when bounding box coordinates and labels alone are insufficient to localize certain referring expressions that require additional visual cues. For example, in Figure 2f, correctly answering the query "White plane in the front" requires knowing the orientation of the planes. We will include more such examples, and the invalid box index failure case, in the revised paper.

R-q6Km-W5: How far are results from SOTA performance on RefCOCO REC?

R-q6Km-A5: At the time of writing the rebuttal, the state-of-the-art (SOTA) performance on RefCOCO/+/g REC remains Florence-2. To be more precise, there are two distinct SOTAs:

• White-box adaptation setting: The SOTA performance occurs when Florence-2 is trained on RefCOCO/+/g data and the REC task, along with other tasks and datasets. For example, the SOTA on RefCOCOg (val-umd) is 90.3 (reported in Table 3, RefCOCOg val-umd). Our work does not aim to replace or outperform this setting, as it focuses on the black-box paradigm (L.27, 64, 88, 352). However, when LLM-wrapper is used to adapt this version of Florence-2, it slightly improves performance to 90.4 (see our answer R-DAad-A3 to reviewer DAad for details).
• Zero-shot setting: When Florence-2 is not trained on the REC task with the RefCOCO/+/g datasets, the SOTA is 68.3 (Table 3) on RefCOCOg (val-umd). We adapt this model with LLM-wrapper, with black-box fine-tuning, and achieve a performance of 77.9 (Table 3).

R-q6Km-W6: Minor comment: The paper is too verbose in many places, e.g., section 3.4 seems unnecessary as the benefits have been explained well in the previous sections. Instead, it would be nice to provide more analysis with SoTA baselines, more diverse qualitative examples, limitations, etc.

R-q6Km-A6: Thank you for this suggestion. We will carefully review the paper again, and Section 3.4 in particular, to ensure conciseness. This adjustment will allow us to allocate space for additional analysis, including:

• New results on zero-shot dataset transfer, VLM transfer, HC-RefLoCo dataset, Kosmos-2 VLM,
• New analyses on the influence of the number of entities in the query, of the LLM reasoning abilities and of the number of trainable parameters, on the P@1 performance,
• More diverse qualitative examples, including success and failure cases,
• Clarifications adressing reviewers' feedbacks.

R-q6Km-Q1: How is GroundingDino (GD) used as zero-shot method for REC task?

R-q6Km-AQ1: In our zero-shot experiments, we use GroundingDino (GD) by selecting the bounding box that has the highest score with respect to any token in the query, from the 900 proposals generated by GD (L.255-256). However, this approach sometimes selects boxes based on parts of the query unrelated to the main object of interest, such as other nouns in the sentence. To mitigate this, we introduce a straightforward alternative, GDrec, which first identifies the query's subject and then selects the bounding box that scores best against it (L.254-257). We will rephrase and clarify in the revised paper.

R-q6Km-Q2: Is the number of fine-tuned parameters an important aspect in achieving better performance?

Please see our answer above (R-q6Km-A3)

R-q6Km-Q3: Does the LLM always adhere to the generated bbox index?

Please see our answer above (R-q6Km-A4)

Answer Part 2/3

R-q6Km-W3: Performance variability depending on the choice of LLM and number of finetuned parameters.

R-q6Km-A3: Our ablation study on LoRA's rank $r$, presented in Figure 4(b), evaluates P@1 across various $r$ values. Since the number of trainable parameters scales linearly with LoRA's rank, this analysis indirectly illustrates how the number of trainable parameters affects performance. To clarify this aspect, and to answer more precisely the reviewer’s comment, we conduct a more detailed analysis of the impact of the number of fine-tuning parameters. LLM-wrapper's results using Llama3 8B and other LLMs (applied to Grounding-DINO (GD) and evaluated on RefCOCOg val-umd), are given in the Table R.2 below, showing the relationship between the number of trainable parameters and performance. We also report the original performance of the respective LLMs on the standard MMLU benchmark and HellaSwag, a benchmark for LLM commonsense reasoning evaluation.

Table R.2. Analysis of the impact of the number of trainable parameters on LLM-wrapper's performances.

As part of our analysis, we computed the Pearson correlation between the P@1 scores and other variables from Table R.2:

Table R.3. Pearson Correlation with P@1 for various variables.

Table R.3 shows that the absolute number of fine-tuned parameters has a loose correlation with performance (pearson=0.08). This supports findings from Table R.2 showing the limited impact of the number of parameters, such as:

• Variations in LoRA's rank $r$ do not have a large effect on the P@1 score of Llama3 8B (cf Table R.2, rows 1 to 3).
• Llama3 8B (rank=64, 176M trainable parameters) and Mixtral 8×7B (rank=128, 113M trainable parameters) yield very similar P@1 scores (77.72 vs. 77.57), despite Llama3 8B fine-tuning a much higher percentage (2.15% vs. 0.24%) of parameters. The similarity of results could then be due to architectural differences, with Mixtral being a Mixture of Experts model.
• Gemma-2 9B (rank=128, 465M trainable params) underperforms compared to Llama3 8B (rank=64, 176M trainable params), even though its setting includes fine-tuning more than double the number of parameters.

These findings suggest that the architecture of the LLM and training specifics may have more influence on performances.

Furthermore, our analysis finds a strong correlation between LLM-wrapper's P@1 scores and the LLM's performances on reasoning tasks, with a Pearson correlation of 0.88 with HellaSwag. This indicates that LLM-wrapper's performance is mostly dependent on the original performance of the LLM on reasoning tasks.

In conclusion, the number of fine-tuned parameters has a small impact, but the LLM's architecture and other training factors play a more significant role regarding performance. We plan to extend this analysis with further studies on the impact of LoRA's rank values and of the number of fine-tuned parameters for other LLMs. We will include these findings in the revised paper.

Answer Part 1/3

We thank the reviewer for the insightful feedback and comments.

R-q6Km-W1:.The method requires constructing new training data to fine-tune the LLM, which partially defeats the purpose of a black-box approach that aims to minimize resource and data preparation efforts.

R-q6Km-A1: Indeed, LLM-wrapper requires REC data to fine-tune the LLM. However, we respectfully disagree that this contradicts the black-box paradigm. The black-box approach focuses on adapting models without access to their internal architecture or parameters, rather than on minimizing resource and data preparation efforts.

Nonetheless, LLM-wrapper remains resource-efficient and practical for several reasons:

• Straightforward data preparation: As described in Section 3.2, constructing training data simply involves converting VLM outputted boxes into textual coordinates, making the process lightweight.
• Efficient training: The training process is highly efficient, taking less than 2 hours on a single A100 GPU.
• Adaptability to new VLMs: As shown in Section 4.3 (L.408-419), a trained LLM-wrapper can be applied to different VLMs without additional fine-tuning.

Additionally, LLM-wrapper generalizes well to new datasets in a zero-shot manner, eliminating the need for creating specific training data for each new target dataset. For instance, in a new experiment, we evaluate LLM-wrapper on the HC-RefLoCo dataset, which includes longer and more complex referring expressions than RefCOCO/+/g. Since HC-RefLoCo lacks a training set, we transfer an LLM-wrapper trained on RefCOCOg to this dataset, following a protocol similar to Section 4.3. Our results demonstrate a significant +18.8 (resp. +18.4) P@1 improvement over zero-shot Florence-2 on the val (resp. test) split, showcasing LLM-wrapper's strong transferability and capacity for complex reasoning:

Table R.1. LLM-wrapper's dataset transfer performance, evaluated on the HC-RefLoCo benchmark (with prior fine-tuning on RefCOCOg), vs zero-shot Florence 2.

We also have new results, included in our response R-f3st-AQ1 to Reviewer f3st, demonstrating further successful transfers (RefCOCOg → RefCOCO and RefCOCOg → RefCOCO+), which align with these findings. Overall, these results show a strong LLM-wrapper’s transferability, significantly reducing the need for new training data for each target application.

We will clarify these points and include these results in the revised paper.

R-q6Km-W2: Comparisons with existing black-box adaptation methods or prompting strategies from the literature.

R-q6Km-A2: The related work section of our paper discusses existing black-box adaptation methods. However, these methods are unsuitable for our scenario for the following reasons:

• Non-applicability of prompting strategies: Methods, such as those by Ouali et al. (2023), Yu et al. (2023), and Liu et al. (2024b), focus on CLIP-based models that rely on prompt templates or require access to intermediate representations of the model. Our work does not use prompt templates as CLIP does, since our target VLMs (e.g., Grounding-DINO and Florence-2) perform open-vocabulary detection and do not require such inputs. Moreover, the need to access intermediate representations of the model does not qualify as a fully black-box setting.

• We focus on task adaptation, not domain shifts: Oh et al. (2023) modify image inputs for visual domain adaptation. However, this approach addresses domain gaps, not task adaptation like adapting a generalist VLM for REC.

Our method is specifically designed to adapt VLMs to the REC task, in a true black-box manner, without requiring intermediate representations, gradients, or multiple API calls. To the best of our knowledge, it is the first proposal aimed at this goal, and we could not identify any baseline method in the literature for direct comparison.

Dear Reviewer,

As promised, we have incorporated your feedback and added clarifications and results to the revised paper, where changes are marked in blue. Below, we point to the specific sections where the changes addressing your remarks and questions have been implemented.

R-q6Km-A1: Our answer has been integrated to the revised paper, with the new Table 6 and new paragraph at the end of Section 4.3 (Paragraph: "Transferring a trained LLM-wrapper to new datasets.").

R-q6Km-A3: Our answer has been integrated to the revised paper with a clearer Section 4.4 and the new Appendix B.1.

R-q6Km-A4: Our answer has been integrated to the revised paper with the new Appendix C, referred to in Section 4.1.3.

R-q6Km-A5: Our answer has been integrated to the revised paper with the new Appendix A, referred to in Section 4.2.

R-q6Km-A6: We have removed Section 3.4 and integrated its key content into the introduction. A discussion on the limitations has been added to the conclusion. Additional results have been included in Section 4.3, highlighting zero-shot dataset transfer, and in Appendix A, demonstrating black-box adaptation for already fine-tuned VLMs. Additional ablations on variables impacting performances are performed in Appendix B. An analysis on failure cases was added in Appendix C and additional qualitative examples are displayed in Appendix D.

R-q6Km-AQ1: Our answer has been integrated to the revised paper with a clearer Section 4.1.2

As the author-reviewer discussion period continues, we would greatly value your feedback on whether our responses have sufficiently addressed your concerns. If you have any further comments, please feel free to share them with us.

Best regards,

Thank you for your feedback and insights.

[Baselines and Pixel-grounding large multimodal models (LMMs)]

We appreciate your suggestion regarding pixel-grounding models, and we will cite these works in the revised paper. LLM-wrapper is a model-agnostic adaptation framework that emphasizes object-level reasoning, complementing the pixel-level reasoning of end-to-end systems like GLaMM or PixelLM. We will include additional baselines for upper-bound comparison, including a semi-black-box adaptation method and state-of-the-art pixel-grounding LMMs.

[More datasets]

We expanded our evaluation beyond the RefCOCO/+/g family and HC-RefLoCo by testing on the Talk2Car REC benchmark [1], which involves commands for driving scenes. This allowed us to assess generalization across a domain that differs from human-centric images and standard textual descriptions. Results are summarized below in Table R.13.

Table R.13. Evaluation on Talk2Car (REC) with LLM-wrapper vs. zero-shot and white-box fine-tuned Florence-2 (LLM-wrapper uses Llama3 8B).

Key findings:

• Zero-shot transfer: LLM-wrapper adapted on RefCOCOg improves P@1 by +9.46 (val) and +8.67 (test) compared to zero-shot Florence-2, showing generalization capabilities despite the domain gap. In contrast, white-box fine-tuning of Florence-2 on RefCOCO/+/g only slightly improves performances in the new domain with +1.89 (val) and +2.45 (test), indicating a lack of generalization.
• Targeted adaptation: LLM-wrapper achieves better results when adapted on Talk2Car (train), boosting P@1 by +25.54 (val) and +27.79 (test).

These results illustrate LLM-wrapper's strength in cross-domain generalization and its effectiveness as a black-box adaptation method. We will include these findings in the final paper.

[1] T. Deruyttere et al., Talk2Car: Taking Control of Your Self-Driving Car, EMNLP/IJCNLP 2019.