Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation

We thank the reviewer for recognizing that the problem is well motivated. Below, we address the concerns:

W1: Size and Overfit

We would like to clarify that our dataset was originally intended for evaluation rather than training. Therefore, overfitting should not be a major concern, assuming standard research practices are followed and models are not trained on the test set.

W2: Gemini 2.5

At the time of our experiments, Gemini 2.5 was not available: its experimental release on March 25 came just days before our submission deadline on March 28. Additionally, the model declines to generate referring expressions when human faces are present in the image, which makes it hard to compare with other models.

W3: Overclaiming

While fine-tuning may mitigate some surface-level issues like verbosity, our results point to a deeper, more structural limitation. Specifically, we observed an average multiple-matches rate of 35.82% and a no-match rate of 14.06%, indicating that models often fail to disambiguate between similar referents, which is a core requirement of pragmatic competence. This failure is not easily addressed by a few iterations of fine-tuning, as it reflects a representational and inferential gap rather than a stylistic one. From the author’s prior experience pre-training VLMs, we found that addressing such failures typically requires more than just lightweight adjustments. Our aim is not to dismiss current VLMs, but to highlight the need to revisit the REG task itself, which has been underexplored despite its foundational role in evaluating referential language understanding.

W1-1: Prompting Diversity

The purpose of randomly sampling one prompt variant per image-object pair is to introduce prompt diversity and reduce potential prompt-specific bias. While each test instance receives a different prompt, the setup remains controlled because we use the same prompt for a given image-object pair across all models, ensuring fair and consistent comparisons.

W1-2: Prompting Length

While the prompts are simple, each one clearly instructs the model to generate unambiguous expressions that uniquely identify the target object. Human subjects receive similar instructions to ensure fairness. Most models did recognize the task and produced relevant descriptions, but often failed at ensuring uniqueness, despite the prompt explicitly requiring it. We also found that longer prompts slightly degraded performance in smaller models, which motivated our choice of concise yet directive phrasing to avoid introducing unnecessary complexity.

W2-1: Redundancy and Length

Yes, we have further annotated the redundant parts in the expressions and found an average redundancy rate of 68.93% (based only on expressions evaluated as successful). Please see the results below.

W2-2: Human Annotations

As the reviewer noted, the task may not appear very difficult. However, we selected a subset of examples for annotation by 3 annotators and observed a Cohen’s kappa of κ = 0.781, indicating substantial agreement and validating the clarity and consistency of the task.

Q1-2: Failure Cases

We discussed failure cases in Section 5, including a representative example in Figure 8. We found that VLMs often fail to capture the most salient features, which are typically spatial expressions that humans rely on in language. Instead, the models tend to produce verbose combinations of visual attributes that are less preferred. On average, we observed a multiple-match rate of 35.82% and a no-match rate of 14.06%. Detailed results are provided below.

Q3-5: CIDEr Score and Typos

This is not a typo. The score was computed using the official pycocoevalcap CIDEr implementation, with 3 human-written expressions as references per instance. Human-spoken expressions were highly consistent with the written ones, it is expected that the final score would exceed 100 in this case. We will revise other typos in the future version of the paper.

W1-1: Dataset Leakage

We would like to clarify that our images are drawn from the validation split of the OpenImages segmentation task, as noted in Section 3.2. We trust VLM developers follow standard research practices and avoid using validation data for training or benchmark construction. In contrast, datasets like RefCOCO are built on the MSCOCO training set, which is known to suffer from data leakage. Our dataset design intentionally avoids this issue.

W1-2: Single-Object Images

Regarding the image selection criteria, we further include single-object images (492 images) in addition to the multi-object subset (995 images). These were evaluated in our extended experiments. Interestingly, we observed a sharp performance drop from an average accuracy of 69.45% on single-object images to 22.36% on images with multiple objects from the same category, highlighting the specific challenges posed by referential ambiguity rather than dataset composition alone.

W2: Annotation

Our pipeline is largely aligned with the ReferItGame setup, with the main difference being that we use red bounding boxes instead of red outlines to mark target objects (see Figure 9). Importantly, we focus on collecting unambiguous referring expressions that uniquely identify a target object in each image. This contrasts with region captioning, which permits general descriptions that may not even mention the object explicitly.

W3: Chain of Thought

As the reviewer noted, “multi-modal reasoning models may not be available at the submission time.” We emphasize our focus is to probe intuitive pragmatics in VLMs, as generating referring expressions is typically a fast, effortless process for humans. That said, we also include additional experiments where GPT-4o is explicitly prompted with CoT.

W4: Not Surprising Results

Our conclusions go well beyond the known limitations of existing evaluation metrics or the observation that VLMs tend to be verbose. As detailed in Section 5.1, we present human studies revealing a misalignment in pragmatic intuition between VLMs and humans. More importantly, our error analysis (see Table below) shows a 35.82% multiple-matches rate and a 14.06% no-match rate, demonstrating that models frequently fail to disambiguate between similar referents, which is a failure that reflects a lack of pragmatic competence, not just verbosity. The goal of our work is to recenter the REG task on its original pragmatic grounding, and to critically assess whether VLMs can handle this core linguistic challenge, which we argue has been overlooked in current evaluation practices.

We thank the reviewer for recognizing our thorough and well-cited analysis of pragmatic competence in VLMs through a carefully designed experimental suite and novel dataset, yielding insightful findings on both model limitations and evaluation metrics. Below, we address the concerns:

Note that the results presented in our original submission were based solely on multi-object images (approximately 1,000). In subsequent experiments, we included an additional set of single-object images (approximately 500), and all updated results now reflect performance on both single- and multi-object images.

W1: Additional Prompting

We emphasize our focus on intuitive pragmatics, as generating referring expressions is typically a fast, effortless process for humans. That said, we also include additional experiments where GPT-4o is explicitly prompted to follow Gricean principles.

W2: Performance Gap in Non-COCO Classes

We have conducted such further analysis and observed performance differences across nearly all models between COCO classes and non-COCO classes. Please see the detailed results below:

W3: Error Analysis

If the annotator chooses the option that they cannot tell where the object is or there are multiple matches, the score will be 0. We observed an average multiple-matches rate of 35.82% and a no-match rate of 14.06%. We also manually annotated the redundant information in the referring expressions and found a redundancy rate of 68.93%. Detailed results are listed below.

W4: Presentations

We apologize for the oversight. We will correct these errors in our final version.

Breakdown Analysis of Main Results

We further provide a detailed breakdown of the main results.

Column Explanations:

• REC: The accuracy of the REC model (CogVLM-Grounding). We use a listener-based evaluation where referential success is defined by whether the model's predicted bounding box has an IoU > 0.5 with the ground truth.
• Human: The accuracy of human listeners.
• Agree: The agreement between the two listeners—the REC model and humans.
• Wrong%: The proportion of failed human guesses, where the clicked location falls outside the target object’s segmentation mask.
• Multi.%: The proportion of cases where the human listener indicated that the expression was ambiguous, i.e., it matched multiple objects in the image.
• No-Mat%: The proportion of cases where the human listener reported that no object matched the expression.
• COCO / No-COCO: Accuracy on objects that are part of COCO classes and those that are outside of it, respectively.
• ΔAcc (Class): The drop in accuracy between COCO and non-COCO objects.
• Coocc. / No-Coocc.: Accuracy on images containing multiple objects of the same class (co-occurring) and those containing only one instance of the target class.
• ΔAcc (Co-occurrence): The drop in accuracy between co-occurring and non-co-occurring cases.