Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models

• It is recommended to include more examples to illustrate the referring expressions of REF-L4, as the current version provides relatively few samples. Additionally, in Figure 1, the expression “It is placed on the floor” appears to be somewhat inaccurate. The current experiments seem insufficient to demonstrate that the dataset descriptions are entirely accurate, and there is limited information on the manual review process. Providing additional examples would help readers gain a fuller understanding of the dataset. 
• Is a longer “referring expression” necessarily better? It would be helpful if the authors could discuss this point. Given that the GPT-4V model generates lengthy descriptions to comprehensively cover object details, such extended expressions may not align with real-world applications, where shorter descriptions are often preferred. There is some concern that this dataset may not fully reflect model performance when working with shorter descriptions.

The weaknesses, from the most significant to the least significant:

• The contribution of the dataset seems incremental. This is my biggest concern. This work majorly presents a better version of RefCOCO datasets for evaluating REC models. The major differences includes larger dataset size, vocabulary size and longer descriptions. These improvements are good, but not essentially significant. The task form is the same; the scale is basically twice of previous works (for number of images, instances, etc.), which is not significant enough to me.
• The relationship between this work and LMMs are not strong enough. The core contribution is a dataset for REC models, with better quality and (not much) larger size. This is not strongly related to LMMs. The only significant relationship between this work and LMMs is the evaluation in Sec. 4. However, this is only an evaluation with experimental results listed and the authors did not provide much insight about LMMs.
• The title "in the era of LMMs" is surprising but the actual contribution is not satisfying. As discussed right above, the core contribution of a REC benchmark with better quality and (not much) larger size, is not very connected with "the era of LMMs". The evaluation with LMMs does not provide much information about how we should tackle REC "in the era of LMMs". I would admit that I am a little disappointed by the contrast between the title and the actual contribution.
• Some related works are not discussed, e.g., Omnilabel (ICCV 2023) and Described Object Detection (NeurIPS 2023). They also propose evaluation benchmarks with more instances and longer expressions. If these benchmarks are considered, then the major differences between this benchmark and previous ones, like scale and description lengths, seem to be invalid. The authors may consider discuss the comparisons with them.

1. The proposed dataset is still a traditional referring dataset. Since there are many advanced referring expression datasets are available, e.g. generalized referring segmentation dataset gRefCOCO, reasoning dataset LISA, sub-object dataset PhraseCut, the value of proposing a new traditional referring expression dataset is relatively limited.
2. Some statistics of the dataset are questionable, e.g. 2.1. The expansion in category count largely stems from the base dataset Object 365, rather than as an inherent contribution of the proposed dataset. 2.2. It appears that “instance area” is calculated in absolute pixel numbers, which may be less informative than the relative area of object pixels compared to the entire image, as the latter provides more practical insight.
3. Author stated that there exist a lot of error cases in test and validation set of previous datasets - why only focus on test and val set, how about training set? And what is your verification process? Table 2 is also confusing - from my understanding, the conclusion of this table is: removing wrong samples from testing set can increase the numeric performance of methods - which could be too obvious and expected.
4. From Introduction, the main motivation of the paper is to “existing REC benchmarks truly capture the comprehensive capabilities of LMMs”, and the solution is to proposed a dataset. However, in my opinion, all of the four main features of the new dataset (L134-L150) are not exclusive to LLMs but can also be well addressed by non-LLM methods. Authors should give more justifications about why these features can help your dataset to “capture the comprehensive capabilities of LMMs” but previous datasets cannot.
5. For example, longer expressions do not necessarily mean harder cases. Actually, as author instructed LLM to “describe various features such as color, size, position, and context”, the generated expressions may tend to contain more information and more details, which in contrast could make them easier. 5.1. Additionally, longer expressions may lack practical relevance. In real-world scenarios, people typically use brief, direct language to refer to objects rather than lengthy descriptions.
6. What is the process of “manually review” of the dataset? Since most of the expressions are generated by LLM, more details are required here to ensure the validity of the annotations.