Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach

releasing data and code: Although the data will not be released, I think the reproducibility of experiments are supported by the experiments with 5 random seeds and also using open-sourced data such as LAION. The community will benefit from reproducing such large-scale entity recognition image data by using LAION and multimodal LLM such as LLaVA.

We intend to release LAION annotations. We also intend to release the implementation of the open source Gemma variant presented in the “General response” of this rebuttal.

I wonder how the REW benchmark generalize to OOD entities of OVEN (landmarks, animals, plants, car, aircraft, etc). Can you show some examples of the model predictions on visual entity categories not covered by the OVEN benchmark such as artwork, shopping items, etc.

We show results for shopping items, such as chairs, lamps and bicycles from the Stanford Online Products dataset with our model in the rebuttal PDF. We can observe that our model is able to predict the correct fine-grained classes, as the corresponding entities are contained in the 6M entity list of Wikipedia, which is much more extensive than the OVEN entities.

It seems using rational data improves query split in OVEN. I wonder if the authors have any intuition about this.

We believe that the rationales help clarify the connections between entities and visual attributes. Furthermore, providing rationales as additional supervision improves the model's language processing capabilities, which is important for deciphering complex text queries. These improvements are particularly valuable for VQA tasks, where an understanding of both visual and linguistic information is essential for accurate question interpretation and answer generation.

In L241, authors state that the trained model is better than MLLM itself which is used to produce filtered datasets. However, I did not see the results for Gemini Pro (in L226 they mention that) in Table 1.

This result is shown in the inline text in Section 4.2: “Finally, we report the zero-shot performance of the multimodal LLM on OVEN: it reaches 13.3 HM top-1 in the entity split and 29.5 HM top-1 in the query split”. Note that this is the zero-shot Gemini Pro model, which does not always generate entities that are sufficiently fine-grained. On the other hand, our fine-tuned GiT-Large model achieves 29.6 top-1 in entity split and 30.9 top-1 in the query split. We will make this more clear in the camera ready.

Lack of references: In L142 and L170, authors state that constrained decoding is used to guarantee the successful grounding of entity rationale generation. Works like GMEL [1] and AutoVER [2] should be the first to introduce such techniques into entity generation in multimodal contexts.

We thank the reviewer for bringing these papers to our attention, we will definitely discuss them in the camera ready. Note, however, that they differ significantly, as they use retrieval to augment their models which is complementary to our approach.

Minor issue: This work mainly proposes a data-centric solution to GER [3], and I assume the base model architecture of GER and this work stays the same with a 0.4B decoder-only model and a CLIP visual encoder. In Table 1, why is GiT-Large with Entity-WebLI training getting a better performance compared to GER-ALD with the same dataset?

The GER paper [6] does not utilize constrained decoding. We opted to apply constrained decoding to both the "GiT-Large with Entity-WebLI" baseline and our model ("GiT-Large with REW-47M") as it yielded a significant performance boost (+5 points) for these captioning models, while it doesn't change the performance of the GER models.

The author's rebuttal was received. Thanks for clarifying the issues.

Considering all the merits and weaknesses, I decided to update my score (6 -> 7) as it seems to be a fair evaluation of this work.

The main contribution of the paper is the creation of an enriched version of the Entity Web-Li dataset, named REW, which however is not released to the public together with the paper (this is mentioned in the Checklist). I find this a major weakness of this project, given the complexity of the conducted work (as clearly described in the paper and the appendix) and would strongly encourage the authors to reconsider this decision.

We thank the reviewer for their comment. We would like to point out that we also conduct experiments with publicly available LAION dataset (Table 5). Our intention is to release the annotations for the LAION dataset. Furthermore, our results are reproducible with the public Gemini-API, and now also with the open source models described in the “General response” of the rebuttal.

Could you clarify how the avg. relative improvement in table 3 is computed?

The relative improvement for each dataset is (new_value - old_value) / old_value. For each row, we compute the relative improvement separately for each of the six datasets in the table and then compute their average.

We thank the reviewer for starting the discussion.

We would like to emphasize that we have already demonstrated the performance of our method using open-source models (PaliGemma+Gemma) in Tables R1, R2, and R3 of the "General Response".

Open-source VLMs like PaliGemma and LlaVa do not support long contexts. To overcome this complexity, as the reviewer rightly points out, we employ a two-stage approach. First, we utilize PaliGemma (similar to LlaVa) to process the visual input and generate a detailed caption. Then, we feed the output from PaliGemma, along with with our prompt (Section A.3.1), to a more powerful LLM, Gemma 27B, chosen for its longer context support and stronger reasoning abilities. This two-stage approach allows us to effectively use our methodology with open-source models.

We can see in Table R1 that in all cases our method gives substantial improvements over the prior work GER-ALD[6] and GIT-Large trained on Webli-Entity[6]. We will include the Gemma results with the full 47M dataset in the camera ready.

We hope that this experiment and explanation addresses the reviewer's concerns. We are happy to elaborate further or discuss any other questions the reviewer might have.

It is regrettable to rely on a private model (Gemini-pro) to build the main contribution of the paper, which is the training dataset. There's no guarantee that this model will be stable over time, such that the proposed method (to build the dataset) may not be reproducible in a couple of months. This issue would nevertheless be partially mitigated if the dataset itself is released, although it still limits the interest of the method itself (e.g maybe another prompt would be required to get comparable results, or another multi-task learning to train the T2I model...).

We agree and appreciate the reviewer's valuable feedback regarding open-source accessibility. To address this, we adapted our method to leverage the open-source Gemma 27B model, renowned for its strong reasoning capabilities and long context support. Please see the “General response” of this rebuttal for the results with PaliGemma and Gemma models. We will release the implementation of this variant.

The authors argue (lines 239-243) that "the zero-shot performance of the multimodal LLM on OVEN" is much lower than those obtained when it is used alone thus it "suggest[s] that we are not merely distilling from the considered multimodal LLM". However, it is hard to believe that the gain in performances over [6] is not essentially due to the use of an additive (large, complex and trained on many data) LLM. Beyond the fact the namely LLM is opaque (see above) the contribution seems to mainly consist of a RAG-like approach to filter and correct the entities associated with the images by [6], relying on a black box which makes the process difficult to actually understand. So yes, "it works", but the scientific contribution nevertheless seems limited in that sense.

We would like to clarify the following points:

1. We compare the results of Gemini to our model (13.3 versus 29.6 top-1 HM in the entity split). The gain is due to the fact that Gemini is a general purpose model, whereas our model is trained on a Gemini curated dataset for the specific task of entity recognition. So Gemini doesn’t work out of the box, but is able to curate the labels when used properly (more on this in point 3 below).
2. We train the same GiT-Large model on two versions of the WebLI datasets, i.e curated and uncurated. We can see that the curation improves the results significantly (20.1 versus 29.6).
3. Our scientific contributions in this work is about how to use LLMs for the curating data for web-scale entity-recognition. As evidenced by the results presented in Table 4, our findings highlight two key contributions for the community:
   • LLMs excel in verification over direct prediction: We show that a naive application of LLMs for direct prediction (Table 4, Row 1) can actually hinder performance compared to [6]. This is because they tend to generate generic entities, rather than very fine-grained ones required for this task. However, utilizing LLMs for verification purposes (Table 4, Row 3) leads to a significant performance boost.
   • External knowledge enhances LLM reasoning for fine-grained entities: We demonstrate that augmenting LLMs with external knowledge sources like Wikipedia (Table 4, Row 4) further improves their ability to reason about and identify fine-grained entities.

To also emphasize the generalizability of our approach beyond a specific LLM, we've included additional experiments with an open-sourced model in the “General response” of this rebuttal. This highlights the adaptability of our methodology across different LLM architectures, even when their internal workings are opaque.

Is it possible to estimate the performance boost/drop with another external "multimodal LLM"?

We report the accuracy of our method with other open source models (PaliGemma + Gemma) in Table R3 of the “General response”. Even though there is a small drop in performance when using open source models, our method still outperforms the prior work.

The authors report that the code is not released (Checklist "Experimental Result Reproducibility") but do they at least plan to release the new "mapping" manually curated and used for fine-grained classification?

Yes, the mapping in the appendix section A.5 of the paper will be released.