VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text

(continue: answer to Q1)

• Transferability validation on other benchmarks: VCR-wiki serves as targeted training data to improve the model's text-image alignment and text recognition capabilities. Although VCR-wiki does not feature variability in style, fonts, and other factors, we believe it is a strong enhancement based on the VLMs' existing text recognition capabilities. Below, we conducted additional experiments to evaluate how well models fine-tuned on VCR-Wiki generalize to other tasks, which has been included in Appendix D in the original submission. This includes tasks with diverse real-world scenarios. Specifically, we fine-tuned CogVLM2 and CogVLM2-Chinese, two of the "finetune-able" VLMs in Section 4.4, on VCR-wiki (English and Chinese) and OKVQA training sets, then assessed their performance across multiple benchmarks. The results are summarized in the table below, where bold entries denote the highest scores for a given base model:

These results demonstrate that models fine-tuned on VCR-wiki not only achieve competitive or superior performance compared to the base model and OKVQA fine-tuning on standard benchmarks but also show significant gains in tasks like OCR-Bench, which feature diverse real-world scenarios. This suggests that VCR-wiki fine-tuning enhances models' ability to generalize to varied text recognition and multimodal reasoning tasks.

• Flexibility of VCR as a framework: The VCR task framework is inherently customizable, allowing researchers to tailor datasets to domain-specific challenges. For example, our open-source codebase enables users to vary data sources, occlusion patterns, fonts, and environmental settings. The current VCR-wiki dataset represents a foundational instantiation of this framework, demonstrating that even simple configurations that mimic the format of an image with a caption still pose significant challenges to state-of-the-art VLMs. This flexibility supports future extensions to more nuanced real-world scenarios. In this paper, as a first step, we focus on providing more insights through a series of controllable experiments to extract the essence behind models' success or failure patterns, which would benefit from a simple yet challenging configuration.

Q2. Robustness of Evaluation Metrics (Concern from Weaknesses Q2)

We appreciate your suggestion regarding the evaluation strategies and offer the following clarifications:

• Characteristics of VCR: Unlike freeform generation tasks such as Masked Language Modeling (MLM), VCR is an objective text recovery task with a single ground truth, determined by partially exposed text in images. This makes evaluation similar to tasks like multiple-choice questions or math problems, where exact-match accuracy is a reliable metric. In this context, generation outputs that deviate even slightly from the ground truth (e.g., incorrect letters) are considered unfaithful to the input, warranting stricter evaluation. This characteristic also allows us to avoid the drawbacks of freeform generation tasks (like MLM or QA tasks), where a proper automatic evaluation is hard to acquire [1]. The accurate and unambiguous evaluation is a strength of the task of VCR.

• Choice of Metrics: To account for partial correctness, we supplement Exact Match (EM) with the Jaccard Index, a cost-effective and widely used metric. This approach leverages the structure of our task, where covered texts are 5-grams, making partial matching straightforward to compute and interpret. The combination of Exact Match and Jaccard Index ensures a balance between precision and nuance in evaluation.

• Inappropriateness of GPT-Based Metrics: While GPT-based metrics can approximate human judgment for freeform generation tasks, they are less suitable for VCR. The high evaluation cost and variability of GPT-based scoring are unnecessary for tasks with a well-defined ground truth, where automatic metrics like Exact Match offer more reliable and consistent evaluation. Therefore, we believe our current approach is robust and aligns well with the nature of the VCR task.

[1] Wang, Cunxiang, et al. Evaluating Open-QA Evaluation. In NeurIPS 2023.

We hope these responses address your concerns and provide additional clarity on the strengths and flexibility of our proposed task and dataset. Thank you again for your valuable feedback, which has helped us further refine and strengthen our submission.

Q1. Generalizability of VCR Models to Real-World Scenarios (Concerns from Weaknesses Q1, Q3, and Questions Q1)

To address this question, we extend our discussion in the following three aspects:

• Real-world applicability of models finetuned on VCR: To evaluate whether fine-tuning on the VCR dataset enhances real-world recognition of partially occluded text, we assess the performance of three models used in Figure 5: MiniCPM-V2.5 8B, CogVLM2 19B, and Qwen2-VL 7B. Since no publicly available datasets for real-world occluded text recognition currently exist, we manually curated a small dataset comprising ten photographs of occluded text for a case study, which is sourced from street maps and collected images. The comparison of pre- and post-finetuning performance is provided below. For clarity, text predictions directly relevant to the results were manually extracted from the model outputs. We define "Incorrect" as not having overlap with the ground truth, and "Partially Correct" as having overlap but not completely correct.

The evaluation of MiniCPM-V2.5 8B, CogVLM2 19B, and Qwen2-VL 7B models before and after fine-tuning on the VCR dataset demonstrates significant improvements in real-world recognition of partially occluded text.

These enhancements indicate that fine-tuning on VCR-wiki markedly boosts the models' accuracy and reliability in interpreting occluded text in real-world scenarios. However, it is important to note that this analysis is based on a case study with only ten examples. Despite the limited sample size, the consistent performance gains across different architectures highlight the potential effectiveness of VCR-based fine-tuning. Future evaluations with larger and more diverse datasets are necessary to further validate and generalize these promising findings, ensuring robust real-world applicability of the fine-tuned models in dynamic and challenging text recognition tasks. We have added this real-world case study in Appendix E in the revised paper.

Thank you for your detailed and thoughtful review. We greatly appreciate your insights and constructive feedback. Below, we address the specific concerns you raised to clarify and strengthen our submission.

Thank you for your thoughtful review and for raising these important points. We have addressed each concern in detail below.

Q1. Regarding the statement: "the architectural design of CogVLM2 is well-suited for VCR"

We appreciate your observation and agree that attributing a model's success to specific design choices requires more comprehensive and well-controlled experiments. Based on your feedback, we will revise this statement in the paper to ensure it is more balanced and reflective of the current evidence. We aim to avoid overstatements and have clarified that this claim is hypothesis-driven and warrants further empirical validation in the revised Section 4.4. Please refer to the purple text in the revision.

Q2. Certain important details about the dataset are missing from the paper.

We address your concerns about the dataset design and experimental setup in the following points:

Q2.1 how does the benchmark ensure that images and captions in the dataset have not been seen during VLM pre-training or SFT?

We acknowledge that Wikipedia is commonly used in the pretraining data for many state-of-the-art VLMs. This, in fact, was one of the motivating factors for building the VCR dataset from Wikipedia. As evidenced in Table 2, even though many VLMs have likely been exposed to similar data during pretraining, their performance on the VCR task is subpar. Notably, models struggle significantly with the Chinese VCR-wiki benchmarks.

This substantial performance gap highlights that state-of-the-art VLMs fail on VCR-wiki not due to unfamiliarity with the underlying text, but because they lack the necessary capabilities to effectively perform the VCR task. Furthermore, we selected Wikipedia due to its general licensing, which enables us to fully open-source the dataset and align with the principles of open science. We have included this analysis in the revised Section 4.4. Please refer to the purple text in the revision.

Q2.2 The difference between easy and hard cases is vague.

To clarify, we do not independently select samples for the easy and hard settings. Instead, both settings share the exact same examples, data splits, and covered text spans. The difference lies solely in the degree of pixel masking applied: the hard setting obscures more pixels compared to the easy setting. This design ensures that the easy and hard settings are directly comparable, as they differ only in the pixel-level masking applied to each instance.

For example, if the word "area" is partially masked in the easy setting, the same word will appear with greater masking in the corresponding hard setting. Thus, the easy setting is always relatively easier than the hard setting for a given example. Figure 3 ("Step 3: Create TEI") illustrates this process. We have changed our narrative to be clearer in the revised Section 3. Please refer to the purple text in the revision.

Q2.3 Would it be possible to manually curate easy samples based on word combinations?

Thank you for this insightful suggestion! As stated above, the easy and hard examples are created from the same image-text pairs, with the difference being the proportion of pixels masked. While we currently do not select examples based on specific word combinations or letter characteristics, your idea to consider letter heights or other semantic attributes is intriguing and represents a promising direction for future work.

We have integrated your suggestion into our codebase, adding support for whitelisting and blacklisting specific words for masking, along with fine-grained masking based on letter heights. Future iterations will explore more nuanced strategies leveraging letter height or other word-level attributes. The features and flexibility of our VCR codebase are highlighted in the revised Section 3.

We hope these clarifications and updates address your concerns and strengthen the contribution of our work. Thank you again for your valuable feedback.

Q2. Why did we choose 5-grams for masking?

The decision to use 5-grams for masking was informed by both linguistic and empirical considerations:

• Linguistic Analysis: We would like the covered text to contain a relatively longer text span that would only capture a local grammatical structure. A 5-gram mask typically captures meaningful grammatical structures, such as noun or verb phrases with determiners, adjectives, prepositional phrases, basic clauses, and subordinating conjunctions. This makes the task sufficiently challenging without rendering it infeasible.

• Empirical Observations: Longer masking spans (e.g., 6-grams) often resulted in instances with limited viable options for masking, reducing the size of the dataset, while shorter spans (e.g., 3-grams) made the task significantly easier according to our observations. Masking at the 5-gram level struck a balance between difficulty and task feasibility.

We emphasize that VCR is a flexible task framework, allowing researchers to customize masking configurations based on specific domains or datasets. We include the options to mask out complete sentences and certain PoS taggings in our codebase. We emphasize the reason behind masking 5-grams and the flexibility of our codebase in the revised version of the paper. Please refer to the purple text added in Section 3.

Q3. How did we determine the visibility threshold of the 'easy' setting?

The visibility threshold in the "easy" setting was chosen to ensure that the task remained easy for human participants while becoming nearly impossible for widely used OCR engines like Tesseract and MMOCR. This threshold was empirically validated, as noted in Section 3 of the paper and calibrated to create a meaningful challenge for current VLMs.

Q4. Why not finetune the models using a larger subset of VCR-wiki?

Our fine-tuning experiments aim to examine models’ "finetune-ability" (i.e., whether the model is able to achieve performance gains after finetuning on VCR), as shown in Figure 5 of the paper. Due to resource constraints, we selected a relatively small but representative subset (16K samples). Despite this limitation, the results successfully highlight different model behaviors, which is the aim of the finetuning experiments:

• Models with minimal improvement after fine-tuning (e.g., MiniCPM-V2.5).
• Models performing well with or without fine-tuning (e.g., Qwen2-VL).
• Models significantly benefiting from fine-tuning (e.g., CogVLM2).

Further exploring the relationship between finetune-ability and model architecture is left for future work.

Q5. Specifying the number of training epochs would be valuable.

Each model was fine-tuned for 1 epoch. This detail, already included in Section 4.1 (Line 271), will be further emphasized in the revised version of the paper for clarity.

Q6. Is there any prompt engineering applied?

We tested 15 task description prompts generated by GPT-4o in our preliminary experiments, but we observed similar results across them. Therefore, we chose to use a single, generic and standardized prompt for simplicity and consistency, avoiding per-model optimization for each VLM. This decision reflects three considerations:

• Optimized prompts for each model would increase experimental costs significantly.
• A universal prompt aligns with our goal of creating VCR-wiki as a standardized and accessible testbed. A different prompt for each model or a prompt optimized specifically for a single model would make the test more costly or biased.
• We call for future works that develop improved prompt-based VLM reasoning frameworks and tailored prompts to enhance models' reasoning capabilities on the VCR task.

Q7. For the performance degradation with VI, did the authors attempt prompts specifically guiding the model to utilize visual cues?

As mentioned in the response for Q6, we tested 15 different prompts in our preliminary experiments. In our tested prompts, we also included prompts that guide the model to "please utilize the visual cues in the image", which also results in a similar performance for the majority of the models. However, the VI setting does not have a visual cue. To ensure comparability between VI+TEI and VI settings, we kept prompts consistent across both settings to not include the "utilize the visual cues" expression. This ensures that the observed performance differences are only attributable to the image input, not prompt design. However, designing prompts or reasoning frameworks explicitly for better visual cue utilization is an interesting avenue for future work.

Q8. Figures are mistakenly linked.

Thank you for catching this oversight. We have corrected the figure references in the revised version.

We appreciate your detailed feedback and believe that addressing these points strengthens the paper. Thank you for your thoughtful review and consideration.

Thank you for raising these important questions. We've provided our responses below.

Q1. How do models fine-tuned on VCR-Wiki perform on other benchmarks?

While VCR serves primarily as a benchmark, the training set is designed to enhance a model's text-image alignment capabilities for partially occluded text. To evaluate its transferability, we conducted additional experiments where we fine-tuned three VLMs—MiniCPM-V2.5, CogVLM2, and CogVLM2-Chinese—on VCR-wiki-en, VCR-wiki-zh, and OKVQA training sets. We then assessed their performance on other tasks. All fine-tuned models follow the same fine-tuning procedure. The table below summarizes the results. The bold scores are the best among the same model but different finetuning datasets. Please note that this experiment has been included in Appendix D in the original submission.

The results demonstrate that fine-tuning on VCR-Wiki generally achieves on-par or superior performance compared to fine-tuning on OKVQA or no fine-tuning, highlighting the transferability of the VCR-Wiki dataset.

Interestingly, fine-tuning CogVLM2 on VCR-Wiki-En-Hard notably improved its performance on VCR-Wiki-Zh-Easy, raising the score from 9.15 to 42.55. Similarly, fine-tuning CogVLM2-Chinese on VCR-Wiki-Zh-Hard enhanced its performance on VCR-Wiki-En benchmarks. These results suggest that models trained on VCR-Wiki can effectively learn to utilize pixel-level hints in text-embedded images (TEI) and apply this knowledge across languages, further validating the dataset's utility.

I thank the authors for the response and clarifications. I will maintain my initial positive score for this work.