Measuring Vision-Language STEM Skills of Neural Models

We are glad that you like the paper! The responses to your key questions are below.

Comparison with other datasets: We agree that comparing STEM with other datasets will help us better understand the difficulty of different datasets. As the goal of our dataset is to measure the fundamental STEM understanding of neural models, we focus on the evaluation of different models on our dataset. We provide the comparison of statistics of different datasets in Figure 1 (a). This follows the typical benchmark paper design such as VQA [1], MMLU [2], IconQA [3]. However, this is a new angle for the community, and among our future investigations.

Limitation and impact: We understand the importance of limitations and potential impact. We have included the discussion in the ethics statement. We plan to expand the discussions to focus on the potential use of models after deep analysis of our dataset. For example, it will follow common limitations of the foundation models such as misuse, bias, and hallucinations [4, 5, 6, 7, 8].

Figure 9 clarification: Thank you for pointing this out. Although the general trend is that the model performance increases when models become larger, as the scaling law indicated [9], the model’s performance does not only depend on the size, other factors could also contribute (e.g., the length of the training). It is possible for a model with fewer parameters to outperform another model, which might explain why ViT-B/16 is outperformed by RN50 and RN101.

Human study: As described in Sec. 2.4 and Sec. 3.3, our exam scores are calculated based on millions of IXL elementary users [10]. This is one of the main advantages of using our dataset, as the human performances are simulated in a really large environment. The IXL SmartScore [10] serves as the main resource for our human performance comparison. The performance of the graduate students mainly serves as an expert performance, setting up an upper bound for human performance, while the IXL exam scores are from elementary students. We agree that the pool of test takes can be expanded to more people, but it seems a common approach in recent literature such as MATH [11]. We will consider adding more expert test takers in future work.

More dataset examples: Due to space limitations, we have included dataset examples for all skills of our dataset in Appendix C and a case study in Section 3.4.

Directions for improvement: Thanks for raising this question. We are also considering approaches to improve the performance on the dataset. One possible direction is to include more high-quality textbook data during training since STEM is knowledge-intensive. Another direction could be allowing models to access relevant knowledge resources during inference.

Ethics statement: As indicated in our statement, the collected data does not contain sensitive data and the copyright restrictively follows the original data sources. We will add more discussion based on your suggestion.

References:

[1] Antol, Stanislaw, et al. "Vqa: Visual question answering." Proceedings of the IEEE international conference on computer vision. 2015.

[2] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

[3] Lu, Pan, et al. "Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning." arXiv preprint arXiv:2110.13214 (2021).

[4] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[5] Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.

[6] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

[7] Anil, Rohan, et al. "Palm 2 technical report." arXiv preprint arXiv:2305.10403 (2023).

[8] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[9] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).

[10] Learning, I. X. L. "The impact of IXL Math and IXL ELA on student achievement in grades pre-K to 12 (pp. 1–27)." (2019).

[11] Hendrycks, Dan, et al. "Measuring mathematical problem solving with the math dataset." arXiv preprint arXiv:2103.03874 (2021).

Thanks for your positive feedback regarding our clarifications. It would be great if you could share your remaining concerns that have not been addressed yet. Despite the short time window left till the end of the discussion, we will try our best to address them. We appreciate your continued engagement and support.

We appreciate your endorsement of the work’s contributions! Please find our response to your questions below.

Other models: We agree that the evaluation of more models will be valuable. The main contribution of our paper is the creation of our dataset to evaluate the multimodal STEM understanding. We choose the foundation models like CLIP for a pilot study on this dataset. We anticipate performance improvements from the recent models and will add the discussion on the models you suggested in the new version.

Caption model: Thank you for the helpful suggestion and we believe that caption models will influence the performance to some extent. As discussed in Appendix B.5, it seems the textual input (e.g., prompt) does not impact the performance significantly. But we will investigate this in future work.

Performance of the finetuned model: We agree that the generalization after finetuning on STEM is important. Our main contribution is the creation of the STEM dataset. The dataset is designed for evaluating the vision-language STEM Skills of neural models. Therefore, the generalization might be out of the scope of the current paper. However, we plan to add the results on VQA (as you suggested) in the new version.

Per your suggestion, we added VQA results of the zero-shot CLIP model and models finetuned on each subject in Appendix B.7. The average increase of the finetuned models over the zero-shot setting is 1.2%. This shows finetuning on our dataset can potentially help improve the generalization of models.

We appreciate your identification of the importance of the work. Our response to your review is below.

Human evaluation clarification: We have included descriptions of the human evaluation e.g., setup, procedure, and metrics in Sec. 2.4 and Sec. 3.3. Overall, not only the average exam score of the tested models is below 60, but also their average accuracy is below 60 (Figure 1b). The results support our findings that model performance falls behind human performance on our dataset. Regarding the sources of the human scores, first, the exam scores of elementary students are sourced based on the IXL SmartScore [1] from grades 1 to 6 and we chose 90 as the reference. So if a skill belongs to grade 1, then 90 is for students at grade 1. According to the official guideline of SmartScore [1], a score higher than 90 is considered excellence for a mastered skill, and this standard also takes into account the grade level of the test takers, which are elementary students if we focus on grades 1 to 6. It means that a score of 90 is achievable and a reasonable goal for elementary students. Note that this score is still at the elementary level, and is considered excellent only for elementary students. Second, for expert-level human performance, we do not consider the grade level information so we test graduate students to get the upper bound performance of humans, which we refer to the expert-level performance. Since the SmartScore is only designed for the corresponding grades, we use graduate students’ test scores for the expert-level evaluation. We will add these clarifications in the new version.

The number of 54.7%: Thank you for pointing this out. We calculate the average exam scores of zero-shot CLIP from grades 1 to 6 which equals 40.8. This is 54.7% lower than the exam score of 90.

GPT-4: The main factor for excluding the GPT-4 results was cost. The approximate cost of evaluating GPT-4 on our benchmark is $6500. Although we don’t include GPT-4, we believe the main contribution of our paper is the multimodal dataset. Given the current results including GPT-3.5 Turbo, we expect the latest models like GPT-4 could improve the performance on our benchmark. This is indeed aligned with the value of contributing our dataset to the research community.

Error analysis: We did include the overall error analysis in Sec. 3.4 and more detailed analysis in the appendix (Appendix B.3) due to space limit. We also investigate the performance variance along many other aspects including questions with different skills (Table 8, Figure 23-28), different numbers of answers (Figure 17), questions with different lengths (Figure 18), different question types (Figure 19), different grades (Figure 21). In addition, we will add more error analysis including the top-5 error rate skills for science and math subjects in the Appendix and give an example for each skill.

References:

[1] Learning, I. X. L. "The impact of IXL Math and IXL ELA on student achievement in grades pre-K to 12 (pp. 1–27)." (2019).