FAMMA: A Benchmark for Financial Multilingual Multimodal Question Answering

1. The paper does not address the possibility of data contamination, considering that some questions may have been sourced from online resources that might appear in the LLM's pretraining corpus.
2. The authors could have evaluated additional inference-time methods, such as self-consistency and program-of-thoughts.
3. The number of questions in Chinese and French is relatively small, which limits the interpretability of results for these languages. Using GPT-4o to translate English questions into Chinese and French (with human verification) can augment the size of multilingual data.
4. The paper lacks an ablation study on the importance of multimodal input. It would be beneficial to show the performance of text-only input and examine the impact of displaying tables as images versus converting them into text.

1. The relatively low human performance (L270), as stated in the paper, raises concerns about the quality of the dataset. What are the reasons for this relatively low human performance? Are the test-takers not familiar with all of the subdomains? Is there any ambiguity in the questions? How might this affect the label quality, which is also determined by human annotators? What is the inter-annotator agreement rate when "each question is cross-verified by 2-3 annotators" (L218)?

2. The results suggest that Chain-of-Thought (CoT) prompting outperforms humans. According to Table 5, CoT corrects 44 errors (out of 100). Does this imply that it would score approximately 100 - [(100 - 42.85) * (1 - 0.44)] = 68% if run on the entire test set, surpassing the human performance of 56% from Table 2? If my understanding is correct, this would undermine the claim that FAMMA poses a significant challenge for LLMs, and would put the benefits of this new dataset into question. In any case, I recommend reporting the full performance of RAG and CoT on the entire test set for comparison, instead of running them on just a subset of the errors.

3. Additionally, the details of the RAG setup are confusing. What embedding model is being used for retrieval, and what is the prompt used for generation? Is OpenAI’s commercial RAG offering being used? If so, there are more performant retrieval models in the literature, which may close the gap between RAG and CoT.

Other Minor Issues:

1. Short-form answers make automatic evaluation easier. However, what if a model arrives at the right answer for the wrong reasons, especially in multiple-choice questions? It might be worth considering requiring models to show their work for some of the questions.

2. "o1-style reasoning" (line 476) should cite [1] as well. The implementation seems to be a modified version of Chain-of-Thought as it was originally proposed in [1], not an "internalized" Chain-of-Thought as in OpenAI’s o1.

3. Please reduce the empty space in tables.

Reference:

1. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

1. Data Quality Assurance: The paper lacks quantitative analysis on data quality control, which raises concerns about the reliability and consistency of the benchmark. A more detailed breakdown of the two-stage data cleaning process, including quantitative results from the initial data cleaning phase and the difficulty-level annotation phase, would improve clarity on data preparation and quality control.

2. Copyright and Usage Rights: A significant concern is potential copyright issues with the data sources, as all data appears to be derived from textbooks, exams, and other educational materials that may have restricted usage for research. The paper should explicitly address these concerns by providing evidence of permissions or free-use provisions to ensure transparency and compliance with copyright regulations.

3. Model Evaluation Scope: The evaluation includes a limited set of multimodal LLMs, mainly focusing on a few proprietary models and open-source model. Expanding the model set to include a broader range of multimodal LLMs would offer a more comprehensive assessment of FAMMA’s benchmarking capabilities and further validate the benchmark’s utility across diverse model architectures.