DOMINO: A Dual-System for Multi-step Visual Language Reasoning

Thank you for your comments and feedback. Please find below the answers to your questions:

W1: The author didn't discuss about the efficiency. How does the inference efficiency of DOMINO compare to the baseline method?

Although there may be multiple calls to the vision module in DOMINO, DOMINO is more efficient compared to the few-shot DePlot model due to two reasons: (1) The vision module in DOMINO only needs to generate the required information based on the image while the vision module in DePlot needs to generate the whole table, which can be arbitrarily long. (2) As a result of (1), DOMINO does not need to take as input the whole table sequence which would consume a large part of the context window in the LM of DePlot. The supervised end-to-end method is more efficient in inference than both DePlot and DOMINO. However, the method generates the answers directly which is prone to error and lacks interpretability for complex questions that require multi-step reasoning. We have included this discussion in Section 6 of the paper.

W2: The template seems relatively limited, more non-chartQA tasks are needed to confirm the potential of this method.

We agree that DOMINO can be applied to other VQA tasks. In this work, we focused on visual language reasoning as it requires accurate information extraction from an image in addition to strong numerical/logical reasoning skills to answer a question based on the extracted information, as also discussed in previous works such as (Liu et al., 2023). As such, we compiled the minimum number of templates that could generally be used across different chart/plot types.

Q1: What types of charts are included in ChartQA and PlotQA? I think adding relevant descriptions can help people have a more intuitive understanding of the capabilities of this method.

The chart types in ChartQA include bar, line and pie charts. PlotQA has bar, line and dot-line charts. Thanks for your suggestion. We have expanded the descriptions of the experimented datasets in Appendix A.1.1 to include this information.

Q2: Does the author consider the dual-system approach to be universally applicable? Can it replace other MLLM methods (such as BLIP2 [1], LLAVA [2]) and become a common solution for solving visual QA problems? For example, besides tasks like chartQA, can DOMINO also generalize to other tasks (such as VQA)?

Yes, DOMINO can be applied to other VQA tasks since the LM can always use language as the vehicle for reasoning and guide the vision module to obtain the required information. We focused on visual language reasoning in this work since it requires the LM to really rely on the visual information to answer a question about a chart and conduct multi-step reasoning while in other VQA tasks, the LM either can rely on language prior (but not necessarily the image) or does not need to conduct multi-step reasoning to answer a question.

Thank you for the comments and suggestions. Answers to your questions as well as clarifications to some of your comments below:

Novelty: The core idea of the paper is very similar to prior work, including “Visual Programming: Compositional Visual Reasoning Without Training, CVPR 2023”, “Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language, arXiv 2022” and “Look, Remember and Reason: Visual Reasoning with Grounded Rationales, ICML workshop 2023”.

DOMINO differs from these works in how the interactions between the language and vision modules are decided. Both “Socratic Models” and “Look, Remember and Reason” pre-define the interactions with templates, which do not generalize to questions with new reasoning processes. “Visual Programming” decides the whole interaction process by generating a program, but the LM does not get to react differently based on the intermediate results returned from the vision module, which limits the question types that could be solved. By contrast, DOMINO learns to compose the atomic operations on the fly based on both the question and the intermediate results from the vision module, which is more flexible.

It is unclear why the performance on PlotQA much worse compared to ChartQA. The paper mentions that PlotQA “is a synthetic dataset with template based and restricted types of questions”. But this should be easier to solve compared to ChartQA, as the proposed approach also follows templated reasoning steps. The paper should make it clear with ample qualitative examples why performance on PlotQA is lacking.

The charts in PlotQA involve extremely large numbers ranging from 0 to 3:50e+15, which pose a challenge to reasoning for both the vision and language models. The fully-supervised method leverages the sufficiently large training set (with over 20M examples) to learn the data bias, which is also pointed out in the DePlot paper (Liu et al., 2023).

Fairness of the comparison to Few-Shot DePlot versions of GPT-4 and LLaMA: The proposed DOMINO version of LLaMA has more information about the chart in question. Therefore, it is unclear if the evaluation is fair.

The few-shot DePlot method includes the entire table associated with the chart, whereas the DOMINO version of LLaMA does not include this information (as illustrated in Table 4) and as such does not necessarily include more information about the chart. We agree that a table is not an authentic representation of the chart, and this is one of the motivations for our work. DOMINO may include other types of information (e.g., color) and can also ask for additional information by interacting with the vision module.

Qualitative examples: The paper is lacking qualitative examples from PlotQA in the main paper. The main paper only includes a single qualitative example from ChartQA in the main paper. Examples of failure cases in Table 7 are hard to follow as the associated charts are not available. The paper should include more quantitative examples which are easier to follow. The format of “GPT-4 Technical Report, arXiv 2023” can serve as a guiding example.

Due to the page limit, we only include two examples in Table 4 in the main paper and attach the associated charts in Figure 4 in the Appendix. We have included the associated charts for the examples in Table 7 in the updated draft for your reference and we will move one PlotQA example to the main paper.

Additional datasets: The paper evaluates performance only on two datasets. There are also more challenging datasets available: SciCap (http://scicap.ai/). The SciCap uses real-world data and requires high-level reasoning along with low-level understanding of scientific figures. It would be an ideal testbed to evaluate the performance of the proposed approach.

Besides ChartQA and PlotQA, we also conducted experiments on two out-of-distribution datasets including DVQA and FigureQA as shown in Table 2 in the main paper. We focused on visual language reasoning in this work since it requires the LM to really rely on the visual information to answer a question about a chart and conduct multi-step reasoning while in other VQA tasks, the LM either can rely on language prior (but not necessarily the image) or does not need to conduct multi-step reasoning to answer a question. Thus, SciCap, which is actually a caption generation task, might not be an ideal testbed for our problem. Based on the examples shown in SciCap, some information mentioned in the target caption is not even grounded by the images. But we view SciCap as a potentially good resource for training our vision module to conduct the Describe operation. Thank you for your suggestion.

Thank you for your comments and feedback. Please find below the answers to your questions:

Q1. Why some of the results are not shown in Table 1?

The results for the baseline models are from their respective papers which are cited in the table. These works have not provided the results for all the datasets and/or dataset splits. We will clarify this in the paper. For GPT-4, we did not run the model on the much larger PlotQA evaluation sets due to high cost and usage limits as pointed out in the footnote on page 7.

Q2: Why is the correct answer for arithmetic in Table 7 is -50752953286.0?

We apologize for not including the original charts for each example in the paper and will include them in the updated version. The correct answer is from the PlotQA dataset. The question-answer pairs in PlotQA are generated by templates automatically and PlotQA considers the “difference” between A and B just as $A-B$ instead of the absolute difference. That is why the answer is negative in Table 7.

Q3: What is the evaluation prompt used when there is no Describe step?

In this case, the reasoning process starts directly with task decomposition and not the Describe step. For example, the first demonstration in the prompt would be:

Q: In which year the private health expenditure per person in Oman is 210.69?
A: Let’s extract the data of Oman.
The data is 183.88 in 2008, 233.80 in 2009, 210.69 in 2010, 195.26 in 2011, 196.32 in 2012,
154.21 in 2013, 153.22 in 2014.
The value 210.69 is in year 2010. So the answer is 2010.

We have added the prompt used for this case to Appendix A-2.

Q4: It would be great if the authors could discuss the applicability of the proposed method on other VQA tasks, e.g. CLEVR.

Our proposed method can be applied to other VQA tasks given an appropriate vision module. We focused on visual language reasoning to necessitate the multiple interactions between language and vision modules since other VQA tasks either do not require multi-step reasoning or can be accomplished by language prior alone. We will highlight the flexibility of our method in the final draft.

Q5: Is there any examples of the model failed at decomposing the problem?

There are examples of incorrect decomposition due to ambiguity in the question. In some cases, the model fails to request the only information required for answering although the model still answers correctly finally:

Q: How many people use daily?
A: Let's describe the figure.
The figure shows the data of: Value (blue). The x-axis shows: Weekly | Daily | Stopped using.
Let's extract the data of Value.
The data is 41.2 in Monthly, 37.8 in Weekly, 18.2 in Daily, 2.8 in Stopped using.
Among [41.2, 37.8, 18.2, 2.8], the value that is 18.2 is the number of people who use daily. So the answer is 18.2.

In other cases, the incorrect decomposition leads to invalid queries for the vision module. The vision module then gives the wrong intermediate result which finally leads to an incorrect answer.

Q: What's the percentage of people who don't believe it can impact us and don't know much about covid?
A: Let's describe the figure.
The figure shows the data of: Value (dark blue). The x-axis shows: Staying alert and to the ground | Don't believe it can impact us or | Don't know much about.
Let's extract the data of Value.
The data is 1.0 in Staying alert and the 2019.
The percentage of people who don't believe it can impact us and don't know much about covid is 1.0. So the answer is 1.0.

We will extend our error analysis in the revised draft to include such examples.