StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding

Q2: What is the relation and difference with the current popular benchmark (e.g., FigureQA)?

A2: Thank the reviewer very much for this valuable comment. FigureQA is also a benchmark for chart understanding task. However, there are still some differences between FigureQA and the mentioned datasets (ChartQA, PlotQA and Chart2Text):

• Overall, the images in FigureQA dataset are all simulated samples based on manual rules, while ChartQA and Chart2Text are sampled from real world.
• For perception stage, ChartQA and Chart2Text provide annotations in CSV format for CIE task. But for FigureQA and PlotQA, we need to preprocess relatively complex retrieval from the raw annotation (JSON file).
• For reasoning stage, the questions in FigureQA dataset are based on fifteen sentence patterns (e.g. "Is X the minimum?", "Does X have the lowest value?"). And all the answer pairs are "yes" or "no". In contrast, the mentioned datasets, such as ChartQA, PlotQA, use human-authored QA annotations. Besides, Chart2Text is designed for chart summarization task.

Furthermore, according to the reviewer's comments, we have supplemented additional experiments for QA task on FigureQA. The output of Matcha is still HTML format (e.g. '('title | title <0x0a> yaxis_label | xaxis_label <0x0a> burlywood | 64 <0x0a> mint | 19.77 <0x0a> dim gray |...'), so the quantitative results of Matcha are almost close to zero. In contrast, our StructChart still works well on FigureQA dataset. For example, our StructChart outperforms Deplot by 9.4% for QA task.

Q1: The experiments should focus on the generalizability of the conclusions/findings derived from the powerful transformer-based models, and it remains a concern.

A1: Thank you for providing such constructive comments! It also gets us to rethink the generalizability of the conclusions/findings. In this regard, we use ChartQA, PlotQA and Chart2Text as the validation sets for the perception stage, and carry out supplementary experiments on the following conclusions to verify the generalization ability from the following two aspects:

1. In Section 4.2 of the main text, we claim that, larger scale of the data will lead to better performance for CIE task, both in real data domain and our simulated domain. The following experimental results show that, the larger the amount of real-world or simulation data, the better the CIE task performance on all the ChartQA, PlotQA, and Chart2Text datasets. "Merging" in the following tables refers to that, training samples are merged from ChartQ, PlotQA, and Chart2Text.

2. In Section 4.3 of the main text, we claim that "We can achieve a high-performance CIE only leveraging few-shot real samples", which has been verified on ChartQA dataset in Table 5 of the main text. To further verify the generalizability of this finding, we also supplement validation experiments with PloatQA and Chart2Text. The results in the following table show that, with the help of SimChart9K, only 10% original real samples in PlotQA and 20% in Chart2Text can basically achieve equivalent CIE performance under the 100% real training samples.

Q1：The approach itself is a bit straightforward as it is not totally end-to-end. However, I don't think it is a drawback, especially given its novel setting compared to other literature.

A1: Thanks for your approval and insightful comment. We are honored that we have reached a consensus that our proposed two-stage pipeline is not a drawback. Furthermore, the main reason why two-stage pipeline is not a drawback is as follows:

• Explicit chart image representations (Such CSV format or the proposed STR format) obtained using perception module can enhance the interpretability of the subsequent reasoning stage.
• The first stage of the proposed perception-reasoing pipeline explicitly extracts the structured text to represent chart image (i.e., CIE task), which can be used as the pre-training corpus of the existing large language models or vision language models.

 

Q2: Some typos: e.g. 4.3 ACHIEVING 100% PERFORMANCE BY ONLY 20% REAL DATA. The ending stop shall be removed.

A2: We thank the reviewer very much for pointing this out, and we have corrected it in the revised manuscript.

 

Q3：Can you compare your method with GPT-4V or other general multi-modality tools?

A3: We would like to thank the reviewer for your interest in our work, and we also have a fascination with the performance of GPT-4V, since OpenAI has just recently opened up their API for GPT-4V.

According to the reviewer's comments, we would like to show three visualization comparisons among StrcutChart, GPT-4V and LLAVA-1.5 in the following link [https://drive.google.com/file/d/1BX7wqS792Z7N4uJNcBagSJbo5622wVYB/view?usp=drive_link]. Note that for QA task, our StructChart is restricted to generating answers only in order to report the quantitative results on ChartQA dataset.

From the visualization results, it can be seen that the proposed StructChart has a strong Chart-based reasoning ability, showing promising performance on multiple downstream tasks. Although GPT-4V also has a strong performance, our model weights, training code, simulation code, and simulation datasets will all be open-sourced for this community! Besides, we have supplemented the visualization comparison results on Page 24-26, Appendix I of the revised manuscript.

Dear Reviewer ojHH,

We are very grateful to the reviewer for the insightful comment about the comparison with GPT-4V or other general vision-language methods.

During the rebuttal period, we have supplemented the visualization comparisons among StrcutChart, GPT-4V and LLAVA-1.5 in the following link https://drive.google.com/file/d/1BX7wqS792Z7N4uJNcBagSJbo5622wVYB/view?usp=drive_link. Also, the comparison results have been included in Page 24-26, Appendix I of the updated manuscript.

Thanks for your effort in improving the quality of our manuscript, and it is appreciated that you could consider these modifications and rapid actions of our manuscript. 

 

Best regards,

Authors of Paper 1125

Dear Reviewer ojHH,

Thanks a lot for your valuable comments, and constructive suggestions, and we have updated the revised paper.

FYI, we provide a summary of the updates inspired by you:

• We emphasize the motivations of designed two-stage pipeline in Sec 3.1, Page 3 of the main text.

• We fix the type error in Sec 4.3, Page 7 in the main text, and check the similar typos.

• We demonstrate visualization comparisons among StructChart, GPT-4V and LLAVA-1.5 in Fig 13,14 and 15 (Page 24-26), Appendix I (Page 17). We also hint at these comparisons in Sec 4.4, Page 9 of the main text.

We all think that your insightful suggestions make our paper more complete and convicing. Thanks again for your precious time.

Best regards,

Authors of Paper 1125

Q7: In Table 4, the authors have omitted a direct comparison with Deplot, which outperforms StructChart with a higher score (i.e., 76.7 compared to 65.3).

A7: Thanks for this valuable comment. We would like to emphasize: 1) Deplot uses closed-source dataset for training. In contrast, our training set including ChartQA and SimChart9K is open-source; 2) Our simulated data, SimChart9k, is scalable and we have verified through experiments that the more simulation data, the better the performance of perception stage in Table 3 of the main text.

Besides, the reason why StructChart has no advantage compared with Deplot (i.e., 76.7 compared to 65.3) can be concluded:

• During inference, Deplot applies Chain-of-Thought (CoT) or Program-of-Thought (PoT) framework as prompt engineering to help the reasoning module boost the performance, but we did not employ any prompt engineering techniques. Since we cannot access specific prompt designs in Deplot, the fairness of result comparison on QA task cannot be guaranteed.
• For evaluation, Deplot adopts Self-Consistency (SC) strategy (generate multiple reasoning paths and answers, and finally select the one with the most frequent answers as the final answer output). However, to ensure consistent output results given the same input condition, we set the temperature parameter of GPT-3.5 as 0.

Finally, for fair comparison, we evaluate StructChart and Deplot with the same reasoning model (GPT-3.5) setting without any prompt engineering strategies nor evaluation tactics. As reported in the following Table, our StructChart has a 12.4% lead over Deplot for QA task on ChartQA.

 

Q8: Confusion about 'merging' in Table 2

A8: Thanks for this valuable comment. Merging-set in Table 2 refers to that, training samples are merged from the real datasets, including ChartQ, PlotQA, and Chart2Text. The purpose of such a merging-set setting is : 1) to better understand the performance scalability given more training data from real-world domain rather than simulation domain, and 2) to provide a baseline that is trained from real-world domain, and such baseline can be used for comparing with that trained from our proposed SimChart9K for chart perception task.

Results from Table 2 in the main text demonstrate that, StructChart continuously improves the perception performance of chart data on each domain given more training samples.

According to the reivewer's comment, we have supplemented some descriptions of the merging setting in the caption of Table 2 in the revised manuscript, to avoid confusion.

Q1: The reasoning process is based on the black box GPT-3.5, which cannot find the relationship between STR and the reasoning process

A1: We are very grateful to the reviewer for the insightful comments. We agree with the reviewer that the reasoning module is based on the black box, but the proposed Structured Triplet Representations (STR) can be regarded as the input of such a reasoning module. As stated in Section 3.1 of the main text, our StructChart employs the two-stage method including perception and reasoning stages, respectively. The perception module generates the STR, and then, the generated STR can be regarded as the input of the reasoning module (GPT-3.5). Besides, due to that the generated STR can effectively represent the positional relations between row and column headers of a given chart, the perception-reasoning task gap can be effectively alleviated.

Experimentally, we have verified that the proposed STR representations are beneficial for improving reasoning ability on different downstream tasks. For example, in QA task, the results in the following Table show that, STR outperforms LCT by 9.0% evaluated on ChartQA validation set (all the settings in GPT-3.5 using LCT input and STR input are consistent, including prompt, code version and parameters). For other downstream tasks including summarization and redrawing tasks, we visually illustrate the comparisons between LCT and STR in Appendix E (please see Figs 11 and 12), where a comma is introduced as noise as illustrated in Fig. 11, and the separator comma itself is included as illustrated in Fig. 12. Both of the case indicate that, STR is more friendly for reasoning process than LCT.

 

Q2: STR may destroy the intrinsic relationship between the original elements, making them independent

A2: Thanks. First of all, the proposed Structured Triplet Representations (STR) does not destroy the intrinsic relationship between the original elements, due to the following reasons:

In Section 3 of the main text, we claim that, "The extracted intermediate CSV text is structured into a triplet form to elucidate the intricate position relationship between the header and index" In this case, "header" indicates column header in CSV format while "index" refers to row header. For STR transformation, each related column header $Entity_{c_m}$ and row header $Entity_{r_n}$ will form a tuple with their relationship $Value_{r_n}^{c_m}$, and the set composed of all tuples aims to represent a complete chart image. Hence, the intrinsic relationship between the original entities will not be destroyed.

 

Q3: Do the authors consider the order of the entity strings during the matching process? Eq (3) lacks a detailed explanation of how this matching process is carried out

A3: Thanks for your valuable comments. In Eq. 3 of the main text, the order of two Entity strings $Entity_{r_n}$ and $Entity_{c_m}$ in each Tuple $(Entity_{r_n}, Entity_{c_m}, Value_{r_n}^{c_m})$ is important for matching. For example, the two Tuples $(Alice, Bob, 100)$ and $(Bob, Alice, 100)$ should be matched exactly. To ensure the order of Entity string does not affect the matching process, we reorder both two Entity strings according to the following pre-process:

• Lowercase all characters in Entity strings. 
• The two Entity strings in each Tuple are sorted by the ASCII of the first letter. 
• If the ASCII of the first letter is the same, the second letter is compared, etc. 

In the revised manuscripts, we have supplemented the detailed explanation of how to perform the matching process in Appendix B on Page 14.

 

Q4: How do the authors determine the correspondence between predictions and specific ground-truth?  

A4: Thanks. For all datasets mentioned in the main text, we have converted Ground-Truth (GT) into the proposed STR format during the matching process. Hence, the matching process is conducted between the STR prediction and the STR GT to determine the correspondence between predictions and specific ground-truth. For example, given that both prediction and GT are sets in the form of Tuples $(Entity_{r_n}, Entity_{c_m}, Value_{r_n}^{c_m})$, the matching process is as follows:

• Query whether each Tuple in GT set has a matching Tuple in prediction set.
• When judging whether two Tuples match, calculate the similarity for Entity string and Value string according to Eqs 3 and 4 in the main text, respectively (the confusion about the order of Entity string has been claimed in the response to Q3).
• Calculate the Intersection over Union (IoU) between prediction set and GT set according to Eq. 7 in the main text. If the calculated IoU is larger than the preset threshold, the prediction and GT are matched.

Q5: In Table 2, why the authors have not included a comparison with Matcha and Deplot on the Chart2Text dataset.

A5: We thank the reviewer very much for pointing this out. The chart images in Chart2Text dataset contain complex background information from the website. As a result, in Table 2, employing the Matcha and Deplot can only achieve very low perception performance (almost close to 0) for Chart-oriented Information Extractor (CIE) task, due to the complex chart background.

For example, on perception task, Matcha can only output the content in the form of HTML format, while Deplot often outputs noisy backgrounds (e.g. website page environment containing other irrelevant text). In contrast, the proposed Structchart can extract accurate information of the chart from complicated background, such as information from the website. Specifically, we have illustrated visualization comparisons among Matcha, Deplot, and our StructChart for CIE task on Chart2Text dataset in Figure 6,7,8 of Appendix F.  

 

Q6: In Table 2, the comparisons between StructChart and the compared methods (Matcha and Deplot) somewhat is unfair, since StructChart leverages powerful GPT-3.5 as the reasoning model, whereas Matcha is based on Pix2Struct and Deplot relies on Codex or GPT-3.  

A6: We thank the reviewer very much for pointing this out. First, we would like to emphasize that In Table 2, we only report the results of chart perception task using our proposed SCRM metric. For perception part, all comparisons with Matcha and Deplot are fair, since we directly evaluate the well-trained Matcha and Deplot on the ChartQA, PlotQA, and Chart2Text, where all evaluation settings are consistent with the proposed StructChart.

Furthermore, for reasoning part, Matcha is an end-to-end model (without the help of GPT-3 or 3.5), and we cannot modify its reasoning module (including network design and language model selection) during the evaluation process. As a result, we only choose to directly compare with Matcha. Experimental results show that our StructChart has a 1.1% lead over Matcha in average accuracy. Besides, according to the Reviewer's comment, to make a fair comparison, we supplement experiments of employing Deplot+GPT-3.5 to perform the QA task on ChartQA dataset, where we all use GPT-3.5 and ensure that the model settings (prompts, temperature, top-k, etc.) are consistent. The results in the following Table show that, our StructChart+GPT-3.5 outperforms Deplot+GPT-3.5 by 12.4% for QA task on ChartQA. Besides, It should be noted that there is no extra prompt engineering strategy (Chain-of-Thought, Program-of-Thought) nor evaluation tactics (self-consistency) during the reasoning process.

In the revised manuscripts, we have supplemented the corresponding experimental results and empirical analyses of Deplot+GPT3.5 in Table 6 of Appendix C on Page 15.

Dear Reviewer mxZb,

We have made a rapid response to your concerns, and hope that our response can address your concerns. As the deadline for discussion is approaching, we would greatly appreciate it if you could let us know if there are any other questions or issues regarding the paper or our response. We are looking forward to further discussion.

Best regards,

Authors of Submission 1125

Dear Reviewer mxZb,

Thanks a lot for your valuable comments, and constructive suggestions, and we have updated the revised paper.

FYI, we provide a summary of the updates inspired by you:

• For confusion may caused by matching process, we explain in detail in Appendix B, Page 13. Also, we make the hint in Sec 3.2, Page 4 of the main text.

• For unfair comparisons between StructChart with other methods, we claim the settings we adopted in GPT-3.5 in Sec 4.1, Page 6 of the main text, as well as further discussion in Appendix C, Page 14. The QA result for Deplot with fair settings is supplemented in Table 4, Page 8 of the main text.

• We redefine "Real maerging" and "Real & Sim Merging" with clear explanation in Table 2, Page 7 of the main text.

We all think that your insightful suggestions make our paper more complete and convicing. Thanks again for your precious time.

Best regards,

Authors of Paper 1125