Synchronous Scene Text Spotting and Translating

We greatly appreciate your valuable time and feedback on our paper. Here are our responses to weakness and questions.

“The proposed method involves a larger set of parameters compared to other methods.”

From Tables 2 and 3, it is evident that our method has a higher number of parameters compared to the baselines. However, by expanding Table 4 and conducting comparative experiments with large models (thank you for your suggestion to include experiments with large models), we have validated our method achieves the best translation performance in a middle level of model size. For specific results, please refer to the table in the "Performance" section of our General Response 1.

“When comparing with multimodal large models, the analysis lacks a comparison against the most recent large multimodal models, such as mplug-owl (CVPR 24), Monkey (CVPR 24), and InternVL (CVPR 24). It would be beneficial to have the performance results of the large multimodal models after they have been fine-tuned using the corresponding dataset.”

It is indeed beneficial to fine-tune. However, considering the need for a quick performance comparison, we opted to directly test the large models with our data. Since these large models do not undergo a fine-tuning process, we designed three testing scenarios to ensure fairness. Besides, we compare our method with AnyTrans, which is the latest large model method trained specifically on scene TIMT tasks. For dataset fairness, we directly test our model on AnyTrans's dataset without any fine-tuning. And in the future research we will add experiment with model fine-tuning.

For specifics of large model comparing and AnyTrans, please refer to our General Response 2.

“The authors claim that “In this pattern, translation performance is affected by propagation of mispredicted reading order and text recognition errors.” However, there is a lack of experiments or evidence to verify this claim.”

For evidence of propagation's effect, please refer to the experiment result of last raw "Remove BAF" in Table 7 in our paper. Remove BAF means that use text spotter(best one in baselines, UNITS) and translation module in a pipeline way, which does not solve the issue of recognition errors and reading order issues and causes the decreasing in translation performance.

What's more, we could see from the image cases in the third and fourth rows of Figure 4 that there are still recognition error and reading order error in pipeline even if we used the state-of-art text spotter. And for their effect in translation, for example:

• Figure 4, row 3, column 3: the text in the image was challenging traditional Chinese art fonts, the text spotter mistakenly recognized "雲南" (Yunnan, a province in China) as "餐满" (which can be loosely translated as "full meal"). And the translation "cuisine" is no doubt wrong.
• Figure 4, row 4, column 3: the complex layout of Chinese characters makes it hard for text spotter to predict correct reading order of "粉" and "面". It wrongly predicted it as "面粉", which was translated into "flour", a basic ingredient. But the correct answer is "粉面", which represents a local food similar to noodles.

“Lack of experiment to verify the effectiveness of proposed BAF Module using different text spotter and Translation module.”

We conducted an ablation experiment on the BAF module to validate its effectiveness. However, to further strengthen the persuasiveness of BAF, it would indeed be a better approach to train various combinations of text spotter and translator while keeping the BAF module fixed. Our subsequent experiments will continue to explore this aspect. Thank you for this valuable suggestion.

“There is a lack of experimental validation to show how a unified framework improves different modules. Does it provide a greater improvement for end-to-end text spotting or for translation?” / “Why is it necessary to unify end-to-end spotting and translation? The author did not emphasize the necessity of unification in the paper.”

Our study focuses on improving translation performance with unification and multi-task training, as evidenced by comparisons with the best pipeline baseline and the best E2E baseline.

• The best pipeline baseline is MCTIT. To eliminate the impact of the OCR tool, we used ground truth coordinates and recognized text as input, solely comparing translation performance with our method. In Table 4 of the paper, our method demonstrated comparable performance on OCRMT30K and achieved a higher score on ReCTS. Notably, the latter dataset features a complex layout level, as shown in the table in the "Data" section of our General Response 1.

• The best E2E baseline is UNITS. We removed BAF module and test in a pipeline pattern by UNITS text spotter + translation module. The result is the last row of Table 7 of paper. The translation performance(BLEU, COMET) of this pipeline is lower than the best row, which uses unification and multi-task training.

We greatly appreciate your valuable time and feedback on our paper. Here are our responses to weakness and questions.

“For text recognition and translation of complex scenes, especially fine-grained and layout complexity is not reflected in the paper.”

Qualitative explanations We present an example in Figure 1 to illustrate the issue of reading order, more examples could be found in Figure 3 and Figure 4. The complex layout is presented in aspects like fonts stype, fonts size, text richness and text reading order.

• For example in Figure 1, the nine Chinese characters use different font styles and sizes, and the reading order is not normal scanning, which could be wrongly detected as "三汤包津" (correct one is "三津汤包", which is a name of a breakfast restaurant). Moreover, "津" could be wrongly recognized as "用" because of art style of font.
• To understand the complexity of our layout, please see another example in Figure 4-row 4-column 3: the Chinese characters are completely scattered, making it hard for text spotter to predict correct reading order of "粉" and "面". It wrongly predicted it as "面粉", which was translated into "flour", a basic ingredient. But the correct answer is "粉面", which represents a local food similar to noodles

Quantitative validations We compare our dataset's layout complexity with the state-of-the-art dataset in scene TIMT field by statistic method. And our dataset shows much more complexity. Please refer to "Data" section in our General Response 1 to see more details.

“The way of synchronized training is not clearly described. The model's training data is large, and the resources consumed are not mentioned. Large datasets lead to more resources consumed for training models.”

The way of synchronized training is described in Chapter 4.3 EXPERIMENTAL SETTINGS, which starts from "Training process includes three steps:...".

We acknowledge that the resources utilized are substantial. This was also noted in Chapter 4.3, specifically in the last sentence which states, "All experiments are done on eight A6000 GPUs." However, we consider this cost to be reasonable, given the current trend of recent state-of-the-art models for E2E text spotting to consume large resources for training. For instance, our baselines, SPTSv2 requires 10 A100 GPUs, and UNITS uses 8 A100 GPUs for training.

“How does the BAF module affect the reliability of the translated output in cases where errors occur in detection or recognition? Could more error analysis be provided to help understand the effects of the fusion of visual and textual features?”

We did ablation experiments on BAF module in Table 7, the translation scores gradually improved with the progressive incorporation of feature fusion, which proves the effect of BAF from experiment aspect.

We also provided cases to compare the result with/without BAF in Figure 4. We compared translation results from both UNITS + translation pipeline and our synchronous method under four different scenarios. Event if we use the best text spotter, there still could be recognition error and reading order error. These issues is solved by our synchronous method, also proving the effectiveness of BAF. Except reading order issues mentioned above, about BAF's effectiveness on recognition error, please see the example below:

• Figure 4, row 3, column 3: the text in the image was challenging traditional Chinese art fonts, the text spotter mistakenly recognized "雲南" (Yunnan, a province in China) as "餐满" (which can be loosely translated as "full meal").

“How about the recognition and translation performance of the model in more complex scenarios?”

We have already compared our layout complexity with the state-of-the-art dataset in the scene TIMT field in the above response. Actually, for current methods, the text in the image is either a single line, like a video caption, or based on a simple scanning reading order, like AnyTrans. Our method demonstrates SOTA translation performance in such complex layout cases. For more details, please refer to General Response 1. It provides a statistical validation of layout complexity and our translation performance in such complexity.

“Can the model be applied to translations in languages other than Chinese and English?”

Yes, our model is able to be do language extension by applying other language pair's data and re-training.

“Can there be a more detailed explanation of the construction of the dataset？”

Yes, our dataset includes synthesized data and real data. The synthesized data is generated by rendering text content(from WMT22) with multiple fonts on the scene images(selected from COCO). The real data is collected from OCRMT30K, ReCTS, CTW1500 and HierText, and we manually re-labeled the reading order and translation with the help of large language model and prompt. Some details could be found in Section 4.1 DATASETS, and we will share our data generation script along with dataset.

We greatly appreciate your valuable time and feedback on our paper. Here are our responses to weakness.

“The related work section needs adjustment. This paper mainly focuses on TIMT, yet the authors provide very little introduction to this field in related work, instead offering a large amount of introduction on text spotting.”

Thank you very much for your advice. In the revised paper, we have modified the related work section by streamlining the text spotting content and expanding the coverage of multi-modal translation.

“The paper is somewhat difficult to read and does not provide some key experimental settings. For example, the setting of alpha in Equation 2 and the size of the Qwen-VL model are not specified. Additionally, all models labeled as 'ours' in the experimental results are bolded, yet they are not the best, which is quite confusing. This result does not align with what the paper claims as SOTA.”

• About alpha in Equation 2, as you mentioned in question, the model training is not sensitive to this parameter and you could set it 0.1, 0.5 or 0.9. We have add it in revised paper to enhance the readability and comprehensibility.

• About Qwen-VL model size, the API we utilized is Qwen-vl-max, which is not open-source. Neither the Qwen-VL paper [1] nor its official documentation [2] provides clear information on the model's size. However, we could use the parameters number 9.6B of Qwen-VL in [1] as a reference.

• About bolded not SOTA, it is accurate that we mistakenly used bolded text to emphasize our method rather than the SOTA score. We have rectified this in the revised paper.

“The novelty is limited. The Bridge & Fusion module essentially extracts visual features based on the predicted text region's coordinates and then obtains multimodal features through cross-attention. This approach is very common in multimodal machine translation.”

Attention is indeed common in feature fusion. However, different from existing attention[3], we did some special optimization on query in our BAF module by using both position and recognition embedding to search visual feature. To see more about our novelty, please refer to our General Response 1. It provide details about our contribution on dataset, translation performance and long-tern impact.

“The paper claims that its model is end-to-end, however, actually the model still requires autoregressively generating coordinates and recognition output first, and then combines them with the image to autoregressively generate the translation. Therefore, it is not fully end-to-end.”

Our method is not end-to-end, and we distinguish it from a pipeline because the text spotter and translator are not entirely independent modules, which we emphasize as a schronous method. The categorization in the first column of Tables 2 and 3 did indeed cause some confusion, and we have corrected it and updated the paper accordingly.

“The paper illustrates the issue of incorrect reading order in the pipeline method shown in Figure 1. However, the proposed method does not address this problem but merely provides such training data. Given this type of data, can the text spotting model in the pipeline also solve this issue?”

The annotation of reading order in the data does indeed allow our method to naturally address this issue, but our approach is not entirely data-driven. We employed a combination of ablation experiments to demonstrate our effectiveness:

• In Table 4 of our paper, we selected MCTIT, which scored the highest among the pipeline baselines, for comparison. By eliminating the influence of external OCR tools and even when provided with ground truth OCR results as input, our model's translation scores surpassed MCTIT on the ReCTS dataset, which features the most complex layout (the complexity of which we explained in the "Data" section of General Response 1). This demonstrates that synchronous method can indeed enhance translation capabilities.

• In Table 7 of our paper, the translation scores gradually improved with the progressive incorporation of feature fusion, which proves the effect of BAF. It is noteworthy that the last row of Table 7 “Remove BAF” represents an experiment where the text spotter of our method, which is based on UNITS (the highest-scoring model in the E2E baselines), is used in a pipeline with the translation module. Even if use the state-of-art text spotter, the pipeline's translation score is still lower than our synchronous method.

[1] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

[2] Chinese web: https://help.aliyun.com/zh/dashscope/developer-reference/tongyi-qianwen-vl-plus-api

[3] PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

“What is the size of the Qwen-VL model being compared, and what are the differences in settings with AnyTrans [1]? It would be better if compared with AnyTrans.”

The model size is answered by weakness part. We add comparison with AnyTrans.

• AnyTrans is also trained on scene TIMT tasks like ours.
• We test our model on AnyTrans's dataset(Chinese-to-English, also called zh2en) without any fine-tuning.
• We use the same calculation method as AnyTrans uses in its paper and compare with its highest score(Qwen-vl-max).

Our model achieves comparable BLEU score and higher COMET score compared with AnyTrans.

We greatly appreciate your valuable time and feedback on our paper. Here are our responses to weakness.

---

“Considering the architecture and pipeline of the proposed model, the novelty of it is limited.”

Our architecture considers the layout issue(reading order, recognition errors, multi-tasks training) and applies optimization on feature fusion module(BAF) by designing query for attention. Besides, please consider our contributions on dataset, translation performance and long-tern impact on combining large multi-modal model with image preprocessing tasks. Details please see our General Response 1.

“The details of casting existing baselines (such as ABCNetv2, SPTSv2 and UNITS) as text spotter and translator are very crucial, but they are absent in the paper.”

We apologize for any inconvenience caused and have now included clarifications about the casting baselines in Section 3.1 OVERVIEW of the revised paper.

“The comparison in Tab. 5 is unfair. The proposed model is pre-trained with data from various sources (such as STST800K and WMT22) and fine-tuned on down-stream datasets, while the Qwen-VL model is not.”

To keep dataset fairness, we add comparison with AnyTrans[1] in our revised paper:

• AnyTrans is also trained on scene TIMT tasks like ours.
• We test our model on AnyTrans's dataset(Chinese-to-English, also called zh2en) without any fine-tuning.
• We use the same calculation method as AnyTrans uses in its paper and compare with its highest score(Qwen-vl-max).

Our model achieves comparable BLEU score and higher COMET score compared with AnyTrans.

About model size, our model has 2.88B parameters. Qwen-vl-max is is not open-source. Neither the Qwen-VL paper [2] nor its official documentation [3] provides clear information on the model's size. However, we could use the parameters number 9.6B of Qwen-VL in [2] as a reference, which is larger than ours.

[1] AnyTrans: Translate AnyText in the Image with Large Scale Models.

[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

[3] Chinese web: https://help.aliyun.com/zh/dashscope/developer-reference/tongyi-qianwen-vl-plus-api