AnyText: Multilingual Visual Text Generation and Editing

Dear Reviewer,

We deeply appreciate your thorough and meticulous review, and we would like to express our sincere gratitude for recognizing the value of our work. Your thoughtful questions have been greatly appreciated, and we are more than willing to provide detailed explanations in response:

Q1: It is not clear whether the improved results come from better training data or the proposed model. It would be best to compare the baseline models trained on the same dataset, or train the proposed model on the previous LAION glyph subset.

The training dataset used for TextDiffuser is MARIO-10M, out of which 9.2M images were filtered from LAION-400M. Similarly, the English data in AnyWord-3M was also filtered from LAION-400M. During the data processing, we indeed found that there were nearly 10 million images containing text in LAION-400M. However, in order to maintain a balance in terms of scale with the Chinese data in AnyWord-3M, we adjusted our filtering rules and retained only 1.3M images. More details about this can be found in Appendix A.2. To some extent, we are using a subset of the training data as for TextDiffuser.

The GlyphControl dataset, LAION-Glyph-10M, was filtered from LAION-2B-en, and it is likely to have a significant overlap with the text-containing images in LAION-400M. Additionally, in our ablation experiments (refer to Table 2 in Supplementary Material for the latest results), the setup of Exp. 2 is very similar to GlyphControl. It involves disabling editing, text embedding, and perceptual loss while retaining only the glyph image that renders tests based on their positions, along with the corresponding text position image. Under the exact same training dataset, our method (Exp. 8) demonstrates a significant advantage.

In addition, we randomly sampled some text lines to assess the accuracy of our OCR annotations. The error rate for the English portion was 2.4%, while for the Chinese portion was significantly higher at 11.8%. This difference may be attributed to the inherent complexity of Chinese character recognition, as well as the performance of PPOCR_v3.

In summary, during model training, we only utilized 1.3 million out of 10 million English data and incorporated over 1.6 million non-English data with noise. While our English metrics have a clear advantage over competing methods, this advantage is certainly not derived from the benefits of the data.

Q2: in the experiments, the ground truth text mask is used as conditional input. It would be interesting to see what if random text position and mask is used, can it still generate reasonable image?

The ground truth text mask is only available during evaluation. In other cases, there is no ground truth mask. In such cases, the user needs to manually specify the position of the text, as shown in Fig. 1 in Supplementary Material, using tools like the Gradio brush. It is true that providing a mask may not be very user-friendly, and if the position of the text is too arbitrary or unreasonable, it can also decrease the quality of the generated image. We also provide a template-based approach to automatically generate masks. However, it is important to acknowledge that the ability for users to specify the position of the text is indispensable. In certain situations, relying solely on methods that automatically predict text positions may not fully meet all users' requirements.

Dear reviewer,

We attempted to compare our model with the method and data used in another work, specifically TextDiffuser. However, we encountered some challenges that made it not as "easy to go" as expected. Following the instructions provided in the Official Code, we found that it only provides rendered images with all text lines. Since AnyText requires the position and corresponding text for each line to make the Auxiliary Latent and Text Embedding modules work, perform inference became impossible. To overcome this issue, we attempted an alternative approach by using our OCR model to generate annotations, followed by generating images and comparing them using their metrics. After spending lots of time configuring the OCR model they used (MaskTextSpotterV3), and made guesses for missing details (presumably modified in the MaskTextSpotterV3 code that was not provided), we roughly replicated their evaluation metrics. Unfortunately, we encountered another issue that our OCR results are not consistent with the ground truth they used in their precision and recall calculations, which prevent us from obtaining reliable and fair evaluation results. Regarding GlyphControl, we were even unable to reproduce their own evaluation results. From the Official Code, they only shared 200+ prompt templates used for evaluation, without providing the texts they randomly selected and the positions of them, and the evaluation code was not provided, either. Consequently, we are unable to use evaluation set and metric from others to assess our model.

In the other question, it is worth considering that other methods might have seen some of our evaluation sets. However, even that if a generated model has seen an image during training, fully restoring the pixels accurately is still challenging. Furthermore, the inputs provided to the model during inference are not the actual images themselves but rather captions and OCR annotations, which could differ from those used by other models. So, we believe that compared to traditional methods such as detection and recognition, the impact of this issue is relatively minor.

Regarding whether the presence of Chinese data improves the performance of English text, our answer is: very likely. Chinese characters encompass thousands of distinct shapes, and if a model can accurately generate these characters, it becomes comparatively much easier to generate the additional 26 concise letterforms of English. This stems from our method's comprehensive approach to designing methods and processing datasets to support multilingual capabilities, which may not be present in other methods. However, even when considering the same training data, if we remove $l_p$ and $l_m$ in the Auxiliary Latent module, disable the Text Embedding module, and exclude the Text Perceptual loss, our method bears resemblance to GlyphControl. In our ablation study (see Table 2 in Supplementary Materials for latest), we have demonstrated performance improvements achieved by each of these self-designed algorithms.

We hope this reply has addressed your concerns, and we are looking forward to further discussion with you. Thank you!

Dear reviewer,

Thank you very much for your affirmation and praise of our work， which has greatly encouraged us, and we will continue to make efforts to open-source and further improve AnyText, advancing the development of text generation technology. Regarding the issues you mentioned, our responses are as follows:

Q1: The architecture seems somewhat reliant on pre-established technologies such as Latent/Stable Diffusion and ControlNet, which slightly shadows its novelty.

Stable Diffusion and ControlNet have been gaining significant popularity in the open-source community recently, and various interesting models have emerged. Our intention was to provide a plug-and-play module that allows more people to work on text generation and editing based on their own models. However, technically speaking, pre-established technologies are not strict dependencies for AnyText. In the future, we may explore other methods such as GANs.

Q2: Encumbered by a complex array of components and a multi-stage training regimen, the model’s re-implementation emerges as a challenging task, compounded further by numerous critical hyperparameters requiring manual assignment.

Text generation and editing can be seen as a multi-task model. It is logical to first focus on "generating" text well before proceeding to "editing" it, which is why we implemented a staged training approach. The current solution does involve several modules and parameters, which is why we built a smaller-scale dataset (mentioned in Sec. 5.3) and conducted extensive validation work. In the future, we will open-source our code and dataset to facilitate the reproducibility of AnyText and consider refining the approach to make it more streamlined.

Q3: Certain aspects, such as token replacement, require a more elaborate discourse for clearer comprehension, primarily concerning the identification of corresponding tokens and their subsequent utility in text-image generation.

The process of token replacement is as follows: First, we obtain the CLIP token index of the placeholder string. For example, the index of '$\*$' is 265. Then, for the input $y'$ (which is processed from the original caption $y$ with all text lines replaced by '$*$'), after tokenization, we obtain a sequence of token indices. At this point, we keep track of the positions where the index is 265. After obtaining all token embeddings through a lookup table, we replace the embeddings at the marked positions, with the embeddings encoded by the OCR model. The replaced tokens then go through the self-attention mechanism of the CLIP text encoder, which combines stroke information with semantic information.

Q4: There exists a potential ambiguity concerning the intermediate generative results (x'_0), where the possible presence of noise or blur could compromise the precision of perceptual loss computation.

Yes, the $x'\_{0}$ may appear blurry, and the blurriness is correlated with the time step t. This is why we introduced the weight adjustment function $\varphi(t) = \bar{\alpha_t}$. We provide a more detailed explanation in Fig. 6 and Fig. 7 in Supplementary Material. According to the original DDPM paper, $\alpha_t=1-\beta_t$, where $\beta_t$ represents the forward process variances that increase linearly from 0.0001 to 0.02. $\bar{\alpha_t}$ is the cumprod of $\alpha_t$, with the range from 0 to 1, this aligns perfectly with our requirements for the weight adjustment. When t is large, the predicted image $x'_{0}$ may be more blurry, but the corresponding weight is also small, and the text perceptual loss has little effect.

Q5: A clearer depiction of computational resource demands (GPU Hours), beyond the ambiguity of epochs, would enhance the paper’s practicability and replicability.

As mentioned in Sec. 5.1, our model was trained for 10 epochs, with the training speed being consistent for the first 8 epochs at approximately 29h/epoch. For the last 2 epochs, it increased to ~40h/epoch due to the enabling of the text perceptual loss. Overall, the total training time for the model amounted to 312 hours on 8XA100 GPUs.

Thanks for the hard work of authors during rebuttal. This is indeed a high quality paper with good performance as demonstrated in the paper. It would be great that authors could open-source the code, models and dataset to contribute the community in the near future. No more questions and will maintain my initial score.

Dear Reviewer,

We express our heartfelt gratitude for your thorough review of our manuscript and the recognition it has received. We have given careful consideration to the issues you raised and offer our responses below, and we would like to engage in further discussion.

Q1: Could you elaborate on how the PP-OCRv3's performance specifically influences the results?

Coincidentally, before utilizing the OCR model, our initial attempt was to employ the CLIP visual encoder. However, we found that the results were unsatisfactory. We suspect that this is because images with only rendered text fall under the Out-of-Distribution data category for the pre-trained CLIP vision model, hindering its text stroke understanding and encoding capabilities. Subsequently, we experimented with a trainable module composed of stacked convolutions and an average pooling layer. However, without strong supervision specifically tailored to this task, this module also struggled to perform well. In this study, we have reorganized our experiments, and the relevant results can be found in Table 2 of Supplementary Material. From Exp. 3 and Exp. 4, we noticed that using the vit or conv module for token replacement in the text embedding resulted in even lower performance compared to the original CLIP text encoder in Exp .2. Comparing with Exp. 5, we can conclude that utilizing a pre-trained OCR model effectively encodes stroke-level information, significantly enhancing text generation accuracy. You can also find qualitative comparisons in Fig. 4 and Fig. 5 in Supplementary Material.

Q2: Have you considered measuring the computational overhead when additional modules are integrated into the vanilla text-to-image diffusion model?

As mentioned in the paper, our framework is based on ControlNet. Despite the addition of some modules, it did not significantly increase the computational overhead. Please refer to the parameter details below:

We compared the computational overhead of both models using a batch size of 4 on a single Tesla V100. The results are as follows:

In summary, compared to ControlNet, AnyText has an increase of 0.34% in parameter size and 1.04% in inference time.

Q3: Why were Korean and Japanese languages included in the qualitative results but not in the quantitative ones?

Generating quantitative results for these languages requires the careful selection of reliable OCR models and the meticulous preparation of benchmark datasets. However, to be honest, none of us are familiar with languages other than Chinese and English. Furthermore, even if we were to calculate quantitative results, there are currently no comparable methods available to the public. Therefore, we have only provided qualitative results for now. Nevertheless, we plan to gradually improve this aspect in the future.

Q4: Is there a reason why FID measurements were not included in the ablation study?

Due to limitations in computational resources, we were unable to conduct ablation experiments on the full dataset and instead opted for a smaller-scale dataset. At this scale, the generated images from the ablation experiments did not exhibit satisfactory realism, making the FID metric less meaningful. In contrast, our primary focus was on the accuracy of the generated text, measured by Sen.ACC and NED. We plan to increase the scale of the ablation experiments in the future.

Q5: Why were qualitative results not provided for each individual ablation experiment?

We have added qualitative results in Fig. 4 and Fig. 5 in Supplemental Material.

Dear reviewer,

We greatly appreciate your time and effort in reviewing our manuscript. Sorry for the oversight of certain details in our paper that have confused you. We are also grateful for the several constructive suggestions which have contributed to the refinement of our work, and we eagerly anticipate further discussions. We have carefully considered each of your concerns and have addressed them thoroughly below:

Q1: Which feature is extracted from PP-OCR to serve as text embedding?

We use the feature map before the last fully connected layer of the PP-OCR model. The feature map mentioned here and the $\hat{m}\_{p},\hat{m'}\_{p}$ referred to in the text perceptual loss(Sec. 3.4) are the same, the difference is that the feature is flattened and linear projected later, to match the token dimension.

Q2: Is the linear projection layer for the OCR encoder trained?

Yes, the linear transformation $\xi$ is randomly initialized and trained during the training process.

Q3: What is the configuration for the fuse layer? A single convolution layer or a stacked convolution?

A single layer, as mentioned in Sec. 3.2 : “...we utilize a convolutional fusion layer $f$ to merge $l_g$, $l_p$, and $l_m$...”. The kernel_size=3, stride=1, in_channels=324(256+64+4, channels for $l_g$, $l_p$, and $l_m$) and out_channels=320.

Q4: What is the input to text controlnet? Based on Figure 2, it seems to be the concatenation of $***z***_a$ and $***z***_t$.

There is a "⊕" in Fig. 2, which means the operation is addition rather than concatenation. To provide an accurate description, we will modify the text in Sec. 3.1 as: "...to control the generation of text, we combine add $z_a$ with $z_t$ and fed them into a ..."

Q5: Why is the image resolution for glyph image 1024x1024 instead of 512x512? This does not align with the final image resolution.

We have observed that when rendering small text onto glyph images at a resolution of 512x512, the text becomes nearly illegible, whereas at 1024x1024 there is no issue (see Supplemental Material Fig. 2, the first and fourth images in the second column). Additionally, we employed 4 downsampling layers for the Glyph block, while the Position block only employed 3. This incurs additional parameter overhead (see our replay to reviewer 5GTp's Q2), but the results demonstrate that it does not affect the alignment with the final image resolution.

Q6: How is the tight position mask $l_p$ annotated? As far as I know, the PP-OCR model does not support arbitrary shape text detection.

Our dataset does not rely solely on OCR models but also includes human annotations. Just as mentioned in Sec. 4, "Except for the OCR datasets, where the annotated information is used directly, all other images are processed using the PP-OCRv3...". The image used in Fig. 2 is sourced from an OCR dataset called Art.

Q7: The glyph image $l_g$ contains rotated text; how is the bounding box annotated from the position mask?

As mentioned in Sec. 3.2: "...we simplify the process by rendering characters based on the enclosing rectangle of the text position...". Specifically, use cv2.minAreaRect( ) on the polygon of the position mask.

Q8: The captions are generated using BLIP-2 model instead of the original captions. Could authors provide some statistics like the length of the captions and show some example captions for images in AnyWord-3M? How does this difference affect the model performance?

The average caption length by BLIP-2 is ~8.26 words per image, excluding the additional description of the text to be rendered. You can find caption examples in Fig. 2 and Fig. 3 in Supplemental Material. We utilize BLIP-2 not due to performance considerations, but because images as in OCR datasets or Noah-Wukong do not have existing English descriptions, and BLIP-2 is one of the most common approaches adopted by many studies. We believe that even perfect captions would have negligible influence on text-generation tasks. Therefore, we did not focus on evaluating it.

Q9: Could authors provide more information about the input for the visual examples? For example, the layouts for Figures 1, 6, 8, 12, the captions for Figures 5, 10, 11.

To provide more information, we have regenerated several images, as shown in Fig. 1 in Supplementary Material. We just use Gradio's brush as input and after a simple process convert it into Glyph and Positon. The layouts and captions for Figures 5, 10, and 11 are available as they are sourced from our benchmark. See Fig. 2 and Fig.3 in Supplemental Material.

Thanks for the clarification, but I'm still confused about the tight position mask annotation. As far as I understand, not all of the OCR datasets contain polygon mask annotation for the scene texts. For example, MTWI only contains rectangle bounding box annotations. Also, the open-sourced PPOCRv3 model only predicts rectangle bounding boxes instead of polygon masks. My major concern is how you collected these tight position masks for Anyword-3M. If such tight position mask annotation is very few in the dataset, how can the model generate texts in irregular/curved regions?